Container logging for ML workloads

Containers are disposable; diagnostic evidence should not be. An ML job may terminate after hours of training, a model server may be replaced during rollout, and a node may disappear before an operator connects. Logs need to leave the process in a form the runtime and platform can collect while preserving enough run context to explain the outcome.

The simplest reliable contract is one event per line on stdout or stderr.

Write application logs to standard streams

Container runtimes capture output from the foreground process. Locally, Docker exposes it through commands such as:

docker logs --timestamps --tail 100 training-container
docker logs --follow --since 10m training-container

The selected Docker logging driver determines how those streams are stored or forwarded. In Kubernetes, the runtime writes logs in the node's expected format and a cluster-level agent commonly collects them. The Kubernetes logging architecture explains the node-level model and why Kubernetes itself does not provide a native long-term log store.

Do not make a file inside the writable container layer the only log destination. It can disappear with the container and contribute to node disk pressure.

Keep one event per line

Multiline stack traces are readable in a terminal but can be split or mis-grouped by collectors. Structured JSON per line gives downstream systems an unambiguous event boundary and searchable fields.

Include stable context:

• timestamp and severity;
• service, component, and version;
• Polyaxon run and project identifiers;
• stage, worker rank, and retry attempt;
• image and code revision;
• error class and safe message.

Configure the application to emit this context. Parsing free-form text in the collector is more fragile and makes every language or team invent a different format.

Separate stdout and stderr by policy

Some teams send normal events to stdout and warnings or errors to stderr. Others emit every structured record to stdout and use a severity field. Either can work if the runtime and collector preserve the distinction and the policy is consistent.

Do not use stderr for every log simply because it is visible. Some platforms interpret stderr as failure or raise alert noise even when the process exits successfully.

Control local retention

Container logs consume node storage before or while they are shipped. A verbose loop can fill a node, trigger DiskPressure, and evict unrelated workloads.

Set runtime-supported rotation and retention limits through the platform's node configuration. Confirm how changes apply to existing versus newly created containers. Monitor log bytes, filesystem capacity, inode use, collector backlog, and dropped records.

Application controls matter too. Avoid one line per training sample, token, or tensor. Emit progress at bounded intervals, use metrics for numeric series, and store large diagnostic reports as artifacts.

Handle backpressure deliberately

Logging paths can block or drop records depending on runtime and driver configuration. Blocking preserves records at the risk of slowing the application. Non-blocking modes can protect workload latency but may lose logs when buffers fill.

Choose based on event importance. Audit or completion records may need a durable application path rather than best-effort stdout alone. High-volume debug events should not be able to stall a latency-sensitive model service.

Measure logging overhead under representative load, including collector disruption. A strategy that works when the backend is healthy may behave differently during the incident you need logs to diagnose.

Protect sensitive ML data

Never dump the complete process environment, authorization headers, cookies, connection strings, cloud credentials, or signing material. Prompts, model responses, labels, and dataset rows can contain personal or proprietary information.

Define an allowlist of fields, redact before emission, and apply access and retention policies downstream. Log identifiers and hashes that let an authorized user locate the governed source without copying its contents into a broader logging system.

Preserve completion evidence

A final log line can be lost during abrupt termination. Persist important outcomes—metrics summaries, evaluation reports, checkpoints, and manifests—as versioned artifacts. Use logs to narrate the attempt and point to those durable objects.

Polyaxon associates container output with each operation and provides logging guidance for run-level inspection. Configured artifact connections preserve outputs that need a lifecycle beyond log retention.

Good container logging has a small, explicit contract: standard streams, structured bounded events, stable run context, safe data, controlled node usage, and a durable home for the results that matter.