Python logging for ML workloads

Useful ML logs explain what a workload attempted, which versioned inputs it used, where time was spent, and why it stopped. They should remain understandable after the container disappears and safe to share with the people responsible for the run.

Python's standard logging package provides the routing and severity model. The design work is deciding which events and context make the execution reproducible without turning logs into an unbounded copy of the dataset or prompt stream.

Use module-level loggers

Application modules should create a named logger and let the entrypoint configure handlers:

import logging

logger = logging.getLogger(__name__)


def load_dataset(dataset_uri: str) -> None:
    logger.info("Loading dataset", extra={"dataset_uri": dataset_uri})

Libraries should not call logging.basicConfig() at import time. That would take control of formatting and destinations away from the application embedding them. The Python Logging HOWTO explains logger levels, handlers, formatters, and configuration patterns.

Use a stable logger hierarchy based on module names. It lets operators raise verbosity for one subsystem without enabling debug output across every dependency.

Write logs to the container stream

For Kubernetes workloads, write application logs to stdout or stderr and let the cluster's logging pipeline collect them. A file inside the container is ephemeral unless a separate mechanism ships or persists it, and unmanaged log files can fill node storage.

Configure logging once in the process entrypoint:

import logging
import sys


def configure_logging(level: int = logging.INFO) -> None:
    handler = logging.StreamHandler(sys.stdout)
    handler.setFormatter(
        logging.Formatter(
            fmt="%(asctime)s %(levelname)s %(name)s %(message)s",
            datefmt="%Y-%m-%dT%H:%M:%S%z",
        )
    )

    root = logging.getLogger()
    root.handlers.clear()
    root.addHandler(handler)
    root.setLevel(level)

Clearing handlers is appropriate in an application-owned entrypoint where configuration must be deterministic. Do not do it inside a reusable library.

Add execution context

A timestamp and message are not enough when many runs share a logging backend. Add stable context such as:

• run identifier and project;
• operation or component version;
• image and code revision;
• dataset or model version, not the full data;
• stage, worker rank, and retry attempt;
• cluster, namespace, and node pool where appropriate.

Use consistent field names across training, evaluation, and serving. A structured formatter can serialize those fields as JSON for reliable querying:

import json
import logging
import sys
from datetime import UTC, datetime


class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "timestamp": datetime.now(UTC).isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
        }
        for field in ("run_id", "stage", "worker_rank", "attempt"):
            value = getattr(record, field, None)
            if value is not None:
                payload[field] = value
        if record.exc_info:
            payload["exception"] = self.formatException(record.exc_info)
        return json.dumps(payload, default=str)


handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())

logger = logging.getLogger("training")
logger.handlers.clear()
logger.addHandler(handler)
logger.setLevel(logging.INFO)
logger.propagate = False

logger.info(
    "epoch_completed",
    extra={"run_id": "run-123", "stage": "train", "worker_rank": 0},
)

In production, populate identifiers from the workload environment rather than hard-coding the example values.

Log exceptions with tracebacks

Use logger.exception() while handling an active exception so the traceback is retained:

try:
    upload_checkpoint()
except Exception:
    logger.exception("checkpoint_upload_failed")
    raise

The final raise preserves failure semantics. Logging an exception and silently continuing can produce a nominally successful run with missing artifacts. If the error is intentionally recoverable, log the retry number, deadline, and eventual outcome.

Control volume and cardinality

Do not emit one log event per sample, token, or tensor element. That can slow the workload, increase storage cost, and hide the events people need. Aggregate frequent progress into metrics and log periodic summaries or state transitions.

Keep identifiers searchable but bounded. Run IDs are useful in logs; user-generated text, full URLs with credentials, raw prompts, and arbitrary labels can create privacy, security, and cardinality problems.

Redact before emission

Assume collected logs will be retained and visible to operators. Never log access tokens, connection strings, cookies, authorization headers, or full environment dumps. Treat prompts, datasets, and model responses according to their data classification.

Redaction is safest at the application boundary before the record leaves the process. Downstream filters are a second defense, not permission to emit secrets.

Connect logs to Polyaxon runs

Polyaxon captures operation logs and associates them with the run's inputs, parameters, status, and artifacts. Use the logging guidance to keep output visible from the run while your platform logging backend handles longer retention and cross-service search.

Log the start and completion of meaningful phases, persist structured reports as artifacts, and use metrics for numeric series. The three signals complement each other: logs explain discrete events, metrics reveal trends, and the Polyaxon run preserves the execution context needed to reproduce the result.