Kubernetes monitoring for ML workloads

Kubernetes can report that a pod is running while the training process inside it is stalled. It can show a fully allocated GPU while the device spends most of its time waiting for data. It can restart an unhealthy container without explaining whether the model service still meets its latency target.

Useful Kubernetes monitoring connects platform health to workload progress and user outcomes. Build the measurement model in layers, then preserve the identifiers needed to move between them.

Monitor five layers

A red control-plane alert can explain many workloads at once. A healthy cluster cannot prove that one model is converging or one inference service is accurate. Keep infrastructure and ML outcome metrics separate, then join them during investigation.

Kubernetes describes observability as collecting and analyzing metrics, logs, and traces in its observability documentation. Events and audit records add important state-change and security context.

Cover control plane and nodes

Monitor API request behavior, scheduler and controller errors, webhook latency, datastore health where accessible, and certificate or quota conditions. Managed Kubernetes may expose a different subset than a self-managed control plane; document what the provider retains and what your team must collect.

At the node layer, collect readiness, pressure conditions, allocatable and requested resources, filesystem capacity, network errors, container-runtime health, and accelerator signals. For GPU nodes, distinguish:

• Allocation: capacity assigned to workloads;
• Activity: whether devices are executing work;
• Throughput: whether the application is completing useful units.

These measurements answer different questions. Continue with GPU utilization metrics for a detailed model.

Observe workload state transitions

Track desired and available replicas, pending and unschedulable pods, container restarts, termination reasons, probe failures, evictions, image pulls, volume attachments, and Job completion. Preserve the controller hierarchy so an alert on a container leads back to its pod, workload, namespace, project, and owner.

Pending time should be segmented. A workload waiting for quota, a node selector, a volume, a missing image, or unavailable accelerators requires a different response. Alerting on “pending pods” without the scheduling reason creates noise.

Kubernetes' resource monitoring guide notes that the resource metrics pipeline is intentionally limited. It supports features such as autoscaling and kubectl top; a complete operational pipeline needs broader system and application telemetry.

Add application and ML progress

Instrument training samples per second, step duration, data-loader time, checkpoint duration, loss or evaluation summaries, and heartbeat age. For batch work, record items completed, remaining work, failures, and retry outcomes. For inference, record request rate, latency distributions, errors, queue depth, batch size, model revision, and resource saturation.

Do not turn every run UUID, prompt, dataset row, or user into a metric label. Use bounded dimensions for aggregate views and place high-cardinality identifiers in logs, traces, or run records. Keep sensitive inputs and outputs out of telemetry by default.

Correlate each signal with environment, cluster, namespace, workload, component, project, and deployment revision. Preserve timestamps consistently so a rollout, node event, and application regression can be compared.

Design logs, events, and traces together

Collect container logs from every relevant node and handle multiline records, rotation, backpressure, and pod deletion. Set log levels by environment and prevent secrets or training data from entering logs.

Retain Kubernetes events long enough for incident investigation and export them if the cluster's default retention is insufficient. Enable and protect audit records appropriate to your security policy. Trace inference and pipeline services across queues, feature stores, model servers, and downstream APIs where distributed latency matters.

Define what happens when the telemetry destination fails. Bounded buffers, rate limits, and drop policies should protect node storage and application performance while making missing data visible.

Alert on symptoms with owners

Page on conditions that require timely action: unavailable serving capacity, repeated workload failure, control-plane impairment, sustained resource pressure, or missed pipeline objectives. Route warnings such as inefficient requests, stale artifacts, and rising telemetry cost to review queues rather than waking an operator immediately.

Every alert needs an owner, severity, evidence links, runbook, and expected response. Test alerts by creating controlled failures and verify both firing and recovery. Review alerts that never fire, always fire, or cannot lead to an action.

Connect Kubernetes to Polyaxon

Use Polyaxon tracking to preserve parameters, metrics, code and environment context for ML runs. Keep detailed logs, checkpoints, and reports in artifacts. Use pipelines to encode recurring monitoring validation and workload checks.

Export stable Polyaxon identifiers into the observability system and link back to the run for high-cardinality detail. Kubernetes monitoring then explains what the platform did, while Polyaxon explains which ML operation ran, what it produced, and whether the result was useful.