Kubernetes alerting practices for ML platforms

An alert is a request for attention. It should describe a meaningful condition, reach an owner with enough context to act, and stop notifying when the condition no longer requires intervention.

Kubernetes can generate thousands of metrics and events, but converting each abnormal value into a page creates noise. ML platforms need alerts that distinguish user impact, workload failure, shared-capacity risk, and useful diagnostic evidence.

Define the response before the threshold

For every alert, answer four questions:

1. Who owns the response?
2. How quickly must they act?
3. What concrete action can change the outcome?
4. What evidence should the notification include?

If no action is available, the signal may belong in a dashboard or report. If the condition can wait for office hours, create a ticket-level notification instead of paging. The Prometheus alerting guidance recommends keeping alerts simple and focused on symptoms.

Use severity to encode response expectations, not how surprising a metric looks. A 95% CPU value can be harmless; a critical inference endpoint returning errors is urgent even when CPU is low.

Separate symptoms from causes

Page on symptoms near the user or business outcome:

• sustained inference error rate or latency outside its objective;
• a production model service with insufficient ready capacity;
• a critical scheduled operation that missed its completion window;
• a priority training queue unable to start eligible work;
• an observability pipeline that can no longer verify platform health.

Use cause-oriented alerts for diagnosis or proactive work: node pressure, a full persistent volume, repeated image pulls, CSI failures, or one container restarting. Causes can fan out during a single outage. Alertmanager inhibition can suppress downstream notifications when a cluster- or service-level alert already explains them.

Account for batch and training lifecycles

Online-service alerts assume continuous traffic and replicas. ML operations have different semantics. A Job that finishes is healthy; a Pod that disappears may be expected; a training process can run for hours without producing a checkpoint every minute.

Alert on lifecycle expectations instead:

• queued longer than the service objective despite eligible capacity;
• startup exceeds the normal image, volume, and initialization window;
• active work makes no domain-specific progress for a sustained period;
• retry count or failure rate crosses a reviewed threshold;
• a scheduled pipeline misses more than one acceptable execution;
• checkpoint age threatens the recovery objective.

Use Polyaxon operation state and metadata to distinguish expected completion from failure. A generic “Pod not running” rule cannot make that distinction.

Add time and volume context

Short spikes and controller transitions are normal. Prometheus alerting rules support a for duration so a condition must remain active before it fires:

groups:
  - name: ml-platform-capacity
    rules:
      - alert: PriorityWorkloadsCannotStart
        expr: ml_priority_queue_oldest_seconds > 900
        for: 10m
        labels:
          severity: page
          team: ml-platform
        annotations:
          summary: "Priority ML work has been waiting for eligible capacity"
          runbook_url: "https://runbooks.example.com/ml/queue-capacity"

Choose the window from the workload's normal behavior and response objective, not an arbitrary round number. Include volume as well as ratios: one error out of one request should not look the same as thousands of failures.

Route, group, and inhibit notifications

Alertmanager groups related alerts, routes them to receivers, applies silences, and inhibits redundant notifications. The Alertmanager concepts describe these controls.

Use stable routing labels such as team, service, cluster, environment, and severity. Do not route on Pod name or unique run ID; those values fragment incidents into many notifications. Put specific affected objects in annotations, dashboards, or linked searches.

Group a node outage rather than sending one notification per affected Pod. Inhibit workload-level symptoms when an unreachable cluster already has an active page, while preserving the details for investigation.

Make every alert self-checking

An alerting system can fail silently. Monitor rule evaluation, scrape health, ingestion delay, Alertmanager availability, notification errors, and an external end-to-end heartbeat. Test that a synthetic rule reaches the intended receiver and can be acknowledged.

Also test alert resolution. A notification that fires correctly but never resolves trains responders to ignore state. Record ownership changes and review stale runbook links.

Include investigation context

A notification should state what is affected, when it started, the observed value, the expected range, and where to investigate. Link to a dashboard with the cluster and namespace preselected, plus a runbook that starts with safe read-only checks.

For a specific Pod, responders can inspect recent events without changing state:

kubectl events \
  --context acme-production \
  --namespace ml-team \
  --for pod/training-job-r8m4x

Events are supplemental and have limited retention. Preserve the relevant event stream in the observability system when incident response depends on historical evidence.

Connect alerts to Polyaxon ownership

Polyaxon connects Kubernetes objects to projects, operations, components, users, queues, and artifacts. Use that context to route platform problems to platform owners and operation-specific problems to the responsible team.

Build alerts around the outcomes the platform promises: timely starts for priority work, reliable execution, checkpoint freshness, service availability, and controlled capacity. Keep low-level signals available for diagnosis, but reserve interruptions for conditions where a person can protect those outcomes.