Export and alert on Kubernetes events

Kubernetes events report notable state changes and controller decisions: a Pod could not schedule, an image pull failed, a volume could not mount, a probe failed, or a container started. They are often the shortest path from a visible symptom to the component that observed it.

They are also best-effort, supplemental data with limited retention. A production platform should consume events promptly, preserve the fields needed for investigation, and avoid paging on every Warning.

Read events in context

Use kubectl events for a resource-focused live view:

kubectl events \
  --context acme-production \
  --namespace ml-team \
  --for pod/training-job-r8m4x

Or inspect recent Warning events in the namespace:

kubectl events \
  --context acme-production \
  --namespace ml-team \
  --types=Warning

Read the reporting controller, reason, note or message, involved object, count or series, and timestamps together. A reason such as FailedMount identifies a stage; the message explains the observed failure.

The Kubernetes Event API reference cautions consumers not to depend on reasons and timing as permanent contracts. Versions and implementations can change event wording.

Export before evidence expires

An event exporter watches the API and sends records to a durable log, event, or observability backend. Deploy it with the narrowest RBAC that covers the required namespaces and event APIs. Monitor watch restarts, resource-version recovery, API throttling, dropped exports, queue depth, and destination failures.

A useful pipeline has four stages:

1. Watch events and recover cleanly after disconnection.
2. Normalize stable resource and cluster metadata.
3. Deduplicate repeated observations without erasing frequency.
4. Route durable records to search, retention, and alert evaluation.

Do not assume the exporter receives exactly one record per underlying condition. Controllers may aggregate repeats, emit several related events, or change messages between releases.

Preserve stable fields

Store fields that let responders reconstruct scope:

• cluster and source API endpoint identity;
• namespace, involved-object kind, name, and UID;
• reporting controller and instance;
• event type and reason;
• first and last observed timestamps;
• repeat count or series information;
• the human-readable note;
• owner references resolved at collection time where practical.

The object UID distinguishes two Pods that reused the same generated name or a resource recreated under the same name. Resolve Deployment, Job, StatefulSet, or operator ownership so incidents can be assigned to the declaring layer.

Control sensitive data and cardinality

Event messages can contain image references, node names, paths, provider identifiers, and fragments returned by external systems. Classify event records, redact known sensitive patterns close to collection, encrypt transport and storage, and restrict search access.

Do not convert object UID, message text, or Pod name into general Prometheus labels. Export counts with bounded dimensions such as cluster, namespace, reason, workload class, or team, while keeping detailed records in a log or event store.

Retention should follow investigation needs and data sensitivity. Longer is not automatically better.

Alert on patterns and impact

A single Warning event is usually too weak for a page. Controllers retry normal transitions, and short-lived failures can self-heal. Create alerts from sustained patterns or user-facing impact:

• repeated scheduling failures for priority operations beyond a queue objective;
• volume errors preventing a production service from reaching ready capacity;
• image-pull failures across a new node pool or release;
• a rising eviction rate combined with node pressure;
• event-export silence when the cluster is otherwise active;
• repeated operation failures sharing one reason and infrastructure boundary.

Group notifications by cluster, workload owner, and reason rather than by Pod. Link to the detailed event search and related metrics.

Correlate events with other signals

Events explain selected Kubernetes decisions; they do not provide continuous resource history or application semantics. Combine them with:

• metrics for duration, rate, saturation, and fleet-wide trends;
• logs for application and component detail;
• traces for request paths;
• audit logs for who changed an API object;
• Polyaxon state for the intended ML operation and its lifecycle.

For example, a Pod may emit FailedScheduling because no node matches its request. Prometheus can show GPU capacity by pool, and Polyaxon can show queue priority, requested preset, and how long the operation has waited.

Build reliable event alerts with Polyaxon

Enrich events from Polyaxon-managed resources with project, operation, workload role, owner, and scheduling class. Keep unique run identity in the durable event record, not a fleet-wide metric label.

Use the combined view to answer: Which ML work was affected? Was the event transient or sustained? Did a retry succeed? Is the cause specific to one operation, image, node pool, storage class, or cluster?

Events are most valuable as timely clues. Export them so those clues survive, then connect them to the telemetry and operation context required to make a safe decision.