Monitor Amazon EKS for ML workloads
Build layered Amazon EKS monitoring for control-plane activity, Kubernetes state, nodes, GPUs, applications, ML runs, and telemetry health.
Amazon EKS manages the Kubernetes control plane, but the customer still owns workload observability, worker capacity, add-ons, applications, and the response to most operational signals.
An effective monitoring design separates these layers and then correlates them around one user-visible outcome: can teams run and serve ML workloads reliably, efficiently, and safely?
Map the responsibility layers
AWS documents the available signals in its Amazon EKS monitoring and logging guidance. Organize them by responsibility:
| Layer | Questions to answer |
|---|---|
| AWS and EKS control plane | Are API, audit, authenticator, controller, or scheduler events showing failures or policy changes? |
| Kubernetes object state | Are desired workloads admitted, scheduled, ready, and complete? |
| Worker infrastructure | Are nodes healthy, eligible, updated, and sufficiently provisioned? |
| Accelerators | Are GPUs allocatable, assigned, active, memory-constrained, or reporting hardware errors? |
| Application | Are requests, batches, retries, latency, throughput, and errors within objectives? |
| ML workflow | Which project, run, code, data, model, parameters, and artifacts explain the outcome? |
Do not rely on the AWS service-health view alone. A healthy managed control plane does not mean a node group can satisfy a GPU request or that a model server is ready.
Enable useful control-plane logs
EKS can export selected Kubernetes control-plane log types to CloudWatch Logs. Enable the categories your security and operating model require, including API, audit, authenticator, controller-manager, and scheduler evidence.
Define retention and access before enabling broad collection. Audit logs can be voluminous and may expose resource names, identities, request bodies, and operational patterns. Route them to a protected destination and alert on meaningful security or availability conditions rather than every event.
Keep AWS CloudTrail for AWS API activity and Kubernetes audit logs for Kubernetes API activity. Correlate identities across both boundaries when investigating changes to clusters, node groups, access, or workloads.
Monitor Kubernetes state separately from usage
Object-state metrics answer whether Deployments have available replicas, Jobs complete, Pods wait, PVCs bind, and nodes report conditions. Resource telemetry answers how much CPU, memory, disk, network, and accelerator capacity is used.
Those are different questions. A Pending Pod may consume no resources while representing a serious capacity or policy problem. A running GPU Pod may hold an accelerator while doing little useful work.
Use kube-state-metrics for ML platforms to expose API object state and keep node or container usage in its appropriate telemetry pipeline.
Watch worker and accelerator capacity
Track node readiness, pressure conditions, allocatable resources, requested resources, instance and availability-zone distribution, autoscaling decisions, launch failures, and interruption notices for capacity types that can be reclaimed.
For GPU nodes, add device-level telemetry and correlate:
- allocatable versus requested GPUs;
- device activity and memory use;
- temperature, power, throttling, and hardware errors;
- interconnect and network behavior for distributed training;
- queue wait and useful application throughput.
Avoid a universal utilization target. Interactive inference, bursty evaluation, and synchronized distributed training have different healthy patterns. Our GPU utilization metrics guide separates allocation, activity, and throughput.
Observe applications and ML outcomes
Instrument services for latency, traffic, errors, saturation, dependency calls, and business or model outcomes. Training and batch workflows need submission-to-start delay, duration, completion, retries, checkpoint progress, input throughput, and accepted evaluation results.
Attach stable dimensions such as cluster, environment, service, workload class, queue, and model family. Keep unique run IDs and high-cardinality experiment parameters in logs, traces, or Polyaxon metadata rather than broad metric labels.
Polyaxon tracks the operation context that infrastructure tools do not: code, inputs, parameters, environment, metrics, lineage, and artifacts. This lets an operator move from a cluster symptom to the affected runs without treating a Pod name as the primary ML identity.
Control collection cost and sensitive data
Cloud telemetry cost depends on ingestion volume, retention, queries, and metric dimensions. Filter low-value logs, bound label cardinality, choose explicit retention by data class, and sample traces with understood consequences.
Use workload identity instead of long-lived shared AWS credentials. Restrict access to raw logs and metrics because Kubernetes and ML metadata can reveal tenant names, image references, storage locations, source paths, prompts, or customer data.
Monitor the collectors themselves: dropped records, export failures, queue growth, throttling, scrape errors, and configuration changes. Missing telemetry must not appear as a healthy zero.
Build alerts around objectives
Page for sustained user-impacting conditions: submissions fail, priority workloads cannot start, serving capacity falls below its objective, artifact access blocks a workload class, or the telemetry and alerting path is unavailable.
Use lower-severity notifications for approaching quotas, uneven capacity, upgrade drift, node pressure, or rising cost. Group alerts by shared cause and include the EKS cluster, namespace, workload owner, and next diagnostic link.
Polyaxon platform observability complements EKS telemetry by covering the services that manage and track workloads. Together, the layers should explain the path from an AWS or Kubernetes event to a specific ML outcome and an owned response.