Use kube-state-metrics for ML platforms
Monitor Kubernetes object state for ML workloads while separating desired state, resource usage, application outcomes, and high-cardinality metadata.
kube-state-metrics listens to the Kubernetes API and exposes metrics derived from object state. It can show that a Deployment wants four replicas but has two available, a Pod is waiting for a scheduling reason, a Job failed, or a node reports a condition.
It does not measure container CPU, GPU activity, request latency, or model quality. Those questions need resource, application, or ML telemetry. The value comes from correlating the layers.
Separate the metric sources
Several Kubernetes components expose “metrics,” but they answer different questions:
| Source | Primary question |
|---|---|
| kube-state-metrics | What state does the Kubernetes API report for objects? |
| Metrics Server | What recent CPU and memory usage supports autoscaling and kubectl top? |
| Kubelet and container metrics | What resources are containers and nodes consuming? |
| Application instrumentation | What requests, queues, errors, and work outcomes occurred? |
| Accelerator telemetry | What devices are allocated and active, and where is time spent? |
| Polyaxon metadata | Which run, project, inputs, configuration, and artifacts produced the workload? |
Do not call a Pod healthy because its phase is Running. Do not call a training run productive because its GPU is allocated. Each layer supplies one part of the explanation.
Use object-state metrics for control loops
The official kube-state-metrics project exposes metrics for native Kubernetes objects and selected metadata. Useful platform questions include:
- Are desired Deployment replicas available?
- Which Pods are Pending, waiting, or repeatedly restarting?
- Did a Job succeed, fail, or exceed its intended duration?
- Are PVCs bound?
- Which nodes report pressure or unavailable conditions?
- Are resource requests and limits present?
- Is a HorizontalPodAutoscaler at a boundary?
Use metrics to find the object, then inspect its events and status for detailed diagnosis. A metric should route an operator to evidence, not attempt to encode every condition in an alert title.
Control labels and cardinality
Kubernetes labels and annotations are open-ended. Exporting all of them into Prometheus can create high cardinality, raise cost, and expose sensitive metadata.
Allowlist only fields required for stable aggregation or routing, such as platform-owned project, workload class, queue, or node-pool labels. Avoid run IDs, Pod UIDs, user-provided annotations, image digests, and arbitrary experiment parameters as metric labels unless the backend and retention design explicitly support that scale.
Keep high-cardinality identifiers in logs, traces, or Polyaxon run metadata, then link to them from a bounded dashboard.
Design alerts around sustained mismatch
Alert on conditions that require action and persist beyond expected reconciliation:
- desired replicas unavailable beyond rollout time;
- Jobs failing or not completing within their deadline;
- Pods unschedulable beyond the queue's expected delay;
- PVCs unbound beyond provisioning time;
- node pressure reducing eligible capacity;
- repeated container waiting or termination reasons.
Use different thresholds for interactive services, short experiments, distributed training, and scheduled batch work. A Pending Pod may be normal while it waits in a controlled queue but urgent when it represents lost serving capacity.
Group alerts by workload or platform owner rather than sending one page per Pod.
Monitor kube-state-metrics itself
An absent series can mean the object disappeared, the scrape failed, the component lacks permission, or a metric changed between versions. Monitor target health, scrape duration, collection errors, memory, CPU, and API-server pressure.
Match the kube-state-metrics release to supported Kubernetes versions and review metric stability before upgrades. Dashboards and rules are consumers of an API; validate them against the new release rather than assuming every metric name and label remains unchanged.
Restrict access
Object-state metrics can reveal namespaces, workload names, image references, labels, and infrastructure shape. Expose the metrics endpoint only to authorized collectors, grant the component the minimum RBAC it needs, and separate tenant-facing dashboards from cluster-wide raw data.
Metric access is part of the platform's information-security model. A user who cannot list Secrets should not gain sensitive operational context through an unrestricted monitoring endpoint.
Connect Kubernetes state to ML outcomes
When a run is slow or fails, correlate:
- queue and admission state;
- Pod scheduling and Kubernetes object conditions;
- node, container, network, storage, and accelerator telemetry;
- application logs and metrics;
- Polyaxon inputs, configuration, run status, and artifacts.
Our guide to Kubernetes metrics for ML covers that broader hierarchy. Polyaxon platform observability adds deployment context around the components serving and scheduling workloads.
kube-state-metrics is most useful when its role stays narrow: translate API object state into queryable time series, then join those series with the signals that explain resource use and useful work.