Trace a failed training run to an unhealthy GPU

A training worker crashes with a device error. Its replacement fails again, but another run using the same image succeeds elsewhere. The useful question is no longer just whether the application failed: which allocation did it use, and what did the platform know about that device at the time?

Kubernetes Dynamic Resource Allocation (DRA) supplies identifiers that connect a workload to its allocated devices. Device health reporting adds another piece of evidence when the driver supports it. It does not make every application error a hardware fault or automatically recover a training job.

Start with DRA claims and capability requests if the allocation model is new to you. This article follows the evidence after allocation.

Establish which telemetry exists

Two DRA mechanisms have different purposes. Drivers can populate ResourceClaim.status.devices with information about allocated devices; this facility is stable in Kubernetes 1.37. Health reporting through ResourceHealthStatus is beta since 1.36 and enabled by default, but the driver must implement the DRAResourceHealth service. The kubelet then exposes health through each container's allocatedResourcesStatus. DRA observability documentation.

Verify the driver, Kubernetes version, and enabled features before interpreting empty fields. A cluster using legacy device-plugin GPU requests follows a different allocation path. It will not necessarily have ResourceClaims to inspect.

Ask the platform team what the installed driver can detect. Memory errors, device disappearance, and vendor-specific failures do not necessarily have identical coverage. Application-level failures still need their own logs and timestamps.

Preserve the allocation before it disappears

For an affected Pod, collect its identity, node, claim references, and container status. These read-only commands use example names; replace them with the actual worker and namespace:

kubectl get pod training-worker-0 -n ml-team \
  -o jsonpath='{.metadata.uid}{"\n"}{.spec.nodeName}{"\n"}{.spec.resourceClaims}{"\n"}{.status.resourceClaimStatuses}{"\n"}'

kubectl get pod training-worker-0 -n ml-team \
  -o jsonpath='{.status.containerStatuses}{"\n"}'

For template-generated claims, use the resolved claim names from Pod status. For directly referenced claims, follow the name in the spec. Then inspect the corresponding allocation:

kubectl get resourceclaim training-device-claim -n ml-team -o yaml

training-device-claim is a placeholder for the name you just found. Preserve the driver, pool, and device identifiers in status.allocation.devices.results, together with the request that selected them. The ResourceClaim API defines these fields.

A device's DRA name is not necessarily the hardware serial number shown by a vendor monitoring tool. Record the driver's documented mapping rather than joining unrelated labels that happen to look similar.

Build a timeline instead of guessing from one status

Use an evidence record such as this one. It is a proposed incident template, not observed output:

Correlate timestamps from systems whose clocks are reasonably aligned. A device report collected after a failure is useful, but timing alone does not establish causation. Compare the application error with driver and node evidence.

Kubernetes marks device health Unknown when reporting times out. It also stops updating health in a terminated Pod's status. Claim device-status entries can disappear when allocation ends. Collect evidence promptly and retain it outside resources that the workload controller may clean up. Health reporting lifecycle.

Distinguish three operational cases

A health signal reports what that mechanism can observe. “Healthy” does not certify a model's correctness, while “Unknown” is not a diagnosis of broken silicon.

Separate detection from action. DRA device taints are a distinct mechanism for affecting allocation or eviction. An unhealthy status alone is not a promise that the device has been excluded or the workload moved. The platform needs a deliberate remediation policy and a controller or operator to carry it out.

Connect the incident to the Polyaxon run

Polyaxon gives the hardware investigation a common reference: the affected training run. Save the redacted Kubernetes events, claim status, and driver observations as an artifact bundle. Record the node, device identity, driver version, model revision, and failure time as run metadata, keeping each observation attributable to its collection time and source.

This makes concrete comparisons possible: the same recipe on different devices, different recipes on one device, or runs before and after a driver update. Save a follow-up run's device identity and outcome too, so the team can assess whether a placement change resolved the failure. The evidence collector or operator supplies the DRA health fields; these steps do not depend on an automatic DRA telemetry import.

Once the platform team has excluded the unhealthy capacity or validated a repair, use Polyaxon's resume or restart workflow with the appropriate placement preset and the application's checkpoint recovery contract. A restart alone does not guarantee a different device. Verify the new allocation and useful training progress, then retain the recovery result alongside the incident evidence.