Understand Kubernetes workload-aware preemption

An urgent training run needs several workers before any of them can make progress. Freeing room for just one worker may interrupt other jobs without making the urgent run useful. The scheduler needs to consider the incoming group and the work it might disrupt.

Workload-aware preemption does that for native Kubernetes PodGroups. It is beta in Kubernetes 1.37 and disabled by default, under the GenericWorkload feature gate. It requires the native workload API; installing a scheduler with its own group objects does not enable the same API automatically. Workload-aware preemption documentation.

This article builds on native gang scheduling. Gang placement describes what must fit; preemption considers whether displacing eligible lower-priority work can make that placement possible.

Think in units of useful progress

Consider a cluster hosting a four-worker synchronous training job and several independent evaluation workers. Losing one evaluation worker delays one shard. Losing one training worker may stop the remaining three from making progress, depending on the training application's recovery model.

The cost of disruption therefore cannot be inferred from the number of terminated Pods alone. Ask what useful work survives, how much work must be repeated, and what capacity remains occupied while recovery happens.

These are application questions to answer before assigning priority:

For a sandbox, “low priority” is an operational choice with a user-visible cost. It should not be inferred merely because the workload is interactive rather than a batch Job.

Understand what the scheduler considers

Workload-aware preemption treats the incoming PodGroup as one preemptor and looks for victims across nodes. It accounts for victim groups and their disruption settings instead of considering every member independently. Priority is central, with additional ordering rules for workload type, group size, and start time. The algorithm does not measure checkpoint freshness or the monetary value of lost training progress. Upstream algorithm.

Do not interpret a priority increase as guaranteed start time. Constraints still have to permit a placement, victims take time to terminate, and competing work can change the situation. Ordinary Pod preemption likewise distinguishes a scheduling decision from capacity becoming available.

That distinction matters for an incident: “victims selected” is not the same milestone as “all training workers ready.” Record both when diagnosing a delayed urgent run.

Choose the victim group's disruption behavior

A PodGroup can use Single, the default, to allow individual member disruption, or All to require its members to be disrupted together. These settings describe scheduler disruption behavior; they do not implement application recovery. PodGroup disruption and priority.

For an application that cannot use surviving workers, All may avoid leaving a broken group consuming capacity. It can also make a larger disruption necessary. For independent shards, allowing a single member to be displaced may preserve more useful work.

This native Kubernetes 1.37 example expresses a four-member training group using an existing platform-defined PriorityClass:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: restart-together-training
  namespace: ml-team
spec:
  priorityClassName: batch-training
  disruptionMode: All
  schedulingPolicy:
    gang:
      minCount: 4

The workload controller must create the member Pods with this membership and the matching priority:

spec:
  priorityClassName: batch-training
  schedulingGroup:
    podGroupName: restart-together-training

These are policy fragments, not a runnable training job. Supply the application, resources, controller, and recovery behavior separately. Kubernetes 1.37 requires the member Pods' priorities to agree with their group; a mismatch can prevent scheduling. The workload scheduling release notes provide the version context.

Keep recovery and protection explicit

Group disruption does not create a consistent checkpoint. A training application must write recoverable state, publish it durably, and restart from a valid checkpoint. The termination grace period is a limited opportunity to finish cleanup, not a guarantee that a large checkpoint will complete.

Likewise, a PodDisruptionBudget is not an absolute preemption shield: Kubernetes documents its handling during preemption as best effort. Avoid promising users that a PDB makes a session or training run non-interruptible.

Evaluate a proposed policy with a controlled failure exercise in an appropriate environment. Record lost work, time until victims release resources, time until the incoming group becomes useful, and the displaced workload's recovery. A lower count of terminated Pods can still produce a worse result if the survivors are stranded.

The training recovery guide explains the application side; the Pod eviction guide distinguishes other disruption paths.

Coordinate priority and recovery with Polyaxon

Polyaxon exposes priority controls at both the queue and Kubernetes scheduling layers. Queue priority determines the relative order in which operations reach Kubernetes; workload environment settings can select a Kubernetes PriorityClass for placement and preemption. In Polyaxon EE and Cloud, queues also provide routing and concurrency controls.

A team might give an interactive debugging notebook higher urgency than a background training sweep, while allowing the sweep's restartable trials to use spare capacity. Polyaxon's resume and restart workflows help recover interrupted operations and reuse saved artifacts. Training code must write and restore checkpoints; priority alone cannot preserve application progress.

For native workload-aware preemption, additionally map the policy to the PodGroup and its members through the supported workload integration. Confirm which component creates the group, assigns member priorities, and reports interruptions, including any Kueue, KAI, or Volcano integration. Start with one checkpointed Polyaxon recipe and measure the work lost and recovered before expanding the policy.