Native gang scheduling reaches beta in Kubernetes 1.37

A distributed training job needs four workers, but the cluster can currently place only three. Starting those three can occupy GPUs while the application waits for a worker that has nowhere to run. Native gang scheduling gives Kubernetes a group-level placement rule for this situation.

The upstream capability has moved beyond its initial preview: Workload and PodGroup APIs, including native gang scheduling, reached beta in Kubernetes 1.37 and remain disabled by default. Kubernetes 1.37 was released on August 26, 2026. This article describes the documented state as of September 20, 2026. See the release record and upstream scheduling update.

Our existing gang-scheduling guide covers the broader problem and scheduler integrations. This follow-up examines the native API, where it could help Polyaxon training and sandbox workflows, and how it compares with KAI, Kueue, and Volcano.

Give the scheduler an explicit group

A native PodGroup holds the scheduling policy and runtime scheduling state for related Pods. A Workload supplies reusable policy templates; controllers can create a PodGroup for each execution. You can also define a standalone PodGroup directly. The beta resources use scheduling.k8s.io/v1beta1. PodGroup API documentation.

For an illustrative training group requiring four worker Pods, the group declaration is:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-workers
  namespace: ml-training
spec:
  schedulingPolicy:
    gang:
      minCount: 4

This declares the scheduling group; it does not create workers. The namespace must already exist. Each worker's Pod specification links to the group:

# Fragment of each worker Pod's spec.
schedulingGroup:
  podGroupName: training-workers

Keep the workers in the same namespace as the PodGroup. Their container images, resources, networking, and training configuration are still required. The Pod's schedulingGroup field is immutable, so membership is decided at creation time. A reference to a missing group leaves the Pod pending. Scheduling group documentation.

minCount counts Pods. Four workers requesting two GPUs each still mean minCount: 4, with eight GPUs requested through the workers' resource specifications. Include any additional required roles deliberately; the group definition should match how the application makes progress.

Wait for membership, then evaluate placement

Native gang scheduling has two distinct checks. First, the referenced PodGroup must exist and enough member Pods must have been created. Then the scheduler evaluates whether enough members can obtain valid placements. During initial placement, it does not bind a smaller group than the configured minimum. Gang-scheduling documentation.

For a fresh group with minCount: 4, consider these illustrative cases:

Created member Pods	Feasible worker placements	Result
3	3	Wait for the fourth member to exist
4	3	The minimum cannot be placed; keep the group pending
4	4	The group satisfies the minimum placement requirement

A cluster's total free GPU count is only part of feasibility. Each worker still needs a node that satisfies its resource requests and placement constraints. Four available GPUs spread across incompatible node pools do not necessarily provide four usable worker placements.

The minimum is a threshold, not necessarily the total group size. For six members with a minimum of four, the scheduling policy does not require all six to fit. Choose a lower minimum only if the workload supports running that way. The alternative basic policy groups Pods without requiring a minimum feasible membership. Scheduling policies.

Scheduling success is only the beginning

After placement, workers still need to pull images, mount data, initialize devices, and join the distributed application. A correct scheduling decision can be followed by an image error or a failed rendezvous. Those failures require different evidence from an unschedulable group.

The native PodGroup status records the initial scheduling result. It does not continuously report the group's operational health or update to reflect every later failed or evicted worker. The PodGroup lifecycle documentation describes this limitation.

Read the group status alongside current Pod states, events, and application logs. A fixed-size training job still needs a controller and recovery policy for losing a worker after startup. Gang scheduling does not restore a checkpoint or make an application elastic.

A useful evaluation of the feature includes both a group that cannot initially fit and a group that fits but subsequently loses a worker. Record whether the remaining processes can progress, how the controller replaces members, and when resources are released. Our training-recovery guide covers the separate application-state problem.

Where this could help Polyaxon workloads

The useful grouping boundary is the smallest set of Pods that must run together for an application to make progress. The following scenarios explain where native gang scheduling could fit; they require the workload controller to create native PodGroups and assign Pod membership. Enabling a Polyaxon runtime or sandbox plugin alone does not establish that integration.

Distributed training and fine-tuning

Consider a Polyaxon PyTorch training run with four worker Pods, each requesting two GPUs. If the training process requires all four workers, placing only three occupies six GPUs without letting training advance. A gang minimum covering all required worker Pods lets the scheduler wait for a feasible placement for the complete training group.

Apply this reasoning to each distributed run, including fine-tuning runs submitted during an experiment sweep. Independent hyperparameter trials can make progress separately, so grouping the entire sweep into one gang would unnecessarily delay useful work. A distributed trial may need its own gang; unrelated trials do not need to start together.

Interactive sandboxes and distributed debugging

Polyaxon's sandbox plugin provides process, filesystem, and terminal access inside a running service. For a sandbox contained in one Pod, Kubernetes already places that Pod as a unit, including its containers. Gang scheduling adds little to that case; selecting the right queue, resources, and GPUs remains the practical concern.

A distributed debugging session presents a different opportunity. A developer might use a sandbox or notebook to work with four GPU worker Pods that reproduce a multi-node training failure. Grouping those workers could prevent a partial allocation from consuming GPUs while the session waits for the remaining worker. Keep the interactive service outside the worker gang when it is useful independently, such as for editing code or inspecting previous logs.

If an environment truly requires the interactive service and its workers together, a controller could group all required Pods. It must create the required membership before waiting for placement. A sandbox that must start first to create its own gang's workers would otherwise wait for members that cannot yet exist. This is an integration design, not an automatic behavior of plugins.sandbox.

In either design, gang scheduling coordinates placement. Service readiness, worker rendezvous, session termination, and idle culling still need their own policies.

The next layer is a group of groups

Some workloads need several coordinated parts. A parameter-server training application might require one server group and a separate worker group, each with its own minimum membership.

Kubernetes 1.37 introduces CompositePodGroup for this hierarchy. It is alpha and disabled by default, uses scheduling.k8s.io/v1alpha3, and requires the CompositePodGroup feature gate. A composite gang policy specifies minGroupCount, which counts child groups rather than individual Pods. CompositePodGroup API.

For example, a parent could require both of these children:

Group	Requirement
Parameter servers	One schedulable server Pod
Workers	Four schedulable worker Pods
Parent composite group	Both child groups must satisfy their policies

This preserves the structure of the application instead of flattening every role into one undifferentiated Pod count. It is a scheduling hierarchy, not a pipeline dependency graph: it coordinates placement, not which training step runs after another finishes.

Upstream identifies core API stability, maturation of hierarchical scheduling, and broader controller integration as next steps. Those are development goals, not a promised availability date for a particular controller or managed Kubernetes service. Upstream roadmap.

How it compares with KAI, Kueue, and Volcano

Native gang scheduling adds a shared Kubernetes API for coordinating Pod placement. Choosing a cluster scheduling system also involves admission, fairness, GPU sharing, and workload lifecycle. Those requirements explain why the native feature does not, by itself, replace the following tools.

Option	Main role	What to evaluate for Polyaxon workloads
Native Kubernetes gang scheduling	Enforce a minimum feasible group through the native PodGroup API and scheduler support	A candidate when coordinated placement is the missing requirement and the workload controller supports the native API. Team quotas and admission still need policies of their own.
KAI Scheduler	GPU-oriented scheduling with gang placement, hierarchical queues, fairness, GPU sharing, and topology-aware placement	Relevant when interactive GPU services and large training runs compete for shared capacity. Check the installed KAI version, queue configuration, and grouping integration for each workload kind.
Kueue	Decide which workloads can consume quota, with admission, fair sharing, resource flavors, and preemption policies	Relevant when teams need controlled access to capacity before work starts. Pod-to-node binding remains the scheduler's responsibility; verify support for the actual Kubernetes objects Polyaxon submits.
Volcano	Batch scheduling with gang policies, queues, resource sharing, preemption, and workload controllers	Relevant when the cluster already relies on Volcano's batch policies or needs its broader scheduling stack. Confirm how the selected training controller creates Volcano's scheduling groups.

Kueue also has topology-aware scheduling and an optional all-or-nothing readiness mechanism. The latter can cancel admission and requeue a workload when its Pods fail to become ready within a timeout. That is different from a scheduler refusing an initial placement below the gang minimum. Admission and placement mechanisms can serve complementary roles, but their controller integrations must support the combination.

For Polyaxon, distinguish configuration from integration. The environment specification lets a workload select an installed scheduler through schedulerName. That selection does not install KAI or Volcano, create the required scheduling groups, or configure Kueue admission. Likewise, a Polyaxon queue is not automatically a Kueue LocalQueue or a KAI/Volcano queue.

Keep a working scheduler integration until the native path covers the policies and workload kinds you depend on. Our GPU scheduling tools comparison goes deeper into the Polyaxon integration boundaries.

Check the controller path before adopting it

For Kubernetes 1.37, the upstream setup requires GenericWorkload on the relevant control-plane components and the native beta API to be enabled. Confirm that the scheduler's gang-scheduling support is active. Job-controller integration has its own WorkloadWithJob feature gate; installing a newer Kubernetes version alone does not connect every workload controller to the API. Follow the version-specific setup guidance.

For an existing ML platform, trace the actual objects:

Which controller creates the worker Pods?
Does it create native PodGroups and set spec.schedulingGroup, or use another scheduler's integration?
Do the group policy, member count, and per-Pod requests match the application's required workers?
Do all members use compatible scheduler, priority, and preemption settings?
Which controller handles worker failure after initial placement?

The native API should not be confused with a similarly named CRD from another scheduling integration. Polyaxon's existing Kubeflow scheduling policy exposes settings such as minAvailable for supported distributed runtimes. That field is a different API contract from native PodGroup.spec.schedulingPolicy.gang.minCount; existing integration settings should not be assumed to generate the new objects automatically.

For Polyaxon workloads, inspect the generated Kubernetes resources and the installed controller's support before changing scheduler configuration. Keep the run's queue and resource settings connected to this inspection so you can distinguish waiting for platform admission from waiting for group placement.

Native gang scheduling provides a common Kubernetes building block for workloads that need several Pods to make progress. Its practical value depends on the complete path from workload controller to scheduling group to healthy application workers. Start with that path, then evaluate the newer hierarchical model where the application actually needs multiple coordinated groups.