Gang scheduling for distributed training

A distributed training job may need four workers to make progress. Starting two workers is not necessarily half a success: those workers can hold GPUs while waiting for the other two. If another job holds the remaining capacity in the same way, neither application can proceed.

Gang scheduling treats related workers as a group with a minimum resource or membership requirement. Its purpose is to avoid committing scarce capacity to an incomplete group that cannot do useful work. The important detail is where that guarantee is enforced: queue admission, pod placement, or a readiness-and-retry mechanism.

Why partial placement wastes capacity

Consider an illustrative cluster with eight GPUs and two fixed-size jobs that each require eight one-GPU worker pods. Without suitable group coordination, four pods from each job might start. The cluster is fully allocated, but both applications are waiting for their remaining workers.

Admitting one complete job first allows progress. The second job waits without holding a partial allocation. This is the resource-management problem gang scheduling addresses; it does not imply that all processes start at the same instant.

Image downloads, initialization, rendezvous, and data loading still happen after placement. A group can satisfy its scheduling policy and then fail at any of those stages. Monitor application readiness as well as scheduling success.

Distinguish three kinds of coordination

These mechanisms can complement one another, but their guarantees are not interchangeable. Ask what the installed controller actually checks and what it does when the check fails.

For example, Kueue's all-or-nothing scheduling documentation separates workload quota reservation, topology-aware admission, and waitForPodsReady. The readiness mechanism can evict and requeue a workload that does not become ready before its timeout; it limits the duration of partial scheduling rather than preventing every partial start.

Minimum membership is not a GPU count

A group policy typically expresses how many member pods must be available, with any additional resource requirements defined separately. Count the members the controller actually creates, not just the number of accelerators in a diagram.

For example, four worker pods requesting two GPUs each represent four members and eight GPUs. A controller may also create a launcher or another role with different requirements. Check whether that role belongs to the group before setting its minimum.

The Kubernetes scheduler-plugins Coscheduling implementation uses a PodGroup with minMember and a pod-group label to identify membership. Those fields belong to that integration; they are not universal keys that every scheduler interprets.

For a fixed-size application, the group must include the workers needed for progress. Reducing the minimum to make a job start sooner is safe only when both the application and its controller support the resulting elastic behavior. A scheduling setting alone does not make a fixed-world-size training script elastic.

Resource shape still matters

Gang scheduling does not combine several small devices into a larger-memory device, and it does not make separate nodes behave like one machine. Each pod still needs an eligible node with the resources it requests.

Check the whole placement contract: accelerator type, per-pod GPU count, CPU and host memory, node selectors, tolerations, storage access, and communication topology. An eight-GPU total spread across several nodes may not satisfy a pod requiring eight GPUs on one node.

Avoid removing genuine locality constraints simply to make admission easier. Instead, compare the expected communication cost and completion time of the alternative placement. The GPU utilization guide explains why using more devices can reduce duration while increasing resource-hours per result.

What can be configured through Polyaxon

Polyaxon's existing Kubeflow runtimes expose scheduling policy for TFJob, PyTorchJob, and MPIJob. With a compatible controller and scheduler integration, schedulingPolicy.minAvailable supplies the relevant minimum-membership setting.

For an existing PyTorchJob whose required group consists of four pods, the policy portion can look like this:

# Fragment of an existing, otherwise complete PyTorchJob component.
# This does not install or configure a gang scheduler.
run:
  kind: pytorchjob
  schedulingPolicy:
    minAvailable: 4

Keep the actual replica definitions and resource requests in the workload, and inspect the generated group to confirm that its members and minimum match the intended application. See the PyTorchJob specification for the surrounding runtime structure.

Do not confuse this existing scheduling policy with Kubeflow Trainer's newer runtime PodGroupPolicy. Upstream Kubeflow Trainer uses that policy to have its controller create the appropriate PodGroup for a TrainJob. Polyaxon's current integration assessment does not include first-class generic runtime PodGroupPolicy support.

KAI can be selected through environment.schedulerName, but scheduler selection alone does not create every required group or queue configuration. Volcano and Coscheduling paths depend on compatible workload-controller integration. The scheduler comparison contains the full support boundaries, including Kueue and Slurm Bridge.

Debug a group that never becomes ready

Follow the workload from its parent object to its member pods:

1. Confirm that the expected job and group objects exist in the intended namespace.
2. Check that the owning controller and scheduler recognize those object versions.
3. Verify member labels, scheduler selection, required pod count, and per-pod resources.
4. Read admission conditions and scheduling events for quota, eligibility, or fit failures.
5. If placement succeeds, inspect image pulls, storage mounts, worker logs, and application rendezvous.
6. Confirm what happens at the scheduling or readiness timeout: release, retry, backoff, and eventual failure reporting.

An impossible resource shape will not become possible through rapid retries. Nor will a longer readiness timeout repair a bad container image. Choose timeout values from observed startup behavior, then retain a clear failure path for persistent problems.

Test the failure path before expanding adoption

A useful validation suite includes a complete group that fits, one that cannot fit, two competing groups, and a group with one worker that fails to initialize. Also test cancellation while waiting and interruption after training has begun.

Measure time until all required workers are ready, resource-hours held before useful progress, and cleanup after failure. Verify that one repeatedly failing workload cannot consume the queue indefinitely.

Finally, test checkpoint recovery independently. A gang policy manages resource coordination; it does not preserve training progress. Pair it with explicit application recovery and a clear orchestration lifecycle so that both successful and interrupted runs release resources predictably.