Keep distributed training workers close together

Four training workers fit in the cluster, but their GPUs sit behind different network paths. The job can start and still spend too much time exchanging gradients. Resource availability and useful placement are different questions.

Native topology-aware workload scheduling lets a Kubernetes PodGroup require a shared topology domain, such as a rack. It is alpha, disabled by default, and available since Kubernetes 1.36. The example here uses the Kubernetes 1.37 scheduling.k8s.io/v1beta1 PodGroup API; administrators must enable the relevant workload APIs and feature gates, including GenericWorkload and TopologyAwareWorkloadScheduling. Topology-aware workload documentation.

This extends native gang scheduling: gang placement asks whether enough workers fit together, while the topology constraint narrows where they may fit.

Describe the communication boundary

“Close” needs an infrastructure definition. Nodes in one zone are not necessarily on one rack, and a common rack label does not prove that a particular high-speed interconnect is configured correctly.

Start with the workload's communication pattern. A tightly coupled training group repeatedly exchanges data during each step. An independent evaluation shard may communicate mainly with storage. They can justify different placement policies even when both request GPUs.

Ask the platform team which labels represent real physical or network boundaries. Then compare end-to-end runs using the same model, dataset, worker count, and software stack. Preserve communication measurements alongside queue delay and training time. A topology policy is useful only if the application benefit warrants the scheduling restriction.

Work through the capacity tradeoff

Consider four workers that each require one GPU. Assume CPU, memory, device compatibility, and all other constraints are satisfied. These counts are an illustrative scheduling exercise:

Initially, rack A can host the group. In the second snapshot, seven GPUs remain free across the cluster, but no single rack can host four workers. A hard same-rack requirement makes the group wait.

That wait may be justified if a dispersed placement would make the training run much slower. It may also be wasteful for a small experiment whose communication needs are modest. Evaluate total completion time from submission, not only training throughput after allocation.

Combine group size and locality

For a Kubernetes 1.37 cluster configured for these preview features, this native PodGroup requests four workers within one rack domain:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-workers
  namespace: ml-team
spec:
  schedulingPolicy:
    gang:
      minCount: 4
  schedulingConstraints:
    topology:
      - key: topology.example.com/rack

The platform must label eligible nodes with accurate values for topology.example.com/rack. This key is an example, not a built-in inventory of racks. The scheduler requires the group to use a common value; it does not infer that value from cabling.

The workload controller must create the four member Pods in the same namespace and include this Pod-spec fragment:

spec:
  schedulingGroup:
    podGroupName: training-workers

These snippets define group membership and placement, not a training application. Each Pod still needs its container, resources, data access, and distributed-training configuration. The controller must also handle worker identity and recovery. The Kubernetes 1.37 workload scheduling announcement explains the native building blocks and their feature states.

Plan for a replacement worker

Once some group members are placed, a replacement must use their topology domain. It can remain Pending if that domain has no room, even when other racks have spare resources. Locality therefore affects recovery as well as initial scheduling.

For a tightly coupled application, define what should happen when one worker disappears. Can the remaining workers continue, must the group restart, or can the application wait for a replacement? Kubernetes placement does not answer those application questions.

Keep checkpoint recovery separate from placement policy. A successful placement cannot reconstruct optimizer state or make partially written checkpoints usable. The training recovery guide covers that contract.

Choose co-location or spreading deliberately

Topology spread constraints distribute matching Pods across domains. They address a different objective from keeping a communicating group within one domain.

For independent inference replicas, spreading can reduce exposure to a rack or zone failure. Within one distributed inference replica, tightly communicating workers may need co-location. Write the policy around the execution unit: a replica composed of multiple workers is different from several independent replicas.

Do not combine constraints by habit. A hard same-rack group rule and an incompatible spreading requirement can eliminate every placement. Record which requirement is essential and which is a preference before adding more rules.

Evaluate it with Polyaxon workloads

Polyaxon is useful for deciding whether tighter placement actually improves a distributed training recipe. Use node scheduling and scheduling presets to keep placement settings consistent between trials. Keep the model, data, worker count, and resource requests fixed while varying the placement policy supported by your scheduler.

Log the observed topology, queue delay, training throughput, total runtime, and checkpoint behavior with each run using Polyaxon tracking. A faster training step may still produce a slower completed experiment if the tighter placement constraint causes a long wait. Those records give the platform team a basis for deciding which recipes merit a locality-oriented preset.

Native PodGroups require an integration that creates the group and attaches its member Pods. If the cluster uses Kueue, KAI, or Volcano, use that system's supported group and topology integration; the workload objects are not interchangeable. The GPU scheduling comparison explains those choices. Polyaxon supplies the repeatable workload configuration and comparison workflow around the selected scheduler.