Validate GPU networking with NCCL and RCCL

Two nodes can each report healthy GPUs while a distributed training run hangs between them. GPU visibility checks the local device path; it does not establish that ranks can exchange data over the intended network.

NVIDIA's NCCL tests exercise collective communication and report correctness and performance. AMD's RCCL tests serve the corresponding purpose on ROCm. Run these checks with the same node placement, interfaces, image dependencies, and security policy that the real workload will use.

Prepare the MPI workload

Install the MPI Operator and enable Polyaxon's MPIJob integration. Connect the target cluster through a Polyaxon Agent (compute cluster), then select the queue and preset for the GPU pool being evaluated.

Use an image containing MPI, the test binaries, and the communication libraries compatible with your drivers and network. CoreWeave publishes NCCL test images with MPI and GPU communication dependencies; the example below selects one of its CUDA 12.9.2 images. Confirm compatibility with your cluster rather than assuming a container supplies the host driver or fabric configuration.

For an inter-node check, place the two GPU workers on distinct physical nodes. Two worker pods alone do not guarantee this. Configure worker affinity or a placement preset, and record the actual node assignments before interpreting bandwidth.

Run a two-rank all-reduce check

Save this as collective-test.yaml. It runs one MPI rank per GPU worker and doubles the message size from 8 bytes through 1 GiB.

version: 1.1
kind: component
name: gpu-collective-test
plugins:
  shm: true
run:
  kind: mpijob
  cleanPodPolicy: All
  slotsPerWorker: 1
  launcher:
    replicas: 1
    container:
      image: ghcr.io/coreweave/nccl-tests:12.9.2-devel-ubuntu22.04-nccl2.31.2-1-8ef4f05
      command: ["mpirun"]
      args:
      - "--allow-run-as-root"
      - "-np"
      - "2"
      - "-bind-to"
      - "none"
      - "-x"
      - "LD_LIBRARY_PATH"
      - "-x"
      - "NCCL_DEBUG=INFO"
      - "/opt/nccl_tests/build/all_reduce_perf"
      - "-b"
      - "8"
      - "-e"
      - "1G"
      - "-f"
      - "2"
      - "-g"
      - "1"
  worker:
    replicas: 2
    container:
      image: ghcr.io/coreweave/nccl-tests:12.9.2-devel-ubuntu22.04-nccl2.31.2-1-8ef4f05
      command: ["/usr/sbin/sshd"]
      args: ["-De"]
      resources:
        limits:
          nvidia.com/gpu: "1"

The binary path follows CoreWeave's MPI example. The root flag permits MPI inside this container; it does not require a privileged pod. Configure CPU, memory, RDMA device requests, network attachments, and interface selection for your environment. Keep the launcher's MPI libraries and the worker libraries consistent.

polyaxon run -f collective-test.yaml
polyaxon ops dashboard

Read correctness before bandwidth

First check that both ranks joined, the expected GPUs were used, and the operation completed without validation errors. Then inspect the selected network transport and interfaces. A completed run using an unintended fallback path is not evidence that the production fabric is configured correctly.

Record the GPU model, node placement, driver and library versions, image digest, message sizes, process count, and command with the result. Compare like-for-like runs. Algorithm bandwidth and bus bandwidth describe different quantities; use the NCCL tests performance explanation when interpreting them.

There is no universal pass bandwidth. Establish a baseline for the intended topology and workload, then investigate regressions against that baseline. Rerun after changes to drivers, firmware, node images, network plugins, or placement policy.

Adapt the check for AMD

The former standalone RCCL-tests repository is retired; use its current location in ROCm's systems repository. Build the tests with MPI support and package them with the matching ROCm and MPI libraries.

Replace both container images, update the executable path to that build, and request amd.com/gpu. Review the AMD device and network configuration as well. Changing only the Kubernetes resource name leaves the NVIDIA software stack in place.

Diagnose failures by stage

No worker pod: inspect admission, queue capacity, operator configuration, and permissions.
Pending worker: inspect GPU allocation, placement constraints, taints, and resource requests.
MPI cannot connect: inspect SSH startup, mounted keys, host discovery, DNS, and network policy.
Ranks connect but a collective hangs: inspect device visibility, interface selection, RDMA access, MTU, and collective-library logs.
Unexpectedly low bandwidth: confirm node placement and transport before changing tuning flags.

Use the broader Kubernetes workload debugging procedure when the failure occurs before MPI starts. Preserve logs and configuration before stopping a stuck diagnostic:

polyaxon ops stop