Polyaxon v3 is coming →
NCCL / RCCL

Validate GPU networking with NCCL and RCCL

A visible GPU is not proof that distributed collectives will work. Polyaxon's MPIJob runtime lets you run NCCL or RCCL tests through the same scheduler, nodes, interfaces, and policies as production workloads.

Outcome

A two-worker MPI diagnostic operation with logs that identify topology, transport, bandwidth behavior, and failures without inventing benchmark targets.

Start the guideRead the MPIJob integration

Infrastructure responsibilities

Your platform defines the topology

The queue and preset select the GPU nodes, interfaces, placement constraints, security rules, and communication stack under test.

Polyaxon runs the diagnostic

The Agent submits an MPIJob through the same scheduler and policy boundary used by production workloads.

NCCL or RCCL measures the path

Launcher and worker logs expose the selected transport, failures, and bandwidth for that exact topology and test configuration.

Prerequisites

  • The Kubeflow MPI Operator and MPIJob CRD installed, with the Polyaxon MPI integration enabled.
  • A validated NCCL-tests or RCCL-tests image containing MPI and the vendor communication stack.
  • Two or more compatible GPU nodes with the intended interfaces, security rules, shared keys, and pod-to-pod connectivity.

Validate the infrastructure

  1. 1

    Define a collective test job

    Use an MPIJob with one launcher and two GPU workers. Select the NVIDIA or AMD diagnostics image through your component library.

    version: 1.1kind: componentname: gpu-collective-testrun:  kind: mpijob  cleanPodPolicy: All  slotsPerWorker: 1  launcher:    replicas: 1    container:      image: ghcr.io/coreweave/nccl-tests:12.9.1-devel-ubuntu22.04-nccl2.30.4-1-2eedd7c      command: ["mpirun"]      args: ["-np", "2", "-bind-to", "none", "/opt/nccl-tests/build/all_reduce_perf", "-b", "8", "-e", "1G", "-f", "2"]  worker:    replicas: 2    container:      image: ghcr.io/coreweave/nccl-tests:12.9.1-devel-ubuntu22.04-nccl2.30.4-1-2eedd7c      resources:        limits:          nvidia.com/gpu: "1"

    For AMD, use a validated RCCL-tests image and request amd.com/gpu. The sample does not publish a target bandwidth because acceptable results depend on topology and hardware.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f collective-test.yaml
  3. 3

    Read transport and bandwidth evidence

    Inspect launcher and worker logs for selected interfaces, transport fallback, errors, and scaling behavior. Compare only runs with the same topology and parameters.

    polyaxon ops dashboard

Production checklist

Keep cluster provisioning and Polyaxon Agent configuration in version-controlled infrastructure code.
Separate general-purpose, training, and inference pools with explicit labels, taints, quotas, and queues.
Validate driver, runtime, device plugin, storage, DNS, and east-west network compatibility after upgrades.
Run communication tests on the exact nodes and interfaces production workloads will use.
Archive test parameters and topology with the result; rerun after driver, firmware, CNI, network, or node-image changes.

Troubleshooting

The GPU is not allocatable

Check node readiness, the vendor driver and device plugin, kubelet capacity, labels, taints, and runtime compatibility.

A multi-node test hangs

Check pod placement, interface selection, DNS, security rules, RDMA devices, MTU, and collective-library logs.

The Agent cannot submit

Verify cluster registration, namespace access, service account permissions, queue routing, and network reachability.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore GPU orchestration