Polyaxon v3 is coming →
Miles

Run Miles post-training on Polyaxon

Miles combines Ray orchestration, SGLang rollouts, Megatron-LM training, and GRPO-style post-training. Polyaxon gives that multi-system workload a stable image, resource topology, storage contract, and run record.

Outcome

A multi-node RayCluster skeleton that separates model preparation from the repeatable Miles training entrypoint.

Start the guideRead the Ray guide

Execution responsibilities

Polyaxon resolves

The component pins the image, inputs, secrets, resources, queue, and output contract.

Kubernetes places

The cluster schedules the requested accelerator topology and enforces namespace policy.

The framework trains

The training stack owns optimization while Polyaxon records status, logs, metadata, and artifacts.

Prerequisites

  • KubeRay installed and enabled for the target Polyaxon Agent.
  • A validated Miles image with its SGLang, Megatron-LM, Ray, PyTorch, CUDA, and NCCL compatibility set.
  • Fast shared or replicated storage sized for source weights, converted checkpoints, rollouts, and training checkpoints.

Run and validate

  1. 1

    Package the Miles topology

    Prebuild model conversion and dataset preparation into a controlled stage, then launch the training script as the Ray entrypoint.

    version: 1.1kind: componentname: miles-post-trainingrun:  kind: raycluster  rayVersion: "2.58.0"  entrypoint: bash /workspace/scripts/train-miles.sh  head:    container:      image: radixark/miles:latest      resources:        limits:          nvidia.com/gpu: "8"  workers:    training-workers:      replicas: 1      minReplicas: 1      maxReplicas: 1      container:        image: radixark/miles:latest        resources:          limits:            nvidia.com/gpu: "8"

    Miles publishes this image. Mount the reviewed launch script with a code connection, then pin the image digest and repository commit before production.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f miles.yaml
  3. 3

    Validate conversion, rollout, and training

    Check the converted checkpoint, Ray membership, rollout throughput, reward signal, optimizer progress, and resumable checkpoints as separate gates.

    polyaxon ops dashboard

Production checklist

Pin the container image, framework versions, model revision, dataset revision, and training configuration.
Resolve registry, dataset, model, and tracking credentials through scoped Polyaxon connections.
Persist checkpoints and final artifacts outside the pod, and define resume behavior before a long run.
Set requests, limits, node selectors, tolerations, queue policy, timeout, and retry ownership explicitly.

Troubleshooting

The pod remains Pending

Inspect quota, queue capacity, node selectors, taints, topology, and the requested accelerator count.

The process runs out of memory

Check model precision, sequence length, batch size, checkpointing, optimizer state, sharding, and actual device memory.

The run cannot read or save

Verify the referenced connection, secret scope, mounted path, object-store permissions, and available storage.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore MLOps