Polyaxon v3 is coming →
Ray

Run Ray on Polyaxon

Polyaxon treats a Ray cluster as a normal operation: the platform resolves workload configuration and lifecycle policy, then KubeRay manages the head and worker pods. This guide starts with a small CPU example and shows where autoscaling, GPUs, storage, and governance fit.

Outcome

A tracked Ray operation with one head pod, two worker pods, a submitted Python entrypoint, archived replica logs, and a clear path to autoscaling.

Start the guideRead the runtime reference

Execution responsibilities

Polyaxon submits

The Polyaxonfile defines the Ray version, entrypoint, head, workers, resources, connections, and lifecycle policy.

KubeRay reconciles

The KubeRay operator creates and manages the RayCluster resource, head pod, worker groups, and optional autoscaler.

Ray distributes

The entrypoint connects to the cluster and Ray schedules remote tasks across the available worker processes.

Prerequisites

  • A Kubernetes namespace managed by Polyaxon CE or a Polyaxon Agent.
  • The KubeRay operator and RayCluster custom resource definition installed in that cluster.
  • The RayCluster operator enabled in the Polyaxon deployment configuration.
  • A configured Polyaxon CLI and a project where you can create operations.

Run and validate

  1. 1

    Enable the RayCluster operator

    Install KubeRay and its CRDs first. Then enable the integration in the Polyaxon CE or Agent configuration that manages the target namespace.

    operators:  raycluster: true

    This is an administrator change. The integration cannot create Ray workloads until both the KubeRay CRDs and this Polyaxon setting are present.

  2. 2

    Create a minimal Ray component

    Save this as ray.yaml. The same Ray version is used in the runtime field and both container images, which avoids a common source of protocol mismatches.

    version: 1.1kind: componentname: ray-square-numbersrun:  kind: raycluster  rayVersion: "2.58.0"  entrypoint: >-    python -c "import ray; ray.init(); square = ray.remote(lambda x: x * x); print(ray.get([square.remote(i) for i in range(8)]))"  head:    rayStartParams:      dashboard-host: "0.0.0.0"    container:      image: rayproject/ray:2.58.0      resources:        requests:          cpu: "500m"          memory: "1Gi"        limits:          cpu: "1"          memory: "2Gi"  workers:    cpu-workers:      replicas: 2      minReplicas: 2      maxReplicas: 4      container:        image: rayproject/ray:2.58.0        resources:          requests:            cpu: "500m"            memory: "1Gi"          limits:            cpu: "1"            memory: "2Gi"

    For production, pin an image version or digest that your team has validated and that contains your application code and dependencies.

  3. 3

    Submit the operation

    Run the component like any other Polyaxon operation. Project defaults, queue settings, presets, connections, and approval policy can be applied through your normal submission path.

    polyaxon run -f ray.yaml
  4. 4

    Inspect the distributed run

    Open the operation dashboard to follow status and replica logs. Polyaxon streams and archives logs from the cluster replicas, while KubeRay remains responsible for reconciling the Ray resources.

    polyaxon ops dashboard

    If you are inspecting a different operation, pass its project and run UUID with the documented dashboard options.

  5. 5

    Turn on demand-based worker scaling

    Enable the Ray in-tree autoscaler when the workload can express demand through Ray resources and the cluster has capacity to add worker pods.

    run:  kind: raycluster  rayVersion: "2.58.0"  enableInTreeAutoscaling: true  autoscalerOptions:    upscalingMode: Default    imagePullPolicy: IfNotPresent  head:    container:      image: rayproject/ray:2.58.0  workers:    cpu-workers:      replicas: 1      minReplicas: 0      maxReplicas: 8      container:        image: rayproject/ray:2.58.0

    Ray autoscaling changes the desired worker count. Kubernetes still has to place the requested pods, so queue capacity, node selectors, quotas, and available CPU or GPUs remain part of the design.

  6. 6

    Stop and clean up deliberately

    Stop the Polyaxon operation when the cluster is no longer needed. Treat shutdown behavior as part of the workload contract, especially for scheduled or failure-prone jobs.

    polyaxon ops stop

Production checklist

Pin the Ray version and image digest, and keep the runtime and container versions compatible.
Bake application code and dependencies into an image or use a controlled, versioned initialization path.
Set requests and limits for the head and every worker group; add node selectors and tolerations for GPU pools.
Resolve datasets, object stores, secrets, and artifact locations through scoped Polyaxon connections.
Decide whether Ray or Polyaxon owns each retry, timeout, schedule, and cleanup boundary.
Validate autoscaling against real tasks and Kubernetes capacity rather than assuming pending pods will create capacity.

Troubleshooting

The operation fails before pods are created

Confirm the KubeRay operator and RayCluster CRD are installed in the managed cluster and operators.raycluster is enabled.

Head starts but workers do not join

Check Ray version compatibility, container logs, image pull access, resource requests, and network policy between head and workers.

Worker pods remain Pending

Inspect Kubernetes events, quotas, node selectors, taints, available CPU or GPUs, and any gang-scheduling policy.

Autoscaling never adds workers

Verify autoscaling is enabled, tasks request Ray resources, maxReplicas is above the current count, and Kubernetes can place new pods.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare Polyaxon and KubeflowExplore GPU orchestration