Polyaxon & RayCluster

How to use Polyaxon and RayCluster together

+

Polyaxon treats a Ray cluster as a normal operation: the platform resolves workload configuration and lifecycle policy, then KubeRay manages the head and worker pods. This guide starts with a small CPU example and shows where autoscaling, GPUs, storage, and governance fit.

See the Ray on Kubernetes for the upstream configuration and requirements.

Prerequisites

• A Kubernetes namespace managed by Polyaxon CE or a Polyaxon Agent.
• The KubeRay operator and RayCluster custom resource definition installed in that cluster.
• The RayCluster operator enabled in the Polyaxon deployment configuration.
• A configured Polyaxon CLI and a project where you can create operations.

Configuration

Enable the RayCluster operator

Install KubeRay and its CRDs first. Then enable the integration in the Polyaxon CE or Agent configuration that manages the target namespace.

operators:
  raycluster: true

This is an administrator change. The integration cannot create Ray workloads until both the KubeRay CRDs and this Polyaxon setting are present.

Create a minimal Ray component

Save this as ray.yaml. The same Ray version is used in the runtime field and both container images, which avoids a common source of protocol mismatches.

version: 1.1
kind: component
name: ray-square-numbers

run:
  kind: raycluster
  rayVersion: "2.58.0"
  entrypoint: >-
    python -c "import ray; ray.init(); square = ray.remote(lambda x: x * x); print(ray.get([square.remote(i) for i in range(8)]))"
  head:
    rayStartParams:
      dashboard-host: "0.0.0.0"
    container:
      image: rayproject/ray:2.58.0
      resources:
        requests:
          cpu: "500m"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "2Gi"
  workers:
    cpu-workers:
      replicas: 2
      minReplicas: 2
      maxReplicas: 4
      container:
        image: rayproject/ray:2.58.0
        resources:
          requests:
            cpu: "500m"
            memory: "1Gi"
          limits:
            cpu: "1"
            memory: "2Gi"

For production, pin an image version or digest that your team has validated and that contains your application code and dependencies.

Submit the operation

Run the component like any other Polyaxon operation. Project defaults, queue settings, presets, connections, and approval policy can be applied through your normal submission path.

polyaxon run -f ray.yaml

Inspect the distributed run

Open the operation dashboard to follow status and replica logs. Polyaxon streams and archives logs from the cluster replicas, while KubeRay remains responsible for reconciling the Ray resources.

polyaxon ops dashboard

If you are inspecting a different operation, pass its project and run UUID with the documented dashboard options.

Turn on demand-based worker scaling

Enable the Ray in-tree autoscaler when the workload can express demand through Ray resources and the cluster has capacity to add worker pods.

run:
  kind: raycluster
  rayVersion: "2.58.0"
  enableInTreeAutoscaling: true
  autoscalerOptions:
    upscalingMode: Default
    imagePullPolicy: IfNotPresent
  head:
    container:
      image: rayproject/ray:2.58.0
  workers:
    cpu-workers:
      replicas: 1
      minReplicas: 0
      maxReplicas: 8
      container:
        image: rayproject/ray:2.58.0

Ray autoscaling changes the desired worker count. Kubernetes still has to place the requested pods, so queue capacity, node selectors, quotas, and available CPU or GPUs remain part of the design.

Stop and clean up deliberately

Stop the Polyaxon operation when the cluster is no longer needed. Treat shutdown behavior as part of the workload contract, especially for scheduled or failure-prone jobs.

polyaxon ops stop

Deployment checks

• Pin the Ray version and image digest, and keep the runtime and container versions compatible.
• Bake application code and dependencies into an image or use a controlled, versioned initialization path.
• Set requests and limits for the head and every worker group; add node selectors and tolerations for GPU pools.
• Resolve datasets, object stores, secrets, and artifact locations through scoped Polyaxon connections.
• Decide whether Ray or Polyaxon owns each retry, timeout, schedule, and cleanup boundary.
• Validate autoscaling against real tasks and Kubernetes capacity rather than assuming pending pods will create capacity.

Troubleshooting

The operation fails before pods are created

Confirm the KubeRay operator and RayCluster CRD are installed in the managed cluster and operators.raycluster is enabled.

Head starts but workers do not join

Check Ray version compatibility, container logs, image pull access, resource requests, and network policy between head and workers.

Worker pods remain Pending

Inspect Kubernetes events, quotas, node selectors, taints, available CPU or GPUs, and any gang-scheduling policy.

Autoscaling never adds workers

Verify autoscaling is enabled, tasks request Ray resources, maxReplicas is above the current count, and Kubernetes can place new pods.

References

• Polyaxon cluster runtimes — Canonical Ray and Dask component examples and operator prerequisites.
• Polyaxon RayCluster specification — Entrypoint, runtime environment, head, workers, and autoscaling fields.
• Ray on Kubernetes — KubeRay architecture, RayCluster, RayJob, RayService, worker groups, and autoscaling.
• KubeRay getting started — Current CRDs and guidance for selecting RayCluster, RayJob, RayService, or RayCronJob.