Polyaxon & DaskCluster

How to use Polyaxon and DaskCluster together

+

Polyaxon submits a DaskCluster resource through the Dask Kubernetes Operator and tracks the resulting scheduler and worker replicas as one operation. The current Polyaxon runtime provisions the cluster; your Python client runs as a separate job, service, or notebook and connects through the scheduler address.

See the Dask Kubernetes Operator for the upstream configuration and requirements.

Prerequisites

• A Kubernetes namespace managed by Polyaxon CE or a Polyaxon Agent.
• The Dask Kubernetes Operator and DaskCluster custom resource definition installed in that cluster.
• The DaskCluster operator enabled in the Polyaxon deployment configuration.
• A configured Polyaxon CLI and a project where you can create operations.

Configuration

Enable the DaskCluster operator

Install the Dask Kubernetes Operator and CRDs first. Then enable the integration in the Polyaxon CE or Agent configuration that manages the target namespace.

operators:
  daskcluster: true

This is an administrator change. Polyaxon cannot submit the DaskCluster resource until both sides are configured.

Define the scheduler and workers

Save this as dask.yaml. The scheduler and workers use the same image so their Dask and distributed protocol versions stay compatible.

version: 1.1
kind: component
name: dask-cpu-cluster

run:
  kind: daskcluster
  worker:
    replicas: 2
    container:
      image: ghcr.io/dask/dask:latest
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "2"
          memory: "4Gi"
  scheduler:
    container:
      image: ghcr.io/dask/dask:latest
      resources:
        requests:
          cpu: "500m"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "2Gi"

The upstream quickstart uses the latest image for convenience. Replace it with a tested version or digest before production.

Create the cluster operation

Submit the component through your normal Polyaxon project, queue, preset, and approval path.

polyaxon run -f dask.yaml

Resolve the scheduler address

The Dask operator creates a scheduler Service for the cluster. Resolve its address from the created Kubernetes resources or your platform's exposed operation details, then pass that address to the client through a scoped environment variable or connection.

DASK_SCHEDULER_ADDRESS=tcp://<scheduler-service>:8786

The exact Service name depends on the resource emitted for your operation. Discover it from the running cluster instead of hard-coding an assumed name in a reusable component.

Submit work from a separate client

Run this client inside a Polyaxon job, notebook, or service that can reach the scheduler Service. Keeping the client separate makes cluster lifetime and task submission explicit.

import os
from dask.distributed import Client

client = Client(os.environ["DASK_SCHEDULER_ADDRESS"])

future = client.submit(lambda value: value * value, 12)
print(future.result())

client.close()

Use the same tested Dask and distributed package versions in the client, scheduler, and workers.

Add worker autoscaling

Set minimum and maximum replicas when the Dask operator should create a DaskAutoscaler for the worker group.

run:
  kind: daskcluster
  minReplicas: 1
  maxReplicas: 10
  worker:
    replicas: 2
    container:
      image: ghcr.io/dask/dask:latest
      resources:
        requests:
          cpu: "1"
          memory: "2Gi"
        limits:
          cpu: "2"
          memory: "4Gi"
  scheduler:
    container:
      image: ghcr.io/dask/dask:latest
      resources:
        requests:
          cpu: "500m"
          memory: "1Gi"

Autoscaling changes worker replicas; Kubernetes placement still depends on available nodes, quotas, selectors, taints, and requested resources.

Inspect and stop the cluster

Use the operation dashboard for status and replica logs, then stop the cluster operation after all clients have finished.

polyaxon ops dashboard
polyaxon ops stop

Deployment checks

• Pin compatible Dask and distributed versions in the client, scheduler, and worker images.
• Size scheduler memory for graph metadata and worker memory for partitions, shuffles, and spill behavior.
• Use scoped Polyaxon connections for object stores, datasets, secrets, and shared artifact locations.
• Keep the scheduler Service private and restrict network access to approved client workloads.
• Make the cluster owner, client owner, idle policy, retries, and shutdown sequence explicit.
• Test autoscaling and worker loss with a production-shaped graph and realistic data movement.

Troubleshooting

No scheduler or worker pods appear

Confirm the Dask operator and CRDs are installed and operators.daskcluster is enabled in the responsible Polyaxon deployment.

The client cannot connect

Verify the discovered scheduler Service, port 8786, namespace and DNS path, network policy, and client-to-cluster reachability.

Workers repeatedly disconnect

Check package compatibility, worker memory limits, scheduler logs, network stability, health probes, and image consistency.

Workers stay Pending during scale-out

Inspect Kubernetes events, quota, node capacity, selectors, taints, CPU or GPU requests, and storage constraints.

References

• Polyaxon cluster runtimes — Canonical Dask and Ray component examples and operator prerequisites.
• Polyaxon DaskCluster specification — Scheduler, worker, replica, resource, and autoscaling fields.
• Dask Kubernetes Operator — Operator installation, scheduler Service, worker groups, Python client, and GPU-capable worker patterns.
• Dask custom resources — DaskCluster, DaskWorkerGroup, DaskJob, DaskAutoscaler, labels, annotations, and full resource shape.