Polyaxon v3 is coming →
Dask

Run Dask on Polyaxon

Polyaxon submits a DaskCluster resource through the Dask Kubernetes Operator and tracks the resulting scheduler and worker replicas as one operation. The current Polyaxon runtime provisions the cluster; your Python client runs as a separate job, service, or notebook and connects through the scheduler address.

Outcome

A tracked Dask cluster with one scheduler, two workers, an explicit client boundary, archived replica logs, and an optional DaskAutoscaler.

Start the guideRead the runtime reference

Execution responsibilities

Polyaxon submits

The cluster component defines scheduler and worker images, replicas, resources, connections, and optional autoscaling bounds.

The operator reconciles

The Dask operator creates the scheduler pod and service, the worker group, worker pods, and an optional autoscaler.

A client connects

A separate Python job, notebook, or service connects to the scheduler address and submits the task graph.

Prerequisites

  • A Kubernetes namespace managed by Polyaxon CE or a Polyaxon Agent.
  • The Dask Kubernetes Operator and DaskCluster custom resource definition installed in that cluster.
  • The DaskCluster operator enabled in the Polyaxon deployment configuration.
  • A configured Polyaxon CLI and a project where you can create operations.

Run and validate

  1. 1

    Enable the DaskCluster operator

    Install the Dask Kubernetes Operator and CRDs first. Then enable the integration in the Polyaxon CE or Agent configuration that manages the target namespace.

    operators:  daskcluster: true

    This is an administrator change. Polyaxon cannot submit the DaskCluster resource until both sides are configured.

  2. 2

    Define the scheduler and workers

    Save this as dask.yaml. The scheduler and workers use the same image so their Dask and distributed protocol versions stay compatible.

    version: 1.1kind: componentname: dask-cpu-clusterrun:  kind: daskcluster  worker:    replicas: 2    container:      image: ghcr.io/dask/dask:latest      resources:        requests:          cpu: "1"          memory: "2Gi"        limits:          cpu: "2"          memory: "4Gi"  scheduler:    container:      image: ghcr.io/dask/dask:latest      resources:        requests:          cpu: "500m"          memory: "1Gi"        limits:          cpu: "1"          memory: "2Gi"

    The upstream quickstart uses the latest image for convenience. Replace it with a tested version or digest before production.

  3. 3

    Create the cluster operation

    Submit the component through your normal Polyaxon project, queue, preset, and approval path.

    polyaxon run -f dask.yaml
  4. 4

    Resolve the scheduler address

    The Dask operator creates a scheduler Service for the cluster. Resolve its address from the created Kubernetes resources or your platform's exposed operation details, then pass that address to the client through a scoped environment variable or connection.

    DASK_SCHEDULER_ADDRESS=tcp://<scheduler-service>:8786

    The exact Service name depends on the resource emitted for your operation. Discover it from the running cluster instead of hard-coding an assumed name in a reusable component.

  5. 5

    Submit work from a separate client

    Run this client inside a Polyaxon job, notebook, or service that can reach the scheduler Service. Keeping the client separate makes cluster lifetime and task submission explicit.

    import osfrom dask.distributed import Clientclient = Client(os.environ["DASK_SCHEDULER_ADDRESS"])future = client.submit(lambda value: value * value, 12)print(future.result())client.close()

    Use the same tested Dask and distributed package versions in the client, scheduler, and workers.

  6. 6

    Add worker autoscaling

    Set minimum and maximum replicas when the Dask operator should create a DaskAutoscaler for the worker group.

    run:  kind: daskcluster  minReplicas: 1  maxReplicas: 10  worker:    replicas: 2    container:      image: ghcr.io/dask/dask:latest      resources:        requests:          cpu: "1"          memory: "2Gi"        limits:          cpu: "2"          memory: "4Gi"  scheduler:    container:      image: ghcr.io/dask/dask:latest      resources:        requests:          cpu: "500m"          memory: "1Gi"

    Autoscaling changes worker replicas; Kubernetes placement still depends on available nodes, quotas, selectors, taints, and requested resources.

  7. 7

    Inspect and stop the cluster

    Use the operation dashboard for status and replica logs, then stop the cluster operation after all clients have finished.

    polyaxon ops dashboardpolyaxon ops stop

Production checklist

Pin compatible Dask and distributed versions in the client, scheduler, and worker images.
Size scheduler memory for graph metadata and worker memory for partitions, shuffles, and spill behavior.
Use scoped Polyaxon connections for object stores, datasets, secrets, and shared artifact locations.
Keep the scheduler Service private and restrict network access to approved client workloads.
Make the cluster owner, client owner, idle policy, retries, and shutdown sequence explicit.
Test autoscaling and worker loss with a production-shaped graph and realistic data movement.

Troubleshooting

No scheduler or worker pods appear

Confirm the Dask operator and CRDs are installed and operators.daskcluster is enabled in the responsible Polyaxon deployment.

The client cannot connect

Verify the discovered scheduler Service, port 8786, namespace and DNS path, network policy, and client-to-cluster reachability.

Workers repeatedly disconnect

Check package compatibility, worker memory limits, scheduler logs, network stability, health probes, and image consistency.

Workers stay Pending during scale-out

Inspect Kubernetes events, quota, node capacity, selectors, taints, CPU or GPU requests, and storage constraints.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare Polyaxon and KubeflowExplore GPU orchestration