V1DaskJob

polyaxon._flow.run.dask.dask.V1DaskJob()

Dask jobs are used to run distributed jobs using a Dask cluster.

Dask Kubernetes deploys Dask workers on Kubernetes clusters using native Kubernetes APIs. It is designed to dynamically launch short-lived deployments of workers during the lifetime of a job.

The Dask job spawn a temporary adaptive Dask cluster with a Dask scheduler and workers to run your container.

YAML usage

run:
  kind: daskjob
  job:
  worker:
  scheduler:

Python usage

from polyaxon.schemas import V1Environment, V1Init, V1DaskJob, V1DaskReplica
dask_job = V1DaskJob(
    job=V1DaskReplica(...),
    worker=V1DaskReplica(...),
    scheduler=V1DaskReplica(...),
)

Fields

kind

The kind signals to the CLI, client, and other tools that this component’s runtime is a job.

If you are using the python client to create the runtime, this field is not required and is set by default.

run:
  kind: daskjob

job

Dask head replica specification

run:
  kind: daskjob
  job:
    container:
      image: "ghcr.io/dask/dask:latest"
      args:
      - python
      - -c
      - "from dask.distributed import Client; client = Client(); # Do some work..."
    ...
  ...

worker

List of worker replica specifications

run:
  kind: daskjob
  worker:
    replicas: 2
    container:
      image: "ghcr.io/dask/dask:latest"
      args:
      - dask-worker
      - --nthreads
      - "2"
      - --name
      - $(DASK_WORKER_NAME)
      - --dashboard
      - --dashboard-address
      - "8788"
  ...

scheduler

Dask scheduler replica specification

run:
  kind: daskjob
  scheduler:
    container:
      image: "ghcr.io/dask/dask:latest"
      args:
      - dask-scheduler
      - --dashboard-address
      - "8787"
  ...