V1DaskReplica

polyaxon._flow.run.dask.replica.V1DaskReplica()

Dask replica is the specification for a Dask job, worker or scheduler.

YAML usage

head/worker:
  replicas:
  environment:
  connections:
  volumes:
  init:
  sidecars:
  container:

Python usage

from polyaxon.schemas import V1Environment, V1Init, V1DaskReplica
from polyaxon import k8s
replica = V1DaskReplica(
    replicas=2,
    environment=V1Environment(...),
    init=[V1Init(...)],
    sidecars=[k8s.V1Container(...)],
    container=k8s.V1Container(...),
)

Fields

replicas

The number of worker replica instances.

executor:
  replicas: 2

environment

Optional environment section, it provides a way to inject pod related information into the replica (executor/driver).

worker:
  environment:
    labels:
       key1: "label1"
       key2: "label2"
     annotations:
       key1: "value1"
       key2: "value2"
     nodeSelector:
       node_label: node_value
     ...
 ...

connections

A list of connection names to resolve for the job.

If you are referencing a connection it must be configured. All referenced connections will be checked:

  • If they are accessible in the context of the project of this run

  • If the user running the operation can have access to those connections

After checks, the connections will be resolved and inject any volumes, secrets, configMaps, environment variables for your main container to function correctly.

worker:
  connections: [connection1, connection2]

init

A list of init handlers and containers to resolve for the replica (executor/driver).

If you are referencing a connection it must be configured. All referenced connections will be checked:

  • If they are accessible in the context of the project of this run

  • If the user running the operation can have access to those connections

worker:
  init:
    - artifacts:
        dirs: ["path/on/the/default/artifacts/store"]
    - connection: gcs-large-datasets
      artifacts:
        dirs: ["data"]
      container:
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
    - container:
      name: myapp-container
      image: busybox:1.28
      command: ['sh', '-c', 'echo custom init container']

sidecars

A list of sidecar containers that will be used as sidecars.

worker:
  sidecars:
    - name: sidecar2
      image: busybox:1.28
      command: ['sh', '-c', 'echo sidecar2']
    - name: sidecar1
      image: busybox:1.28
      command: ['sh', '-c', 'echo sidecar1']
      resources:
        requests:
          memory: "128Mi"
          cpu: "500m"

container

The main Kubernetes Container that will run your experiment training or data processing logic for the replica (executor/driver).

worker:
  init:
    - connection: my-code-repo
  container:
    name: tensorflow:2.1
    command: ["python", "/plx-context/artifacts/my-code-repo/model.py"]