V1Plugins

polyaxon._flow.plugins.V1Plugins()

Plugins section provides a way to customize extra Polyaxon utilities.

By default, Polyaxon injects some information for example an auth context and triggers some mechanisms for collecting logs and outputs.

Plugins section provides more control to the end user to enable/disable some of these utilities.

• Args:
  • auth: bool, optional, default: True
  • docker: bool, optional, default: False
  • shm: bool, optional, default: True
  • tmux: bool, optional, default: False
  • sandbox: bool, optional, default: False
  • ssh: bool, optional, default: False
  • mount_artifacts_store: bool, optional, default: True
  • collect_artifacts: bool, optional, default: True
  • collect_logs: bool, optional, default: True
  • collect_resources: bool, optional, default: True
  • sync_statuses: bool, optional, default: True
  • auto_resume: bool, optional, default: True
  • external_host: bool, optional, default: False
  • log_level: str, optional
  • sidecar: V1PolyaxonSidecarContainer, optional

YAML usage

plugins:
  auth:
  docker:
  shm:
  tmux:
  sandbox:
  ssh:
  mountArtifactsStore:
  collectArtifacts:
  collectLogs:
  collectResources:
  syncStatuses:
  autoResume:
  externalHost:
  logLevel:
  sidecar:

Python usage

from polyaxon.schemas import V1Plugins
plugins = V1Plugins(
    auth=False,
    docker=True,
    shm=True,
    tmux=True,
    sandbox=True,
    ssh=True,
    mount_artifacts_store=True,
    collect_artifacts=False,
    collect_logs=False,
    collect_resources=False,
    sync_statuses=True,
    auto_resume=False,
    external_host=False,
    log_level="INFO",
)

Fields

auth

This plugin is enabled by default in Polyaxon deployments with user management.

By default, Polyaxon will create an auth context for each operation, this removes the overhead to think about how you can pass tokens to your runs, or the need to create secrets to load the token from during the run time.

The auth context that Polyaxon provides is not only specific to the user who executed the run, but it also impersonates similar user access rights, it has the same scopes and restrictions the user usually has within the context of the project managing the run. This is important to make sure the API calls made during the run by the user's code have the right access to the resources requested.

Oftentimes, users might not need to use an authenticated client inside their containers, in that case they can disable this plugin.

To disable this plugin:

plugins:
  auth: false

docker

This plugin is disabled by default.

This plugin exposes a docker socket volume to your run container.

N.B. use this plugin carefully, you might also need to check with your devops team before using it, it requires docker.sock of the host to be mounted which is often rejected by OPA.

To enable this plugin:

plugins:
  docker: true

shm

This plugin is enabled by default.

This plugin mounts an tmpfs volume to /dev/shm. This will set /dev/shm size to half of the RAM of node. By default, /dev/shm is very small, only 64MB. Some experiments/jobs will fail due to the lack of shared memory, such as some experiments running on Pytorch.

To disable this plugin:

plugins:
  shm: false

tmux

This plugin is disabled by default.

This plugin ships a statically-linked tmux into the user container so that the in-cluster shell session opened by polyaxon ops shell can attach to a persistent terminal. Without it, shell falls back to /bin/bash and the session ends as soon as the websocket disconnects.

To enable this plugin:

plugins:
  tmux: true

sandbox

This plugin is disabled by default.

This plugin enables the sandbox daemon for programmatic exec, filesystem, and PTY access from SDKs, agents, and UIs.

Sandbox is supported for service runs only. When enabled, Polyaxon wraps the main container command with /opt/polyaxon/bin/bootstrap-sandbox.sh. If no command is provided, plx-exec runs as PID 1. If a workload is needed, set an explicit command; args without command are not supported in v0. The user image must provide /bin/sh.

To enable this plugin:

plugins:
  sandbox: true

ssh

This plugin is disabled by default.

This plugin enables SSH access to a service run for local development tools such as OpenSSH, VS Code Remote SSH, JetBrains Gateway, SFTP, SCP, and rsync.

SSH is supported for service runs only. When enabled, Polyaxon injects the OpenSSH runtime into the user container, starts sshd, and exposes the SSH port through the service. In this version, the CLI prepares access and tunnels the SSH connection through the sandbox transport, so SSH-enabled runs also need the sandbox control plane to be reachable.

To enable this plugin:

plugins:
  ssh: true

mountArtifactsStore

This plugin is disabled by default.

this plugin allows to request the default artifacts store and mount it to the main container without adding the connection reference name to the connections section.

This is usually very useful than setting the connections section as it make the component more generic and will not break if the artifacts store name changes.

collectArtifacts

This plugin is enabled by default.

By default, Polyaxon will collect all artifacts and outputs that you share in the plx-context/artifacts/run-uuid/outputs to the default artifacts store that you configured during the deployment.

This plugin is important if you want to have an agnostic code to the type of storage backend your are using, by changing the environment variable you can test your code with tmp file locally and the artifacts path in production.

To disable this plugin:

plugins:
  collectArtifacts: false

Sometimes you might want to access the artifacts path in your polyaxonfile, Polyaxon expose a context that get resolved during the compilation time, you can just use {{run_artifacts_path}} global variable and it will be resolved automatically.

• Example:

run:
  kind: job
  container:
    command: "cp /some/know/path/file {{run_artifacts_path}}/file"

For more information about the context, please check context

collectLogs

This plugin is enabled by default.

By default, Polyaxon will collect all logs related to your runs before deleting the resource on the clusters. This ensures that your cluster(s) are kept clean and no resources are actively putting pressure on the API server.

Sometimes you might want to avoid collecting logs for some runs, for example test or debug jobs.

To disable this plugin:

plugins:
  collectLogs: false

collectResources

This plugin is enabled by default.

By default, Polyaxon will collect all Mem/CPU/GPU resources for your runs that use the python client.

Sometimes you might want to avoid collecting this information for some runs, for example test or debug jobs.

To disable this plugin:

plugins:
  collectResources: false

autoResume

This plugin is enabled by default.

By default, Polyaxon will resume from collecting metrics/outputs/artifacts if a run fails and retries or if the user resume a run.

To disable this plugin:

plugins:
  autoResume: false

externalHost

plugins:
  externalHost: true

logLevel

Default is None.

If you want to control the log level of your runs in a similar way locally and on the cluster, you can either use env vars or this plugin to share the same log level with all containers running in your operation.

plugins:
  logLevel: warning

sidecar

Default is None.

To override the default global sidecar configuration.

plugins:
  sidecar:
    syncInterval: 60