Pipelines & OrchestrationOverview
Pipelines & Orchestration
Polyaxon orchestration lets you compose multi-step ML workflows with DAGs, run hyperparameter sweeps, and schedule recurring pipelines—all with a declarative Polyaxonfile.
- Build multi-step pipelines with the DAG flow engine.
- Run hyperparameter optimization with grid, random, Bayesian, and Hyperband strategies.
- Schedule recurring jobs with cron, interval, or datetime triggers.
- Chain operations with hooks and events for automation.
Run automated hyperparameter sweeps with grid, random, Bayesian, or Hyperband strategies.
# polyaxonfile.yaml
version: 1.1
kind: operation
matrix:
kind: bayes
numInitialRuns: 5
maxIterations: 30
metric:
name: val_accuracy
optimization: maximize
params:
learning_rate:
kind: logspace
value: [0.0001, 0.1, 10]
dropout:
kind: uniform
value: [0.1, 0.9]Chain multiple operations into a DAG pipeline with dependencies between steps.
# pipeline.yaml
version: 1.1
kind: operation
run:
kind: dag
operations:
- name: preprocess
component_ref:
name: data-prep
- name: train
component_ref:
name: train-model
dependencies: [preprocess]
- name: evaluate
component_ref:
name: eval-model
dependencies: [train]Schedule recurring jobs with cron expressions, intervals, or specific datetime triggers.
# scheduled-training.yaml
version: 1.1
kind: operation
schedule:
kind: cron
cron: "0 2 * * *" # Run daily at 2 AM
component:
run:
kind: job
container:
image: my-team/retrain:latest
command: [python, retrain.py]Trigger actions based on run lifecycle events — send notifications, start downstream operations, or call webhooks.
# polyaxonfile.yaml
version: 1.1
kind: operation
hooks:
- trigger: succeeded
connection: slack-notifications
params:
channel: "#ml-alerts"
message: "Training completed with accuracy: {{ outputs.accuracy }}"
- trigger: failed
connection: pagerduty