Now that our component works, we want to be able to run it continuously. Polyaxon provides an abstraction to put operations on a schedule.

Running components on schedule

Schedules can be used at the operation level, and similar to the matrix section, they create a pipeline to follow the scheduling of a component or an operation.

The file schedules/recurrent_training.yaml contains an operation definition with a schedule attached:

version: 1.1
kind: operation
name: every-60-minute
schedule:
  kind: interval
  frequency: 3600
  dependsOnPast: true
params:
  optimizer: {value: sgd}
  epochs: {value: 1}
urlRef: https://raw.githubusercontent.com/polyaxon/polyaxon-quick-start/master/scheduling/experimentation/typed.yaml

Run this schedule:

polyaxon run --url https://raw.githubusercontent.com/polyaxon/polyaxon-quick-start/master/scheduling/schedules/recurrent_training.yaml

This interval schedule starts an experiment every 60 minutes. With dependsOnPast: true, Polyaxon waits for the previous execution to finish before scheduling the next one. If you need to start this schedule at a specific date you can set the startAt field, and to provide a stopping condition you can provide endAt or maxRuns.

It's also possible to put the complete DAG that we created in the previous guide on schedule, similarly to this operation, you can add a valid schedule section, this way, not only you automate the journey of creating, training, and validating a model, but also you do it continuously, the file schedules/recurrent_dag.yaml contains an example of such operation:

version: 1.1
kind: operation
name: every-monday-check
schedule:
  kind: cron
  cron: "0 0 * * MON"
params:
  max_loss: { value: 0.01 }
  top: { value: 5 }
cache:
  disable: true
urlRef: https://raw.githubusercontent.com/polyaxon/polyaxon-quick-start/master/scheduling/dags/dag.yaml

Run the scheduled DAG:

polyaxon run --url https://raw.githubusercontent.com/polyaxon/polyaxon-quick-start/master/scheduling/schedules/recurrent_dag.yaml

N.B.: In this second example we are disabling the cache completely.

Learn More

You can check the schedules section for more details about all the possible schedules and their specifications.