Note: Requires Polyaxon v1.9.2 or higher.

Polyaxon provides a section to automate the process of building new containers and setting the destination image automatically on the main container section. If an operation or a component has a build section attached to it, Polyaxon will create the main operation, start the build process, and set an upstream dependency until the build process is done.

Let’s see how we can modify the simple.yaml to build a new container before starting the main logic.

Using a build preset

The easiest way to test new requirements or a new image definition is by attaching a preset without any changes to the polyaxonfile itself.

build:
  runPatch:
    init:
    - dockerfile:
        image: "tensorflow/tensorflow:2.2.0"
        run:
        - pip install keras
        - pip3 install --no-cache-dir -U polyaxon
        langEnv: 'en_US.UTF-8'
  hubRef: kaniko
  connection: docker-connection

Let’s save this file as build-override.yaml, you should notice that this includes the same content as in the standalone build operation, but it does not set a destination param. Although it’s possible to set a static destination or a templated destination based on the context of the operation, the intention behind of the per-operation build section is to completely automate the destination image and it’s usage in the job or service by replacing the predefined image.

Now we can start simple.yaml file with this override file (we can also use this override with the typed component and the hyperparameter tuning operation):

polyaxon run -f experimentation/simple.yaml -f build-override.yaml

After submitting this operation, if you check the UI to see the progress of the run you will notice that this operation is waiting for a build process, there’s also a link in the sidebar to the build run:

You can also see in the lineage page the type of the upstream link:

Using an operation

If we expect our operation to always require a build and if we decide to reduce the amount of logic with the CLI with the override flag, we can create an operation that sets the build and depends on the component:

version: 1.1
kind: operation
pathRef: ./experimentation/simple.yaml
build:
  runPatch:
    init:
    - dockerfile:
        image: "tensorflow/tensorflow:2.2.0"
        run:
        - pip install keras
        - pip3 install --no-cache-dir -U polyaxon
        langEnv: 'en_US.UTF-8'
  hubRef: kaniko
  connection: docker-connection

This is similar to the result of the previous section with the preset, the build section can be defined both on the component or the operations level similar to several other sections in the Polyaxonfile.

Single file

In some situations where the operation is expected to change very frequently, or when the user intends to package several dependencies(requirements, config files, artifacts, …) into a single container, the build section can live in the same polyaxonfile:

version: 1.1
kind: component
name: simple-experiment
description: Minimum information to run this TF.Keras example
tags: [examples]
build:
  runPatch:
    init:
    - dockerfile:
        image: "tensorflow/tensorflow:2.2.0"
        run:
        - pip install keras
        - pip3 install --no-cache-dir -U polyaxon
        langEnv: 'en_US.UTF-8'
  hubRef: kaniko
  connection: docker-connection
run:
  kind: job
  init:
  - git: {url: "https://github.com/polyaxon/polyaxon-quick-start"}
  container:
    command: [python3, "{{ globals.artifacts_path }} + /polyaxon-quick-start/model.py"]

You can notice here that we completely removed the image because this component will automatically default to whatever image generated by the build process.

Moving the git init to the build section

Since we are building a container and because the quick-start project has a already a valid Dockerfile, we can use it for building the new image:

version: 1.1
kind: component
build:
  runPatch:
    init:
    - git: {url: "https://github.com/polyaxon/polyaxon-quick-start"}
  params:
    context:
      value: "{{ globals.artifacts_path }}/polyaxon-quick-start"
  connection: docker-connection
  hubRef: kaniko
run:
  kind: job
  container:
    command: [python3, model.py]

Using a local folder with upload

Similar to the behavior above, if we clone the quick-start repo and initialize it locally, we can iterate and upload the content, which will trigger a new build and then a new experiment:

version: 1.1
kind: component
build:
  params:
    context:
      value: "{{ globals.run_artifacts_path }}/code"
  connection: docker-connection
  hubRef: kaniko
run:
  kind: job
  container:
    command: [python3, model.py]

You should notice that we are setting the build context to code, which means that we need to upload our repo to the code folder:

polyaxon run -f new-experiment.yaml -u-to code