Passing params with args

Passing the parameters as args requires that you create and expose your programs as simple command-line applications.

Creating command-line applications is a powerful way of exposing work, it forces users to think about how to structure and organize programs, it provides documentation for the expected inputs and performs checks that other users can benefit from when interacting with a program.

Passing params to a simple program

Your program can be written in any language, the only requirements to use this method for passing configurations (inputs/outputs), is that you need to provide an interface to consume arguments.

Let's consider the echo command. This first version is available as passing-params/args-static.yaml:

version: 1.1
kind: component
name: static-message
run:
  kind: job
  container:
    image: busybox:stable
    command: ["echo", "This is a test"]

This is a simple program that prints information, you can run it using Polyaxon CLI:

polyaxon run -f passing-params/args-static.yaml -l

In order to run this program with multiple values, we can keep changing the message manually or we can expose the message to print as an input and pass it as an argument:

version: 1.1
kind: component
name: message-parameter
inputs:
- name: message
  type: str
run:
  kind: job
  container:
    image: busybox:stable
    command: ["echo", "{{ message }}"]

Now you can run multiple versions of this example without changing the Polyaxonfile:

polyaxon run -f passing-params/args-message.yaml -P message="test 1" -l

polyaxon run -f passing-params/args-message.yaml -P message="test 2" -l

You can also notice that Polyaxon will track the input and will show it in the UI and the CLI:

polyaxon ops get -uid UUID

Run inputs:

-------  ------
message  test 1
-------  ------

....

Multirun

Sometimes users might need to run the same job and pass different params, one way to do that is by invoking the CLI multiple times, another way is to use -HP(--hparams) instead of -P(--params).

To pass test 1 and test 2 to our program without invoking the CLI multiple times:

polyaxon run -f passing-params/args-message.yaml \
  -HP message='choice:["test 1","test 2"]'

This command will automatically create a grid search with the following matrix configuration:

matrix:
  kind: grid
  params:
    message:
      kind: choice
      value:
        - test 1
        - test 2
  concurrency: 1

You should notice that the CLI uses the following format: kind:value to pass hyperparameters, in this case it passes the choice kind. Another important aspect to notice is that matrix is of kind grid search and it runs the operations sequentially by setting the concurrency to 1. You can configure those options via CLI as well by passing the following extra arguments --matrix-kind, --matrix-concurrency, and --matrix-num-runs.

The CLI arg -HP is a nice way to avoid creating configuration files when iterating, however if you are to create a complex operation with multiple inputs/outputs and complex matrix definition, we suggest that you use a proper Polyaxonfile. See the intro for the hyperparameter tuning in this section and the optimization engine reference.

Matrix runs can repeat a component with inputs that Polyaxon has already executed. When that happens, Polyaxon may reuse the cached result. Add --cache=f if you want every generated run to execute again:

polyaxon run -f passing-params/args-message.yaml \
  -HP message='choice:["test 1","test 2"]' \
  --cache=f

Creating a custom program

Since most Polyaxon's users are data-scientists or machine learning engineers, they generally write their programs in Python, so the content of these tutorials will be in Python as well.

This is a simple application that prints your input. In this example we use the argparse package to consume the parameter, but the same logic can be used with python-fire, click, or any other library of your choice.

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--message", type=str)
args = parser.parse_args()
print(args.message)

We can adjust our previous Polyaxonfile to run this Python program. To avoid uploading code or cloning a repository, we will pass the Python code inline:

Let's save these changes under passing-params/args-inline-python.yaml:

version: 1.1
kind: component
name: inline-python-parameter
inputs:
  - name: message
    type: str
run:
  kind: job
  init:
    - file:
        content: |
          import argparse

          parser = argparse.ArgumentParser()
          parser.add_argument("--message", type=str)
          args = parser.parse_args()
          print(args.message)
        filename: message.py
  container:
    image: python:3.11
    workingDir: "{{ globals.artifacts_path }}"
    command: [python3, message.py]
    args: ["--message={{ message }}"]

To run this example:

polyaxon run -f passing-params/args-inline-python.yaml -P message="test 1" -l

How the argument is passed

The component passes the resolved input through its container args:

args: ["--message={{ message }}"]

At runtime, Polyaxon resolves {{ message }} and passes one argument such as --message=test 1 to message.py. See the params specification for other ways to transform inputs.

Upload the Python file

The inline example creates message.py during initialization. The quick-start repository also includes the same example as separate files:

passing-params/
  args-python/
    message.py
    message.yaml

passing-params/args-python/message.py contains:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--message", type=str)
args = parser.parse_args()
print(args.message)

passing-params/args-python/message.yaml contains:

version: 1.1
kind: component
name: uploaded-python-parameter
inputs:
  - name: message
    type: str
run:
  kind: job
  container:
    image: python:3.11
    workingDir: "{{ globals.run_artifacts_path }}/uploads/passing-params/args-python"
    command: [python3, message.py]
    args: ["--message={{ message }}"]

Use --upload/-u to upload message.py with the component:

polyaxon run -u -f passing-params/args-python/message.yaml -P message="test 1" -l

Note: You can learn more about uploads, Git, and inline scripts in the iterative process section.

Multiple params

Let's extend the previous example to require multiple parameters:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--message1", type=str)
parser.add_argument("--message2", type=str)
args = parser.parse_args()
print(args.message1)
print(args.message2)

The explicit component is available as passing-params/args-python/messages-explicit.yaml:

version: 1.1
kind: component
name: explicit-message-parameters
inputs:
  - name: message1
    type: str
  - name: message2
    type: str
run:
  kind: job
  container:
    image: python:3.11
    workingDir: "{{ globals.run_artifacts_path }}/uploads/passing-params/args-python"
    command: [python3, messages.py]
    args:
      - "--message1={{ message1 }}"
      - "--message2={{ message2 }}"

Run it with the two inputs:

polyaxon run -u -f passing-params/args-python/messages-explicit.yaml \
  -P message1="test 1" \
  -P message2="test 2" \
  -l

For a component with many inputs, params.as_args generates the argument list in one line. The checked-in version is passing-params/args-python/messages-all.yaml:

version: 1.1
kind: component
name: all-message-parameters
inputs:
  - name: message1
    type: str
  - name: message2
    type: str
run:
  kind: job
  container:
    image: python:3.11
    workingDir: "{{ globals.run_artifacts_path }}/uploads/passing-params/args-python"
    command: [python3, messages.py]
    args: "{{ params.as_args }}"

For these two inputs, "{{ params.as_args }}" resolves to the same two command-line arguments used by the explicit component.

polyaxon run -u -f passing-params/args-python/messages-all.yaml \
  -P message1="test 1" \
  -P message2="test 2" \
  -l