V1IO

polyaxon._flow.io.io.V1IO()

Each Component may have its own inputs and outputs. The inputs and outputs describe the expected parameters to pass to the component and their types. In the context of a DAG, inputs and outputs types are used to validate the flow of information going from one operation to another.

The final value of an input/output can be resolved from params, or from other values in the context.

  • Examples:
  • A build component may have a git repository as input and a container image as output.
  • A training component may have a container image, data path, and some hyperparameters as input and a list of metrics and artifacts as outputs.

An input/output section includes a name, a description, a type to check the value passed, a flag to tell if the input/output is optional, and a default value if it’s optional.

Sometimes users prefer to pass a param to the command or args section only if the value is not null. Polyaxon provides a way to do that using {{ params.PARAM_NAME.as_arg }}, this will be empty if the value is null or --PARAM_NAME=value if the value is not null. If the value is of type bool, it will be --PARAM_NAME. It’s also possible to control how the param should be converted to an argument using the arg_format.

To learn how to pass valid values, please check the param section.

  • Args:
    • name: str
    • description: str, optional
    • type: str, any python type hint, pydantic built-in types, and gcs, s3, wasb, dockerfile, git, image, event, artifacts, path, metric, metadata, date, datetime.
    • value: any, optional
    • is_optional: bool, optional (Deprecated)
    • is_list: bool, optional
    • is_flag: bool, optional
    • arg_format: str, optional
    • connection: str, optional
    • to_init: bool, optional
    • to_env: str, optional
    • validation: V1Validation, optional
    • delay_validation: bool, optional (Deprecated: please see valiation.delay)
    • options: List[any], optional (Deprecated: please see valiation)

YAML usage

inputs:
  - name: loss
    type: str
    isOptional: true
    value: MeanSquaredError
  - name: preprocess
    type: bool
outputs:
  - name: accuracy
    type: float
  - name: outputs-path
    type: path

Python usage

from polyaxon import types
from polyaxon.schemas import V1IO
inputs = [
    V1IO(
        name="loss",
        type='str',
        description="Loss to use for training my model",
        is_optional=True,
        value="MeanSquaredError"
    ),
    V1IO(
        name="preprocess",
        type='bool',
        description="A flag to preprocess data before training",
    )
]
outputs = [
    V1IO(
        name="accuracy",
        type='float',
    ),
    V1IO(
        name="outputs-path",
        type=types.PATH,
    )
]

These inputs/outputs declarations can be used to pass values to our program:

 ... --loss={{ loss }} {{ params.preprocess.as_arg }}

Fields

name

The input/output name. The name must be a valid slug, and cannot include dots ..

inputs:
  - name: learning_rate

description

An optional description for the input/output.

inputs:
  - name: learning_rate
    description: A short description about this input

type

The type of the input/output. The type will be used to validate the value

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float

for more details about composite type validation and schema, please check the types section, possible types include any python type hint, pydantic built-in types, and:

  • URI: “uri”
  • LIST: “list”
  • GCS: “gcs”
  • S3: “s3”
  • WASB: “wasb”
  • DOCKERFILE: “dockerfile”
  • GIT: “git”
  • IMAGE: “image”
  • EVENT: “event”
  • ARTIFACTS: “artifacts”
  • PATH: “path”
  • METRIC: “metric”
  • METADATA: “metadata”

value

If an input is optional you should assign it a value. If an output is optional you can assign it a value.

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float
    value: 1.1

isOptional

A flag to tell if an input/output is optional.

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float
    value: 1.1
    isOptional: true

isList

Deprecated: Please use type: List[TYPING] instead.

In v2 you should:

inputs:
  - name: learning_rates
    type: List[float]

A flag used in v1 to tell if an input/output is a list of the type passed.

inputs:
  - name: learning_rates
    type: float
    isList: true

In this case the input name learning_rates will expect a value of type List[float], e.g. [0.1 0.01, 0.0001]

argFormat

A key to control how to convert an input/output to a CLI argument if the value is not null, the default behavior:

  • For bool type: If the resolved value of the input is True, "{{ params.PARAM_NAME.as_arg }}" will be resolved to "--PARAM_NAME" otherwise it will be an empty string "".
  • For non-bool types: If the resolved value of the input is not null, "{{ params.PARAM_NAME.as_arg }}" will be resolved to "--PARAM_NAME=value" otherwise it will be an empty string "".

Let’s look at the flowing example:

inputs:
  - name: lr
    type: float
  - name: check
    type: bool

This manifest:

container:
   command: ["{{ lr }}", "{{ check }}"]

Will be transformed to:

container:
   command: ["0.01", "true"]

This manifest:

container:
   command: ["{{ params.lr.as_arg }}", "{{ params.check.as_arg }}"]

Will be transformed to:

container:
   command: ["--lr=0.01", "--check"]

Changing the behavior with argFormat:

inputs:
  - name: lr
    type: float
    argFormat: "lr={{ lr }}"
  - name: check
    type: bool
    argFormat: "{{ 1 if check else 0 }}"

This manifest:

container:
   command: ["{{ params.lr.as_arg }}", "{{ params.check.as_arg }}"]

Will be transformed to:

container:
   command: ["lr=0.01", "1"]

connection

A connection to use with the input/outputs.

toInit

If True, it will be converted to an init container.

toEnv

N.B: Requires Polyaxon CLI and Polyaxon Agent/CE version >= 1.12

If passed, it will be converted automatically to an environment variable.

inputs:
  - name: learning_rate
    type: float
    value: 1.1
    toEnv: MY_LEARNING_RATE

validation [v2]

A schema to use to validate the input/output value or to delay the validation to runtime.

To instruct the parser to only validate the input/output at compilation or resolution time:

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float
    value: 1.1
    validation:
      delay: true

This flag is enabled by default for outputs, since they can only be resolved after or during the run. To request validation at compilation time for outputs, you need to set this flag to False.

To validate an input/output value:

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float
    value: 1.1
    validation:
      min: 0.01
      max: 0.1

Options allow to pass a list of values that will be used to validate any passed params.

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float
    value: 0.1
    validation:
      options: [0.1, 0.2, 0.3]

delayValidation

Deprecated: Please use validation: ... instead.

A flag to tell if an input/output should not be validated at compilation or resolution time.

This flag is enabled by default for outputs, since they can only be resolved after or during the run. To request validation at compilation time for outputs, you need to set this flag to False.

options

Deprecated: Please use validation: ... instead.

Options allow to pass a list of values that will be used to validate any passed params.

inputs:
  - name: learning_rate
    description: A short description about this input
    type: float
    value: 1.1
    options: [1.1, 2.2, 3.3]

If you pass the value 4.4 for the learning rate it will raise a validation error.

Example

version: 1.1
kind: component
inputs:
  - name: batch_size
    description: batch size
    isOptional: true
    value: 128
    type: int
  - name: num_steps
    isOptional: true
    default: 500
    type: int
  - name: learning_rate
    isOptional: true
    default: 0.001
    type: float
  - name: dropout
    isOptional: true
    default: 0.25
    type: float
  - name: num_epochs
    isOptional: true
    default: 1
    type: int
  - name: activation
    isOptional: true
    default: relu
    type: str
run:
  kind: job
  image: foo:bar
  container:
    command: [python3, model.py]
    args: [
        "--batch_size={{ batch_size }}",
        "--num_steps={{ num_steps }}",
        "--learning_rate={{ learning_rate }}",
        "--dropout={{ dropout }",
        "--num_epochs={{ num_epochs }}",
        "--activation={{ activation }}"
    ]

Running a typed component using the CLI

Using the Polyaxon CLI we can now run this component and override the inputs’ default values:

polyaxon run -f polyaxonfile.yaml -P activation=sigmoid -P dropout=0.4

This will generate a manifest and will replace the params passed and validated against the inputs’ types.

Required inputs

In the example all inputs are optional. If we decide for instance to make the activation required:

...
inputs:
  ...
  - name: activation
    type: str
  ...
...

By changing this input, polyaxon can not run this component without passing the activation:

polyaxon run -f polyaxonfile.yaml -P activation=sigmoid