ReferenceOverview
OrchestrationMatrixOverview

Overview

Hyperparameters selection is crucial for creating robust models, since they heavily influence the behavior of the learned model. Finding good hyperparameters can be very challenging, and requires to efficiently search the space of possible hyperparameters as well as how to manage, schedule, and monitor a large set of experiments for hyperparameter tuning,

The way Polyaxon performs hyperparameter tuning is by providing a selection of customizable search algorithms. Polyaxon supports both simple approaches such as random search and grid search, and provides a simple interface for advanced approaches, such as Hyperband and Bayesian Optimization, it also integrates with tools such as Hyperopt, and provides an interface for running custom iterative processes.

All these search algorithms run in an asynchronous way, and support concurrency and routing to leverage your cluster(s)'s resources to the maximum.

Some of these approaches are also iterative and improve based on previous experiments.

Features

  • Easy-to-use: Polyaxon's search algorithms are built-in and can be used by adding a matrix section to your operations. You can run hyperparameter tuning using the CLI, client, and dashboard.
  • Scalability: Tuning hyperparameters or neural architectures requires leveraging a large amount of computation resources, using Polyaxon you can run hundreds of trials in parallel and intuitively track their progress.
  • Flexibility: Besides the rich built-in algorithms, Polyaxon allows users to customize various hyperparameter tuning algorithms, neural architecture search algorithms, early stopping algorithms, etc.
  • Efficiency: We are intensively working on more efficient model tuning from both system level and algorithm level. For example, leveraging early feedback to speedup tuning procedure.
  • Agnostic: The interface that we provide is agnostic to the framework, and even the language, used to define the model or function to optimize, hence you can use any machine learning framework, including PyTorch, XGBoost, MXNet, Tensorflow, and Keras.
  • Visualization and Dashboards: Polyaxon's Optimization Engine reuses the same logic and core features for jobs, tracking, checkpoint management, and integration with TensorBoard.

Workflow

  • Define a search space.
  • Define a search algorithm.
  • Define a model to optimize.
  • Optionally define the queuing, routing, concurrency, and early stopping.

Algorithms

In order to search a hyperparameter space, all search algorithms require a matrix section, they also share some subsections such as: params definition of hyperparameters, earlyStopping, and concurrency. Each one of these algorithms has a dedicated subsection to define the required options.

Pipeline helpers

All optimization algorithms can leverage the pipeline helpers for managing concurrency, early stopping, caching, ...

Run names

Each run created by a matrix is named after the matrix run, followed by a suffix. By default the suffix is a short id taken from the run's uuid, for example train-a1b2c3d4.

Use nameSuffix to set your own suffix from the values of each run:

name: train
matrix:
  kind: grid
  nameSuffix: "{{ lr }}-{{ batch_size }}"
  params:
    lr:
      kind: choice
      value: [0.01, 0.1]
    batch_size:
      kind: choice
      value: [32]

This creates the runs train-0_01-32 and train-0_1-32. The matrix run itself stays train.

  • The suffix is rendered once for each run, when the run is created. It can use the run's params and the globals that are set at that point, including globals.iteration.
  • Creating the runs fails if the suffix uses an unknown variable.
  • Each run of characters other than letters, digits, _ and - becomes one _. Creating the runs fails if nothing is left.
  • A plain text suffix works too. Names can repeat; the run's uuid identifies it.
  • nameSuffix: "" turns the suffix off, and every run takes the name of the matrix run.
  • A name longer than 128 characters is cut to fit, and the suffix is kept.
  • This applies to all matrix kinds, including mapping.

Search Space

In order to define a search space, users must define how to generate the values that will be used to create the parameters combination for running the component, The params are defined as {key: value} object where the key is the name of the parameter you are defining and the value is one of these options:

Discrete params

  • choice
  • range
  • logspace
  • linspace
  • geomspace

Distributions params

  • pchoice
  • uniform
  • quniform
  • loguniform
  • qloguniform
  • normal
  • qnormal
  • lognormal
  • qlognormal

You can check all the options available in the params section reference.

Example usage

matrix:
  kind: random
  params:
      lr:
        kind: logspace
        value: 0.01:0.1:5

      loss:
        kind: choice
        value: [MeanSquaredError, AbsoluteDifference]

These values can be accessed in the following way:

--lr={{ lr }} --loss={{ loss }}