Since the last stable release, we have been talking with Polyaxon’s users to gather feedback, pain points and possible enhancements to bring to the platform. Polyaxon has seen a steady growth, our API has over 735K installs (via public Docker Hub) and collectively our tools have been downloaded more than 3.5M times (via public Docker Hub). Today we are releasing a new version to address some of the feedback we have been hearing.

v0-5

New features in Polyaxon v0.5 include:

  • Updated dependencies for higher availability and better monitoring.
  • Plugins oriented architecture.
  • Improved deployment and management experience.
  • Local run experience.
  • Polyaxonfiles typing.
  • Catalogs and resources management.
  • Improved security.
  • Improved search filters.
  • Automation and pipelining with Polyflow.

Up-to-date dependencies

When we started working on Polyaxon, several dependencies have been built inside Polyaxon’s Helm chart to insure reproducible deployments. As the Helm packages ecosystem is evolving fast, we start to see the shortcomings of this decision, several upstream dependencies became stable and start receiving further enhancements. v0.5 has now completely updated dependencies, with no customization in the Polyaxon chart.

This decision pushed several feature requests high in the priority, such as allowing teams to deploy Polyaxon with managed version of these dependencies.

Polyaxon now supports high available deployment of all third party requirements, every dependency can be fully customized, and users can disable the dependency and opt for a completely managed solution either in-cluster or on their favorite cloud provider. We also published several guides for integrating cloud solutions with Polyaxon.

Plugins oriented architecture

This release includes several internal enhancements that will allow the platform to be easily extended with backend and frontend plugins. Currently only plugins developed by Polyaxon will be supported, and in future releases we will document how anyone can write their own plugins.

Some of the direct benefits of this change:

  • Single port for Polyaxon’s APIs.
  • Support for all JupyterLab versions

Before the v0.5 release, Polyaxon was exposing 2 ports, which meant that any team deploying Polyaxon had to secure 2 ports. With this change we addressed this issue and provided a solution that will be used for other plugins that need to be run as services.

In Polyaxon v0.4, several users were limited by the version of JupyterLab they can deploy in their projects, some users were opting for Notebooks to circumvent the issue that Polyaxon had deploying some versions of JupyterLab. This architectural change, fixes this issue and users can now deploy any installable JupyterLab version on Polyaxon.

Improved deployment and management experience

Dynamic options

Several Polyaxon configuration options are now stored in the database, not in a configuration file. This change will have a huge impact on the deployment and management process of Polyaxon.

Before this change, Polyaxon CE was completely based a configuration file, a team managing Polyaxon had to redeploy Polyaxon every time they make a configuration change, some of these configuration changes could be just typos or a misconfigured secrets.

Polyaxon has a now a completely dynamic configuration process with a UI accessible to admins. Several options such as Github, GitLab, Bitbucket authentication, default node scheduling for runs, default resources, default annotations, default backends, and many more options are completely managed by the UI.

Polyaxon admin

Polyaxon cli has now an admin command for managing the lifecycle of a Polyaxon deployment, polyaxon admin command takes care of deploying, upgrading, and managing the correct version and correct namespace specified in the deployment file.

Users can still opt to use Helm for managing a Polyaxon deployment, but often times they might upgrade a Polyaxon deployment to a newer version unintentionally. Using the admin command will make sure that you always deploy/upgrade to the version in the deployment file.

In future versions, Polyaxon-cli will allow to manage and authenticate to multiple Polyaxon deployments.

Local runs

Polyaxon v0.5 comes also with the possibility to run and track a job or an experiment locally. We think that this is an important feature for data-scientists to have faster feedback on their code before scheduling longer runs on GPUs/TPUs or a hyperparameters search with several parallel runs.

Local runs are easy to start, users just need to add--local flag:

polyaxon run -f polyaxonfile.yaml --local

Polyaxonfile typing

Although Polyaxon is good at guessing and validating the types of parameters in Polyaxonfiles, several users had issues running some polyaxonfiles, in the v0.5 users can declare their Polyaxonfiles with typed inputs/outputs that validate that the params are of certain types, users can also provide default values for these inputs.

Polyaxon-cli allows to override params in very simple way, before v0.5 overriding params in a polyaxonfile required another polyaxonfile polyaxon run -file1 -f file2, now users can just pass the params directly to the CLI:

polyaxon run -f polyaxonfile.yaml -P param1=value1 -P param2=value2

This will validate that the values correspond to the declared types and override the default values if they exist.

N.B. Typing is required for creating Polyflow components.

Catalogs

Polyaxon v0.5 introduces some new concepts for managing resources, connexions, git access, registry access, data access, and artifacts: Catalogs.

With catalogs users can declare what data access they want their users to have access to with all secrets and authentication requirements:

feature-1

In v0.5 users will be using some access types, more access types will be generally available in v0.6/v0.7.

Improved security

Polyaxon can be deployed with security contexts, this is a first iteration, and it will allow several teams to deploy Polyaxon on their cluster without breaking any compliance requirements.

Automation and pipelining

Improving machine learning workflows means automation and reusability. Polyaxon’s important feature is increasing the speed of developing machine learning applications.

This release introduce a new abstraction to enhance the speed of machine learning iteration: Polyflow, an event/action engine.

Polyflow is an open source project that provides container-native engine for running machine learning pipelines on Polyaxon. Each operation in a Polyflow pipeline is defined as pod running a main container defined by the user.

Often times, Polyaxon’s users tend to run complex jobs and experiments that depend on each other. Polyaxon defines several primitives that can run independently, each one of these primitives is a mini-pipeline. For example:

  • An experiment requires a build to prepare a docker image before running
  • A hyperparameters group run several experiments and follow their progress

Polyflow is an extension to these primitives to allow the users to automate workflows and define complex interactions.

Polyflow Features

  • Easy-to-use: Polyflow is easy to use and does not add any extra complexity or extra dependencies to a cluster already running Polyaxon.
  • Scalability: Polyflow scales similarly to Polyaxon, defines extra API endpoints, and reuses same logic for scheduling and monitoring runs.
  • Flexibility: Similar to Polyaxon, Polyflow allows users to run anything that can be run in a container.
  • Reusability: Polyflow introduces new concepts that will allow several teams to operationalize and reuse several logic and package them as templates/actions/events.
  • Kubernetes and Polyaxon Native: Polyflow integrates natively with Polyaxon and reuses it’s components, and it also allows to leverage Kubernetes services such as volumes, secrets, and RBAC.
  • Interoperability: Similar to Polyaxon, Polyflow has a native backend, but can be used to leverage other backends: e.g. airflow.

Upgrading to v0.5

If you are upgrading from Polyaxon v0.4, please follow the migration guide.

Learn More about Polyaxon

To learn more about all the features, fixes, and enhancements in v0.5, please visit the [release notes]/docs/releases/0-5/).

Polyaxon continues to grow quickly and keeps improving and providing the simplest machine learning abstraction. We hope that these updates will improve your workflows and increase your productivity, and again, thank you for your continued feedback and support.