Polyaxon v1.9 release has several new featues and improvements, we shared several blog post to instroduce some notable featues, like:

Improved upload command

We are very happy to finally announce that the upload command is now stable and generally available. For the last couple of months, the upload command was in a beta phase and had several edge cases. In this release we reworked several aspects of the upload logic to handle:

  • Upload and eager mode: Polyaxon CE users can now start a grid or random search or a mapping in eager mode with the upload command.
  • Restarting operations where the code or some artifacts were initialized using the upload command.
  • Auto-tracking of lineage information of the uploaded artifacts.

Improved restart logic

Polyaxon restart command has also received several new quality-of-life enhancements, users can restart operations with new names, descriptions, and tags. The restart command will also automatically forward any artifacts that were necessary during the initialization process (previously the restart logic would fail for runs started with the upload command).

The restart command with the copy mode was also significantly improved; previously the copy mode copied all artifacts, but we have heard from users that they sometimes need more control over the artifacts to copy during the restart process, for example copying a specific checkpoint in case of a deep learning experiment.

The CLI exposes new arguments that allows users to specify the directories and/or files to copy during the restart process:

polyaxon ops restart --copy-dir=dir1 --copy-dir=path/dir2 --copy-file=path/dir3/file1

And to copy everything, they can use the --copy flag:

polyaxon ops restart --copy

New plugin to mount the artifacts store

Several users request the default artifacts store for their operations, whether to have direct access to previous runs’ outputs or to manage new outputs manually. Previously, in order to request the default artifacts store, users needed to add store name to the connections list in their manifests. This is still possible, but for most use-cases, this means that the components and operations are decorated with specific values and dependent on the deployment configuration of that Polyaxon instance.

Starting with v1.9.1, users can request the artifacts store without setting any connection name, which should improve portability and readability of the polyaxonfiles:

plugins:
  mountArtifactsStore: true

Additionally, requesting the artifacts store via the plugins section will automatically inject a new global variable in the context: globals.store_path which should abstract the need to access the schema or to know beforehand if the artifacts store is a volume, a host path, or blob storage bucket.

New Tensorboard versions

Based on the new mountArtifactsStore plugin, we are now distributing two new Tensorboard versions tensorboard:single-run-storepath and tensorboard:multi-run-storepath that will start Tensorboard based on the artifacts store directly instead of using the initializer.

These versions are useful to start a service for experiments still running since Tensorboard will keep syncing the data.

Note: These versions will not work with Azure blob, Minio artifacts stores, or other unsupported backends.

Docs Improvements

We refactored the intro section of the documentation to provide a comprehensive tutorial and guides to get started with Polyaxon. This refactoring was based on the feedback of several users, the idea is to provide a simple introduction to several aspects of the platform and only reserve the reference sections for users who need to customize specific aspects of their manifests.

We also added new sections and tutorials to host and serve in-cluster REST APIs based on Streamlit, FastAPI, and Flask for ML models, these tutorials do not intend to replace platforms that are built for production serving and deployment, rather they intend to give a simple infrastructure and guidelines for hosting internal API and services or for testing purposes before moving to a system like KFServing.

Upcoming automatic build

We are making a good progress on reintroducing an automatic build process, and we intend to release a beta version on Polyaxon Cloud next week or the week after. This initial release will provide the following feature:

  • It will not replace the ad-hoc build operations, users can still create independent polyaxonfiles with a kaniko/dockerize hub ref.
  • The no-build requirement that the platform provides does not change, users who have stable pipelines that do not require frequent changes to their images can safely ignore this feature.
  • A new section called build allows to define the necessary fields for creating a container as well as other flags for the queue, preset, resources, node selectors, … specific to the build.
  • An image based on the project and the run’s uuid, i.e. project:build-uuid, is generated automatically and set on the main container.
  • When the build and matrix sections are used together, a single build operation will be scheduled and will be used for all runs.

Learn More about Polyaxon

This blog post just goes over a couple of features that we shipped since our last product update, several other features and fixes are worth checking. To learn more about all the features, fixes, and enhancements, please visit the release notes and the short term roadmap.

Polyaxon continues to grow quickly and keeps improving and providing the simplest machine learning abstraction. We hope that these updates will improve your workflows and increase your productivity, and again, thank you for your continued feedback and support.