Polyaxon monthly updates
As many of you know, at Polyaxon, we maintain a fast pace of development and ship several features on a weekly or bi-weekly basis. Polyaxon Cloud is released almost daily, and sometimes multiple times per day. While we try to keep our constantly growing user-base up-to-date on our latest and greatest features, many have requested a better change log location to look for updates. Currently, we keep up-to-date release notes, but it lacks visual elements to give more context to UI and UX features and highlights the most important changes.
From this release on, we will use the announcements tag to go over interesting new features or enhancements that we think might go unnoticed in the release notes on a monthly basis.
New hyperparameter tuning capabilities
This release comes with several new features and enhacenements to the hyperparameter tuning capabilities that we talked about in details in this blog post.
Run info and metadata
The main page that displays the run metadata was reworked to surface more metadata and information, this blog post goes through the new enhancements.
New search and visualization interface
Polyaxon v1.3 introduced several quick filters and multi-run visualizations and comparison features, you can explore the new enhancements in this blog post.
Better cache hit compilation and UX
We have reworked the caching logic to improve the performance of time consuming computations, we also intorduced new heuristics to detect cache hits faster.
Polyaxonfile UI
In Polyaxon v0 and the first release of Polyaxon v1, we only displayed the raw content of the Polyaxonfile used for creating an operation and starting a run. We later added the compiled specification to help users have a better idea about the compiled version of their Polyaxonfile and how the parameters and the context were resolved and injected in their template. In v1.2 we added another tab that contains a split view where it’s easier to view the component and the operation.
-
Split mode view
-
Compiled op view
Updating the last status from the UI
It was possible to use the Python Client, the API, or one of the language SDKs to update/override the last status of a run. In Polyaxon v1.3 we added a new possibility to do the same procedure directly from the UI.
New pricing and plans structure
Polyaxon cloud delivers all the key features of the Polyaxon suite but in a completely managed cloud-native environment. Users only need to deploy our fault-tolerant agent on their Kubernetes cluster, Polyxon Control Plane delivers all orchestration and scheduling abstractions and lets you seamlessly connect your compute resources in minutes.
We refactored the pricing for Polyaxon Cloud entry plans to enable more features for all plans. We are also displaying transparent pricing for additional agents. And we start offering a 14-days free trial on all Cloud plans, no credit card required.
While Polyaxon always delivered the power needed to do data science at scale, teams also needed strong system administration to manage and keep a stateful service healthy such as Polyaxon running on Kubernetes in production. Polyaxon needs monitoring and management, backup, redundancy, capacity and upgrade planning, to name just a few. That represents a real challenge for some organizations. In addition to the complexity of Kubernetes, users running Polyaxon in production need to keep up with the updates and migrations that we ship continuously.
With Cloud, we’ve removed the barriers to entry and decoupled the control plane from the workload, code. datasets, and artifacts that stay on the users’ cluster. We only provide a smaller subset of our services to run on the user’s cloud or on-prem cluster, which makes it possible for smaller teams to start doing data science faster.
Polyaxon Cloud includes all of the features that were challenging for teams to set up, scaling over multiple namespaces and clusters, security and isolation, backup, and automation.
Upcoming features
With our new offering, we want to deliver on our ultimate vision for a fully collaborative, shareable, scallable, portable, privacy preserving, and reproducible data science platform.
Over the next couple of releases, we are working on more features to achieve that vision, some of these features include:
Component Hub
Component Hub is coming out of beta in the next release and will be enabled to all accounts following their quotas.
Model registry
Model Registry is still in beta, but we will soon start enabling some of the features that are getting stable such as experiment promotion, locking, and versioning in the registry. The more complex features such as feedback metrics and monitoring will take a bit more time until we finalize the events interface to avoid breaking changes for a large number of users.
Control plane connections management
Currently updating a connection in an agent requires a redeploy of the agent with the updated configuration. We are currently working on a new feature to allow organizations using Polyaxon Cloud or Polyaxon EE to opt to manage connections via the control plane UI, which should essentially eliminate the need for a redeploy when a connection definition needs to change. This should:
- Improve testing new connections
- Reduce friction
- Improve adoption of more granular connections for datasets, git repos, …
This change also does not impact in any way the isolation, security, and privacy we strive to provide to our customers both Cloud and EE, since all secrets and actual logic is still hosted and runs on the users’ clusters.
More lineage information
Currently, Polyaxon UI shows lineage information related to artifacts. In the upcoming releases, Polyaxon’s lineage page will show more information about:
- Related runs (restarts and copies).
- Upstream and downstream runs when an operation is running in the context of a DAG or if an operation requires artifacts or inputs from previous runs.
- Connections lineage information.
Upload command
This is one of the requests that keep coming, especially from users of Polyaxon v0, we will provide a simple way to upload code and also artifacts that need to be provisioned before a run starts. We are also thinking about making the build-and-run recipe as simple as in v0.
Adding the schedule_at information to the compiler’s context
Polyaxon schedules are fault-tolerant and highly available, although they provide a similar interface to cron, they run within our scheduler and they are decoupled from the cron system to handle several use cases related to time zones, availability, and conditioning. When there’s an issue on the user’s agent deployment, and the agent is not picking queued operations, a scheduled run might not run at the DateTime when it was supposed to. As soon as the agent is healthy again and can connect to the control plane, Polyaxon will not only schedule the upcoming runs on time but it will also schedule all runs that were supposed to run before. This is both a feature and a bug and in the upcoming releases we would like to:
- Give the user the option to specify if the scheduler should start those run or not.
- Pass the
schedule_atinformation to the compiler, so that the user code can use it to handle extra logic on their code.
Add graph visualization for DAG and Matrix run
Currently, Polyaxon UI shows a grouped view for dags, hyperparameter tuning, and mapping using the comparison table.
In future releases, we would like to bring a visual view using dagre-d3 to show the dependency between runs in the context of a pipeline.
Learn More about Polyaxon
This blog post just goes over a couple of features that we shipped since our last product update, there are several other features and fixes that are worth checking. To learn more about all the features, fixes, and enhancements, please visit the release notes.
Polyaxon continues to grow quickly and keeps improving and providing the simplest machine learning abstraction. We hope that these updates will improve your workflows and increase your productivity, and again, thank you for your continued feedback and support.
