Building machine learning models is an experimental process that requires several iterations. You change different model parameters or data preprocessing steps at each iteration to obtain an optimal model. It is vital to keep track of the processing steps and the parameters at each iteration. Failure to do this leads to repetition of these steps, which leads to loss of time and computational resources.
The datasets and models should also be tracked to enable comparison between different experiments. Information about the models and datasets is referred to as metadata. In this article, we’ll explore the idea of experiment tracking and metadata storage in machine learning. We’ll also discuss why they are essential in your machine learning workflow.
What is experiment tracking?
Building machine learning models involves various steps such as data processing and model definition. Each step could result in new data and model versions. Furthermore, you may also try different parameters at each step. Other information you may be interested in tracking in your experiments include:
- Model evaluation metrics.
- Model evaluation charts.
- Model code.
- Sample predictions.
- Images, for example, in computer vision projects.
The process of tracking all this information is known as experiment tracking.
What is a machine learning metadata store?
Success in experiment tracking requires a tool that can keep track of all the information mentioned above and store it, i.e., a metadata store. A metadata store saves data that is generated in the process of building machine learning models. Storing experiment metadata enables comparability and reproducibility of ML experiments.
Using Polyaxon as a metadata store
Polyaxon is an MLOps platform that comes with a built-in metadata store that lets you track models and datasets and their resulting metadata in a central repository. Polyaxon also offers semantic versioning, extensive artifact logging, and dynamic reporting with seamless local and cloud integration.
Polyaxon is used by either individuals or organizations to manage machine learning projects. Polyaxon organizations enable a team to collaborate on different projects. The projects can either be public or private.
Polyaxon’s tracking module is an open-source tool that allows data scientists to log, store, view, and compare different experiments.
Polyaxon is different from other platforms because it allows for smooth local and remote development integration. For instance, you can use your local resources to run a few simple experiments and quickly transition to using a Kubernetes cluster managed by a Polyaxon Agent to train the model with more powerful computing resources. Polyaxon logs and versions metadata whether you are using local or remote mode.
Some items you can log using Polyaxon include:
- Charts.
- Images.
- Markdown
- Model metrics.
- Model parameters.
- DataFrames.
- Primitives.
- Gradio/Streamlit/Dash Apps
Polyaxon automatically versions datasets and models at each run. Further, Polyaxon allows you to log information about your data. For instance, you may be interested in recording:
- The data source.
- The data description.
- Summary statistics of the data using charts.
Collaboration is a very crucial part of any model development process. Documenting projects is important because it makes it easy for other people to work together on the project. It also makes it easy for you to return to the project in the future. For this reason, Polyaxon provides project, model, artifact, and run pages to make it easy to document your work. The documentation is provided using a README file with Markdown format. Polyaxon also allows for dynamic reporting by pasting links to your Polyaxon model, dataset, charts, etc in the README.
The documentation and community page provides multiple example projects and datasets to get you started quickly. There are other examples on the GitHub repo.
Tracking experiments with Polyaxon
You can easily add Polyaxon to new or existing ML projects. Polyaxon provides two methods to log metadata, a tracking module and a Python class with several methods that result in a rich model and data registry.
tracking.log_inputs, tracking.log_outputs, tracking.log_metrics, …
These methods enable logging params, results, metrics, tables, images, markdown/HTML content, and everything from your machine learning experiment. Then compare the results with your previous versions of your experiments.
tracking.log_model
Allows wrapping your model training function and stores and versions the resulting model to Polyaxon. This helps you manage the lifecycle of your model with semantic versioning. The model can then be fetched and used for making predictions immediately.
tracking.log_data_ref, tracking.log_artifact
Tracking datasets, artifacts, and other assets to save the resulting information. Polyaxon will semantically version the data/artifact making your operations reproducible.
Tracking and executing operations and pipelines
Polyaxon is an advanced ML metadata store and pipeline runner. You can use Polyaxon infrastructure to seamlessly build your datasets, train your models, and store your resulting model and dataset artifacts. This is especially useful when:
- Your training data is too big to fit in your local machine.
- Your model requires special infra like a high-end GPU that is too expensive to provision permanently.
Final thoughts
In this article, you have learned what experiment tracking is and how you can implement it in your project. We have also covered:
- What is a metadata store?
- How to use Polyaxon to keep track of your machine learning projects’ metadata.
- How to improve your projects with dynamic project documentation.
Visit Polyaxon to start building now! Don’t forget to join our Slack community, and follow us on Twitter and Linkedin for resources, events, and much more.
