Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
All blog articles
Practical guides to building, running, and improving ML and AI in production.
All articles
Page 2 of 20
Keep distributed training workers close together
Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.
KubernetesSchedulingDistributed Training
Resume interrupted training without losing progress
Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.
MLOpsKubernetesPyTorch
Work interactively with Polyaxon sandboxes
Choose between sandbox shells and exec, understand PTY lifetime and output replay, and manage terminal attachment explicitly through the Python SDK.
ProductCliGuides
Did you find better hyperparameters—or a lucky seed?
Confirm hyperparameter finalists across matched training seeds, compare variation and paired differences, and reserve compute for a defensible final choice.
Hyperparameter TuningMLOpsExperimentation
Connect your local tools to Polyaxon with SSH
Connect native terminals and IDEs to a Polyaxon service, forward application ports, and use tmux explicitly when you want to resume a shell.
ProductCliGuides
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
Keep long-running workloads authenticated through token rotation
Use projected ServiceAccount tokens and refresh-aware clients so long-running training, notebooks, and services can keep authenticating without static credentials.
KubernetesSecurityReliability
Reconnect to your Polyaxon shell
Learn how ordinary ops shells differ from tmux sessions, then detach and reconnect to the same Polyaxon shell from the CLI or UI.
ProductCliGuides
Enforce ML workload rules with CEL admission policies
Use Kubernetes ValidatingAdmissionPolicy to check workload ownership and image references, with scoped warning and enforcement stages for ML namespaces.
KubernetesGovernanceSecurity
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Combine run filters with AND and OR
Use OR to combine metric thresholds, negated conditions, and independent groups of filters in Polyaxon queries.
ProductCliGuides
Control notebook and inference service lifetimes
Reuse Polyaxon's termination specification to bound notebook and inference service lifetimes, stop idle services, and distinguish inactivity from ongoing work.
PolyaxonSchedulingNotebooks
Automate multiple runs with async Python clients
Monitor several Polyaxon runs concurrently with async Python clients, retrieve recent logs, and manage concurrency, timeouts, and client cleanup.
ProductPythonGuides
Tune LLM hyperparameters within a compute budget
Design a small LoRA fine-tuning sweep, control its execution in Polyaxon, and compare validation quality, resource use, and repeatability.
Hyperparameter TuningLLMOpsGuides
Control Kubernetes Job retries with pod failure policies
Stop retrying permanent errors, preserve the retry budget for marked disruptions, and inspect Kubernetes Job failure decisions with a concrete example.
KubernetesSchedulingTroubleshooting
Write shorter Polyaxonfiles
Use direct parameter values and let input and output defaults imply optionality, with a look ahead at simpler workload definitions planned for Polyaxon 2.18.
ProductCliGuides
Connect LLM production traces to evaluation runs
Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.
LLMOpsObservabilityEvaluation
Mount model artifacts with Kubernetes image volumes
Separate immutable model files from the serving image with Kubernetes image volumes, and choose when an OCI artifact fits better than a download or PVC.
KubernetesStorageInference
Route ML jobs to regional infrastructure with Polyaxon
Configure regional execution with Polyaxon agents, queues, and storage connections, then measure data-transfer costs without assuming same-region traffic is free.
InfrastructureSchedulingGuides
Unify hybrid AI platform operations
Operate AI workloads across clusters and clouds with central intent, local execution, placement policy, consistent identity, connected evidence, and failure-aware control.
Hybrid CloudInfrastructureMLOps
Request GPUs by capability with Kubernetes DRA
Use Dynamic Resource Allocation to describe the accelerator a workload needs, understand device claims, and plan the integration with Polyaxon.
KubernetesGPUScheduling
Upload multiple files and folders with Polyaxon CLI
Upload code, configuration, and small datasets in one Polyaxon CLI command, then save the same path mappings in a declarative mount section.
ProductCliGuides
Reduce the cost of self-hosted ML workloads
Reduce self-hosted ML costs with outcome-based accounting, right-sized resources, elastic capacity, interruption-ready workloads, local data paths, and deliberate retention.
InfrastructureKubernetesScheduling