Multi-cluster GPU orchestration
Design multi-cluster GPU orchestration around workload eligibility, data locality, queue routing, recovery, and clear dispatch ownership.
OrchestrationKubernetes
The Polyaxon blog
Build better ML & AI systems.
Practical guides to building, running, and improving ML and AI in production.
Kubernetes
GPU orchestration, distributed training, and multi-tenant AI workloads.
Gang scheduling for distributed training
Understand how gang scheduling prevents partial distributed jobs from holding GPUs, how minimum membership works, and what Polyaxon supports.
SchedulingKubernetes
GPU cluster scheduling tools compared
Compare Kueue, KAI Scheduler, Volcano, Coscheduling, and Slurm Bridge by responsibility, workload fit, and Polyaxon integration path.
SchedulingKubernetes
Keep distributed training workers close together
Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.
KubernetesSchedulingDistributed Training
Design multi-tenant AI agent infrastructure on Kubernetes
Separate identity, network access, data, queues, budgets, and evidence when multiple teams or customers run AI agents on shared Kubernetes infrastructure.
AI AgentsKubernetesSecurity
MLOps
Track experiments, tune models, manage compute costs, and recover training.
ML Metadata Store: Examples, Artifacts, and Lineage
Understand ML metadata with examples of runs, datasets, and model artifacts. Learn how a metadata store connects them and how to query them in Polyaxon.
MLOps
ML Experiment Tracking: What to Log and How to Compare Runs
Learn what to record for each ML experiment, how to compare runs fairly, and how to log parameters, metrics, datasets, and model artifacts with Polyaxon.
MLOps
Hyperparameters tuning on Kubernetes
hyperparameters tuning is very important concept in order to choose the optimal hyperparameters for a given algorithm!
ProductMLOps
Optimize multi-cloud costs for ML workloads
Control multi-cloud ML spending with normalized allocation, workload unit economics, placement policies, elastic capacity, and reproducible efficiency measurements.
FinopsCost OptimizationMLOps
Resume interrupted training without losing progress
Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.
MLOpsKubernetesPyTorch
AI Agents
Agent runtimes, Kubernetes isolation, and parallel coding workflows.
The Polyaxon agent sandbox roadmap
Our plans for persistent sandbox workspaces, suspend/resume, activity-aware idle suspension, and ready pools on Kubernetes—and where AX and Substrate differ.
RoadmapSandboxesAI Agents
Kubernetes for AI agents: A platform engineering guide
Design a Kubernetes platform for AI agents with clear workload boundaries, durable state, scoped access, resource controls, and end-to-end evidence.
AI AgentsKubernetesInfrastructure
Choose an AI agent architecture for production
Map agent architectures to Polyaxon jobs, services, and DAGs, with concrete workload configuration and a repeatable architecture-comparison workflow.
AI AgentsOrchestrationGuides
Sandbox AI agents on Kubernetes
Design isolated Kubernetes execution for AI agents with a threat model, stronger runtimes, scoped identity, controlled egress, bounded storage, and auditable cleanup.
AI AgentsKubernetesSecurity
Orchestrate parallel coding agents with Polyaxon
Run parallel coding agents in separate Polyaxon workspaces, collect candidate patches, evaluate independently, and merge only reviewed changes.
AI AgentsSandboxesCollaboration
Designing the runtime layer for AI agents
Design agent execution around durable state, safe tool retries, approval waits, and recovery, with clear responsibilities for Polyaxon and your agent framework.
AI AgentsOrchestrationGuides
Operate dynamic agent plans with bounded execution
Operate dynamic AI agent plans with Polyaxon execution, explicit policy checks, cumulative budgets, durable request state, and independent evaluation.
AI AgentsOrchestrationGovernance
LLMOps
Evaluate, observe, and improve LLM applications in production.
Turn production traces into regression tests
Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.
LLMOpsObservabilityEvaluation
What is LLMOps? From prototype to production
LLMOps applies repeatable development, evaluation, deployment, and observability practices to production LLM applications and AI agents.
LLMOpsMLOps
LLM-as-a-judge: Design and validate model-based evaluators
Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.
EvaluationLLMOps
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
DataOps
Build reliable data pipelines and preserve the evidence behind your models.
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
DataOps vs. MLOps: Differences and How They Work Together
DataOps delivers reliable data; MLOps delivers reliable ML systems. Compare their responsibilities, checks, and handoffs using a practical example.
MLOpsDataOps
Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
Evaluate with only the data available at decision time
Reconstruct evaluation features using event time and actual availability time, excluding late arrivals and later corrections while preserving the source versions in Polyaxon.
DataOpsEvaluationData
Verified data pipelines for reliable AI agents
Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.
AI AgentsDataOpsEvaluation
Recent articles
The latest guides, tutorials, and product updates.
Let an agent drive ML experiments on Polyaxon
Host an agent controller on Polyaxon to launch training and evaluation jobs, inspect results, and choose the next experiment.
AI AgentsMLOpsExperimentation
One replica model for inference, training, and RL
Polyaxon's roadmap for named replica groups, heterogeneous services, coordinated training, and RL workloads across Kubernetes scheduling backends.
RoadmapPolyaxonKubernetes
Diagnose NCCL timeouts in Kubernetes training
Find the first failed or stalled training rank, preserve Kubernetes evidence, and use PyTorch collective diagnostics in Polyaxon.
KubernetesPyTorchDistributed Training
Durable execution for AI agents
Build agent recovery around committed checkpoints, versioned inputs, safe side effects, and approval state so interrupted tasks can continue reliably.
AI AgentsOrchestrationGuides
Docker build caching for ML workloads on Kubernetes
Reduce container build and startup time for ML jobs with reusable dependency layers, persistent build caches, and deliberate image and model caching.
DockerKubernetesInfrastructure
Deploy Qwen3 Reranker on Polyaxon
Deploy Qwen3-Reranker-0.6B as a Polyaxon GPU service, rerank a text-search shortlist, and compare relevance against added latency.
LLMOpsInferenceRAG
Compare runs across synchronized charts
Compare six experiments at the same epoch, zoom across metric charts together, and configure tooltips with multiple live metric columns and run parameters.
ProductExperimentation
Finish Indexed Jobs after enough attempts succeed
Use Kubernetes Job success policies for chosen indexes or success counts, preserve durable results, and distinguish them from Polyaxon metric early stopping.
KubernetesPipelinesScheduling
What is data lineage in machine learning?
Trace a model back to its data and transformations, find consumers of a changed dataset, and inspect recorded inputs and outputs in Polyaxon.
DataOpsMLOpsLineage
Spread inference replicas across Kubernetes zones
Use topology spread constraints for inference availability, understand minDomains and GPU capacity, and preserve placement intent in Polyaxon.
KubernetesSchedulingInference
Run PyTorch on GKE TPUs with Polyaxon
Deploy TorchTPU through a simplified Polyaxonfile, train with eight PyTorch workers on four TPU7x chips, and track metrics and model weights.
MLOpsPyTorchKubernetes
Align CPU and GPU resources with Kubernetes NUMA policies
Combine CPU Manager eligibility, full-core allocation, and Topology Manager policy, then compare CPU/GPU workloads with Polyaxon.
KubernetesGPUPerformance
Set up a Polyaxon code execution workspace
Enable sandbox execution on a Polyaxon service, reuse its image and compute settings, run commands through the SDK, and preserve results before cleanup.
SandboxesPythonTutorials
Save and reuse an experiment comparison
Save a Polyaxon comparison's query, columns, sorting and charts, reopen it for later runs, and share reusable searches and dashboards with your team.
ProductExperimentationCollaboration
Package the evidence behind a model release
Build an attributable model release record, reuse unchanged pipeline outputs, and recompute reports and decisions when their dependencies change.
MLOpsModel RegistryOrchestration
Diagnose a Pending GPU Pod across compute and storage
Work through a GPU placement case where available devices, CPU reservations, and volume topology leave no eligible node.
KubernetesGPUScheduling
Recover agent actions after a lost tool response
Reconcile uncertain tool effects, preserve approval and cancellation across task changes, and compare recovery policies using final-state evidence.
AI AgentsOrchestrationReliability
Reuse Hugging Face weights across Polyaxon runs
Prepare a pinned model revision in a CPU job, then reuse its files from read-only storage in Polyaxon training, notebook and inference workloads.
LLMOpsHugging FaceStorage
Budget PyTorch CPU threads on Kubernetes
Set PyTorch CPU threads, DataLoader workers, and native-library thread counts together within a Kubernetes container's CPU budget.
KubernetesPyTorchPerformance
Fix PyTorch DataLoader bus errors on Kubernetes
Inspect /dev/shm, size DataLoader prefetching, and configure shared memory for PyTorch jobs on Kubernetes and Polyaxon.
KubernetesPyTorchTroubleshooting
Start runs without a Polyaxonfile
Try container images, package versions and environment changes from the Polyaxon CLI, or start a sandbox without writing a Polyaxonfile.
ProductCliSandboxes
Define jobs and services with simpler Polyaxonfiles
Define jobs and services with top-level container, cmd, env, resources, and service in Polyaxon 2.18, while keeping existing Polyaxonfiles supported.
ProductGuides
Clean up finished Kubernetes Jobs without losing results
Use Kubernetes Job TTLs to bound finished workload retention, preserve logs and artifacts separately, and configure the same policy through Polyaxon.
KubernetesSchedulingStorage
Optimize a small-model harness with autoresearch on Polyaxon
Build a measured research loop around Jev or a self-hosted classifier using Polyaxon jobs, sandboxes, experiment matrices, and per-case evidence.
LLMOpsAutoresearchClassification