Multi-cluster GPU orchestration
Design multi-cluster GPU orchestration around workload eligibility, data locality, queue routing, recovery, and clear dispatch ownership.
OrchestrationKubernetes
The Polyaxon blog
Build better ML & AI systems.
Practical guides to building, running, and improving ML and AI in production.
Kubernetes
GPU orchestration, distributed training, and multi-tenant AI workloads.
Gang scheduling for distributed training
Understand how gang scheduling prevents partial distributed jobs from holding GPUs, how minimum membership works, and what Polyaxon supports.
SchedulingKubernetes
GPU cluster scheduling tools compared
Compare Kueue, KAI Scheduler, Volcano, Coscheduling, and Slurm Bridge by responsibility, workload fit, and Polyaxon integration path.
SchedulingKubernetes
Keep distributed training workers close together
Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.
KubernetesSchedulingDistributed Training
Design multi-tenant AI agent infrastructure on Kubernetes
Separate identity, network access, data, queues, budgets, and evidence when multiple teams or customers run AI agents on shared Kubernetes infrastructure.
AI AgentsKubernetesSecurity
MLOps
Track experiments, tune models, manage compute costs, and recover training.
ML Metadata Store: Examples, Artifacts, and Lineage
Understand ML metadata with examples of runs, datasets, and model artifacts. Learn how a metadata store connects them and how to query them in Polyaxon.
MLOps
ML Experiment Tracking: What to Log and How to Compare Runs
Learn what to record for each ML experiment, how to compare runs fairly, and how to log parameters, metrics, datasets, and model artifacts with Polyaxon.
MLOps
Hyperparameters tuning on Kubernetes
hyperparameters tuning is very important concept in order to choose the optimal hyperparameters for a given algorithm!
ProductMLOps
Optimize multi-cloud costs for ML workloads
Control multi-cloud ML spending with normalized allocation, workload unit economics, placement policies, elastic capacity, and reproducible efficiency measurements.
FinopsCost OptimizationMLOps
Resume interrupted training without losing progress
Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.
MLOpsKubernetesPyTorch
AI Agents
Agent runtimes, Kubernetes isolation, and parallel coding workflows.
The Polyaxon agent sandbox roadmap
Our plans for persistent sandbox workspaces, suspend/resume, activity-aware idle suspension, and ready pools on Kubernetes—and where AX and Substrate differ.
RoadmapSandboxesAI Agents
Kubernetes for AI agents: A platform engineering guide
Design a Kubernetes platform for AI agents with clear workload boundaries, durable state, scoped access, resource controls, and end-to-end evidence.
AI AgentsKubernetesInfrastructure
Choose an AI agent architecture for production
Map agent architectures to Polyaxon jobs, services, and DAGs, with concrete workload configuration and a repeatable architecture-comparison workflow.
AI AgentsOrchestrationGuides
Sandbox AI agents on Kubernetes
Design isolated Kubernetes execution for AI agents with a threat model, stronger runtimes, scoped identity, controlled egress, bounded storage, and auditable cleanup.
AI AgentsKubernetesSecurity
Orchestrate parallel coding agents with Polyaxon
Run parallel coding agents in separate Polyaxon workspaces, collect candidate patches, evaluate independently, and merge only reviewed changes.
AI AgentsSandboxesCollaboration
Designing the runtime layer for AI agents
Design agent execution around durable state, safe tool retries, approval waits, and recovery, with clear responsibilities for Polyaxon and your agent framework.
AI AgentsOrchestrationGuides
LLMOps
Evaluate, observe, and improve LLM applications in production.
Turn production traces into regression tests
Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.
LLMOpsObservabilityEvaluation
What is LLMOps? From prototype to production
LLMOps applies repeatable development, evaluation, deployment, and observability practices to production LLM applications and AI agents.
LLMOpsMLOps
LLM-as-a-judge: Design and validate model-based evaluators
Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.
EvaluationLLMOps
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
DataOps
Build reliable data pipelines and preserve the evidence behind your models.
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
DataOps vs. MLOps: Differences and How They Work Together
DataOps delivers reliable data; MLOps delivers reliable ML systems. Compare their responsibilities, checks, and handoffs using a practical example.
MLOpsDataOps
Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
Evaluate with only the data available at decision time
Reconstruct evaluation features using event time and actual availability time, excluding late arrivals and later corrections while preserving the source versions in Polyaxon.
DataOpsEvaluationData
Verified data pipelines for reliable AI agents
Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.
AI AgentsDataOpsEvaluation
Recent articles
The latest guides, tutorials, and product updates.
Save and reuse an experiment comparison
Save a Polyaxon comparison's query, columns, sorting and charts, reopen it for later runs, and share reusable searches and dashboards with your team.
ProductExperimentationCollaboration
Package the evidence behind a model release
Build an attributable model release record, reuse unchanged pipeline outputs, and recompute reports and decisions when their dependencies change.
MLOpsModel RegistryOrchestration
Diagnose a Pending GPU Pod across compute and storage
Work through a GPU placement case where available devices, CPU reservations, and volume topology leave no eligible node.
KubernetesGPUScheduling
Recover agent actions after a lost tool response
Reconcile uncertain tool effects, preserve approval and cancellation across task changes, and compare recovery policies using final-state evidence.
AI AgentsOrchestrationReliability
Reuse Hugging Face weights across Polyaxon runs
Prepare a pinned model revision in a CPU job, then reuse its files from read-only storage in Polyaxon training, notebook and inference workloads.
LLMOpsHugging FaceStorage
Budget PyTorch CPU threads on Kubernetes
Set PyTorch CPU threads, DataLoader workers, and native-library thread counts together within a Kubernetes container's CPU budget.
KubernetesPyTorchPerformance
Fix PyTorch DataLoader bus errors on Kubernetes
Inspect /dev/shm, size DataLoader prefetching, and configure shared memory for PyTorch jobs on Kubernetes and Polyaxon.
KubernetesPyTorchTroubleshooting
Run PyTorch on GKE TPUs with Polyaxon
Deploy TorchTPU through a simplified Polyaxonfile, train with eight PyTorch workers on four TPU7x chips, and track metrics and model weights.
MLOpsPyTorchKubernetes
Start runs without a Polyaxonfile
Try container images, package versions and environment changes from the Polyaxon CLI, or start a sandbox without writing a Polyaxonfile.
ProductCliSandboxes
Define jobs and services with simpler Polyaxonfiles
Define jobs and services with top-level container, cmd, env, resources, and service in Polyaxon 2.18, while keeping existing Polyaxonfiles supported.
ProductGuides
Clean up finished Kubernetes Jobs without losing results
Use Kubernetes Job TTLs to bound finished workload retention, preserve logs and artifacts separately, and configure the same policy through Polyaxon.
KubernetesSchedulingStorage
Compare runs across synchronized charts
Compare six experiments at the same epoch, zoom across metric charts together, and configure tooltips with multiple live metric columns and run parameters.
ProductExperimentation
Optimize a small-model harness with autoresearch on Polyaxon
Build a measured research loop around Jev or a self-hosted classifier using Polyaxon jobs, sandboxes, experiment matrices, and per-case evidence.
LLMOpsAutoresearchClassification
Build multimodal search with EmbeddingGemma 2 on Polyaxon
Deploy EmbeddingGemma 2 as a Polyaxon GPU service, then use text or an image to search a mixed collection of documents and photos.
LLMOpsInferenceRAG
Start large ML containers with lazy image pulling
Use eStargz and containerd for lazy image pulls, then measure a Polyaxon GPU job through its first completed CUDA operation.
KubernetesContainersPerformance
Autoresearch as a controlled experiment
A design note on autonomous experimentation: fixed protocols, selection bias, independent evaluation, and durable research campaigns in Polyaxon.
RoadmapAI AgentsExperimentation
Build a coding agent with Polyaxon sandboxes
Give an open model a bounded read, edit, and check loop in a Polyaxon workspace, then retain the patch and evidence behind its completion claim.
AI AgentsSandboxCode Execution
Extract invoice fields with Qwen-VL on Polyaxon
Serve an open vision-language model on your GPU cluster, extract typed invoice fields, and evaluate whether the values actually match the document.
LLMOpsInferenceEvaluation
Measure MIG and GPU time-slicing under co-load
Run synchronized training and inference workers on Polyaxon, compare MIG with time-slicing, and measure the shared interval rather than misleading solo utilization.
KubernetesGPUPerformance
Size GPU memory for LLM serving
Calculate Qwen3-4B weight and KV-cache storage, understand what the estimate excludes, and use a Polyaxon service to check the actual capacity.
LLMOpsInferenceGPU
Profile a slow PyTorch training run on Polyaxon
Capture a short CPU/CUDA trace, identify whether input loading or GPU work is delaying training, and measure a targeted change without profiler overhead.
MLOpsPyTorchGPU
Prioritize GPU jobs without preempting running workloads
Give urgent GPU evaluations or notebooks higher Kubernetes scheduling priority without evicting running training Pods, and select the policy in Polyaxon.
KubernetesGPUScheduling
Benchmark vLLM speculative decoding on Polyaxon
Compare ordinary decoding with prompt-lookup proposals under the same model, workload, and GPU allocation, including latency, failures, and output checks.
LLMOpsvLLMInference
Tune PyTorch models with Polyaxon
Tune a PyTorch model's depth, width, dropout, optimizer, and learning rate with native TPE, GPU trials, and validation accuracy in Polyaxon.
PolyaxonHyperparameter TuningPyTorch