Multi-cluster GPU orchestration
Design multi-cluster GPU orchestration around workload eligibility, data locality, queue routing, recovery, and clear dispatch ownership.
OrchestrationKubernetes
The Polyaxon blog
Build better ML & AI systems.
Practical guides to building, running, and improving ML and AI in production.
Kubernetes
GPU orchestration, distributed training, and multi-tenant AI workloads.
Gang scheduling for distributed training
Understand how gang scheduling prevents partial distributed jobs from holding GPUs, how minimum membership works, and what Polyaxon supports.
SchedulingKubernetes
GPU cluster scheduling tools compared
Compare Kueue, KAI Scheduler, Volcano, Coscheduling, and Slurm Bridge by responsibility, workload fit, and Polyaxon integration path.
SchedulingKubernetes
Keep distributed training workers close together
Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.
KubernetesSchedulingDistributed Training
Design multi-tenant AI agent infrastructure on Kubernetes
Separate identity, network access, data, queues, budgets, and evidence when multiple teams or customers run AI agents on shared Kubernetes infrastructure.
AI AgentsKubernetesSecurity
MLOps
Track experiments, tune models, manage compute costs, and recover training.
ML Metadata Store: Examples, Artifacts, and Lineage
Understand ML metadata with examples of runs, datasets, and model artifacts. Learn how a metadata store connects them and how to query them in Polyaxon.
MLOps
ML Experiment Tracking: What to Log and How to Compare Runs
Learn what to record for each ML experiment, how to compare runs fairly, and how to log parameters, metrics, datasets, and model artifacts with Polyaxon.
MLOps
Hyperparameters tuning on Kubernetes
hyperparameters tuning is very important concept in order to choose the optimal hyperparameters for a given algorithm!
ProductMLOps
Optimize multi-cloud costs for ML workloads
Control multi-cloud ML spending with normalized allocation, workload unit economics, placement policies, elastic capacity, and reproducible efficiency measurements.
FinopsCost OptimizationMLOps
Resume interrupted training without losing progress
Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.
MLOpsKubernetesPyTorch
AI Agents
Agent runtimes, Kubernetes isolation, and parallel coding workflows.
Kubernetes for AI agents: A platform engineering guide
Design a Kubernetes platform for AI agents with clear workload boundaries, durable state, scoped access, resource controls, and end-to-end evidence.
AI AgentsKubernetesInfrastructure
Choose an AI agent architecture for production
Map agent architectures to Polyaxon jobs, services, and DAGs, with concrete workload configuration and a repeatable architecture-comparison workflow.
AI AgentsOrchestrationGuides
Sandbox AI agents on Kubernetes
Design isolated Kubernetes execution for AI agents with a threat model, stronger runtimes, scoped identity, controlled egress, bounded storage, and auditable cleanup.
AI AgentsKubernetesSecurity
Orchestrate parallel coding agents with Polyaxon
Run parallel coding agents in separate Polyaxon workspaces, collect candidate patches, evaluate independently, and merge only reviewed changes.
AI AgentsSandboxesCollaboration
Designing the runtime layer for AI agents
Design agent execution around durable state, safe tool retries, approval waits, and recovery, with clear responsibilities for Polyaxon and your agent framework.
AI AgentsOrchestrationGuides
LLMOps
Evaluate, observe, and improve LLM applications in production.
Turn production traces into regression tests
Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.
LLMOpsObservabilityEvaluation
What is LLMOps? From prototype to production
LLMOps applies repeatable development, evaluation, deployment, and observability practices to production LLM applications and AI agents.
LLMOpsMLOps
LLM-as-a-judge: Design and validate model-based evaluators
Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.
EvaluationLLMOps
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
DataOps
Build reliable data pipelines and preserve the evidence behind your models.
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
DataOps vs. MLOps: Differences and How They Work Together
DataOps delivers reliable data; MLOps delivers reliable ML systems. Compare their responsibilities, checks, and handoffs using a practical example.
MLOpsDataOps
Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
Evaluate with only the data available at decision time
Reconstruct evaluation features using event time and actual availability time, excluding late arrivals and later corrections while preserving the source versions in Polyaxon.
DataOpsEvaluationData
Verified data pipelines for reliable AI agents
Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.
AI AgentsDataOpsEvaluation
Recent articles
The latest guides, tutorials, and product updates.
Set GPU quotas for shared Kubernetes namespaces
Limit aggregate GPU requests with ResourceQuota, supply CPU and memory defaults, and align Polyaxon resource profiles and queues with namespace budgets.
KubernetesGPUScheduling
Tune hyperparameters with native TPE in Polyaxon
Use completed trial results to guide hyperparameter choices with native TPE, run GPU training trials, and compare validation loss in Polyaxon.
PolyaxonHyperparameter TuningExperimentation
Query runs and model versions across projects
Use Polyaxon's organization and team APIs to inspect GPU workloads by agent, count model versions, and calculate job failure rates across projects.
PolyaxonPythonMonitoring
Keep inference available during Kubernetes node drains
Set a PodDisruptionBudget around usable inference capacity, inspect blocked evictions, and account for replacement GPUs and model warmup.
KubernetesInferenceReliability
Optimize one model three ways with NVIDIA Model Optimizer on Polyaxon
Compare INT8 quantization, FastNAS pruning, and distillation against one tracked baseline, then measure the candidate that meets your accuracy floor.
MLOpsModel OptimizationTraining
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
Run persistent Dask and Ray clusters on Polyaxon
Launch Dask or Ray clusters with Polyaxonfiles, submit repeated work, and control worker capacity, durable outputs, and cluster lifetimes.
ProductDaskRay
Finish Indexed Jobs after enough attempts succeed
Use Kubernetes Job success policies for chosen indexes or success counts, preserve durable results, and distinguish them from Polyaxon metric early stopping.
KubernetesPipelinesScheduling
Transfer files to Polyaxon workspaces over SSH
Use Polyaxon's generated SSH host with native SCP and SFTP to move files into and out of a running service workspace.
ProductCliRemote Development
Spread inference replicas across Kubernetes zones
Use topology spread constraints for inference availability, understand minDomains and GPU capacity, and preserve placement intent in Polyaxon.
KubernetesSchedulingInference
Train a decision classifier on Polyaxon
Fine-tune an open Qwen model for bounded support-routing decisions with Polyaxon jobs, track the adapter, and serve the reviewed version through vLLM.
LLMOpsFine TuningClassification
Deploy Laya typed decisions on Polyaxon
Serve the open Laya typed-decisions checkpoint through Polyaxon, with a CPU classifier endpoint, explicit model scope, and a review path for uncertain classifications.
LLMOpsInferenceClassification
Serve a Jev-style Qwen classifier on Polyaxon
Deploy an open Qwen model behind a typed classification API on Polyaxon, then check the decision schema and calibration before routing real work.
LLMOpsInferenceClassification
Serve DiffusionGemma on Polyaxon
Adapt Google's Gemma-on-Kubernetes deployment choices to Polyaxon with a DiffusionGemma vLLM service, a batch evaluation job, and an explicit compatibility checklist.
LLMOpsInferenceKubernetes
Align CPU and GPU resources with Kubernetes NUMA policies
Combine CPU Manager eligibility, full-core allocation, and Topology Manager policy, then compare CPU/GPU workloads with Polyaxon.
KubernetesGPUPerformance
Run background commands in Polyaxon sandboxes
Start a background process, save its execution ID, reconnect to status and logs, and distinguish command limits from the lifetime of a Polyaxon sandbox.
PolyaxonSandboxesPython
Schedule mapped runs in the order you specify
Use an ordered Polyaxon mapping for a curated list of experiments, control concurrency, and distinguish scheduling order from dependencies and completion order.
PolyaxonOrchestrationExperimentation
Separate evaluator changes from application improvements
Compare old and new application outputs under both evaluator versions, inspect changed decisions, and retain the four comparisons in Polyaxon.
LLMOpsEvaluationExperimentation
Make vLLM prefix caching work for repeated prompts
Understand exact prefix reuse in vLLM, structure repeated context, account for replica routing, and compare cold and warm workloads with Polyaxon.
LLMOpsInferencevLLM
Resize CPU and memory without replacing Kubernetes Pods
Use Kubernetes in-place resource resizing for running workloads, inspect whether changes took effect, and understand the limits for Polyaxon services.
KubernetesSchedulingResources
Native gang scheduling reaches beta in Kubernetes 1.37
Explore native gang scheduling in Kubernetes 1.37, its potential for Polyaxon training and sandboxes, and how it compares with KAI, Kueue, and Volcano.
KubernetesSchedulingDistributed Training
Keep failed candidates visible in pipeline reports
Reconcile evaluation results against an expected candidate manifest, preserve failed and missing outcomes, and separate pipeline reporting from release approval.
MLOpsOrchestrationEvaluation
Work inside a team from Python and the CLI
Scope Polyaxon Python clients and CLI commands to a team, review runs across its projects, and keep explicit project targets and saved defaults clear.
PolyaxonTeamsPython
Design a search space before launching the sweep
Use Polyaxon matrices to choose parameter scales, separate meaningful combinations, and control the size of a hyperparameter sweep.
Hyperparameter TuningMLOpsExperimentation