Multi-cluster GPU orchestration
Design multi-cluster GPU orchestration around workload eligibility, data locality, queue routing, recovery, and clear dispatch ownership.
OrchestrationKubernetes
Kubernetes
Learn Kubernetes fundamentals, troubleshoot GPU workloads, and monitor ML infrastructure with practical guides for data scientists and platform engineers.
Start here
Essential guides, selected for this topic.
Gang scheduling for distributed training
Understand how gang scheduling prevents partial distributed jobs from holding GPUs, how minimum membership works, and what Polyaxon supports.
SchedulingKubernetes
GPU cluster scheduling tools compared
Compare Kueue, KAI Scheduler, Volcano, Coscheduling, and Slurm Bridge by responsibility, workload fit, and Polyaxon integration path.
SchedulingKubernetes
Keep distributed training workers close together
Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.
KubernetesSchedulingDistributed Training
Design multi-tenant AI agent infrastructure on Kubernetes
Separate identity, network access, data, queues, budgets, and evidence when multiple teams or customers run AI agents on shared Kubernetes infrastructure.
AI AgentsKubernetesSecurity
Latest articles
The latest guides, tutorials, and product updates.
Prioritize GPU jobs without preempting running workloads
Give urgent GPU evaluations or notebooks higher Kubernetes scheduling priority without evicting running training Pods, and select the policy in Polyaxon.
KubernetesGPUScheduling
The Polyaxon agent sandbox roadmap
Our plans for persistent sandbox workspaces, suspend/resume, activity-aware idle suspension, and ready pools on Kubernetes—and where AX and Substrate differ.
RoadmapSandboxesAI Agents
Protect notebook volumes with ReadWriteOncePod
Restrict a Kubernetes notebook workspace PVC to one Pod, mount it in a Polyaxon service, and hand access to a replacement notebook.
KubernetesStorageNotebooks
Check agent access before starting workloads
Polyaxon checks Kubernetes access and the default artifact store before managed agents begin syncing. Find failed checks and correct credentials, permissions, or mounts.
ComputeKubernetesAdministration
Set GPU quotas for shared Kubernetes namespaces
Limit aggregate GPU requests with ResourceQuota, supply CPU and memory defaults, and align Polyaxon resource profiles and queues with namespace budgets.
KubernetesGPUScheduling
Keep inference available during Kubernetes node drains
Set a PodDisruptionBudget around usable inference capacity, inspect blocked evictions, and account for replacement GPUs and model warmup.
KubernetesInferenceReliability
Finish Indexed Jobs after enough attempts succeed
Use Kubernetes Job success policies for chosen indexes or success counts, preserve durable results, and distinguish them from Polyaxon metric early stopping.
KubernetesPipelinesScheduling
Spread inference replicas across Kubernetes zones
Use topology spread constraints for inference availability, understand minDomains and GPU capacity, and preserve placement intent in Polyaxon.
KubernetesSchedulingInference
Serve DiffusionGemma on Polyaxon
Adapt Google's Gemma-on-Kubernetes deployment choices to Polyaxon with a DiffusionGemma vLLM service, a batch evaluation job, and an explicit compatibility checklist.
LLMOpsInferenceKubernetes
Align CPU and GPU resources with Kubernetes NUMA policies
Combine CPU Manager eligibility, full-core allocation, and Topology Manager policy, then compare CPU/GPU workloads with Polyaxon.
KubernetesGPUPerformance
Resize CPU and memory without replacing Kubernetes Pods
Use Kubernetes in-place resource resizing for running workloads, inspect whether changes took effect, and understand the limits for Polyaxon services.
KubernetesSchedulingResources
Native gang scheduling reaches beta in Kubernetes 1.37
Explore native gang scheduling in Kubernetes 1.37, its potential for Polyaxon training and sandboxes, and how it compares with KAI, Kueue, and Volcano.
KubernetesSchedulingDistributed Training
Keep distributed training workers close together
Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.
KubernetesSchedulingDistributed Training
Resume interrupted training without losing progress
Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.
MLOpsKubernetesPyTorch
Keep long-running workloads authenticated through token rotation
Use projected ServiceAccount tokens and refresh-aware clients so long-running training, notebooks, and services can keep authenticating without static credentials.
KubernetesSecurityReliability
Enforce ML workload rules with CEL admission policies
Use Kubernetes ValidatingAdmissionPolicy to check workload ownership and image references, with scoped warning and enforcement stages for ML namespaces.
KubernetesGovernanceSecurity
Control Kubernetes Job retries with pod failure policies
Stop retrying permanent errors, preserve the retry budget for marked disruptions, and inspect Kubernetes Job failure decisions with a concrete example.
KubernetesSchedulingTroubleshooting
Mount model artifacts with Kubernetes image volumes
Separate immutable model files from the serving image with Kubernetes image volumes, and choose when an OCI artifact fits better than a download or PVC.
KubernetesStorageInference
Request GPUs by capability with Kubernetes DRA
Use Dynamic Resource Allocation to describe the accelerator a workload needs, understand device claims, and plan the integration with Polyaxon.
KubernetesGPUScheduling
Reduce the cost of self-hosted ML workloads
Reduce self-hosted ML costs with outcome-based accounting, right-sized resources, elastic capacity, interruption-ready workloads, local data paths, and deliberate retention.
InfrastructureKubernetesScheduling
Design multi-tenant AI agent infrastructure on Kubernetes
Separate identity, network access, data, queues, budgets, and evidence when multiple teams or customers run AI agents on shared Kubernetes infrastructure.
AI AgentsKubernetesSecurity
Restart containers selectively with Kubernetes restart rules
Use exit-code-based container restart rules, observe what stays within the Pod, and distinguish local recovery from retrying a complete ML operation.
KubernetesReliabilityWorkloads
Sandbox AI agents on Kubernetes
Design isolated Kubernetes execution for AI agents with a threat model, stronger runtimes, scoped identity, controlled egress, bounded storage, and auditable cleanup.
AI AgentsKubernetesSecurity
What is sovereign AI? Control across the AI lifecycle
Define sovereign AI as control over data, models, compute, operations, providers, and evidence, then implement it with Kubernetes.
InfrastructureKubernetesSecurity