Kubernetes Guides for ML Teams

Serve DiffusionGemma on Polyaxon

Adapt Google's Gemma-on-Kubernetes deployment choices to Polyaxon with a DiffusionGemma vLLM service, a batch evaluation job, and an explicit compatibility checklist.

Resize CPU and memory without replacing Kubernetes Pods

Use Kubernetes in-place resource resizing for running workloads, inspect whether changes took effect, and understand the limits for Polyaxon services.

Native gang scheduling reaches beta in Kubernetes 1.37

Explore native gang scheduling in Kubernetes 1.37, its potential for Polyaxon training and sandboxes, and how it compares with KAI, Kueue, and Volcano.

Keep distributed training workers close together

Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.

Resume interrupted training without losing progress

Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.

Keep long-running workloads authenticated through token rotation

Use projected ServiceAccount tokens and refresh-aware clients so long-running training, notebooks, and services can keep authenticating without static credentials.

Enforce ML workload rules with CEL admission policies

Use Kubernetes ValidatingAdmissionPolicy to check workload ownership and image references, with scoped warning and enforcement stages for ML namespaces.

Control Kubernetes Job retries with pod failure policies

Stop retrying permanent errors, preserve the retry budget for marked disruptions, and inspect Kubernetes Job failure decisions with a concrete example.

Mount model artifacts with Kubernetes image volumes

Separate immutable model files from the serving image with Kubernetes image volumes, and choose when an OCI artifact fits better than a download or PVC.

Request GPUs by capability with Kubernetes DRA

Use Dynamic Resource Allocation to describe the accelerator a workload needs, understand device claims, and plan the integration with Polyaxon.

Reduce the cost of self-hosted ML workloads

Reduce self-hosted ML costs with outcome-based accounting, right-sized resources, elastic capacity, interruption-ready workloads, local data paths, and deliberate retention.

Design multi-tenant AI agent infrastructure on Kubernetes

Separate identity, network access, data, queues, budgets, and evidence when multiple teams or customers run AI agents on shared Kubernetes infrastructure.

Restart containers selectively with Kubernetes restart rules

Use exit-code-based container restart rules, observe what stays within the Pod, and distinguish local recovery from retrying a complete ML operation.

Sandbox AI agents on Kubernetes

Design isolated Kubernetes execution for AI agents with a threat model, stronger runtimes, scoped identity, controlled egress, bounded storage, and auditable cleanup.

What is sovereign AI? Control across the AI lifecycle

Define sovereign AI as control over data, models, compute, operations, providers, and evidence, then implement it with Kubernetes.

Fine-tune Mistral 7B with LoRA on Kubernetes

Plan a Mistral 7B LoRA fine-tuning workflow on Kubernetes with versioned data, GPU scheduling, Polyaxon tracking, evaluation, and adapter packaging.

Retry failed evaluation shards with Indexed Jobs

Give evaluation shards stable indexes and independent retry budgets with Kubernetes Indexed Jobs, while keeping incomplete results visible.

Run Promptfoo evaluations on Kubernetes with Polyaxon

Package a Promptfoo suite as a Polyaxon job, export evaluation reports, track artifacts, and verify that failed checks fail the workload.

What are your ML jobs connecting to?

Trace image pulls, Git clones, S3 and GCS access, Hugging Face downloads, and artifact uploads across the lifecycle of Kubernetes jobs and sandboxes.

Pods, Jobs, or Services for AI agents?

Choose the Kubernetes execution shape for an AI agent by separating logical task identity from Pods, Jobs, Services, workflows, and sandbox lifecycles.

Restore notebook workspaces from volume snapshots

Create a CSI snapshot of a notebook workspace, restore it into a separate PVC, and verify files and application state before switching workspaces.

Kubernetes for AI agents: A platform engineering guide

Design a Kubernetes platform for AI agents with clear workload boundaries, durable state, scoped access, resource controls, and end-to-end evidence.

Hold Pods until they are ready for scheduling

Use Kubernetes scheduling gates to wait for external prerequisites before placement, with a controlled example and clear ownership of gate removal.

Design reliable self-hosted ML infrastructure

Build self-hosted ML infrastructure around explicit failure domains, durable queues and artifacts, eligible failover, actionable telemetry, and tested recovery.