When to migrate ML workloads to Kubernetes

Kubernetes can give ML teams a common execution model for training, evaluation, notebooks, batch processing, and serving. It can also replace a simple system with a control plane, networking layer, storage integrations, identity model, and upgrade program that the organization must operate.

The right question is not whether Kubernetes is popular. It is whether the workloads and platform team need its scheduling, reconciliation, isolation, and extensibility enough to justify the operational cost.

Start with the problem, not the platform

Write down the limits of the current environment:

• deployments depend on manual host preparation;
• teams compete for accelerators without consistent allocation;
• failed jobs require manual recovery;
• environments drift between development and production;
• workload identity and access are difficult to govern;
• scaling or multi-tenancy has exceeded the existing scheduler;
• several runtime types need one observable control surface.

Kubernetes is a reasonable candidate when these are recurring platform problems. A small team with a few stable services may be better served by a managed application platform, batch service, or simpler orchestrator.

Do not use migration to avoid fixing unclear ownership, missing resource requirements, or unreproducible code. Kubernetes will expose those problems at larger scale.

Evaluate the production obligation

The official Kubernetes production environment guidance covers availability, scale, access management, control-plane design, node lifecycle, and workload limits.

Decide which responsibilities a provider will manage and which remain with the platform team. A managed control plane still leaves node pools, GPU drivers, storage, networking, identity, admission policy, upgrades, observability, and workload support to someone.

Estimate the operating model, not only infrastructure cost. Include on-call coverage, version upgrades, security response, capacity planning, backup validation, and incident recovery.

Inventory workload contracts

Classify each workload before moving it:

Record images, commands, ports, resources, inputs, outputs, credentials, health behavior, deadlines, and failure semantics. This inventory becomes the acceptance criteria for the Kubernetes version.

Build a portable container contract

Package the application and runtime dependencies into an immutable image. Keep datasets, credentials, environment policy, and durable output outside the image.

Run as a non-root user where possible, handle termination signals, write logs to standard output, and make startup and readiness meaningful. Declare CPU, memory, ephemeral storage, and accelerator requirements based on measurements.

A container that works only with a privileged host mount or mutable local directory is not yet portable. Resolve that dependency before scaling the migration.

Separate execution from durable state

Pods are replaceable. Checkpoints, models, datasets, metadata, and business outputs are not.

Define which data belongs in:

• object storage for portable, durable artifacts;
• persistent volumes for filesystem semantics;
• databases or queues for transactional state;
• container-local storage for bounded scratch data only.

Test interruption during writes and verify restart from the last valid checkpoint. A successful Pod restart does not prove application recovery.

Polyaxon artifact connections provide controlled durable destinations while keeping outputs linked to the run that produced them.

Design scheduling before adding scale

Accurate requests are the foundation of scheduling and capacity planning. Define node pools, taints, tolerations, affinity, topology, priority, quotas, and queue policy from workload classes rather than team-by-team YAML.

GPU workloads also require device plugins, drivers, compatible images, memory-aware placement, and often gang or topology-aware scheduling. Validate representative distributed training before declaring the platform ready.

Polyaxon scheduling presets centralize reusable environment policy, while queues can control access to shared capacity.

Establish security and tenancy boundaries

Map human and workload identities separately. Use namespaced RBAC, dedicated service accounts, admission policy, network policy, scoped secrets, and audit evidence.

Namespaces organize and help isolate resources, but they do not automatically provide hard tenant isolation. Decide whether some teams, environments, or regulated workloads require separate clusters or accounts.

Move credentials out of repository variables and images. Polyaxon connections let administrators expose approved integrations to eligible operations without sharing raw credentials broadly.

Migrate in vertical slices

Do not move every workload type at once. Choose one representative, non-critical path and migrate it end to end:

1. build and scan the image;
2. submit with measured resources;
3. schedule on the intended nodes;
4. access required data and services;
5. capture logs, metrics, and artifacts;
6. interrupt it and recover;
7. compare cost and performance with the current system.

Then migrate a different workload shape. A short CPU Job does not validate a multi-GPU training run, and a stateless API does not validate notebook persistence.

Run old and new paths in parallel long enough to compare outcomes. Define rollback and data-compatibility rules before production traffic moves.

Use Polyaxon as the ML control plane

Kubernetes manages infrastructure resources and Pod execution. Polyaxon adds projects, components, operations, tracking, orchestration, scheduling policy, lineage, artifacts, and team workflows above it.

The operations quick start shows how a versioned workload definition becomes a tracked run. That separation lets teams preserve ML intent while platform operators evolve clusters, node pools, and policies underneath.

A migration is complete when the new platform is reproducible, observable, recoverable, secure, and operable by the responsible team—not when the first container reaches Running.