Single vs. multi-cluster Kubernetes for ML platforms

A single Kubernetes cluster concentrates capacity and administration. Multiple clusters create stronger infrastructure boundaries and more placement options, while multiplying control planes, policies, upgrades, credentials, and failure modes.

ML platforms often need both patterns over time. The useful decision is not an abstract preference for consolidation or isolation; it is which boundaries each workload, tenant, region, and accelerator pool requires.

Start with a single-cluster baseline

One shared cluster can simplify:

• identity and access configuration;
• observability and inventory;
• scheduling across a larger capacity pool;
• platform installation and upgrades;
• service discovery and internal networking;
• policy and cost allocation.

Shared capacity can improve bin packing, particularly when workloads have varied CPU, memory, and GPU shapes. It also reduces duplicated system components and operational routines.

The tradeoff is a larger shared failure and contention domain. A control-plane incident, unsafe admission policy, network failure, exhausted quota, or noisy tenant can affect many teams unless isolation controls are carefully designed.

Add clusters for explicit boundaries

Separate clusters are justified when they provide a required boundary:

• region or data-residency constraints;
• production isolation from experimental workloads;
• incompatible Kubernetes, driver, or accelerator stacks;
• dedicated regulated or high-trust environments;
• independent team ownership and change schedules;
• provider or region failure isolation;
• capacity that exists in different clouds or data centers.

Avoid creating a cluster for every team by default. Each new cluster needs an owner, lifecycle, security baseline, observability, backup plan, access model, upgrade path, and capacity policy.

Compare isolation honestly

Kubernetes namespaces, RBAC, quotas, network policies, service accounts, and admission controls support shared-cluster multi-tenancy. The official multi-tenancy guidance explains the isolation and fairness considerations.

A separate cluster creates a stronger control-plane and data-plane boundary, but it is not absolute isolation if clusters share cloud accounts, identity providers, registries, secrets, networks, or storage.

Define the threats and failure modes first. Use the smallest boundary that satisfies them and verify every shared dependency outside Kubernetes.

Design failure domains and recovery

Multiple clusters do not automatically create high availability. A workload is only recoverable elsewhere if the second cluster has:

• compatible images and runtime APIs;
• required accelerator and storage capabilities;
• access to the same logical inputs and artifacts;
• sufficient quota and capacity;
• equivalent identity and policy;
• a tested routing and state-recovery procedure.

Active-active service traffic, warm standby, and resubmitting a checkpointed batch job are different recovery models. Choose one per workload.

Test loss of a node pool, availability zone, cluster control plane, region, registry, and artifact store. Cross-cluster orchestration cannot repair a dependency shared by every cluster.

Put data locality before free capacity

A cluster with idle GPUs may still be ineligible if it cannot access the dataset, model, registry, or output store with acceptable latency, cost, and authorization.

Record data region, storage topology, sovereignty, network path, and expected transfer volume as placement inputs. Copying a multi-terabyte dataset for each run can cost more and take longer than waiting for nearby compute.

Use logical connection identities so workloads remain portable while endpoints and credentials differ by environment. Track the actual cluster, storage, image digest, and data version used by every run.

Treat accelerator pools as capabilities

GPU count alone is not enough. Model, memory size, interconnect, driver, runtime, topology, sharing mode, and supported scheduling objects determine eligibility.

A single cluster can contain several specialized node pools. Multiple clusters may be necessary when providers, regions, security domains, or incompatible driver stacks differ.

Maintain a capability inventory and route work only to environments that satisfy the entire request. A global “available GPU” total can hide fragmentation that prevents the workload from scheduling.

Account for the operational multiplier

With multiple clusters, standardize:

• provisioning and deletion;
• Kubernetes and add-on versions;
• identity and access;
• admission, quota, and network policy;
• observability and audit retention;
• secrets and connection distribution;
• vulnerability and configuration scanning;
• upgrade, backup, and incident procedures.

Central templates reduce drift but do not remove environment-specific differences. Detect drift continuously and document which differences are intentional.

Budget for minimum node capacity, duplicated system workloads, cross-region traffic, idle failover capacity, and the people required to operate the fleet.

Separate routing from local scheduling

In a multi-cluster platform, two decisions happen:

1. select an eligible execution environment;
2. schedule the resulting Pods within that cluster.

The global router understands workload class, policy, data, accelerator capability, quota, and availability. The local Kubernetes scheduler understands nodes and current placement constraints.

Do not make users switch kubeconfig contexts and manually guess the least-busy cluster. That bypasses policy and makes retries, ownership, and auditability inconsistent.

Use Polyaxon across execution environments

Polyaxon agents connect approved clusters and namespaces to one AI engineering control plane. Queues can express priority, concurrency, quota, and routing, while scheduling presets carry environment-specific placement and resource policy.

The multi-cluster GPU orchestration guide covers eligibility, dispatch ownership, recovery, and avoiding duplicate attempts. Polyaxon run metadata preserves which agent, cluster, namespace, image, code, inputs, parameters, and artifacts produced an outcome.

Start with one well-operated cluster when it satisfies the required boundaries. Add another cluster only with a written reason and a repeatable lifecycle. The goal is not more clusters; it is dependable access to the right environment for each workload.