Place ML workloads close to their data

A training job can spend less on compute in a remote region and still cost more overall. It may wait for a large container image, copy a dataset across a regional boundary, fetch model weights through a gateway, and upload checkpoints back to the original store. The nominally cheaper worker is then slower, less reliable, and responsible for a transfer bill that does not appear in the compute comparison.

Region-aware ML infrastructure starts with the complete data path. Place work where its required data and services can be accessed legally, securely, and efficiently, then measure whether that placement improves time and cost per accepted result.

Inventory every byte the workload moves

Do not reduce “the data” to the training dataset. Follow a cold execution from submission to completion and record each source, destination, volume, and owner.

Include failed attempts and retries. A workload that repeatedly pulls the same image or stages the same partial dataset can move more data while producing no usable output. The ML job network-dependency guide shows how to assign each connection to the container or node component that actually makes it.

Measure bytes on the billed network path rather than estimating from file size alone. Compression, cache hits, multipart retries, cross-zone traffic, network address translation, replication, and provider-specific service paths can all change what appears on an invoice.

Separate residency from locality

Data residency is a policy constraint about where data may be stored or processed. Data locality is an engineering property describing proximity between compute and its dependencies. They often point in the same direction, but they are not interchangeable.

A job and object store can share a region while logs or model requests leave it. Conversely, an approved multi-region design can satisfy its policy while accepting a measured transfer cost for resilience. Define the permitted boundary first, then optimize placement inside it.

The sovereign AI guide covers jurisdiction, operators, providers, supply chain, and continuity across the complete lifecycle. This article focuses on the narrower placement decision: which eligible execution location gives the workload the best end-to-end path.

Define hard eligibility before preferences

Create a workload placement contract. Hard rules remove destinations that cannot run the work; preferences rank the remaining options.

Hard eligibility may include:

• approved region, jurisdiction, cloud account, or on-premises environment;
• access to a specific immutable dataset and artifact destination;
• accelerator type, memory, interconnect, and driver compatibility;
• required storage class, network path, identity mechanism, and runtime;
• permission to call model, database, or internal service endpoints; and
• support for the workload controller, scheduler, and recovery mechanism.

Preferences can then compare queue delay, startup time, data-staging time, expected execution duration, output-transfer time, cost, carbon policy, and recovery options. An empty cluster should not win if it cannot access the input or if staging it makes completion later.

Record why a target was eligible and why it was selected. This turns routing into an inspectable policy decision rather than a label the user must choose from memory.

Co-locate the high-volume dependencies

The most valuable locality improvement usually comes from the largest or most frequently repeated path. For one workload that may be a multi-terabyte dataset; for another it may be a large base image pulled hundreds of times or checkpoints written every few minutes.

Keep container registries, artifact stores, model repositories, and frequently used datasets near the compute that consumes them when the security and availability model permits it. Use private service paths or provider endpoints where they meet the requirements, and confirm which traffic remains billable.

Avoid turning local caches into unversioned sources of truth. Cache keys should include the immutable image digest, dataset version, model revision, preprocessing configuration, and other inputs that affect the result. A warm cache can reduce startup time without proving that the correct content was used.

Replicate deliberately. A replica can improve repeated execution in another region, but it introduces storage, synchronization, access-control, deletion, and provenance work. Record whether it is authoritative, read-only, eventually consistent, or valid for a bounded period.

Align Kubernetes scheduling with storage topology

Kubernetes nodes commonly carry topology labels for zones and regions. Node affinity can require or prefer eligible topology values. Use required rules for real constraints and preferred rules for optimizations that may yield under pressure.

Volume topology must agree with Pod placement. Kubernetes StorageClass volume binding documents WaitForFirstConsumer, which delays volume selection or provisioning until scheduling constraints are known. This helps avoid binding storage in a topology that cannot satisfy the consuming Pod.

Do not hard-code a node name to express regional intent. Nodes are replaceable, and direct binding can bypass useful scheduling behavior. Label infrastructure from a trusted inventory, then expose stable workload classes through presets or policy.

For distributed training, locality includes communication between workers. Putting data near each Pod does not compensate for a topology that makes synchronization slow or unsupported. Validate storage bandwidth, worker-to-worker networking, and failure behavior under the same placement.

Model startup, execution, and output separately

Break completion time into queueing, image pull, initialization, data staging, active execution, checkpointing, and final output transfer. Some phases can overlap; measure the observed critical path instead of adding estimates mechanically.

Track cold and warm cases. A region with a warm image and dataset cache may look dramatically faster until an autoscaling event creates new nodes. Re-run comparisons with empty caches, replacement nodes, and realistic concurrent demand.

For each candidate location, compare:

• time to an executing application process;
• time to first training step or first completed item;
• total time to an accepted output;
• bytes transferred by path and billing category;
• failed or repeated transfer volume;
• compute, storage, network, and managed-service cost; and
• whether all residency and access rules remained satisfied.

The useful outcome is not the lowest transfer count. It is a correct, accepted result delivered within the required time and policy at a sustainable total cost.

Make failover respect the same contract

A fallback region is useful only if the workload can run there. Confirm that it has compatible compute, approved data, container images, credentials, storage destinations, quotas, and checkpoint access. A routing rule that changes the cluster without changing these dependencies produces a different failure rather than recovery.

Decide whether inputs are pre-positioned, copied after failure, or unavailable outside the primary location. Define recovery-point and recovery-time expectations for stateful work. Test whether a replacement job can restore its checkpoint and whether two regions can accidentally advance the same logical task.

Residency rules apply during incidents too. A silent failover to an unapproved region can violate the workload contract precisely when operators are under pressure. Prefer an explicit outcome—approved fallback, delayed queue, or fail-closed—over an undocumented route.

Apply region-aware routing with Polyaxon

Polyaxon can run on an existing Kubernetes environment and, in the commercial offering, use agents and queues to manage multiple namespaces or clusters. A queue can route operations to an execution environment while also applying priority, concurrency, resource, or cost constraints. See the queue documentation and agent setup.

Represent each region or environment with verified capabilities rather than a generic location name. Use project or organization presets to package node selectors, tolerations, resource requests, connections, and other repeatable settings. Polyaxon's node-scheduling guide covers the available placement fields.

Attach region-appropriate connections for datasets, registries, repositories, and artifact stores. Keep the same logical input and output identities across environments while allowing the underlying endpoints to differ. Record the agent, queue, namespace, cluster, image digest, data version, and artifact destination with the run.

The multi-cluster GPU orchestration guide expands on eligibility and recovery when accelerator pools span environments. This locality model applies to CPU jobs, builds, evaluations, data processing, and services as well.

Start with one high-volume workload

Select a repeatable workload whose data path is understood. Measure a baseline, move only the execution and dependencies required by the proposed design, then compare the complete path under cold and warm conditions.

Verify access denial from an ineligible environment, interruption during staging, artifact persistence after node loss, and the documented regional fallback. Preserve the measurements and effective configuration with the run.

Region-aware execution works when placement follows a verifiable contract and the large data paths remain visible. Run the workload where its dependencies make sense—not merely where an instance appears cheapest or idle capacity happens to exist.