Route ML jobs to regional infrastructure with Polyaxon

A nightly evaluation can run on an available cluster while repeatedly downloading its dataset from somewhere else. Moving that evaluation to a regional execution environment can improve the data path, but the change needs more than a different queue name.

Polyaxon lets teams express the execution destination through agents and queues, configure storage access for each environment, and retain the resulting run history. The practical task is to connect those controls to the actual infrastructure: which cluster runs the job, which endpoints it uses, and where its outputs are stored.

This guide turns a regional placement decision into an explicit operation configuration. For the broader architecture, start with placing ML workloads close to their data.

Establish the regional execution environment

In Polyaxon Cloud and Enterprise Edition, compute agents can manage execution in separate Kubernetes clusters or namespaces. A single control plane can manage those agents through their queues. Multi-cluster management is part of the commercial offering; it is not enabled simply by adding a queue field to a Community Edition deployment.

Provision the Kubernetes environment in the intended location and configure its agent before exposing a regional queue to users. Polyaxon routes work to configured infrastructure; a region-like name does not provision a cloud region or verify its location.

Maintain a small, reviewed mapping for each destination:

Each agent can have its own artifact store and connection catalog. Use that capability to give regional workloads the intended endpoints rather than assuming every cluster should inherit the same configuration. See connections and deployment strategies.

Route an operation explicitly

Suppose an agent named eu-ml manages a queue named batch-evaluation. Your team has registered an evaluation component that accepts a dataset revision and reads it through an approved connection.

Save an operation such as the following as regional-evaluation.yaml:

version: 1.1
kind: operation
name: regional-agent-evaluation
hubRef: YOUR_ORG/agent-evaluation:v2
queue: eu-ml/batch-evaluation

params:
  dataset_revision:
    value: regression-set-v5
  candidate_revision:
    value: agent-v12

cache:
  disable: true

The organization, component, agent, and queue are examples that your team must create. The component must declare both string inputs, contain the evaluation implementation, and reference connections available on the selected agent. Its image and input data must also be accessible there.

The documented queue reference uses agent-name/queue-name. Including the agent avoids relying on default-agent resolution. Queue routing selects the execution environment; node scheduling then selects suitable nodes within that environment. A node selector cannot move an operation to a different cluster.

Caching is disabled here because the evaluation is intended to execute afresh. That setting does not disable container image caching, storage caches, or provider-side caching.

Make project defaults deliberate

Once the environment is reviewed, administrators can set a project's default queue and restrict the queues available to that project. This reduces repeated configuration while keeping approved destinations visible.

A default is a convenience, not an access boundary. Review explicit queue choices, component defaults, and project restrictions together. For workloads with mandatory placement, enforce the required infrastructure and access policies rather than depending on users to remember a naming convention.

Queue concurrency and priority can organize demand within the regional environment. They do not create capacity or guarantee a completion time. If the approved destination is saturated, decide whether the workload should wait or whether an independently approved alternative exists.

Measure the transfer bill by path

Do not budget regional execution as automatically having zero network cost. The bill depends on the provider, services, source and destination, zones, and actual network route. A private endpoint, gateway, replication process, or external model API may introduce a different charge even after compute moves closer to storage.

For one representative workload, assemble a before-and-after worksheet:

Use provider billing records for charged usage and the applicable rates for that period. Application-level byte counts help explain the workload but may not match billing units or account for every network hop. Avoid adding a generic transfer rate to a charge that already includes it.

Compare total compute, storage, and network cost per accepted result. A lower transfer bill can still accompany higher total cost if the regional hardware is slower or queue delays cause additional retries.

Record run identifiers and workload revisions so the comparison can be reproduced. Polyaxon's run timeline helps separate execution stages; provider billing remains the source for actual network charges.

Review the control plane and output path separately

Selecting a regional queue establishes where the configured agent executes the workload. It does not establish where every related record is processed or retained.

Review the control-plane deployment and the metadata sent to it, the agent's artifact store for outputs and logs, external model calls, notifications, backups, and downloaded reports. Keep sensitive payloads out of run names, parameters, and ordinary diagnostic messages unless their handling is explicitly approved.

The data-residency guide covers that broader review. Regional routing is one control within the design, not a compliance certification or a guarantee that all data stays inside a boundary.

Roll out one regional workload class

Start with a repeatable evaluation or preprocessing job. Keep the component version, image digest, dataset, and acceptance checks fixed while comparing eligible execution destinations. Include cold starts and realistic concurrency, not just an already-warm worker.

Before making the regional queue the default, confirm that inputs resolve correctly, approved outputs persist, disallowed destinations are rejected, and the documented behavior during a capacity shortage is acceptable. Preserve the placement mapping and measurements with the rollout record.

With Polyaxon, the useful result is an explicit, reusable execution route backed by configured storage and observable runs. Regional infrastructure then becomes an operational choice the team can inspect and measure, rather than an assumption hidden behind the location of a worker.