One replica model for inference, training, and RL
Polyaxon's roadmap for named replica groups, heterogeneous services, coordinated training, and RL workloads across Kubernetes scheduling backends.
An inference service can need a CPU router, GPU workers that process prompts, and another GPU pool that generates tokens. A distributed training job can need four worker Pods placed together. A reinforcement-learning run can need both learner and rollout capacity before training can advance.
We are building a shared replica-group model in Polyaxon to describe these workloads. Each named group has its own instance count and workload configuration, while the operation owns the groups and their lifecycle. The goal is consistent configuration and visibility across inference, training, and RL, with scheduling policies that preserve each workload's requirements.
This is a roadmap preview. The shared model and heterogeneous service support are in development. Group-aware scheduling integrations are part of the direction described here; their APIs, backend coverage, and release timing are not final.
Illustrative deployment layouts. An RL framework may colocate roles or run several actors inside one worker Pod.
Start with a service that has several roles
The first concrete target is disaggregated inference with SGLang and Shepherd Model Gateway (SMG):
| Group | Example replicas | Responsibility |
|---|---|---|
| Router | 1 | Accept requests and route work to the appropriate prefill and decode workers. |
| Prefill | 4 | Process prompts and prepare the state needed for generation. |
| Decode | 8 | Generate output tokens using that state. |
Prefill and decode can have different capacity needs. SGLang's disaggregated serving design separates those phases; SMG's deployment guide describes routing and the transfer setup between workers. The counts above are a deployment example, not a sizing recommendation.
Polyaxon's current service model provides one workload template and a replica count. That is useful when replicas run the same process. This deployment needs several templates: the router and workers have different commands, resource requests, ports, and initialization requirements.
Our initial routed-service design will designate one router group, constrained to one replica, and expose only that group's Pods through the public endpoint. Prefill and decode groups will have separate replica counts and configuration. Increasing decode capacity should update that group without changing the desired router or prefill counts.
Creating the Pods is one part of this work. Worker discovery, registration, prefill bootstrap addresses, and readiness must also be wired correctly. A running router with no usable workers should not make the whole service ready.
Give each group a stable identity
A replica group describes instances that share a workload template. Its name identifies the role; its replica count describes how many instances to create. The common model will carry the familiar configuration for containers, cmd, resources, environment, connections, initialization, sidecars, and volumes.
The planned authoring direction is a root-level groups section. We are defining how shared defaults, group overrides, presets, and patches compose. In particular, updating decode must continue to target decode when another group is added or removed. Group identity cannot depend on its position in a list.
The model will also sit beneath the existing TFJob, PyTorchJob, DaskCluster, and RayCluster replica schemas. Those runtimes will retain their framework-specific roles and public configuration shapes, including Ray's scaling bounds and startup parameters. Existing distributed workloads should keep their behavior as the common implementation is extracted.
Resource accounting must follow the same structure. For a one-GPU-per-worker version of the inference example, four prefill replicas and eight decode replicas request twelve GPUs, plus the CPU and memory required across all thirteen Pods. GPU quantity and GPU model selection remain separate: requesting one GPU does not select an H200. Hardware placement continues to use the cluster's scheduling configuration and presets.
Extend the model to training and RL
For synchronous distributed training, the useful unit can be a worker group that must fit together. Placing three of four required workers can reserve GPUs while the application waits for the fourth. The replica model describes the workers; a gang-placement policy describes which set must be schedulable before they start. The training framework still establishes ranks, rendezvous, and collectives.
RL introduces several kinds of work. Rollout workers generate trajectories, learners update the policy, and some algorithms use separate reward, critic, or reference-model processes. Their resource requirements and scaling behavior can differ. Adding rollout capacity does not necessarily increase the number of learner workers that must participate in a training step.
These are deployment choices. For example, verl supports workers that combine actor, rollout, and reference roles. A Ray cluster can host multiple actors inside its worker Pods. The shared abstraction should represent the chosen infrastructure layout while the framework manages its actors, policy updates, and weight synchronization. Our existing Ray and RAGEN walkthrough shows that division of responsibility today.
For a separated layout, role-specific requirements become essential. Suppose an RL step needs four learner Pods and at least two rollout Pods. A flat minimum of six Pods could be satisfied by rollout capacity alone if group membership is discarded. The workload needs a policy that preserves both requirements.
Constructed example, not scheduler output. The placement policy needs the required roles, not just their summed replica count.
A replica group and a scheduling gang therefore need separate meanings. Some groups can scale independently; others need coordinated placement. A single model replica may itself span several worker Pods. The scheduling integration must preserve that structure, including required locality between workers where the application depends on it.
Map the topology to the cluster's scheduling system
Some coordinated layouts will need native Kubernetes workload-aware scheduling or an additional scheduling integration. Our target is to keep the workload description consistent while translating supported policies to the selected backend.
| Backend | Relevant responsibility | Planned integration work |
|---|---|---|
| Native Kubernetes scheduling | Group placement through Workload and PodGroup APIs. | Create group resources, assign Pod membership, and preserve the required group thresholds. |
| Volcano | Gang placement within its batch scheduling system. | Map groups and resource requirements to compatible Volcano workload and queue policies. |
| KAI Scheduler | GPU scheduling, coordinated Pod groups, and topology constraints. | Preserve role membership and supported subgroup/locality requirements in KAI's grouping model. |
| Kueue | Quota admission, resource flavors, and topology-aware admission decisions. | Integrate the emitted workload with Kueue's admission lifecycle; the selected Pod scheduler still performs binding. |
These are integration targets, not a claim of interchangeable APIs or completed Polyaxon support. A scheduler name or queue label alone does not establish group creation, admission, placement, and cancellation behavior. The scheduling tools comparison explains the existing integration boundaries.
For the native path, Kubernetes 1.37 provides beta Workload/PodGroup APIs and gang scheduling, disabled by default. The relevant feature gate, API resources, scheduler plugin, and controller integration must be enabled. Hierarchical scheduling through CompositePodGroup is a separate alpha capability in 1.37, also disabled by default. A topology requiring that hierarchy has additional requirements beyond the basic beta feature. Upstream scheduling update.
Kubernetes 1.37 is one route to coordinated placement. Ordinary replication and existing distributed runtimes do not acquire that version requirement. Clusters using Volcano or KAI may provide another route for supported layouts; Kueue can govern admission alongside a compatible placement system. We will document the supported combinations rather than assume that installing several controllers makes them work together.
Make the operation reflect all of its groups
The operation must remain understandable when only part of the workload is healthy. We plan to associate Pod events and diagnostics with both the owning operation and its named group, resolve connections and initialization for every group, and reconcile group updates independently. Removing a group must remove its owned resources.
Readiness and completion need runtime-specific rules. A router exiting successfully must not mark an inference service succeeded. Losing a learner may require coordinated training recovery, while a serving deployment may be able to replace a worker. The framework determines whether recovery can continue from existing state; gang placement alone does not restore a checkpoint or synchronize a policy.
This also keeps three decisions distinct: whether the workload may consume quota, whether its required Pods can be placed, and whether the application is ready to make progress. Each needs visible state and a clear owner.
How we plan to deliver it
We are starting by extracting the common replica model while preserving existing runtime behavior. Next comes service-group compilation and operator support, including the router/worker wiring. We will then complete root-level authoring, configuration composition, and operation-wide lifecycle handling. Group-aware training and RL scheduling will build on that foundation through backend-specific integrations.
We will judge those integrations against concrete cases: scaling decode without changing the other desired groups; holding a four-worker training group when only three workers fit; keeping an RL workload unready when learner capacity is missing; reporting a failed group accurately; and releasing all owned resources when an operation is cancelled.