Roll out a new model when every GPU is occupied

Three inference replicas occupy the cluster's three available GPUs. A new model revision is ready, but the first replacement Pod stays Pending. The old replicas remain healthy, and the rollout makes no progress.

This can be the expected result of a rollout configured to preserve every old replica until a new one is available. Kubernetes can create an additional Pod, but it cannot create an additional GPU. The team must supply temporary capacity, accept reduced serving capacity, or choose a maintenance window.

The following scenario assumes independent replicas, one whole GPU per Pod, compatible nodes, and no spare eligible devices. A distributed model replica spanning multiple Pods needs a controller that coordinates the entire replica.

Calculate the overlap

A Deployment's rolling-update settings control allowed surge and unavailability. maxSurge permits extra replicas; maxUnavailable permits fewer available replicas during replacement. Neither is a hardware reservation. Deployment documentation.

For the three-replica scenario, compare these choices:

These are illustrative choices, not measured availability guarantees. Other constraints can still block placement: host memory, topology, device type, volume attachment, or namespace quota.

Before selecting the second policy, establish that two replicas can handle the expected traffic while a replacement loads. If they cannot, slowing down the rollout does not solve the missing capacity. Arrange temporary capacity or reduce demand through an agreed operational procedure.

Express the chosen availability tradeoff

This Deployment-spec fragment permits one of three replicas to be unavailable, with no surge. Merge it into the existing manifest only after validating the reduced-capacity case:

spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  minReadySeconds: 15
  progressDeadlineSeconds: 1200

The readiness interval and deadline are example values. Choose them from measured startup and stability behavior. A Deployment progress deadline reports stalled progress; it does not automatically roll back a model or fix an unschedulable Pod. The Deployment API defines these controls.

Also identify who owns the replica count. If HPA or another controller is active, its changes affect the capacity calculation. Coordinate the rollout with that ownership model instead of letting a release process continually overwrite the autoscaler's desired count. See the inference autoscaling guide for that boundary.

Follow one replacement through the system

The first old replica begins termination. It may still be serving in-flight work and holding its GPU allocation. Only after that capacity is released can the replacement use it. The replacement then obtains its image and model files, initializes the runtime, and becomes ready.

Treat these as separate milestones:

1. The old replica stops accepting new requests and drains existing work.
2. Its process and device allocation are released.
3. The replacement receives an eligible allocation.
4. Model loading and any required warmup complete.
5. Readiness succeeds and the replica remains available for the configured interval.

During part of this sequence, two replicas carry the traffic. Record queue delay, errors, and latency throughout, rather than comparing only the steady state before and after deployment.

Terminating Pods can continue consuming resources beyond the Deployment's ordinary replica accounting. A no-surge policy is therefore not a promise of instantaneous GPU reuse. The rollout lifecycle documentation explains the terminating-Pod distinction.

Make readiness and draining meaningful

Readiness should reflect whether the model server can accept the intended requests. An open TCP socket or a live process can precede completed model loading. Use a startup probe where initialization is slow, and keep the responsibilities of startup and readiness probes distinct.

During shutdown, the application and traffic layer must handle in-flight requests, especially streaming generation. Endpoint updates are not instantaneous, and removing a backend from new routing decisions does not finish its existing streams.

If the server uses a preStop hook, its time is included in the Pod's termination grace period. A long sleep can consume time that the server needs to drain. Use the serving stack's supported shutdown behavior and measure it. Container lifecycle hooks.

Diagnose a stalled rollout by stage

For an existing Deployment named model-api in inference, inspect the controller and its Pods:

kubectl describe deployment model-api -n inference
kubectl get replicasets -n inference
kubectl get pods -n inference -o wide

Follow owner references to the relevant old and new ReplicaSets. If the new Pod is unscheduled, inspect its placement events. If it has a node but is not ready, investigate image retrieval, model loading, application initialization, and probes. Increasing the progress deadline only hides the distinction when the underlying condition cannot resolve.

Plan rollback capacity too. Returning to an older image still requires that image and model to start, pass readiness, and take traffic under the same GPU constraints. Retain both runtime and model identifiers for each revision.

Qualify the model release with Polyaxon

Polyaxon can organize the evidence needed to choose a GPU-constrained rollout. Run a repeatable load evaluation against the serving endpoint at its normal capacity and at the reduced capacity the rollout requires. Retain the model revision, runtime image, resource requirements, load profile, warmup duration, latency, and errors with each evaluation, using the inference benchmarking workflow.

Use resource scheduling and presets to keep benchmark placement and resource requests consistent. A Polyaxon pipeline can place an explicitly implemented release operation behind manual approval, giving the reviewer the candidate's results before authorizing the capacity tradeoff. The release operation still needs its own deployment logic and appropriate permissions.

The serving controller owns replica replacement and traffic handling. For a Polyaxon-managed service, verify that the deployed integration exposes the relevant Deployment settings; for an external Deployment, apply the policy through its existing release mechanism. The native Deployment fragment above is not automatically a Polyaxon service specification.

Choose the rollout policy after deciding what the remaining replicas can safely serve. The meaningful result is a completed model transition with acceptable request outcomes, not simply three Pods reporting Running again.