Resize CPU and memory without replacing Kubernetes Pods

A notebook needs more CPU after a developer starts a larger preprocessing task. An inference service has enough GPU capacity, but its host-side workers are throttled. Replacing the Pod to adjust resources may interrupt a useful session or force a model to load again.

Kubernetes can change container CPU and memory requests and limits while preserving the Pod. Container-level in-place resizing has been stable since Kubernetes 1.35. The distinction that matters is whether the container also keeps running: preserving a Pod does not guarantee preserving its process. Upstream graduation announcement.

This guide covers that stable container-level mechanism, with a disposable example and practical implications for Polyaxon workloads. For choosing resource values in the first place, start with right-sizing Kubernetes resources.

Decide whether a resize fits the problem

An in-place change acts on the Pod's current node. It can be useful when a running application needs a different host-resource envelope and the node can accommodate it. It does not move the Pod to a larger machine.

For ML workloads, distinguish these cases:

This mechanism resizes CPU and memory, not nvidia.com/gpu. Check the current restrictions for your node configuration: Windows Pods and Pods using static CPU or Memory Manager policies are outside this walkthrough.

Separate Pod identity from process continuity

Each container can declare a resizePolicy for CPU and memory. NotRequired requests resizing without a container restart; RestartContainer requires one for that resource change. Set these policies when creating the Pod: the resize policy itself is immutable. If a change includes a resource requiring restart, do not expect the other resource's policy to preserve the process. Upstream resize design.

A restart can discard a notebook's in-memory variables or make an inference server reload its model even though the Pod UID stays the same. Applications may also read CPU counts, heap sizes, or thread-pool settings only at startup. A larger resource limit does not automatically reconfigure that application.

For a stateful session, decide how to preserve work before requesting a change. NotRequired is not protection from unrelated crashes, OOM kills, or node failures.

Try a small CPU increase

Use a disposable Linux cluster or namespace, Kubernetes 1.35 or later, a compatible container runtime, and a kubectl version within the supported skew for your cluster. The resize flag requires kubectl 1.32 or later. Your account needs permission to create the example resources and patch pods/resize; namespace policies must permit the requested resources.

Save this as resize-demo.yaml. The container only sleeps: this demonstrates the resource transition, not a performance improvement.

apiVersion: v1
kind: Namespace
metadata:
  name: ml-resize-demo
---
apiVersion: v1
kind: Pod
metadata:
  name: session
  namespace: ml-resize-demo
spec:
  restartPolicy: Always
  containers:
    - name: workspace
      image: python:3.12-slim
      command: ["python", "-c", "import time; time.sleep(86400)"]
      resizePolicy:
        - resourceName: cpu
          restartPolicy: NotRequired
        - resourceName: memory
          restartPolicy: RestartContainer
      resources:
        requests:
          cpu: "250m"
          memory: "128Mi"
        limits:
          cpu: "500m"
          memory: "256Mi"

The image tag is a convenience for this disposable example; use your approved image digest for a reproducible workload. Create the resources and capture their initial state:

kubectl apply -f resize-demo.yaml
kubectl wait -n ml-resize-demo --for=condition=Ready pod/session --timeout=120s
kubectl get pod session -n ml-resize-demo -o yaml

Record the Pod UID, container ID, restart count, and resource values. Then increase the CPU request to half a core and its limit to one core:

kubectl patch pod session -n ml-resize-demo \
  --subresource=resize --type=strategic \
  --patch '{"spec":{"containers":[{"name":"workspace","resources":{"requests":{"cpu":"500m"},"limits":{"cpu":"1"}}}]}}'

The named-container strategic merge patch changes the CPU fields and retains the existing memory settings. Both configurations remain Burstable: requests stay below limits. A resize must preserve the original Pod QoS class, so this is not a way to convert a BestEffort Pod into a Guaranteed one.

Confirm the applied state

Inspect the Pod again:

kubectl get pod session -n ml-resize-demo -o yaml
kubectl describe pod session -n ml-resize-demo

spec.containers[*].resources records the desired values. Compare them with status.containerStatuses[*].resources to see the configured resources, and inspect generation tracking and resize conditions. An accepted patch alone does not prove that the runtime applied it. Resize status documentation.

Read these conditions with container state and events, as described in the Pod lifecycle documentation. Do not treat a stale condition from an earlier generation as the result of the newest request.

The example deliberately makes memory changes restart the container. Other policies can allow a memory resize without restart, but lowering a limit is not a reliable way to make an application release memory. Preserve headroom and inspect actual use before attempting a decrease.

Apply the lesson to Polyaxon workloads

Polyaxon sandbox resource settings and scheduling presets define resource requirements for runs. The Kubernetes feature creates an opportunity for platform-managed live resizing; it does not establish that editing a Polyaxon preset resizes an already running Pod. Presets participate in operation compilation.

For a Polyaxon-managed workload, first identify the owning controller and the platform's supported update path. A direct patch to a generated Pod is not a durable replacement for the component or controller template. Keep the validated resource settings in that source configuration so a future run or replacement receives them.

Record the run identity, old and requested resources, applied state, restart evidence, and application measurements. For a sandbox, verify that the session remains usable. For inference, compare latency and errors. For training, confirm throughput and worker health; additional host CPU does not change the distributed application's world size.

Use the disposable example to understand the mechanism, then evaluate a supported update path for one representative workload. Clean up only the namespace created for this exercise:

kubectl delete namespace ml-resize-demo