Autoscale inference services using workload metrics

Requests wait for an inference server while its CPU usage looks unremarkable. The bottleneck may be GPU execution, a growing request queue, or a workload with highly variable prompt lengths. CPU utilization alone can be a poor signal for how much serving capacity users need.

Kubernetes HorizontalPodAutoscaler (HPA) can use custom or external metrics through the autoscaling/v2 API. It adjusts a compatible workload's replica count; it does not make a new replica ready instantly. The metric pipeline, workload controller, and model startup behavior all belong in the design. HPA documentation.

The example below assumes a Deployment of independent inference replicas. Each Pod can serve requests on its own. Scaling individual workers inside one tensor-parallel or otherwise distributed engine requires a controller that understands the complete serving unit.

Choose a signal tied to user demand

Start with a measured service objective, such as acceptable queue delay or time to first token. Then identify a signal that changes early enough for additional capacity to help.

For a first experiment, a queue-depth gauge is understandable and inspectable. Validate that it responds to overload in your workload before using it as the production control signal. The repeatable inference benchmark guide helps define the traffic mix and comparison.

Expose the metric HPA actually reads

An application metric in a Prometheus dashboard is not automatically available to HPA. A custom-metrics adapter must expose it through custom.metrics.k8s.io, associated with the correct namespace and Pods. Metrics Server supplies resource metrics; it does not by itself provide an application's request queue. The HPA walkthrough explains the different metrics APIs.

For this example, inference_requests_waiting is a reader-supplied metric name, not a built-in Kubernetes or universal inference-server metric. It reports each selected Pod's own waiting requests as a gauge.

If your queue lives at a shared gateway, model it as the appropriate object or external metric instead. Do not copy the same global queue count onto every Pod and treat the copies as independent measurements. Check missing samples, scrape delay, and behavior when a Pod is starting or shutting down.

Set a bounded scaling policy

This HPA assumes an existing model-api Deployment in the inference namespace, a working custom-metrics adapter, and permission to manage autoscalers. Adapt the target and metric mapping before using it.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-api
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-api
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_requests_waiting
        target:
          type: AverageValue
          averageValue: "4"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

The four-request target and replica limits are illustrative, not recommendations for every model. AverageValue is a raw per-Pod metric target, not a CPU percentage. The behavior section limits replica changes and retains a five-minute scale-down stabilization window. See the HPA v2 API for these fields.

With two replicas averaging twelve waiting requests each, the basic ratio calculation suggests six replicas: ceil(2 × 12 / 4). Actual changes also depend on metric availability, tolerance, stabilization, and rate limits. In this example, the scale-up policy prevents immediately adding four replicas in one unrestricted jump.

Measure usable capacity, not just desired replicas

New Pods need suitable nodes, images, model files, initialization, and successful readiness checks. If every new replica needs a GPU and the cluster has none free, increasing the desired count creates Pending Pods. Node autoscaling is a separate process and can itself be limited by configuration or provider capacity.

Record the time from overload detection to the first request served by added capacity. Include cold-node and cold-model cases. Keep a minimum warm pool when the measured startup delay is longer than the service can tolerate.

Scale-down also needs application behavior: stop accepting new work and handle in-flight requests during termination. A stabilization window reduces rapid reversals in replica count; it does not drain requests or preserve a server's in-memory cache.

Inspect the autoscaler and Deployment together:

kubectl describe hpa model-api -n inference
kubectl get deployment model-api -n inference
kubectl get pods -n inference

Compare current metrics, desired replicas, ready replicas, and request outcomes. A missing metric, insufficient GPU capacity, and slow model loading call for different fixes. If removing this example HPA, delete only hpa/model-api in inference after arranging who will own the Deployment's desired replica count.

Compare scaling policies with Polyaxon

Polyaxon can turn a scaling-policy decision into a repeatable serving evaluation. Package the load generator as an operation, keep the prompt or request dataset fixed, and compare candidate HPA settings against the same demand increase and decrease. Use resource scheduling for the benchmark workloads and run tracking for latency, throughput, errors, and observed ready capacity.

Retain the model revision, runtime image, metric definition, load profile, and scaling configuration with every result. This works whether the load generator targets a compatible Polyaxon-managed deployment or an externally managed inference endpoint. The inference benchmarking guide shows how to organize that comparison.

HPA needs a scalable target whose controller permits it to manage replicas; a bare service Pod is not such a target. Verify ownership for the deployed serving integration. The HPA metric adapter supplies live scaling signals, while Polyaxon records the benchmark evidence used to choose the policy.

Adopt the policy when it improves the measured service objective through both a demand increase and a later decrease. More requested replicas alone are not evidence of a healthier inference service.