Load-test ML services on Kubernetes

A load test should answer a capacity or reliability question. Sending as many requests as possible and reporting an average latency rarely does that for an ML service. Model size, input shape, batching, queueing, accelerator memory, cold starts, and autoscaling all influence the result.

Design the experiment so another engineer can repeat it, explain the bottleneck, and decide what to change.

Write the question before the script

Useful questions are specific:

• How many requests per second can this model sustain while keeping p99 latency below the service objective?
• How does a scale-from-zero deployment behave during a sudden burst?
• Does dynamic batching improve throughput without exceeding the latency budget?
• How much traffic can the service absorb when one node or zone becomes unavailable?
• Does a new runtime reduce GPU time per accepted request?

Each question implies a workload shape, duration, metrics, and acceptance criteria. A single test configuration cannot answer all of them.

Model arrivals and inputs

Choose between an open and closed workload model deliberately. A closed model starts a fixed number of virtual users who wait for responses before sending more work. As the service slows, their request rate falls. An open model schedules arrivals independently, which can reveal queue growth and overload more realistically.

Represent production input diversity as well:

• prompt or payload size distribution;
• response length or compute variability;
• model and endpoint mix;
• streaming versus non-streaming requests;
• cache-hit distribution;
• authenticated tenant or priority classes;
• valid error and cancellation behavior.

Synthetic inputs should be safe, but they must exercise comparable tokenization, preprocessing, and execution paths.

Separate test phases

Run explicit phases instead of one undifferentiated traffic block:

1. Warmup: initialize clients, connections, caches, and model paths without recording the main result.
2. Ramp: increase arrivals in controlled steps to locate the operating range.
3. Steady state: hold a target long enough to observe queueing, autoscaling, and thermal or memory behavior.
4. Overload: exceed planned capacity to verify rejection, backpressure, and recovery.
5. Recovery: reduce traffic and confirm queues, latency, and replicas return to an acceptable state.

Record the exact timestamps for every phase so service and cluster telemetry can be aligned with the generator.

Measure useful work

Request rate alone can reward fast failures. Track accepted and successfully completed work together with quality and latency:

Our GPU utilization metrics guide explains why allocation, device activity, and throughput must be read together. A busy GPU is not automatically producing useful results.

Isolate the load generator

The generator must not become the hidden bottleneck. Run it on separate capacity from the system under test, especially when testing cluster saturation. Measure its CPU, memory, network, open connections, event-loop lag, and error rate.

For distributed generators, verify that their clocks and phase control are consistent. Keep request identifiers so a client observation can be correlated with gateway, service, and model-runtime telemetry.

Store the generator version, test definition, data profile, and target configuration with the result. A chart without those inputs is not a reproducible benchmark.

Test autoscaling as a control loop

Kubernetes autoscaling is not instantaneous. Metrics are sampled, recommendations are computed, Pods are scheduled, images are pulled, models are loaded, and readiness must succeed before capacity serves traffic. The Horizontal Pod Autoscaler is one part of that sequence.

Measure:

• time from increased arrivals to the first scaling decision;
• time from decision to scheduled Pod;
• model initialization and readiness time;
• queue growth during the gap;
• overshoot and oscillation;
• scale-down behavior and cost after the burst.

If Pods cannot schedule because accelerator capacity is absent, a perfect HPA signal will not help. Include node autoscaling and quota behavior when they are part of production capacity.

Exercise failure during load

Capacity without recovery evidence is incomplete. During a controlled test, remove a replica, drain eligible capacity according to policy, delay a dependency, or introduce a known error. Confirm that traffic shifts, retries remain bounded, and the service recovers without duplicating unsafe work.

Define stopping conditions before the test. Protect shared clusters with traffic ceilings, namespace quotas, cost limits, and an owner who can terminate the run. Never point an unbounded load generator at production by accident; make the target environment explicit in both configuration and review.

Track load tests as experiments

A load test has the same reproducibility needs as an ML experiment: versioned code, parameters, environment, input data, metrics, and artifacts. Run generators as controlled Polyaxon operations, attach their configuration and reports, and compare changes under equivalent conditions.

Finish with a decision, not merely a dashboard. State the highest validated operating point, the limiting resource, the behavior beyond capacity, recovery time, and cost per useful unit. That turns load testing from a one-time demonstration into evidence for capacity planning and release control.