Size GPU memory for LLM serving

A language model's weights can fit on a GPU while its serving workload still runs out of memory. Requests bring their own state: the keys and values retained for attention as the context grows. Longer contexts and more live sequences can make that state a substantial part of the budget.

We will calculate the raw KV-cache storage for a pinned Qwen3-4B configuration, then separate that calculation from the capacity a serving engine actually exposes. This gives you a way to reason about context length and concurrency before changing a Polyaxon GPU service.

Count the state for each live token

For a dense transformer with full attention, raw KV tensor storage is:

bytes = 2 × layers × KV heads × head dimension
          × bytes per element × total live tokens

The factor of two accounts for keys and values. total live tokens is the sum of the stored prompt and generated tokens across the live sequences. If eight sequences each hold 4,096 tokens, that population is 32,768 tokens.

Use the number of KV heads, not query heads. Grouped-query attention lets multiple query heads share a smaller set of key/value heads. This changes cache storage without reducing every dimension of the model by the same factor. GQA, Ainslie et al., 2023.

Qwen3-4B at revision 1cfa9a7208912126459214e8b04321603b3df60c specifies 36 layers, 8 KV heads, and a head dimension of 128. With FP16/BF16 cache elements:

2 × 36 × 8 × 128 × 2 = 147,456 bytes per token = 144 KiB

Eight 4,096-token sequences therefore require 4.5 GiB of raw KV tensors. These numbers are derived from the pinned model configuration, not measured GPU memory.

Raw tensor calculation with no shared prefixes or page padding. Prompt and generated tokens both count. These cells are capacity estimates, not serving benchmark results.

The checkpoint index records 8,044,936,192 serialized weight bytes, approximately 7.49 GiB. Adding those weight bytes to the 4.5 GiB cache estimate gives about 11.99 GiB. That is a useful subtotal, but insufficient evidence that this workload fits on a 12 GiB GPU.

Add the engine's costs

Serving also needs temporary activations, kernel workspaces, CUDA graph pools, allocator reserve, and other runtime state. The weight layout and dtype loaded by the engine can differ from serialized checkpoint storage. Image encoders, adapters, or a draft model add their own requirements.

PagedAttention improves how a serving engine manages KV blocks: a sequence's logical blocks can map to noncontiguous physical blocks, and eligible prefixes can share storage. The physical block pool can be reserved in advance even while individual requests occupy only part of it. That is why an idle nvidia-smi reading does not directly reveal the incremental memory of one request. PagedAttention, Kwon et al., SOSP 2023.

The simple formula assumes independent token storage in a full-attention model. Shared prefixes can reduce it; partially filled blocks add waste. Sliding-window or hybrid-attention models need a layer-aware calculation. Tensor parallelism introduces partitioning details. Treat the formula as a model of raw tensors with stated assumptions, not a universal GPU-capacity rule.

Weight quantization and KV-cache quantization are separate choices. Reducing weight precision does not automatically halve this cache estimate. vLLM exposes a distinct --kv-cache-dtype setting with hardware and scale requirements. Recheck output quality when adopting a lower-precision cache. vLLM quantized KV cache.

Keep length, scheduling, and admission distinct

Three controls are easy to conflate:

max_num_seqs=8 does not mean the HTTP service can have only eight clients. Other requests can wait. Neither that flag nor gpu_memory_utilization defines an application admission policy or guarantees acceptable latency. vLLM v0.11.2 engine arguments.

Use the memory estimate to choose a plausible starting point, then inspect the engine's logged KV capacity and measure requests at the lengths and concurrency you intend to support. A service that remains alive while requests wait too long has passed a memory check but failed a latency requirement.

Try a controlled Polyaxon deployment

The companion contains a standard-library calculator and a GPU service pinned to Qwen3-4B and the vLLM v0.11.2 image digest:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/llm-memory-budget
python budget.py --tokens 4096 --sequences 8 --kv-bytes 2
polyaxon run -f service.yaml -p YOUR_PROJECT --queue YOUR_AGENT/YOUR_GPU_QUEUE -P max_model_len=4096 -P max_num_seqs=8

Use a configured project, service permissions, a dedicated NVIDIA GPU, compatible drivers, and model-download access. Approve the run if required. Polyaxon applies the GPU request and queue selection and retains the service configuration; vLLM determines how its memory budget is used.

Read startup logs before sending traffic:

polyaxon ops logs -p YOUR_PROJECT -uid SERVICE_UUID
polyaxon ops service -p YOUR_PROJECT -uid SERVICE_UUID --external --url

Record the GPU SKU, driver, loaded dtype, model/image revisions, engine limits, and logged cache capacity. Use a frozen prompt inventory with known token lengths and an output cap. Change one limit at a time, retain the run UUID for each configuration, and observe failures, waiting, and latency as well as device memory. Our repeatable inference benchmark guide describes that comparison.

If the model fails during loading, examine weights and initialization headroom first. If it loads but struggles under long concurrent requests, investigate the live-token population and cache capacity. If reducing concurrency fixes latency without changing memory much, inspect scheduling and compute saturation. Those are different observations and justify different changes.

Stop the service after the experiment:

polyaxon ops stop -p YOUR_PROJECT -uid SERVICE_UUID

The companion is source-reviewed and unexecuted. The figures are exact calculations under the stated tensor-storage assumptions; no GPU fit, throughput, or latency result is claimed.

References

• Pinned Qwen3-4B configuration and checkpoint index: inputs to the calculation.
• GQA, Ainslie et al., EMNLP 2023: the distinction between query heads and stored KV heads.
• PagedAttention, Kwon et al., SOSP 2023: block allocation and prefix sharing.
• vLLM engine arguments and KV quantization: the version-specific controls used in the deployment.