Benchmark vLLM speculative decoding on Polyaxon

Speculative decoding can produce several tokens in one target-model iteration. Whether it saves time depends on the cost of proposing and checking those tokens, how often they are accepted, and what the GPU is already doing.

This tutorial compares ordinary decoding with vLLM's n-gram prompt-lookup method on Polyaxon. We use one pinned Qwen3-4B model, one GPU allocation, and a paired prompt inventory. The outcome is a measured comparison for your workload, rather than a speedup borrowed from another model's benchmark.

What the target model still decides

In the original speculative-sampling algorithm, a cheaper proposal mechanism suggests tokens and the target evaluates them together. Acceptance and correction preserve the target distribution under the algorithm's assumptions. A rejected proposal is replaced through the correction rule; it is not simply emitted because it was cheap. For greedy decoding, the intuition is straightforward: keep the matching prefix and continue with the target's choice. Leviathan, Kalman, and Matias, ICML 2023.

Fewer target iterations do not guarantee lower latency. Proposals and verification add work, and the benefit depends on available compute and accepted tokens. The paper's analysis separates acceptance from proposal cost; its measurements describe its own models and hardware. We need to measure ours.

We start with prompt lookup because it does not require deploying a second draft checkpoint. It searches for matching token sequences in the existing context and proposes their continuation. Copying or lightly editing source text is a plausible fit; a short answer or unrelated continuation may offer little reusable material. This is a hypothesis about the workload, not a promise. vLLM v0.11.2 speculative decoding.

Fix the deployment and vary one method

The companion supplies the service, launcher, nine teaching prompts, streaming harness, and comparison helper:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/vllm-speculative-decoding
python -m pip install -r requirements.txt
polyaxon run -f service.yaml -u . -p YOUR_PROJECT --queue YOUR_AGENT/YOUR_GPU_QUEUE -P mode=baseline

Use a configured project with service permissions, a dedicated NVIDIA GPU queue, compatible drivers, and model-download access. Both modes pin Qwen3-4B to revision 1cfa9a7208912126459214e8b04321603b3df60c and the vLLM v0.11.2 image digest. The deployment keeps a 2,048-token maximum length, an eight-sequence scheduler limit, and a GPU-memory budget of 0.85.

Both modes disable automatic prefix caching. The repeated prompts otherwise make cache reuse a second changing factor. If your real deployment uses prefix caching, perform another paired comparison with it enabled in both modes. vLLM engine arguments.

The n-gram launcher adds this configuration:

{
  "method": "ngram",
  "num_speculative_tokens": 5,
  "prompt_lookup_min": 2,
  "prompt_lookup_max": 4
}

These are starting values. The companion explicitly sets both lookup bounds because the pinned implementation derives omitted values; leaving defaults implicit can make an upgrade change the experiment. The configuration is checked against the v0.11.2 source.

Measure the same requests at several loads

Approve the baseline service if required. Once healthy, obtain its URL:

polyaxon ops service -p YOUR_PROJECT -uid BASELINE_UUID --external --url
export SERVICE_URL='PASTE_THE_RETURNED_BASELINE_URL'
export POLYAXON_TOKEN='YOUR_CONFIGURED_POLYAXON_TOKEN'
python bench.py --concurrency 1 --repeats 10 --service-run BASELINE_UUID --out baseline-c1
python bench.py --concurrency 4 --repeats 10 --service-run BASELINE_UUID --out baseline-c4
python bench.py --concurrency 8 --repeats 10 --service-run BASELINE_UUID --out baseline-c8

The client preserves the URL prefix and uses Polyaxon's token authentication. Each request uses temperature zero, disables Qwen's thinking mode, and caps output at 128 tokens. The harness warms up every case, repeats the inventory ten times, and shuffles it with a fixed seed. Concurrent workers submit the next request when an earlier one completes: this is a closed-loop load.

Experiment design. The diagram contains no measured latency or throughput values.

Stop the baseline and confirm its Pod has terminated before launching the other mode on the same isolated GPU:

polyaxon ops stop -p YOUR_PROJECT -uid BASELINE_UUID
polyaxon run -f service.yaml -u . -p YOUR_PROJECT --queue YOUR_AGENT/YOUR_GPU_QUEUE -P mode=ngram
polyaxon ops service -p YOUR_PROJECT -uid NGRAM_UUID --external --url
export SERVICE_URL='PASTE_THE_RETURNED_NGRAM_URL'
python bench.py --concurrency 1 --repeats 10 --service-run NGRAM_UUID --out ngram-c1
python bench.py --concurrency 4 --repeats 10 --service-run NGRAM_UUID --out ngram-c4
python bench.py --concurrency 8 --repeats 10 --service-run NGRAM_UUID --out ngram-c8
python compare.py baseline-c1 ngram-c1

Compare the other concurrency pairs too. Repeat the whole deployment pair in reverse order with fresh output directories. Preserve GPU SKU, driver, server configuration, client location, and network path with both Polyaxon run UUIDs. A stopped run request alone does not establish that the device is free.

Read latency, throughput, and output together

The client stores each request and a summary:

An SSE chunk can contain several tokens. The harness uses the final usage count and does not treat chunks as token arrivals. Its decode interval includes stream-completion and client overhead; it is not a kernel timing or a precise inter-token latency distribution.

The nine prompts contain copy/edit tasks, explanations, and short answers. They make the interface easy to inspect, but 90 repeated requests do not establish a reliable production tail. Replace them with a frozen representative sample and inspect each family separately. The complete request records also retain output text, length, and termination reason, so a capped or shorter output does not quietly become evidence of faster equivalent work.

For these greedy requests, the comparison helper checks paired case identities and successful output text. A mismatch warrants investigation before using the performance result. Equality on this corpus is a smoke check, not proof of distribution equivalence for stochastic sampling or other configurations. A production task also needs its own quality evaluation.

N-gram proposals may help a copying workload at low load while adding little value to short answers. At higher load, verification can compete with useful target work already filling the GPU. Select the method that meets latency and quality requirements at the intended traffic mix; aggregate throughput alone can conceal a worse experience for one request family.

Stop the n-gram service after measurement:

polyaxon ops stop -p YOUR_PROJECT -uid NGRAM_UUID

The companion is source-reviewed and unexecuted. This article supplies the experiment and its interpretation; it makes no measured speedup claim.

References

• Fast inference from transformers via speculative decoding, Leviathan et al., ICML 2023: acceptance/correction, proposal cost, and the conditions in the speed analysis.
• vLLM v0.11.2 speculative decoding: n-gram proposals and version-specific support.
• Pinned speculative configuration source: explicit lookup bounds.
• vLLM engine arguments: the common serving and cache controls.
• Qwen3-4B model card: checkpoint and thinking-mode behavior.