Polyaxon v3 is coming →
vLLM

Serve models with vLLM on Polyaxon

vLLM owns the inference engine and API behavior. Polyaxon turns the model server into an operated Kubernetes workload with explicit inputs, GPU placement, cache, secrets, access, and run history.

Outcome

A reusable vLLM component whose model, tensor parallelism, context length, and GPU request can be reviewed together.

Start the guideServing documentation

Service responsibilities

Polyaxon schedules

The service component fixes the image, model, port, accelerator request, connections, and health boundary.

The runtime serves

The inference engine loads the model and exposes its native or OpenAI-compatible API.

Clients validate

Smoke tests and evaluation operations exercise the endpoint while Polyaxon retains the deployment context.

Prerequisites

  • A vLLM image compatible with the selected model, GPU architecture, and driver.
  • A persistent model cache and Hugging Face credentials when required.
  • A representative request set for smoke, quality, and load validation.

Deploy and validate

  1. 1

    Define the vLLM service

    Expose vLLM's API through a Polyaxon service and keep the main memory and parallelism settings visible as component inputs.

    version: 1.1kind: componentname: vllm-openaiinputs:- name: model  type: str  value: Qwen/Qwen3.6-27Brun:  kind: service  ports: [8000]  rewritePath: true  connections: [hf-token, model-cache]  container:    image: vllm/vllm-openai:latest    command: ["vllm", "serve"]    args: ["{{ model }}", "--host", "0.0.0.0", "--port", "8000", "--tensor-parallel-size", "4"]    resources:      limits:        nvidia.com/gpu: "4"

    vLLM publishes this image for its OpenAI-compatible server. Replace the moving tag with a tested release tag or digest before production.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f vllm.yaml
  3. 3

    Probe the vLLM API

    Resolve the service URL and verify model discovery before running an evaluation or performance workload.

    SERVICE_URL=$(polyaxon ops service --external --url)curl "$SERVICE_URL/v1/models"

Production checklist

Pin the runtime image and model revision; warm the model cache before measuring startup or latency.
Keep the service private by default and add authentication, TLS, rate limits, and network policy deliberately.
Define readiness, liveness, timeout, graceful shutdown, and rollback behavior before production traffic.
Measure latency distributions, throughput, quality, errors, and accelerator memory under representative load.
Increase context length only with measured KV-cache headroom; do not assume model weight fit is sufficient for production traffic.

Troubleshooting

The service never becomes ready

Inspect model download, credentials, disk space, GPU memory, runtime flags, port binding, and readiness behavior.

Requests fail through the URL

Verify the service port, rewrite-path setting, authentication header, ingress, network policy, and API path.

Latency degrades under load

Measure queueing, batch settings, context length, KV cache pressure, replica saturation, and storage or network contention.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore LLMOps