Polyaxon v3 is coming →
TensorRT-LLM

Serve models with TensorRT-LLM

TensorRT-LLM performance depends on a tightly matched model, engine or PyTorch backend, container, CUDA stack, precision, and GPU topology. Polyaxon keeps those choices attached to the service and its benchmark operations.

Outcome

A TensorRT-LLM service component with an explicit model, backend, tensor-parallel size, GPU count, cache, and validation endpoint.

Start the guideServing documentation

Service responsibilities

Polyaxon schedules

The service component fixes the image, model, port, accelerator request, connections, and health boundary.

The runtime serves

The inference engine loads the model and exposes its native or OpenAI-compatible API.

Clients validate

Smoke tests and evaluation operations exercise the endpoint while Polyaxon retains the deployment context.

Prerequisites

  • A TensorRT-LLM container compatible with the model, GPU architecture, CUDA driver, and selected backend.
  • Enough cache disk and accelerator memory for the model and KV cache.
  • A separate engine-build operation when the selected deployment does not use the PyTorch backend directly.

Deploy and validate

  1. 1

    Define the TensorRT-LLM service

    Keep runtime flags and GPU topology in the Polyaxon component; treat any engine build as a versioned upstream artifact.

    version: 1.1kind: componentname: tensorrt-llm-serverrun:  kind: service  ports: [8000]  rewritePath: true  connections: [hf-token, model-cache]  container:    image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc7    command: ["trtllm-serve"]    args: ["TinyLlama/TinyLlama-1.1B-Chat-v1.0"]    resources:      limits:        nvidia.com/gpu: "1"

    This is NVIDIA's documented quick-start model and published runtime image. Pin the image digest and choose a larger model only after checking its build and hardware requirements.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f tensorrt-llm.yaml
  3. 3

    Validate the served model

    Probe model discovery and then compare latency, throughput, memory, and quality against the exact runtime revision.

    SERVICE_URL=$(polyaxon ops service --external --url)curl "$SERVICE_URL/v1/models"

Production checklist

Pin the runtime image and model revision; warm the model cache before measuring startup or latency.
Keep the service private by default and add authentication, TLS, rate limits, and network policy deliberately.
Define readiness, liveness, timeout, graceful shutdown, and rollback behavior before production traffic.
Measure latency distributions, throughput, quality, errors, and accelerator memory under representative load.
When using serialized engines, record their build inputs and compatibility alongside the engine artifact; do not treat an engine as portable across unmatched GPUs or software stacks.

Troubleshooting

The service never becomes ready

Inspect model download, credentials, disk space, GPU memory, runtime flags, port binding, and readiness behavior.

Requests fail through the URL

Verify the service port, rewrite-path setting, authentication header, ingress, network policy, and API path.

Latency degrades under load

Measure queueing, batch settings, context length, KV cache pressure, replica saturation, and storage or network contention.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore LLMOps