Polyaxon v3 is coming →
SGLang

Serve models with SGLang on Polyaxon

SGLang owns model loading, batching, parallelism, cache behavior, and the OpenAI-compatible API. Polyaxon owns placement, secrets, service lifecycle, access, logs, and the connection to evaluation workloads.

Outcome

A tracked SGLang service with a fixed model revision, explicit GPU request, model cache connection, and testable endpoint.

Start the guideServing documentation

Service responsibilities

Polyaxon schedules

The service component fixes the image, model, port, accelerator request, connections, and health boundary.

The runtime serves

The inference engine loads the model and exposes its native or OpenAI-compatible API.

Clients validate

Smoke tests and evaluation operations exercise the endpoint while Polyaxon retains the deployment context.

Prerequisites

  • A validated SGLang image compatible with the selected model and accelerator.
  • A model cache and a scoped Hugging Face token connection for gated weights.
  • Enough aggregate device memory for the chosen precision, context length, and parallelism.

Deploy and validate

  1. 1

    Define the SGLang service

    Expose the SGLang port through the Polyaxon service runtime and keep runtime flags in the component revision.

    version: 1.1kind: componentname: sglang-serverrun:  kind: service  ports: [30000]  rewritePath: true  connections: [hf-token, model-cache]  container:    image: lmsysorg/sglang:v0.5.18    command: ["python3", "-m", "sglang.launch_server"]    args: ["--model-path", "Qwen/Qwen3.6-27B", "--host", "0.0.0.0", "--port", "30000", "--tp", "4", "--reasoning-parser", "qwen3"]    resources:      limits:        nvidia.com/gpu: "4"

    SGLang publishes this release tag. Pin its digest and confirm model and GPU compatibility before production.

  2. 2

    Submit the operation

    Run the component through the target Polyaxon project, queue, preset, and approval path.

    polyaxon run -f sglang.yaml
  3. 3

    Call the OpenAI-compatible endpoint

    Resolve the Polyaxon service URL and send a small request before running quality or load evaluations.

    SERVICE_URL=$(polyaxon ops service --external --url)curl "$SERVICE_URL/v1/models"

Production checklist

Pin the runtime image and model revision; warm the model cache before measuring startup or latency.
Keep the service private by default and add authentication, TLS, rate limits, and network policy deliberately.
Define readiness, liveness, timeout, graceful shutdown, and rollback behavior before production traffic.
Measure latency distributions, throughput, quality, errors, and accelerator memory under representative load.
For prefill/decode disaggregation, operate the router and worker groups as a tested multi-service topology rather than extending this single-pod starter blindly.

Troubleshooting

The service never becomes ready

Inspect model download, credentials, disk space, GPU memory, runtime flags, port binding, and readiness behavior.

Requests fail through the URL

Verify the service port, rewrite-path setting, authentication header, ingress, network policy, and API path.

Latency degrades under load

Measure queueing, batch settings, context length, KV cache pressure, replica saturation, and storage or network contention.

Sources

Official platform, library, model, and Polyaxon references used by this guide.

Continue

Compare AI platformsExplore LLMOps