Serve Qwen 3.6 on Polyaxon

A model server can start successfully and still be unsuitable for your application. A long context window can leave too little memory for concurrent requests; a missing parser can change the structure of reasoning or tool-call responses.

This article packages Qwen3.6-27B as a Polyaxon service using SGLang. Keep the model revision, serving image, context limit, and GPU allocation together when comparing deployments.

Prepare the GPU node and model cache

You need a GPU compute cluster connected to Polyaxon, an installed NVIDIA device plugin, and enough memory for the model and the selected request load. See the cluster connection instructions if the cluster is not connected yet.

The Qwen model card's SGLang example uses eight GPUs for a 262,144-token context. The configuration below follows that allocation; it is not a claim that every workload needs eight GPUs or that every eight-GPU node has enough memory. Use a queue and preset that select the hardware you evaluated.

Create a model-cache connection that mounts writable persistent storage at /mnt/model-cache. The component sets HF_HOME to that location so model downloads use the mount. Size its disk capacity for the checkpoint and any revisions you retain. Add a secret connection if your selected model requires credentials.

Configure the service

Save this as qwen-3-6.yaml. The example uses the published SGLang 0.5.18 release. Resolve the image to a digest and pin the model revision before a reproducibility-sensitive deployment.

version: 1.1
kind: component
name: qwen-3-6-service
plugins:
  shm: true
run:
  kind: service
  ports: [30000]
  rewritePath: true
  connections: [model-cache]
  container:
    image: lmsysorg/sglang:v0.5.18
    command: ["python3", "-m", "sglang.launch_server"]
    env:
    - name: HF_HOME
      value: /mnt/model-cache
    args:
    - "--model-path"
    - "Qwen/Qwen3.6-27B"
    - "--host"
    - "0.0.0.0"
    - "--port"
    - "30000"
    - "--tp-size"
    - "8"
    - "--mem-fraction-static"
    - "0.8"
    - "--context-length"
    - "262144"
    - "--reasoning-parser"
    - "qwen3"
    - "--tool-call-parser"
    - "qwen3_coder"
    resources:
      limits:
        nvidia.com/gpu: "8"

Set CPU and memory requests through your preset or the container resources. Shared memory, tokenizer work, and checkpoint loading also need host memory. For an AMD deployment, use a supported ROCm image and AMD resource requests; changing only nvidia.com/gpu to amd.com/gpu does not make a CUDA image compatible.

Submit and inspect the endpoint

polyaxon run -f qwen-3-6.yaml
polyaxon ops dashboard

Wait for model loading to finish, then inspect the API. For a service protected by Polyaxon authentication, export an authorized token in POLYAXON_TOKEN first:

: "${POLYAXON_TOKEN:?Export an authorized Polyaxon token first}"
QWEN_SERVICE_URL=$(polyaxon ops service --external --url)
curl --fail-with-body "$QWEN_SERVICE_URL/v1/models" \
  --header "Authorization: token $POLYAXON_TOKEN"

Use the authentication scheme required by your deployment if it sits behind a different gateway. The service integration covers chat-completion requests and access options. A model-list response confirms that the API responds, not that generation is correct.

Evaluate the behavior your application needs

Keep a small fixed request set with ordinary chat, long inputs, and any image or tool-call cases your application sends. Inspect reasoning and tool fields as structured API output, not just rendered text. Include invalid requests and requests near the chosen context limit.

Measure startup time separately from steady-state latency. Compare memory use and request failures at the intended concurrency, then reduce the context budget or adjust the hardware if necessary. The article on repeatable inference benchmarks explains how to keep those comparisons fair.

For load failures, inspect the image's model support, checkpoint access, cache capacity, and GPU memory. For malformed responses, compare the tokenizer, chat template, parser flags, and client request format. Keep these settings with the Polyaxon operation so a later image update is a deliberate comparison.

Stop an evaluation deployment when you no longer need it:

polyaxon ops stop