Polyaxon & SGLang

How to use Polyaxon and SGLang together

+

Polyaxon can schedule SGLang as a GPU-backed service and expose its OpenAI-compatible API through the service URL.

This guide uses the verified SGLang recipe for GLM-5.2 on H200 with FP8, low-latency mode, and a single node.

Overview

The linked SGLang recipe runs zai-org/GLM-5.2-FP8 with tensor parallelism across 8 H200 GPUs and enables EAGLE/MTP speculative decoding for lower latency.

Use this integration when you want to:

• run an OpenAI-compatible model endpoint on your own Kubernetes GPU nodes,
• keep model serving runs tracked and controlled by Polyaxon,
• expose the endpoint through the Polyaxon service abstraction.

Polyaxon component

The following component keeps the SGLang H200/FP8/low-latency flags intact and exposes the server on port 30000.

version: 1.1
kind: component
name: sglang-glm-5-2
tags: ["sglang", "glm", "llm-serving"]
run:
  kind: service
  ports: [30000]
  rewritePath: true
  container:
    image: lmsysorg/sglang:latest
    command: ["python3", "-m", "sglang.launch_server"]
    args:
      - "--model-path"
      - "zai-org/GLM-5.2-FP8"
      - "--tp"
      - "8"
      - "--speculative-algorithm"
      - "EAGLE"
      - "--speculative-num-steps"
      - "5"
      - "--speculative-eagle-topk"
      - "1"
      - "--speculative-num-draft-tokens"
      - "6"
      - "--mem-fraction-static"
      - "0.8"
      - "--cuda-graph-max-bs"
      - "32"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "30000"
    resources:
      limits:
        nvidia.com/gpu: "8"

If your cluster has dedicated H200 nodes, add an environment.nodeSelector or preset matching your H200 pool. GPU SKU labels are cluster-specific, so use the labels configured by your Kubernetes provider or platform team.

For production, use a model cache backed by local NVMe or a persistent volume. Repeatedly pulling a large checkpoint from shared storage or over the network can dominate service startup time.

GLM parsers

SGLang exposes optional parser flags for GLM reasoning and tool calls:

    args:
      - "--reasoning-parser"
      - "glm45"
      - "--tool-call-parser"
      - "glm47"

Add those flags to the component if your clients need structured reasoning or tool call fields. Otherwise, keep the base recipe unchanged.

Scheduling the service

Save the component as sglang-glm-5-2.yaml and run:

polyaxon run -f sglang-glm-5-2.yaml

To get the service URL:

polyaxon ops service --external --url

Calling the endpoint

SGLang exposes an OpenAI-compatible /v1/chat/completions endpoint:

curl HOST:PORT/v1/chat/completions \
  --request POST \
  --header "Content-Type: application/json" \
  --data '{
    "model": "zai-org/GLM-5.2-FP8",
    "messages": [
      {"role": "user", "content": "Hello"}
    ]
  }'

For Polyaxon EE or Polyaxon Cloud services protected by Polyaxon auth, include a valid token:

curl HOST:PORT/v1/chat/completions \
  --request POST \
  --header "Authorization: token AUTH_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "model": "zai-org/GLM-5.2-FP8",
    "messages": [
      {"role": "user", "content": "Hello"}
    ]
  }'

If you expose the service without Polyaxon auth or place it behind a compatible gateway, standard OpenAI-compatible clients can point their base_url to HOST:PORT/v1.

Tuning notes

The example is intentionally scoped to the SGLang H200/FP8/low-latency/single-node recipe:

• --tp 8 maps the model across 8 GPUs.
• --speculative-algorithm EAGLE with 5-1-6 is the low-latency MTP setting from the recipe.
• --mem-fraction-static 0.8 leaves memory headroom instead of trying to fill the GPU to the edge.
• --cuda-graph-max-bs 32 targets latency, not maximum throughput.

For higher throughput, use the SGLang recipe selector and change the Polyaxon component deliberately. The balanced and high-throughput recipes add different batching, DP-attention, and MoE communication choices.

Deployment checks

• Pin the runtime image and model revision; warm the model cache before measuring startup or latency.
• Keep the service private by default and add authentication, TLS, rate limits, and network policy deliberately.
• Define readiness, liveness, timeout, graceful shutdown, and rollback behavior before production traffic.
• Measure latency distributions, throughput, quality, errors, and accelerator memory under representative load.
• For prefill/decode disaggregation, operate the router and worker groups as a tested multi-service topology rather than extending this single-pod starter blindly.

Troubleshooting

The service never becomes ready

Inspect model download, credentials, disk space, GPU memory, runtime flags, port binding, and readiness behavior.

Requests fail through the URL

Verify the service port, rewrite-path setting, authentication header, ingress, network policy, and API path.

Latency degrades under load

Measure queueing, batch settings, context length, KV cache pressure, replica saturation, and storage or network contention.

References

• SGLang documentation — Current server, model, parallelism, and deployment behavior.
• Install SGLang with Docker — Published image names, release tags, and server command.
• Polyaxon service runtime — Service ports, replicas, connections, volumes, and external access.
• Polyaxon model serving — Patterns for deploying APIs, loading models, and operating inference workloads.