Serve DeepSeek V4 on Polyaxon

DeepSeek V4 deployments depend on more than the model name. The checkpoint format, GPU architecture, serving kernels, and parallelism settings must agree. A configuration for one GPU family is not automatically portable to another.

This article uses the original DeepSeek-V4-Pro checkpoint with SGLang on an eight-GPU B200 node. The SGLang DeepSeek V4 cookbook documents the hardware-specific recipes and distinguishes this checkpoint from the later Pro-0813 release.

Prepare the deployment

• Connect the target Kubernetes cluster through a Polyaxon Agent (compute cluster).
• Select a queue or preset that schedules onto your B200 nodes using their actual labels and taints. A request for eight NVIDIA GPUs does not select a GPU model.
• Configure a writable model-cache connection at /mnt/model-cache, with enough disk for the checkpoint and retained revisions.
• Check the model's license, access requirements, image compatibility, host memory, and shared-memory capacity before reserving the node.

For another accelerator or checkpoint variant, start with its own cookbook recipe. Do not copy B200 flags into a Hopper, Grace Blackwell, or ROCm deployment and assume the kernels will match.

Configure a B200 service

Save this as deepseek-v4.yaml. It uses the published SGLang 0.5.18 release, which includes DeepSeek V4 support. Pin its digest and the model revision for repeatable deployment.

version: 1.1
kind: component
name: deepseek-v4-service
plugins:
  shm: true
run:
  kind: service
  ports: [30000]
  rewritePath: true
  connections: [model-cache]
  container:
    image: lmsysorg/sglang:v0.5.18
    command: ["sglang", "serve"]
    env:
    - name: HF_HOME
      value: /mnt/model-cache
    args:
    - "--model-path"
    - "deepseek-ai/DeepSeek-V4-Pro"
    - "--tp"
    - "8"
    - "--tool-call-parser"
    - "deepseekv4"
    - "--reasoning-parser"
    - "deepseek-v4"
    - "--host"
    - "0.0.0.0"
    - "--port"
    - "30000"
    resources:
      limits:
        nvidia.com/gpu: "8"

This establishes a tensor-parallel service configuration; it does not claim a throughput target. Set CPU and memory requests in the component or preset, and select a context and concurrency budget for your application. If you add data-parallel attention, expert parallelism, or speculative decoding, take the complete compatible recipe from the cookbook rather than changing one flag in isolation.

Launch and inspect the service

polyaxon run -f deepseek-v4.yaml
polyaxon ops dashboard

Check model download, kernel initialization, and memory allocation in the logs before sending requests. For a Polyaxon-authenticated service, export an authorized token in POLYAXON_TOKEN and inspect the model endpoint:

: "${POLYAXON_TOKEN:?Export an authorized Polyaxon token first}"
DEEPSEEK_SERVICE_URL=$(polyaxon ops service --external --url)
curl --fail-with-body "$DEEPSEEK_SERVICE_URL/v1/models" \
  --header "Authorization: token $POLYAXON_TOKEN"

Use your gateway's authentication scheme if it differs. See the SGLang integration for chat requests. Add a scoped secret connection if the selected checkpoint needs download credentials.

Check reasoning and tool-call behavior

Evaluate ordinary chat, reasoning responses, and structured tool calls separately. Retain the exact requests, sampling settings, expected response fields, and results. Parser changes can break a client even when the generated text looks reasonable.

Measure cold startup independently from request latency after warmup. Increase concurrency using representative prompt lengths and record memory headroom, errors, and tail latency. Do not treat the first successful response as evidence that the deployment can carry production traffic.

If loading fails, inspect checkpoint format, cache space, image compatibility, and the actual GPU SKU. If responses change after an update, compare model revision, tokenizer, parser settings, precision, and sampling before attributing the change to infrastructure.

Use repeatable inference benchmarks to compare configurations. Stop the temporary service after evaluation:

polyaxon ops stop