Serve a Jev-style Qwen classifier on Polyaxon

A support queue may need one of three labels—billing, technical, or other—rather than another paragraph of generated text. NobodyWho's Jev-in-25-lines experiment shows how a small open Qwen model can score candidate answers from next-token logits. It is a useful demonstration of the mechanism, and its author explicitly calls it a parody rather than a trained or calibrated Jev equivalent.

This tutorial takes the operational step the short experiment leaves open: run an open classifier server as a Polyaxon service, keep its model and server revision identifiable, and test its decisions against reviewed examples. We use Simple Jev's open Hugging Face server, which supports Qwen and exposes a typed /v1/classifier API. This is a Jev-style workflow, not the hosted Jev model or its API.

Choose the decision contract

The server accepts shared state plus one or more typed questions. A choice question names the allowed labels and returns their scores. The server constructs JSON from the scores; it does not ask Qwen to generate a JSON paragraph. Its API reference also supports ordinal score and yes/no noul questions.

That distinction matters when a downstream program needs a bounded answer. It does not make the probabilities calibrated or the label correct. Candidate wording, the prompt template, tokenizer, input length, and model version can all change the ranking. Keep an other or review path for requests that do not fit your label set.

We will serve the upstream project's documented Qwen/Qwen3.5-0.8B CPU example. A CPU deployment makes the first integration simple; select a GPU allocation only after evaluating its behavior and load.

Use a PyTorch runtime image

You can run this example without building or pushing a custom image. Use the official PyTorch runtime image, have Polyaxon fetch a reviewed Simple Jev commit, and install the server package when the service starts. The selected 2.14.0-cuda12.6-cudnn9-runtime tag includes Python 3.12, which Simple Jev requires. This example uses CPU even though the published runtime image includes CUDA libraries, so its image pull is larger than a purpose-built CPU image.

Replace YOUR_REVIEWED_FULL_COMMIT_SHA in the component below with a full upstream commit. Keep that commit, the PyTorch image digest, resolved package versions, and the model revision with the deployment record. A tag alone can move. Runtime pip installation needs package-index access and adds time to each start; the first model load also downloads weights unless the cache is prepopulated.

Run the classifier as a Polyaxon service

Create a model-cache connection that mounts persistent writable storage at /mnt/model-cache. Save this component as qwen-classifier.yaml. The memory and CPU values are starting allocations to evaluate, not measured capacity requirements.

version: 1.1
kind: component
name: qwen-typed-classifier
run:
  kind: service
  ports: [8000]
  rewritePath: true
  connections: [model-cache]
  init:
    - git:
        url: https://github.com/featherless-ai/simple-jev
        revision: YOUR_REVIEWED_FULL_COMMIT_SHA
  container:
    image: pytorch/pytorch:2.14.0-cuda12.6-cudnn9-runtime
    workingDir: "{{ globals.artifacts_path }}/simple-jev"
    command: ["sh", "-c"]
    args:
      - >-
        python -m pip install --break-system-packages --no-cache-dir -e './hf-server'
        && exec simple-jev
        --model Qwen/Qwen3.5-0.8B
        --device cpu
        --dtype float32
        --max-model-len 4096
        --max-batch-size 4
        --max-batch-tokens 4096
        --host 0.0.0.0
        --port 8000
    env:
      - name: HF_HOME
        value: /mnt/model-cache
    resources:
      requests:
        cpu: "4"
        memory: "12Gi"
      limits:
        cpu: "4"
        memory: "12Gi"

The PyTorch image is based on Ubuntu, whose Python installation is marked externally managed; pip's --break-system-packages option permits installation into this disposable container. Polyaxon clones the server before the main container starts; pip then installs the API. Schedule it on a queue with the requested CPU and memory:

polyaxon run -f qwen-classifier.yaml
polyaxon ops dashboard

Wait for model loading. The service URL belongs to the current operation:

CLASSIFIER_URL=$(polyaxon ops service --external --url)
: "${POLYAXON_TOKEN:?Export an authorized Polyaxon token first}"
curl --fail-with-body "$CLASSIFIER_URL/health" \
  --header "Authorization: token $POLYAXON_TOKEN"

The upstream server has no authentication of its own. Keep the endpoint behind your Polyaxon access controls or an approved gateway; do not expose a raw public Pod port. Use the authentication scheme required by your installation. A healthy response means the model loaded, not that it makes good decisions.

Send one typed request

Use a synthetic message first. The API model ID must match the one used to start the server:

curl --fail-with-body "$CLASSIFIER_URL/v1/classifier" \
  --header "Authorization: token $POLYAXON_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "model": "Qwen/Qwen3.5-0.8B",
    "state": "Customer says the latest invoice includes a duplicate charge.",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Which team should review this request?",
        "criteria": {
          "billing": "Invoices, charges, and refunds",
          "technical": "Product errors and outages",
          "other": "Requests that do not match either team"
        }
      }
    }
  }'

Inspect answers.route.choice and answers.route.probabilities in the response. No particular label or probability is promised for this example. The response shape comes from Simple Jev; Polyaxon schedules the service, controls resources and access, and keeps the operation's configuration.

Check the classifier before routing work

Build a small, reviewed evaluation set with examples for every label, ambiguous requests, out-of-scope messages, and input lengths near your chosen limit. Record the expected label and a human review rule. Then compare the chosen label, distribution, and latency across the same cases whenever the model, server commit, template, or label descriptions change. Polyaxon runs and artifacts can retain the case set and response files, while repeatable inference comparisons help separate behavior from serving performance.

Measure calibration on held-out cases before treating 0.9 as a 90% real-world chance. A softmax over allowed labels is conditional on those labels and can be confident when all of them are wrong. Human review should remain the default action until your own thresholds and failure handling are supported by evidence.

When the review is over, stop the selected service operation with polyaxon ops stop. If you want a native encoder decision model rather than scoring an instruction model's tokens, the companion Laya deployment guide uses the same API with a different backend.