Deploy Qwen3 Reranker on Polyaxon

A documentation search for “keep notebook files after the session ends” may return pages about restarting kernels, idle shutdown and persistent storage. They share words, but only some answer the question.

A reranker reads the query together with each candidate passage and scores how well the passage answers it. Use your existing search to find a shortlist, then rerank that shortlist before displaying results or passing context to an LLM.

This walkthrough deploys Qwen3-Reranker-0.6B as a private Polyaxon GPU service. A small client retrieves six passages with TF-IDF and asks the service to return three. You can replace that first stage with your existing lexical, embedding or hybrid search.

Keep retrieval and reranking separate

The first stage searches the corpus efficiently. An embedding retriever can compare a query vector against precomputed document vectors; a lexical retriever can use an inverted index. A reranker jointly processes the query and each candidate, so its work grows with the shortlist size and passage length. Scoring the whole corpus this way would be expensive.

The query therefore goes to both stages. The reranker needs the original question as well as the retrieved text. It can reorder candidates, but it cannot recover a relevant passage that the retriever omitted. This is the retrieve-and-rerank pattern.

Qwen supplies an Apache-2.0 text reranker with about 0.6 billion parameters. Its Sentence Transformers configuration includes the query/document template and scoring head. The example fixes the checkpoint revision and instruction so later comparisons can identify exactly what ran.

Create the reranking API

You need a Polyaxon project, an AMD64 NVIDIA GPU queue with a compatible CUDA driver, and access to Hugging Face and PyPI. The resource values below are a starting allocation; GPU memory use depends on sequence length and batch size.

Create a service/ directory. Save this as service/requirements.txt:

sentence-transformers==6.2.0
transformers==5.19.0
fastapi==0.119.1
pydantic==2.12.3
uvicorn==0.38.0

Save the following as service/app.py. The API accepts the query and candidate texts, then returns their original indexes in score order:

from threading import Lock
from typing import Annotated

import torch
from fastapi import FastAPI
from pydantic import BaseModel, Field
from sentence_transformers import CrossEncoder

MODEL_ID = "Qwen/Qwen3-Reranker-0.6B"
MODEL_REVISION = "e61197ed45024b0ed8a2d74b80b4d909f1255473"
INSTRUCTION = (
    "Given a technical documentation question, retrieve passages "
    "that directly answer the question."
)
MAX_LENGTH = 2048

model = CrossEncoder(
    MODEL_ID,
    revision=MODEL_REVISION,
    device="cuda",
    max_length=MAX_LENGTH,
    model_kwargs={"torch_dtype": torch.float16},
)
model.eval()
model_lock = Lock()
app = FastAPI()


class RerankRequest(BaseModel):
    query: str = Field(min_length=1, max_length=512)
    documents: list[Annotated[str, Field(min_length=1, max_length=4000)]] = Field(
        min_length=1, max_length=64
    )
    top_n: int = Field(default=3, ge=1, le=64)


@app.get("/health")
def health():
    return {"model": MODEL_ID, "revision": MODEL_REVISION}


@app.post("/rerank")
def rerank(request: RerankRequest):
    pairs = [(request.query, text) for text in request.documents]
    with model_lock:
        scores = model.predict(
            pairs,
            prompt=INSTRUCTION,
            batch_size=4,
            show_progress_bar=False,
        ).reshape(-1)
    order = sorted(
        range(len(scores)), key=lambda index: float(scores[index]), reverse=True
    )
    return {
        "model": MODEL_ID,
        "revision": MODEL_REVISION,
        "instruction": INSTRUCTION,
        "max_length": MAX_LENGTH,
        "results": [
            {"index": index, "score": float(scores[index])}
            for index in order[:request.top_n]
        ],
    }

The checkpoint's saved activation is an identity function, so these scores are raw logit differences. Use their order to rank passages; a score is not a calibrated probability that a passage is correct. The model card describes this scoring behavior and instruction handling.

The service limits each request to 64 passages and processes four pairs at a time. Pair tokenization is capped at 2,048 tokens, independently of the character limits on the request. Long passages can be truncated; split documents into passages upstream and evaluate the length limit with your data.

Schedule the service on a GPU

Save this as reranker.yaml, beside service/:

name: qwen3-reranker
service:
  ports: [8000]
  rewritePath: true
container:
  image: pytorch/pytorch:2.14.0-cuda12.6-cudnn9-runtime@sha256:a77983eb7a3042ccf41a94da547a7f54a47fb70da15c36d67a8e8a329ebaa098
  workingDir: "{{ globals.run_artifacts_path }}/uploads"
  readinessProbe:
    httpGet:
      path: /health
      port: 8000
    periodSeconds: 10
resources:
  cpu: "2..4"
  memory: "8Gi..16Gi"
  gpu: 1
env:
  HF_HOME: /tmp/huggingface
cmd: |
  python -m pip install --break-system-packages --no-cache-dir -r requirements.txt
  exec python -m uvicorn app:app --host 0.0.0.0 --port 8000 --workers 1

This uses the official PyTorch runtime and installs the API dependencies at startup. One Uvicorn worker loads one model copy. The lock serializes inference calls; this small service does not combine requests into dynamic batches. Retain a resolved dependency lock or prebuild the environment when repeating a deployment.

From the directory containing reranker.yaml, submit it. Replace the project and queue with your own:

polyaxon run -p acme/search -q inference/gpu \
  -f reranker.yaml -u -u-from service

The upload flags place the contents of service/ in the run's uploads directory. The queue selects the configured compute destination, while gpu: 1 requests a device. If you use the installation's default agent, omit -q.

Keep the returned run UUID. Inspect the startup logs and obtain the service URL:

export SERVICE_RUN_UUID=YOUR_RUN_UUID
polyaxon ops logs -p acme/search -uid "$SERVICE_RUN_UUID"
polyaxon ops service -p acme/search -uid "$SERVICE_RUN_UUID" --external --url

The readiness check succeeds after the model loads. Set RERANK_URL to the printed URL and use an authorized POLYAXON_TOKEN:

export RERANK_URL=YOUR_SERVICE_URL
: "${POLYAXON_TOKEN:?Export an authorized Polyaxon token first}"
curl --fail-with-body "$RERANK_URL/health" \
  --header "Authorization: token $POLYAXON_TOKEN"

The service access controls remain enabled; --external --url prints a URL without making it public. A new container downloads the checkpoint into its local cache. To reuse downloaded weights, follow the shared Hugging Face cache guide.

Rerank a search shortlist

On your machine, install the client dependencies:

python -m pip install requests scikit-learn

Save this as search.py. The eight short passages are teaching examples. TF-IDF supplies a first-stage lexical ranking; the service scores its top six candidates and selects three.

import os
from time import perf_counter

import requests
from sklearn.feature_extraction.text import TfidfVectorizer

documents = [
    {"id": "persistent-workspace", "text": "Mount a persistent volume claim at the notebook workspace path. Files on that volume remain available after the notebook service stops, subject to the volume's retention policy."},
    {"id": "restart-kernel", "text": "Restarting a notebook kernel clears Python variables and releases process memory. The notebook service can remain running while a new kernel starts."},
    {"id": "temporary-storage", "text": "An emptyDir volume belongs to a Pod. Its files are removed when that Pod is deleted. Use it for temporary work rather than files you need after the Pod is gone."},
    {"id": "idle-shutdown", "text": "An idle notebook service can be stopped to release compute. Save work to persistent storage before the service is stopped."},
    {"id": "model-registry", "text": "A registered model version links a selected model package to its producing training run and the artifacts chosen during registration."},
    {"id": "gpu-queue", "text": "Choose a GPU queue and request a GPU when a notebook session needs accelerator access. Queue configuration controls where the workload can run."},
    {"id": "notebook-image", "text": "Select a notebook container image with the packages needed for your session. Changing the image changes the environment used when the service starts."},
    {"id": "download-files", "text": "Download selected files from a run's Files view when you need a local copy. A downloaded copy and a mounted persistent workspace serve different workflows."},
]
query = "How do I keep notebook files after the session ends?"

vectorizer = TfidfVectorizer()
matrix = vectorizer.fit_transform([doc["text"] for doc in documents])
retrieval_scores = (vectorizer.transform([query]) @ matrix.T).toarray()[0]
candidate_indexes = retrieval_scores.argsort()[::-1][:6]
candidates = [documents[int(index)] for index in candidate_indexes]
print("Baseline top three:", [doc["id"] for doc in candidates[:3]])

started = perf_counter()
response = requests.post(
    os.environ["RERANK_URL"].rstrip("/") + "/rerank",
    headers={"Authorization": "token " + os.environ["POLYAXON_TOKEN"]},
    json={"query": query, "documents": [doc["text"] for doc in candidates], "top_n": 3},
    timeout=120,
)
response.raise_for_status()
elapsed_ms = (perf_counter() - started) * 1000

for rank, result in enumerate(response.json()["results"], start=1):
    document = candidates[result["index"]]
    print(rank, document["id"], f"score={result['score']:.3f}")
print(f"Rerank HTTP round trip: {elapsed_ms:.1f} ms")

Run it with the URL and token set above:

python search.py

The returned index refers to the submitted candidate list. Map it through candidates, as the code does, to recover the document ID. Mapping it directly into the full corpus would return the wrong documents after retrieval reordered or filtered them.

The printed latency covers the HTTP request, server processing and any queueing. It excludes the first-stage retrieval performed before the timer and is not an isolated GPU measurement. Compare the printed baseline and reranked document IDs; this tiny example demonstrates the connection, rather than establishing a quality improvement.

For an embedding-based first stage, the EmbeddingGemma search walkthrough provides a deployment example. This Qwen service scores text pairs: supply textual passages when connecting the two.

Decide whether the extra stage helps

Use a held-out set of realistic queries with relevance judgments fixed before inspecting the reranker's scores. Keep the corpus snapshot, retrieved candidates and final result count the same when comparing the first-stage order with the reranked order. For binary relevance, MRR@3 averages the reciprocal rank of the first relevant result, using zero when none appears in the first three. Use nDCG at the same cutoff when your judgments have graded relevance. Inspect queries where relevant passages moved down as well as the aggregate change.

Measure candidate recall separately: what fraction of the judged relevant passages reached the shortlist? If the useful passage is missing, increase or improve first-stage retrieval before expecting reranking to help. When comparing candidate counts such as 10, 20 and 50, record both ranking quality and the additional work each setting creates.

After warmup, measure p50 and p95 request latency under representative concurrency, passage lengths and batch sizes. Count timeouts and failed requests alongside successful timings. Keep startup/model-loading time separate from warm serving latency, and include the whole search path when checking the application's latency target.

Run those comparisons as Polyaxon evaluation jobs. Record the corpus and query-set revisions, model revision, instruction, candidate count, result count, maximum length and precision with each run. Save per-query rankings and timings as artifacts, log aggregate quality/latency metrics, and use run comparison to review the tradeoff. Polyaxon retains the experiment evidence; your relevance judgments and evaluation code define the quality measure.

When you finish trying the service, stop it to release the GPU:

polyaxon ops stop -p acme/search -uid "$SERVICE_RUN_UUID"