Run batch LLM evaluations on Kubernetes

Batch LLM evaluation on Kubernetes works best when the evaluation dataset, execution units, and result identities are explicit. Split work into reproducible shards, bound the load on the model endpoint, and accept the final report only after every required result is accounted for.

The model does not have to run inside every evaluation worker. Workers calling a remote endpoint usually need CPU, memory, and network access. A self-hosted model server may need a separate GPU allocation. Keeping those roles distinct helps avoid reserving a GPU for a process that only sends HTTP requests.

Freeze the evaluation manifest

Before starting jobs, record the dataset revision, case IDs, prompt and evaluator versions, model identifier, generation settings, and repetition count. Include retrieval or tool-fixture revisions when a case depends on them.

Define a result identity such as (dataset revision, case ID, candidate, repetition). Transport retries for that identity should not become additional independent samples. If the application intentionally samples the model again, record a new attempt with its own identity.

A minimal application-specific manifest might look like this:

{
  "dataset_revision": "support-eval-v3",
  "candidate": "support-app-r17",
  "case_ids": ["summary-001", "refund-002", "retrieval-003"],
  "repetitions": 2,
  "shards": 2,
  "evaluator_revision": "rubric-v4"
}

This example expects six logical results. It is not a native Polyaxon configuration; your evaluation program reads and enforces the manifest.

Use stable shard assignment

For an immutable ordered dataset, assigning row index modulo shard count is simple and reproducible. If you need assignment to remain stable when rows are reordered, use a stable hash of the case ID. Do not use a language's process-randomized hash as a persistent partition function.

This self-contained Python example produces the same assignment across processes:

import hashlib

def shard_for(case_id: str, shard_count: int) -> int:
    if shard_count < 1:
        raise ValueError("shard_count must be positive")
    digest = hashlib.sha256(case_id.encode("utf-8")).digest()
    return int.from_bytes(digest[:8], "big") % shard_count

case_ids = ["summary-001", "refund-002", "retrieval-003"]
for case_id in case_ids:
    print(case_id, shard_for(case_id, 2))

Hashing balances counts approximately, not execution cost. Long contexts, multi-turn agents, and expensive judges can make one shard much slower. For predictable completion time, build an explicit shard manifest using estimated case cost, then version that manifest too.

Bound total endpoint load

Total concurrency is approximately worker count multiplied by in-worker concurrency, plus any separate judging or retrieval traffic. Ten workers each allowing eight requests can create roughly 80 simultaneous target calls before those extra stages are counted.

Set a global request and token budget based on your endpoint's limits. Use bounded retries with backoff for transient failures, honor provider retry signals, and stop retrying permanent configuration or authorization errors. Preserve the error category and number of transport attempts.

For a self-hosted model, measure tokens per second, latency, queueing, and memory at representative input lengths. Increasing evaluator replicas beyond model capacity usually adds queueing rather than useful throughput. For GPU placement and sharing decisions, see GPU orchestration and MIG versus time-slicing.

Make workers safe to restart

Write results incrementally to a shard-specific location. Each row should contain its logical identity, outcome, error state, timings, and evaluator revision. Commit only complete records; an interrupted partial write should not look like a finished case.

Kubernetes Jobs can replace or retry Pods, so evaluation code must tolerate re-execution. A worker that resumes should inspect completed logical identities rather than simply appending duplicate rows. For tools with side effects, use an isolated test service and an explicit idempotency strategy.

Keep endpoint errors separate from evaluation failures. Retrying an HTTP timeout is different from rerunning a completed answer until it happens to pass. The latter changes the experiment and can bias the final score.

Aggregate only complete, compatible results

The aggregation step should verify that every expected identity appears exactly once in the accepted result set. Reject missing shards, conflicting duplicates, mismatched dataset or evaluator versions, and malformed rows.

Combine counts using their denominators. Suppose one shard passes 9 of 10 cases and another passes 10 of 20. The combined rate is 19/30, about 63.3%, not the unweighted average of 90% and 50%. Also report performance by scenario category so a large easy category does not hide a consequential regression.

If partial reporting is useful operationally, label it as partial and show completed versus expected counts. Do not let the partial report authorize a release that requires the full suite.

Orchestrate execution and review in Polyaxon

Use components for the evaluation worker and aggregation program, then connect them with pipelines. Pass shard identity and manifest revision as inputs. Store each shard's rows and the final report as artifacts.

Separate CPU-based evaluation workers from GPU model-serving workloads in resource requests and queues. Record image digests and endpoint revisions alongside results. A scheduled evaluation should identify what changed since its baseline rather than reporting only a new aggregate score.

The Promptfoo tutorial provides a small tested runner before you add sharding. Use continuous red teaming for security release gates, and the LLM evaluation learning path for choosing the right quality checks.