How to evaluate LLM routers for cost, quality, and latency

An LLM router chooses a model for each request using signals such as task type, predicted difficulty, latency, availability, and cost. The routing rule can reduce spend or improve responsiveness, but only if the selected model still completes the task at the required quality.

Evaluating the candidate models independently is not enough. The router is a decision system: it classifies traffic, applies a policy, invokes a model, may fall back, and returns an outcome. The complete path needs a versioned test and an inspectable decision record.

Write the routing contract per task

Start with task classes that reflect the application, not generic model rankings. Structured extraction, short factual answers, code repair, and multi-step tool use can have different acceptable models and failure costs.

For each class, define:

• Required output schema and task-success check.
• Semantic quality dimensions and minimum acceptable values.
• Actions or policy violations that fail the case regardless of quality.
• End-to-end latency and cost budgets.
• Models and providers permitted for the data involved.
• Conditions for fallback, escalation, or refusal.

Treat the quality requirement as a gate rather than an average that cost can offset. A cheap route that violates a required JSON schema or chooses an unauthorized tool has failed even if its prose score is high.

The RouteLLM paper demonstrates learning a routing decision between stronger and weaker models from preference data. Application teams still need to evaluate any router against their own tasks, prompts, tools, and traffic distribution.

Record every routing decision

Keep enough data to reproduce and segment the choice without retaining sensitive prompt content unnecessarily:

{
  "request_id": "req-8e12",
  "task_class": "support-summary",
  "router": {
    "version": "router-12",
    "policy": "quality-gated-cost-v4",
    "decision": "balanced-model",
    "confidence": 0.82
  },
  "served": {
    "provider": "provider-a",
    "model": "model-release-2026-07",
    "fallback_attempt": 0
  },
  "result": {
    "latency_ms": 1840,
    "input_tokens": 912,
    "output_tokens": 146,
    "estimated_cost_usd": 0.0041
  }
}

Use provider-reported usage when available and document the pricing snapshot used for estimates. Record the model that actually served the response; the router's intended destination and a provider alias are not sufficient when fallbacks or provider-side changes occur.

Store the router version, features available at decision time, selected route, score or confidence where meaningful, fallback chain, final model, prompt and application versions, and evaluation outcome. Redact or hash user identifiers according to your data policy.

Evaluate models and policies in two layers

First, run every eligible model against the same versioned case set. This creates a model-by-case result matrix with task success, quality dimensions, latency, token usage, and estimated cost. Preserve failures and timeouts rather than dropping them from averages.

Second, replay each proposed routing policy over that matrix. For a deterministic policy, select the corresponding model result for each case. For a learned router, evaluate its predictions without letting the test labels leak into its training data. Execute an end-to-end sample as well, because a matrix simulation will not reveal live provider errors, serialization differences, or fallback latency.

Compare the proposed policy with useful baselines:

• Always use the least expensive eligible model.
• Always use the strongest approved model.
• Route by a simple task-class rule.
• Use the current production policy.
• Use an oracle that selects the least expensive passing result after seeing all outcomes.

The oracle is not deployable, but the gap to it shows how much opportunity the policy leaves on the table. Report the gap by task class so high-volume easy cases do not hide weak routing on difficult or high-risk requests.

Measure cost per successful task

Total model spend can fall while completed work becomes more expensive. Use quality and cost together:

Keep the component metrics even if a release gate uses a composite. A single score cannot explain whether a regression came from route classification, model quality, provider reliability, or evaluator drift. The LLM cost monitoring guide describes task-level cost attribution in more detail.

Build a router-specific dataset

Sample the real mix of task classes, then add cases around the decision boundary. Include short and long contexts, ambiguous classification, multiple languages, tool-dependent tasks, strict output schemas, unavailable providers, rate limits, and cases that require a high-quality model even when the input appears simple.

Keep train, calibration, and test cases separate for learned routers. Version the task taxonomy and route labels. When the set of eligible models changes, preserve the old result matrix and create a new comparison rather than silently overwriting the baseline.

Evaluate semantic criteria with deterministic rules where possible and calibrated judges where necessary. Track evaluator agreement and cost. If every routed response is graded by another expensive model, include that evaluation expense in the system comparison. See LLM-as-a-judge for calibration and disagreement checks.

Test fallback behavior as part of the policy

Fallbacks change both quality and cost. Test timeouts, rate limits, malformed outputs, content-policy refusals, context limits, and partial streaming failures. Specify which conditions may retry the same model, choose another model, or stop.

Use idempotent test tools or a simulator when routed agents can take actions. A fallback must not repeat a completed purchase, ticket, or data update. Preserve the first attempt and its outcome even when the final response succeeds.

Set a total task budget for attempts, latency, and cost. Otherwise each individual model call can remain within its limit while the fallback chain exceeds the user's expectation.

Run the comparison as a reproducible workflow

Create one containerized evaluator that accepts the dataset revision, application version, model candidate, and routing policy as inputs. Run candidate models in parallel, aggregate the per-case results, simulate policies, and publish a comparison report.

Polyaxon matrix and DAG operations can coordinate those jobs on Kubernetes. Track model and policy identifiers as run inputs, log per-case scores and operating metrics, and preserve raw responses, route decisions, and reports as controlled artifacts. The batch LLM evaluation guide covers stable shards, bounded concurrency, and complete aggregation.

Polyaxon provides the experiment and evidence workflow; the application or routing layer continues to make live route decisions. Keep provider credentials in managed connections and isolate evaluation outputs that contain protected prompts or responses.

Monitor each route after release

Release with a shadow comparison or a bounded canary where practical. Shadow candidates must not execute real tool actions or expose data to an unapproved provider. For a canary, define rollback conditions for task success, policy violations, latency, cost per success, and fallback rate before sending traffic.

In production, segment by router version, task class, intended route, actual model, provider, fallback path, prompt version, and application release. Compare route distribution with the evaluated distribution. A change in traffic mix can move aggregate quality even when every model behaves the same.

Sample outcomes for evaluation and retain a stable control group when policy allows. Feed reviewed failures back into the versioned case set. When a provider changes a model behind an alias, rerun qualification or route to a pinned approved version where the provider supports it.

A router earns its savings when the complete system keeps passing the task contract. The decision log, result matrix, and production segments make that claim reviewable instead of assuming the least expensive response was good enough.