Separate evaluator changes from application improvements

Your evaluation pass rate rises after a release. The application changed, but so did the judge prompt. How much of the increase reflects better answers, and how much reflects a more permissive evaluator?

Comparing yesterday's score with today's cannot resolve that question. You need to score the same frozen outputs with both evaluator versions, then compare application versions under a fixed evaluator. Polyaxon can organize these runs and retain their inputs, decisions, and summary metrics so the explanation survives beyond a dashboard screenshot.

This is a diagnostic workflow for an evaluator migration. For rubric design, human calibration, and judge bias, start with LLM-as-a-judge. Here, the goal is narrower: explain a score change before using it to justify an application release.

Freeze the evidence before rescoring

Create one versioned case manifest and two output bundles: one generated by the old application and one by the candidate. Each bundle must cover the same case IDs. Preserve the context the evaluator actually sees, including retrieved passages, tool results, references, or policy text.

Do not regenerate answers separately for each evaluator. That would add generation variability to the comparison. If repeated application samples are part of the study, generate them first and give every sample a stable identity that both evaluators receive.

Record enough information to identify each comparison:

An evaluator version includes more than its model. A parser that changes how malformed responses are handled or a new acceptance threshold can move the score without changing the judge's text. A mutable provider alias also limits reproducibility; record the resolved version when available and the evaluation time.

Run all four comparisons

Call the old and new applications A0 and A1, and the old and new evaluators E0 and E1. Score both frozen output bundles under both evaluators:

Read down a column to compare applications under one evaluator. Read across a row to inspect how an evaluator revision changes its assessment of the same outputs.

Suppose a synthetic teaching example has 100 cases with valid decisions in every comparison:

The diagonal comparison, 70% to 86%, suggests a 16-percentage-point increase. But A1 improves over A0 by 8 points under E0 and only 1 point under E1. E1 raises the old application's score by 15 points and the new application's score by 8 points.

The evaluator revision changes the measured application advantage. The difference between those advantages is 1 - 8 = -7 percentage points. That is an interaction in this comparison, not proof that either evaluator is correct. These invented values illustrate the arithmetic; they are not a benchmark or an observed Polyaxon result.

The two useful paths through the table are 8 + 8 = 16 and 15 + 1 = 16. Neither supports a unique statement that a fixed share of the diagonal increase is "real quality." The interpretation depends on the evaluator contract and human evidence.

Inspect the cases that changed decisions

Aggregate rates can hide opposite changes. For each frozen output bundle, count both E0-fail → E1-pass and E0-pass → E1-fail transitions, then inspect the case-level reasons.

The following standard-library helper compares one application's decisions under both evaluators. It requires exactly the manifest's IDs and refuses to score invalid or missing decisions. Save it as compare-evaluators.py; the inline data are a synthetic four-case example.

from collections import Counter
import json


def compare_evaluators(case_ids, old_rows, new_rows):
    expected = set(case_ids)
    if not expected or len(expected) != len(case_ids):
        raise ValueError("The manifest must contain unique case IDs")

    def index(rows):
        result = {}
        for row in rows:
            case_id = row["case_id"]
            if case_id in result or case_id not in expected:
                raise ValueError("Duplicate or unexpected case ID")
            if row.get("valid") is not True:
                raise ValueError("Resolve invalid judgments before comparing")
            if row.get("decision") not in ("pass", "fail"):
                raise ValueError("Expected a pass/fail decision")
            result[case_id] = row["decision"]
        if set(result) != expected:
            raise ValueError("Missing case decisions")
        return result

    old, new = index(old_rows), index(new_rows)
    transitions = Counter(f"{old[c]}_to_{new[c]}" for c in case_ids)
    return {
        "case_count": len(case_ids),
        "old_pass_rate": sum(old[c] == "pass" for c in case_ids) / len(case_ids),
        "new_pass_rate": sum(new[c] == "pass" for c in case_ids) / len(case_ids),
        "transitions": dict(transitions),
        "changed_case_ids": [c for c in case_ids if old[c] != new[c]],
    }


case_ids = ["c1", "c2", "c3", "c4"]
old_rows = [
    {"case_id": c, "valid": True, "decision": d}
    for c, d in zip(case_ids, ["pass", "pass", "fail", "fail"])
]
new_rows = [
    {"case_id": c, "valid": True, "decision": d}
    for c, d in zip(case_ids, ["pass", "fail", "pass", "pass"])
]
report = compare_evaluators(case_ids, old_rows, new_rows)
print(json.dumps(report, indent=2))

The expected rates are 50% and 75%, with two fail-to-pass transitions and one pass-to-fail transition. The net gain is one case, while three cases change labels. Run this comparison separately for A0 and A1 using actual reviewed results; do not mix their rows. In production, the loader must also verify dataset, application-output, and evaluator identities against the study manifest. The helper only checks coverage and decision shape.

Keep invalid judgments in a separate report with their counts and causes. Do not silently discard them to produce a cleaner percentage. Resolve them under a documented policy before claiming a complete paired comparison.

Organize the study in Polyaxon

Use a Polyaxon mapping to describe the four combinations. This fragment belongs in an operation whose evaluator component declares application_bundle and evaluator_revision inputs:

matrix:
  kind: mapping
  concurrency: 2
  values:
    - application_bundle: a0-frozen
      evaluator_revision: e0
    - application_bundle: a0-frozen
      evaluator_revision: e1
    - application_bundle: a1-frozen
      evaluator_revision: e0
    - application_bundle: a1-frozen
      evaluator_revision: e1

These are study labels, not predefined Polyaxon assets. Your component must resolve them to immutable files and evaluator configurations, read the same manifest, call your evaluator, validate its output, and write per-case judgments. Give the job the required read-only data connection and, for an external judge, its configured provider credentials. A shared evaluator component keeps that contract consistent across runs.

Record study_id, dataset_revision, application_revision, output_bundle_hash, and evaluator_revision as run inputs or outputs. Log pass_rate, case_count, and invalid_count as tracking metrics and metadata. Save the complete case-level decisions, reasons, and resolved evaluator configuration as artifacts.

An aggregation step should require all four combinations exactly once before building the comparison. A missing run is an incomplete study, not a zero score. Polyaxon provides the run records, orchestration, and artifacts; your evaluator and aggregation code define validity and the interpretation of the results.

Decide whether to adopt the evaluator

Review changed decisions against an independently labeled reference set, preferably with reviewers blinded to application and evaluator identity. Include important slices, such as policy questions, unsupported claims, and tool failures, rather than relying only on the average.

For model judges, repeat judgments where variability could change the decision. Keep the same frozen inputs, record attempts, and report agreement and uncertainty. Research on MT-Bench and Chatbot Arena documents position, verbosity, and self-enhancement biases; a stronger aggregate score does not remove the need to inspect those failure modes.

If E1 changes the rubric intentionally, document the new meaning of "pass" and reassess the release threshold. Preserve the old score series and publish an overlap comparison instead of joining E0 and E1 scores into one apparently continuous trend. If you tune E1 after inspecting these results, use fresh independent validation before treating the next comparison as confirmation.

The release review should answer two separate questions: is A1 acceptable under the approved evaluation contract, and is E1 a justified replacement for E0? Keeping both decisions explicit lets you improve the application and its measurement without confusing one for the other.