Find retrieval regressions before changing the prompt

Your RAG assistant starts answering a familiar question incorrectly after an index update. Rewriting the final prompt is tempting, but first find out what evidence the generator actually received. A relevant passage might never have been retrieved, might have fallen below the rerank cutoff, or might have disappeared when the context was trimmed to fit the token budget.

The practical investigation has two parts: trace the evidence through each stage, then replay frozen context with a fixed generator. Polyaxon can organize those controlled runs and retain the intermediate artifacts so a failed answer leads to a reproducible diagnosis.

This complements RAG evaluation and retrieval tuning with experiment matrices. Those establish the metrics and search space. Here, the goal is to find where an observed regression enters the pipeline before starting another tuning sweep.

Capture the boundary between each stage

Save more than the final answer and its citations. For the affected cases, retain the submitted question, rewritten query, access filters, index revision, initial candidates, reranked list, and exact context passed to generation. Include passage text or an immutable reference to it, source revisions, ordering, and truncation boundaries.

Inspect authorization and freshness alongside relevance. If a source was correctly excluded by access policy, bypassing the filter is not a retrieval fix. The correct answer may be an abstention. If the corpus lacks the needed source, adding more retrieved results cannot recover it.

For changing chunk layouts, compare stable evidence spans or reviewed acceptable alternatives rather than assuming old chunk IDs still exist. A document ID match is only a starting point: the retrieved chunk must contain the fact needed for this question.

Follow one piece of missing evidence

Consider a fictional documentation assistant answering: "Can I renew an expired export link, or do I need to create a new export?" The approved source says expired links cannot be renewed and a new export is required. Its decisive passage is labeled export-links-v3:expiry.

These two synthetic traces suggest different investigations:

Neither trace justifies blaming the generator for missing a fact it never received. Conversely, an answer that contradicts the decisive passage when that passage is present needs a generation-focused investigation. Ragas research separates context retrieval, faithful use of context, and response quality; preserving stage boundaries makes that separation actionable for a specific incident.

The following small helper identifies the earliest recorded stage without a required evidence span. Save it as locate-evidence-loss.py. It uses trusted diagnostic labels, not semantic matching or a live retrieval system.

import json


def locate_loss(required_ids, stages):
    if not required_ids or not stages:
        raise ValueError("Provide required evidence and ordered stage snapshots")
    required = set(required_ids)
    for stage_name, evidence_ids in stages:
        missing = sorted(required - set(evidence_ids))
        if missing:
            return {"first_missing_stage": stage_name, "missing_ids": missing}
    return {"first_missing_stage": None, "missing_ids": []}


required = ["export-links-v3:expiry"]
traces = {
    "rerank-loss": [
        ("retrieved", ["export-links-v3:expiry", "export-format-v2"]),
        ("reranked_top5", ["export-format-v2"]),
        ("assembled_context", ["export-format-v2"]),
    ],
    "assembly-loss": [
        ("retrieved", ["export-links-v3:expiry", "export-format-v2"]),
        ("reranked_top5", ["export-links-v3:expiry", "export-format-v2"]),
        ("assembled_context", ["export-format-v2"]),
    ],
}
for trace_id, stages in traces.items():
    print(json.dumps({"trace_id": trace_id, **locate_loss(required, stages)}))

The expected first missing stages are reranked_top5 and assembled_context. No benchmark is implied, and the example has not been executed. In real traces, label the assembled stage from text that actually survives serialization and truncation. Carrying an ID beside a truncated-away passage would give false reassurance.

This helper requires all listed evidence spans. For questions with alternative sufficient passages, represent acceptable evidence groups in your evaluator instead. If all required IDs survive but the answer fails, inspect the actual text, contradictions, ordering, and generation; ID coverage alone cannot establish correctness.

Replay context while holding generation fixed

Build three context bundles for the same question and conversation history:

Use the same generation prompt, model version, settings, and answer evaluator across these replays. Do not rerun retrieval during this step. Preserve document wrappers, citation identifiers, and serialization conventions; otherwise the context intervention also changes the interface presented to the generator.

If the current context fails while the previous and reviewed contexts succeed, that supports investigating the context change. It does not identify a particular embedding or reranker defect on its own. Combine replay outcomes with the stage snapshots.

If all three fail, confirm that the reviewed evidence is actually sufficient and the evaluator is applying the intended contract before changing the prompt or model. If all three succeed, the original failure may depend on nondeterminism, a missing conversational condition, or a dependency that changed since capture.

Repeat model calls where variability could affect the conclusion, keeping each attempt identifiable. The reviewed context is a diagnostic intervention, not a realistic estimate of deployed retrieval quality. Do not present its success rate as the application's production score.

Run the replay study in Polyaxon

Use a mapping to run the three context bundles through the same replay component. The fragment below belongs in an operation whose component declares context_bundle as a string input:

cache:
  disable: true

matrix:
  kind: mapping
  concurrency: 2
  values:
    - context_bundle: previous
    - context_bundle: current
    - context_bundle: reviewed

Your team supplies the replay component; these bundle names are study labels, not predefined Polyaxon assets. The component must load an immutable case manifest, resolve the selected saved context, invoke the fixed generator, evaluate the response, and write case-level outputs. Put provider credentials in an appropriate configured connection, and limit provider concurrency inside each worker as well as matrix concurrency.

Record study_id, case_set_revision, context_bundle_hash, generator_revision, prompt_revision, and evaluator_revision as run inputs or outputs. Save the context snapshots, stage traces, answers, and evaluation records using artifact tracking. Log evidence coverage and answer correctness separately through tracking metadata and metrics.

Disabling operation caching makes each submitted replay execute. If a model provider or application has its own response cache, control and record that separately. For repeated trials, give each trial a distinct identity and retain the same context bundle rather than rebuilding it.

Before comparing, confirm that every selected case has a valid result in every required bundle. Keep missing or failed calls visible. Run comparisons help inspect recorded differences; they do not decide whether two runs used the same evidence contract.

Fix the responsible stage, then confirm end to end

Choose the smallest change supported by the evidence. If retrieval misses an indexed exact identifier, inspect query handling and candidate generation. If reranking removes the useful passage, investigate ranking features and cutoffs. If assembly discards it, inspect token accounting, deduplication, truncation, and passage ordering.

Restore the live retrieval path after the diagnostic replay and evaluate the corrected configuration end to end. Confirm on held-out cases, including abstention, freshness, permissions, and multi-passage questions. Improving one known incident is not sufficient evidence for changing the default configuration.

Keep the failing stage trace and replay bundles as regression artifacts. The next index or prompt update should be able to answer two concrete questions: did the necessary evidence reach the generator, and did the generator use it correctly?