Turn production traces into regression tests

Production traces contain the cases that matter most: real user intent, changing context, tool behavior, latency, cost, and the exact path that produced an outcome. Yet many teams inspect a failure, patch the prompt or code, and move on without preserving the evidence.

Turning a production trace into a regression test makes the fix durable. The test reproduces the important conditions, defines the expected behavior, and runs against future models, prompts, retrieval configurations, tools, and application releases.

What is a trace-derived regression test?

A trace-derived regression test is a versioned evaluation case created from a production interaction or workflow. It preserves the minimum context needed to reproduce a confirmed behavior while removing or protecting sensitive and unstable details.

For an LLM application, the case may include the user request, conversation context, retrieved evidence, prompt variables, model configuration, output, and expected quality criteria.

For an agent, it may also include tool schemas, initial state, simulated tool results, expected actions, prohibited actions, and the final environment state.

The objective is not to replay every production request verbatim. It is to convert a meaningful failure into a controlled test with a clear reason for existing.

Why production traces improve evaluation coverage

Designed test sets reflect what the team expects users and systems to do. Production exposes what they actually do.

Trace-derived cases add:

• Natural phrasing, ambiguity, and multi-turn context.
• Real combinations of tasks and constraints.
• Tool, retrieval, and provider failure behavior.
• Long-tail inputs that synthetic generation may miss.
• Interactions between application versions and live data.
• Examples connected to user feedback or business outcomes.
• Concrete evidence behind latency and cost regressions.

These cases help keep offline evaluation representative as the application, users, and external dependencies change.

Start with an observable production path

The trace needs enough lineage to support attribution and reproduction. Record:

• Trace, session, task, and application-release identifiers.
• Model provider, model, and inference parameters.
• Prompt or message template and version.
• Retrieval query, index version, ranked document identifiers, and selected context.
• Tool definitions, sanitized arguments, results, errors, and retries.
• Agent, router, policy, and state-schema versions.
• Evaluator results, user feedback, and task outcome.
• End-to-end latency, tokens, cost, and important span timing.

AI agent tracing provides the causal structure. Without version and state context, a trace can show that a failure happened without making it reproducible.

Decide which traces should become tests

Not every trace deserves a permanent case. Prioritize evidence with durable learning value.

Confirmed quality or safety failures

Promote unsupported answers, incorrect actions, permission violations, bad citations, missed escalations, or other behavior the team has verified as unacceptable.

High-impact incidents

Preserve rare cases with a large user, operational, financial, or compliance consequence even when traffic volume is low.

New task types and segments

Add production behavior that is not represented in the existing dataset, including new languages, tools, document sources, workflows, or user groups.

Evaluator disagreements

Cases where an automated evaluator conflicts with a knowledgeable reviewer are useful for improving both the application and the evaluator calibration set.

Expensive or slow trajectories

Promote traces that complete successfully but use unnecessary model calls, large context, repeated tools, or avoidable retries.

Repeated failure clusters

When many traces share the same underlying defect, select representative cases rather than storing every duplicate. Keep frequency as metadata so priority is not lost.

Convert a trace into a safe, stable test

1. Confirm and categorize the failure

Have an owner verify what failed and where it originated. Assign a category such as retrieval recall, groundedness, tool selection, argument construction, handoff context, policy, loop, latency, or cost.

Do not create a regression test from negative feedback until the expected behavior is understood.

2. Minimize the case

Remove steps and context that do not influence the failure. A smaller case is easier to understand, cheaper to run, and less likely to become brittle.

Minimization should preserve the causal condition. If the failure depends on a long conversation or a specific sequence of tool results, retain that structure.

3. Sanitize sensitive data

Remove or replace personal data, credentials, customer content, internal identifiers, and confidential tool results. Use realistic synthetic substitutes when the value's shape matters.

Record the sanitization method and restrict access to any original trace. The test dataset should have its own retention and permission policy.

4. Stabilize external dependencies

Snapshot required documents, use a versioned test index, or reference immutable fixtures. Replace state-changing tools with controlled test implementations or recorded responses where appropriate.

Keep contract behavior realistic. A mock that always succeeds cannot reproduce a retry or partial-failure bug.

5. Define expected and prohibited behavior

Specify the outcome rather than one exact string or reasoning path unless exact behavior is required.

An agent case might define:

• Required tool or state transition.
• Allowed alternative trajectories.
• Prohibited actions.
• Required clarification or escalation.
• Final response requirements.
• Latency, call-count, and cost limits.

6. Attach complementary evaluators

Use deterministic assertions for state, schemas, tool arguments, permissions, citations, and budgets. Use calibrated model-based or human evaluation for semantic properties.

7. Record provenance

Store the source incident or trace identifier, date added, owner, failure category, severity, affected release, and fix that motivated the case. Preserve the relationship without copying sensitive content into broadly accessible metadata.

Define a regression-case schema

A consistent schema makes cases portable across local testing, CI, scheduled evaluation, and release review.

Keep mutable review notes separate from the executable case definition where possible.

Avoid brittle regression tests

Generative systems can produce several valid responses and trajectories. Exact string matching or one required chain of reasoning often creates false failures.

Prefer:

• Structured claims or required facts instead of exact prose.
• Verified final state instead of a confirmation sentence.
• Allowed and prohibited tool sets instead of one exact sequence.
• Semantic rubrics calibrated with examples.
• Range or percentile budgets instead of a single observed latency.
• Invariants that remain meaningful across model providers.

Use exact matching for identifiers, structured fields, policy text, tool arguments, and other properties where variation is not acceptable.

Example: a failed refund-agent trace

A customer asks whether a delayed order is eligible for a refund. The agent retrieves the correct policy, calls the refund tool, and reports success. The user later discovers that no refund occurred.

Trace review shows that the tool returned pending_approval, but the agent interpreted every non-error response as completion.

The regression case contains:

• A sanitized customer request and order state.
• The relevant policy fixture.
• A refund tool that returns pending_approval.
• An assertion that the agent must not claim completion.
• An expected escalation or status explanation.
• A check that no duplicate refund call occurs.
• A task-level latency and call-count budget.

The fix updates the tool-result handling and prompt instructions. Future changes must pass the case even if the response wording or safe trajectory differs.

Work through a minimized approval case

Consider a synthetic cluster of 12 failures attributed to the same result-handling defect. The number is illustrative, not a production measurement. Keep a cluster review record separate from the executable fixture:

{
  "cluster_id": "synthetic-approval-cluster-01",
  "source_kind": "synthetic teaching example",
  "occurrence_count": 12,
  "representative_trace_ids": ["synthetic-trace-03", "synthetic-trace-09"],
  "failure_category": "pending_status_reported_as_complete",
  "review_status": "accepted_for_example",
  "owner": "example-application-team",
  "case_id": "refund-pending-approval"
}

Twelve observed occurrences would still produce one representative fixture here. Preserve the count for prioritization, but do not count twelve independent test cases or infer a production failure rate without its traffic denominator.

Keep the state that causes the defect

Save this synthetic case as approval-case.json. It targets the application code that interprets a tool result, so it retains the tool response and required outcome. The real case's review record should identify which source spans justify those expectations.

{
  "case_id": "refund-pending-approval",
  "case_version": 1,
  "source_cluster": "synthetic-approval-cluster-01",
  "initial_state": {"applied_refunds": 0},
  "tool_response": {
    "status": "pending_approval",
    "request_id": "synthetic-request-01"
  },
  "expected": {
    "refund_completed": false,
    "next_action": "await_approval",
    "refund_requests": 1,
    "applied_refunds": 0
  }
}

Customer names, order addresses, and payment credentials are unnecessary for this handler-level defect. The pending_approval status is essential. Replacing it with a generic successful response would remove the condition the regression needs to preserve.

The fixture fixes the tool result and state. It does not reproduce the whole agent: a separate end-to-end case should verify tool selection and confirm that the user-facing response reflects the structured outcome correctly.

Compare a faulty handler with a corrected handler

Save the following as approval-regression.py beside the JSON file. It uses only the Python standard library, makes no provider or tool-network calls, and never issues a real refund. The fake tool records a request and returns the fixture; the application handler decides what it can report.

import json
from copy import deepcopy
from pathlib import Path


def faulty_handler(request_refund):
    result = request_refund()
    completed = result["status"] != "error"
    return {
        "refund_completed": completed,
        "next_action": "done" if completed else "review_error",
    }


def corrected_handler(request_refund):
    result = request_refund()
    status = result["status"]
    return {
        "refund_completed": status == "completed",
        "next_action": {
            "completed": "done",
            "pending_approval": "await_approval",
        }.get(status, "review_error"),
    }


def evaluate(case, handler):
    if case["tool_response"]["status"] != "pending_approval":
        raise ValueError("This fixture models only pending approval")
    state = deepcopy(case["initial_state"])
    calls = []

    def request_refund():
        calls.append("request_refund")
        # Pending approval records a request; no refund is applied.
        return deepcopy(case["tool_response"])

    outcome = handler(request_refund)
    expected = case["expected"]
    checks = {
        "completion_claim": (
            outcome.get("refund_completed") is expected["refund_completed"]
        ),
        "next_action": outcome.get("next_action") == expected["next_action"],
        "request_count": len(calls) == expected["refund_requests"],
        "refund_state": state["applied_refunds"] == expected["applied_refunds"],
    }
    return {
        "outcome": outcome,
        "calls": calls,
        "final_state": state,
        "checks": checks,
        "passed": all(checks.values()),
    }


case = json.loads(Path("approval-case.json").read_text(encoding="utf-8"))
report = {
    "case_id": case["case_id"],
    "case_version": case["case_version"],
    "source_cluster": case["source_cluster"],
    "evaluator_version": "pending-approval-v1",
    "results": {
        "baseline": evaluate(case, faulty_handler),
        "candidate": evaluate(case, corrected_handler),
    },
}
Path("approval-report.json").write_text(
    json.dumps(report, indent=2) + "\n", encoding="utf-8"
)
print(json.dumps(report, indent=2))

By inspection, the faulty handler should fail the completion-claim and next-action checks. The corrected handler should pass all four checks. Those are expected outcomes of this synthetic example, not results from an executed benchmark. The state check verifies the fixture's simulated environment, not an external payment system.

Review the individual checks, not just the aggregate result. A handler that says it will wait but requests the refund twice must still fail. A handler that returns a safe response for this fixture may still fail other tool statuses, so add completed, error, and unknown-status cases separately when qualifying the real application.

Build a production-to-evaluation workflow

1. Detect a quality, safety, reliability, latency, or cost signal.
2. Query traces from the affected release and segment.
3. Group similar failures and choose representative examples.
4. Assign an owner to confirm cause and expected behavior.
5. Sanitize and minimize the case.
6. Create stable fixtures and evaluators.
7. Run the case against the failing version to confirm reproduction.
8. Run it against the candidate fix and broader evaluation suite.
9. Add the case to the appropriate release gate.
10. Monitor production for the same failure category after release.

The failing version should fail the new test. Otherwise, the case or evaluator may not preserve the defect.

Manage the regression suite over time

Organize by task and risk

Use tags for failure category, task, language, source, tool, user segment, severity, and policy domain. This supports targeted evaluation and segmented reporting.

Track duplicates and clusters

Maintain the production frequency and impact associated with a representative case. A deduplicated suite should not make a common failure look rare.

Review stale cases

Products, policies, tools, and acceptable behavior change. Periodically verify that the expected outcome remains valid and that fixtures still model production realistically.

Separate release blockers from diagnostics

Critical safety and policy cases may block every release. Broader quality cases may use regression budgets. Expensive simulations can run on a schedule rather than for every small change.

Measure suite effectiveness

Track how many production incidents correspond to an existing case, how often tests catch a regression before release, evaluator disagreement, runtime and cost, and the age of unreviewed production candidates.

Common trace-to-test mistakes

Copying raw production content into the dataset

This creates privacy and retention risk. Sanitize, minimize, and control access before promotion.

Preserving the symptom without the cause

A final response alone may not reproduce a retrieval, tool, or state failure. Keep the relevant trace context and fixtures.

Requiring one exact agent path

Allow safe alternative trajectories unless the sequence itself is a requirement.

Adding cases without ownership

Unreviewed trace dumps create a large, low-trust suite. Require a failure definition, expected behavior, and responsible owner.

Never retiring or revising cases

Stale policy and tool assumptions can produce false regressions. Version changes and document why expectations changed.

Tracking pass rate without production impact

Connect cases with the frequency and severity of the failures they represent. Prioritize evaluation work using user impact, not test count.

Keep the reviewed case and results in Polyaxon

Run the evaluator as a Polyaxon job with a versioned case file and application revision. Mapping operations can expand an explicit list of case and candidate combinations, while DAGs can connect evaluation, report aggregation, and release review. Your evaluator owns the assertions and acceptance rules; Polyaxon schedules the workload and retains its evidence.

For this teaching example, append the following block to approval-regression.py when running it inside a Polyaxon job with the tracking library, authentication, and artifact storage configured. It uses the case and report produced above. The standard-library example remains usable without this optional integration.

from polyaxon import tracking
from polyaxon.schemas import V1ArtifactKind

tracking.init()
tracking.log_outputs(
    case_id=case["case_id"],
    case_version=case["case_version"],
    source_cluster=case["source_cluster"],
    evaluator_version=report["evaluator_version"],
)
tracking.log_metrics(
    baseline_passed=int(report["results"]["baseline"]["passed"]),
    candidate_passed=int(report["results"]["candidate"]["passed"]),
)
tracking.log_artifact(
    name="reviewed-case", path="approval-case.json", kind=V1ArtifactKind.FILE
)
tracking.log_artifact(
    name="approval-regression-report",
    path="approval-report.json",
    kind=V1ArtifactKind.FILE,
)

This records two fixture outcomes, not a population pass rate. A successful evaluator job means that the evaluator finished; a release decision must inspect its checks. In a real suite, retain the expected-case manifest and reconcile failed or missing evaluations before aggregation. Metadata logging and artifact logging provide the run records; keep application, case, and evaluator revisions distinct when comparing candidates.

After review, a manual approval can hold the downstream release operation until an owner accepts the evidence. Keep sensitive source traces in their original access-controlled system and retain only permitted provenance references with the sanitized fixture. Native integrated LLM observability and evaluation remain described as active development on the Polyaxon roadmap; this workflow uses the existing job, tracking, and orchestration capabilities.

Make every failure teach the system

A production incident that ends with a one-time patch is likely to return in another form. A trace-derived regression test converts the team's investigation into reusable evidence.

Capture the path, minimize and sanitize the case, define the expected outcome, and attach the right evaluators. Use offline and online evaluation to keep the release and production loops connected. Over time, the suite becomes a practical memory of production behavior and a guardrail for faster, safer change.