Keep failed candidates visible in pipeline reports

A pipeline evaluates three model candidates. One finishes, one fails, and one never produces a usable result. A report that loads only successful evaluations shows a tidy comparison with a single row. It has also lost the experiment's original question.

The missing candidates belong in the report. Start with the inventory of work you intended to evaluate, then attach the evidence you actually collected. This preserves the difference between a complete comparison, an execution failure, and an unresolved gap.

Reliable pipeline orchestration establishes the dependency and failure-handling principles. Here, we will make the completeness rule concrete with an expected-candidate manifest, a small Python reconciliation function, and a reporting step that can run after failed evaluations.

Freeze the expected comparison

Write the candidate inventory before launching the evaluations. Give the comparison an immutable identifier tied to its candidate artifacts, dataset revision, evaluator revision, required cases, and evaluation protocol. A later change to any of those inputs creates a new comparison revision.

This abbreviated manifest shows the fields used by the reconciliation example:

{
  "comparison_id": "comparison-2026-09-19-01",
  "candidate_ids": ["baseline", "candidate-a", "candidate-b"]
}

In the full manifest, bind each candidate ID to the exact model artifact and configuration being evaluated. Preserve the manifest with the pipeline's artifacts, independently of the evaluation workers. A worker that never starts must not be able to erase its candidate from the inventory.

A query for successful runs can help build a ranking, but it cannot define this inventory. Neither can a list of files that happen to exist. Both describe observed work, which may be smaller than the work you intended. The leaderboard guide explains where joins help assemble results and where completeness remains the reporting application's responsibility.

Collect outcomes independently of worker success

Have a collector reconcile submitted run IDs, execution status, and readable evaluation artifacts against the manifest. A failed process may leave useful logs without a final report; a killed process may leave nothing. Use orchestration records as well as worker-written files.

The collector emits one normalized receipt per selected candidate attempt. Use an explicit state:

A valid report with a poor quality score is still complete. It may fail qualification later. Likewise, a successful process exit is insufficient to mark a receipt complete if required cases or artifacts are missing.

These are application-defined receipt states, not a replacement for Polyaxon run statuses. The collector must validate the report before issuing complete: check its comparison and candidate identity, expected case IDs without duplicates, metric definitions, and finite metric values. Preserve the report's immutable location or digest and the execution record with the receipt. The function below reconciles those validated receipts; it does not download or validate model reports itself.

Use bounded collection time and retain retrieval errors. An API timeout should become unavailable evidence or a failed reporting operation, never an empty list that is presented as a complete comparison.

Keep one row for every expected candidate

This standard-library helper preserves missing candidates and rejects ambiguous duplicate receipts. It assumes receipts came from the collector described above. Save it as reconcile.py if you want to adapt the example:

from collections import Counter


def reconcile(expected, receipts):
    comparison_id = expected["comparison_id"]
    candidate_ids = expected["candidate_ids"]
    if not isinstance(comparison_id, str) or not comparison_id:
        raise ValueError("A comparison ID is required")
    if (
        not isinstance(candidate_ids, list)
        or not candidate_ids
        or any(not isinstance(cid, str) or not cid for cid in candidate_ids)
        or len(set(candidate_ids)) != len(candidate_ids)
    ):
        raise ValueError("Expected candidate IDs must be nonempty and unique")

    grouped = {cid: [] for cid in candidate_ids}
    unassigned = []
    for receipt in receipts:
        cid = receipt.get("candidate_id") if isinstance(receipt, dict) else None
        if not isinstance(cid, str) or cid not in grouped:
            unassigned.append(receipt)
        else:
            grouped[cid].append(receipt)

    allowed = {"complete", "failed", "stopped", "skipped", "unavailable", "invalid"}
    rows = []
    for cid, matches in grouped.items():
        state = "missing"
        if len(matches) > 1:
            state = "duplicate"
        elif matches:
            receipt = matches[0]
            attempt_id = receipt.get("attempt_id")
            reported_state = receipt.get("state")
            valid = (
                receipt.get("comparison_id") == comparison_id
                and isinstance(attempt_id, str)
                and bool(attempt_id)
                and isinstance(reported_state, str)
                and reported_state in allowed
            )
            state = reported_state if valid else "invalid"
        rows.append({"candidate_id": cid, "state": state, "receipts": matches})

    return {
        "comparison_id": comparison_id,
        "expected_count": len(candidate_ids),
        "counts": dict(Counter(row["state"] for row in rows)),
        "evidence_complete": not unassigned and all(
            row["state"] == "complete" for row in rows
        ),
        "rows": rows,
        "unassigned_receipts": unassigned,
    }

An unknown candidate or malformed receipt remains in unassigned_receipts and prevents completeness. A receipt for the wrong comparison becomes invalid. If two receipts name the same expected candidate, the function keeps both and marks the row duplicate rather than silently overwriting one.

An invalid expected manifest raises an error because there is no trustworthy inventory to reconcile. Preserve that input and error as reporting diagnostics; do not substitute an empty successful report.

The following fixture demonstrates the intended failure case. These records are illustrative, not results from a model evaluation:

import json
from pathlib import Path

from reconcile import reconcile

expected = {
    "comparison_id": "comparison-2026-09-19-01",
    "candidate_ids": ["baseline", "candidate-a", "candidate-b"],
}
receipts = [
    {
        "comparison_id": expected["comparison_id"],
        "candidate_id": "baseline",
        "attempt_id": "baseline-attempt-1",
        "state": "complete",
    },
    {
        "comparison_id": expected["comparison_id"],
        "candidate_id": "candidate-a",
        "attempt_id": "candidate-a-attempt-1",
        "state": "failed",
        "reason": "Evaluator exited before producing a report",
    },
]

report = reconcile(expected, receipts)
Path("comparison-report.json").write_text(
    json.dumps(report, indent=2, allow_nan=False), encoding="utf-8"
)

By construction, the report retains three rows: baseline is complete, candidate-a failed, and candidate-b is missing. evidence_complete is false. Missing means no receipt was collected; it does not diagnose whether the candidate was never submitted, remained pending, or lost its artifact. Keep that investigation attached to the row.

For retries, preserve the full attempt history and define which attempt is authoritative before producing this input. Do not choose the attempt with the highest score merely because several exist. A failed first attempt and a successful retry can both matter when reviewing reliability and cost.

Let reporting run after evaluation failures

In Polyaxon, separate evidence collection from qualification. This fragment appends three operations to an existing DAG whose evaluation operations are named baseline, candidate-a, and candidate-b:

- name: report
  pathRef: ./report.yaml
  dependencies: [baseline, candidate-a, candidate-b]
  trigger: all_done
  skipOnUpstreamSkip: false

- name: qualify
  pathRef: ./qualify.yaml
  dependencies: [report]
  trigger: all_succeeded

- name: release
  pathRef: ./release.yaml
  dependencies: [qualify]
  trigger: all_succeeded

This is dependency wiring, not a complete runnable pipeline. Supply the three component definitions, their images and code, and explicit inputs identifying the comparison manifest and upstream runs. The report component collects receipts and saves the reconciliation; qualification reads that exact report and its manifest; release consumes the qualified candidate identity. File and artifact handoff must be configured in those components.

The dependency rules let the report wait until its upstream operations reach final states with all_done. Setting skipOnUpstreamSkip: false permits reporting when an upstream evaluation was skipped. Avoid making the reporter require an output parameter that a failed or skipped evaluation never produced; collect through known run identities and explicit artifact availability checks instead.

This trigger does not make the reporter immune to pipeline cancellation or early stopping. If those actions terminate pending work, use a separately submitted reconciliation run to inspect the preserved manifest and available execution evidence.

Save the manifest, receipts, and comparison report under the reporting run's outputs directory using the documented artifact workflow. Confirm that the report is readable before passing it to qualification. Reference-only lineage logging records a pointer; the application still needs to save the referenced file.

Treat a saved report and an approved release separately

The reporting operation can succeed after producing an honest report that says evidence is incomplete. Its success means the collection and reconciliation work finished. It does not approve a model.

Under a strict policy, qualification rejects the report unless every expected candidate has complete evidence and there are no unassigned or duplicate records. It then applies the declared quality, latency, resource, and other acceptance criteria. Complete evidence can still produce a decision to release nothing.

If the experiment intentionally permits a subset, define that rule before execution: required baseline, minimum eligible candidates, allowed exclusions, and the way exclusions appear in the decision record. Preserve all expected rows even when the policy accepts fewer candidates. The helper above implements the strict completeness rule; a subset policy requires an explicit additional decision, not deleting failed rows until the flag becomes true.

Start with a report that always names the intended candidates. That gives the next reviewer enough information to distinguish a weaker model, a broken evaluation, and an experiment that has not yet answered its question.