Build evaluation leaderboards with Polyaxon joins

A leaderboard is useful only when the candidates were measured under comparable conditions. Sorting every successful run by its highest score can mix datasets, evaluator versions, and incomplete experiments.

Polyaxon joins let a downstream operation query and collect values from upstream or parallel runs. Use them to assemble a well-defined comparison cohort, then let a reviewed reporting component produce the leaderboard.

Define eligibility before ranking

Choose a benchmark identifier that represents a fixed dataset, evaluator revision, and execution protocol. Tag the candidate runs with that identifier and record the constituent revisions as inputs.

Have each evaluator report whether it processed the required cases. A process can succeed while producing a low score; incomplete coverage should be a distinct outcome, not a candidate that happens to rank lower.

For example, the evaluator can record quality, latency_p95_ms, and completed_cases as outputs. Publish the final ranking score only after validating the report's completeness.

Use tracking metadata consistently across every candidate. A leaderboard cannot repair incompatible metric definitions after the runs finish.

Select a precise cohort with a join

The following fragment belongs at the operation level for an application-owned reporting component:

joins:
  - query: "project.name: agent-evals, tags: benchmark-v7, kind: job, status: succeeded, inputs.dataset_revision: questions-v2, inputs.evaluator_revision: grounded-answer-v3, metrics.quality: >=0"
    sort: "-metrics.quality"
    limit: 20
    params:
      run_ids:
        value: globals.uuid
      quality_scores:
        value: outputs.quality
      latency_values:
        value: outputs.latency_p95_ms

Replace the project, tag, and revisions with the intended cohort. The metrics. query prefix supports numerical comparisons, while joined values can come from the recorded outputs.

The report component must declare matching list inputs:

inputs:
  - name: run_ids
    type: str
    isList: true
  - name: quality_scores
    type: float
    isList: true
  - name: latency_values
    type: float
    isList: true

These are fragments, not a complete executable component. Your report image supplies the code that validates the lists and writes the result. See the join specification and query syntax for the underlying interfaces.

Coordinate timing separately from selection

A query selects runs that exist and match when it is evaluated. It does not wait until the whole experiment is complete.

Place the report after the evaluation group in the workflow, using the dependency and trigger behavior appropriate to the experiment. If failed candidates are expected, decide explicitly whether reporting should still run.

Capture the complete candidate inventory separately from the successful top-ranked subset. The example's limit of 20 is a presentation choice, not evidence that only 20 candidates were attempted.

A useful report states how many runs were submitted, completed, failed, stopped, and excluded, with reasons for exclusion.

Validate the joined result before publishing

Check that the lists have equal lengths, required values are present and finite, and run identifiers are unique. Treat an empty result as “no eligible candidates,” not as a successful blank leaderboard.

For important decisions, inspect the referenced reports to verify coverage and metric definitions. Do not infer completeness solely from a nonnegative quality score.

Use artifact tracking to persist the report, selected run identifiers, query, and generation timestamp. That freezes the comparison even if more matching runs appear later.

Separate rank from promotion

Show quality alongside latency, resource consumption, and relevant failure categories. Two candidates with similar scores may have very different operating costs or risks.

Use run comparisons for deeper inspection. A top-ranked candidate should still pass held-out evaluation and the team's release process.

Joins automate cohort assembly; they do not define the benchmark or approve a deployment. Keeping those responsibilities separate produces a leaderboard that is both convenient and defensible.