Build a shared library of agent evaluators

Teams often write the same evaluation plumbing repeatedly: load predictions, check a schema, calculate a score, save a report, and decide whether a release is acceptable.

A shared evaluator library can standardize that work without forcing every agent to use the same quality metric. Polyaxon's Component Hub provides versioned, reusable workload definitions; your team supplies the evaluator implementations and acceptance policy.

Standardize the contract, not every score

Define a common input envelope containing the candidate revision, dataset revision, evaluator revision, and prediction artifact reference. Include a benchmark identifier so results can be grouped safely.

Define outputs for coverage, quality, and a clear decision. Keep operational errors separate from a valid low-quality result.

A classification evaluator, citation checker, and tool-policy evaluator can share this envelope while computing different metrics. Avoid labeling an exact-match score as a general measure of answer quality.

Start with one deterministic evaluator

Clone the examples repository and open the evaluator directory:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/blog/agent-evaluators

Use the complete evaluate.py companion for a small ticket-routing evaluator. It performs case-insensitive exact matching after whitespace normalization, and checks coverage before scoring. Expected labels come from the trusted manifest.json; prediction files contain only the candidate revision, case IDs, and actual labels.

The evaluator requires every manifest ID exactly once, rejects unexpected IDs, and verifies the candidate revision. It saves one of three decisions: accepted, rejected, or invalid_input. Invalid or incomplete evidence receives no quality score. The core call is:

import json
from pathlib import Path

from evaluate import evaluate  # Use evaluate.py from the cloned example directory.

report, exit_code = evaluate(
    manifest=json.loads(Path("manifest.json").read_text(encoding="utf-8")),
    predictions=json.loads(Path("accepted.json").read_text(encoding="utf-8")),
    candidate_revision="agent-v12",
    minimum_score=0.90,
)
print(json.dumps(report, indent=2))
raise SystemExit(exit_code)

If you save the short calling example, use a separate file such as inspect-evaluation.py in this directory and run it from there; keep the repository's evaluate.py as the implementation. Include that implementation in your evaluator image at /app/evaluate.py. The image needs the Polyaxon Python package, and the job needs configured tracking and artifact collection. Its command-line interface saves the summary under the run's outputs path and logs candidate, dataset, and evaluator revisions.

The example intentionally makes a rejected candidate exit unsuccessfully. If your experiment distinguishes technical completion from quality rejection, return a successful process status and consume accepted explicitly in the promotion workflow instead.

Keep the manifest outside the candidate's writable workspace. Matching a candidate's own list of expected cases would not establish authoritative coverage. The evaluator checks the supplied manifest; your trusted workflow must select the correct dataset revision and protect its contents.

The directory also contains the accepted, rejected, incomplete, and duplicate prediction fixtures. The expected outcomes at a 0.90 threshold are:

These are expected fixture outcomes derived from the source, not reported execution results. To inspect one fixture locally without contacting Polyaxon, the companion accepts:

python evaluate.py --manifest manifest.json --predictions accepted.json --candidate-revision agent-v12 --minimum-score 0.90 --local-output summary.json

Register a reusable workload definition

Define a component that runs the evaluator image and declares its inputs. For example, save the following as evaluator.yaml:

version: 1.1
kind: component
name: exact-match-evaluator

inputs:
  - name: manifest_file
    type: str
  - name: predictions_file
    type: str
  - name: candidate_revision
    type: str
  - name: minimum_score
    type: float

run:
  kind: job
  connections: [evaluation-data]
  container:
    image: registry.example.com/team/evaluators:reviewed
    command: ["python", "/app/evaluate.py"]
    args:
      - "--manifest={{ manifest_file }}"
      - "--predictions={{ predictions_file }}"
      - "--candidate-revision={{ candidate_revision }}"
      - "--minimum-score={{ minimum_score }}"

Replace the example image with your built image, pinned to a reviewed digest. It must contain the repository's evaluator script at /app/evaluate.py and the Polyaxon Python package. This example assumes an administrator-configured evaluation-data connection mounts the manifest and prediction files read-only; supply paths inside that mount. Configure the connection and artifact collection for your installation before running the component.

Register that definition using the documented component-version workflow. For example:

polyaxon components register --version YOUR_ORG/exact-match-evaluator:v2 -f evaluator.yaml

The organization, component project, and evaluator.yaml are resources your team creates. They are not predefined public components.

Consumers can then save this operation as evaluate-candidate.yaml, replacing the organization and mounted paths with their configured values:

version: 1.1
kind: operation
name: evaluate-agent-v12
hubRef: YOUR_ORG/exact-match-evaluator:v2
params:
  manifest_file: {value: /evaluation-data/manifest.json}
  predictions_file: {value: /evaluation-data/accepted.json}
  candidate_revision: {value: agent-v12}
  minimum_score: {value: 0.90}
cache:
  disable: true

Submit it to an existing project with polyaxon run -p agent-evaluations -f evaluate-candidate.yaml. The operation uses registered evaluator code and mounted inputs; it does not generate predictions itself. Inspect evaluation/summary.json, the logged revisions, and decision before promoting a candidate. A process exit code alone loses the distinction between quality rejection and invalid evidence.

Version behavioral changes deliberately

Changing normalization, missing-value behavior, or the acceptance threshold can change the meaning of a score. Record those choices and create a new reviewed evaluator revision when the contract changes.

Do not assume a version tag is inherently immutable: the management interfaces allow privileged replacement. Establish a policy that preserves released versions, and record the underlying source and image digests.

Maintain fixtures for valid, malformed, incomplete, and duplicate inputs. Include examples that demonstrate the evaluator's limits, such as semantically equivalent answers that exact matching rejects.

Reuse the evaluator across workflows

The same component can participate in a candidate sweep, a nightly schedule, or an independent release check. Keep dataset and candidate revisions explicit in each use.

Preserve reports through artifact tracking and compare compatible results in run comparisons.

A useful evaluator library grows from small, well-defined checks. Component Hub makes them discoverable and reusable; consistent contracts and reviewed implementations make their results trustworthy.