Stop expensive agent evaluation sweeps early

An agent evaluation sweep can consume substantial resources after it has stopped producing useful information. A broken dependency may cause every candidate to fail, or one candidate may already meet the experiment's predefined acceptance criterion.

Polyaxon early-stopping rules can terminate a workflow based on failures or recorded metrics. Use them deliberately: stopping the whole search is different from pruning one weak candidate, and neither is the same as a command timeout.

Choose the stopping scope

The early-stopping reference documents the group-level rules and available policies. A metric threshold without a policy can stop running and pending candidates; it does not merely skip the next suggestion.

Stop a systematically failing sweep

Add a failure rule to a matrix when continuing after widespread execution failures has little value:

matrix:
  kind: grid
  concurrency: 2
  params:
    strategy:
      kind: choice
      value: [baseline, retrieval-first, verify-first, compact]
  earlyStopping:
    - kind: failure_early_stopping
      percent: 50

This is a matrix fragment for an operation whose component declares the strategy input. The failure percentage is evaluated by the scheduler; do not treat it as a guarantee that exactly half the candidates will finish before stopping.

Define failure semantics in the harness. A missing dataset or invalid provider configuration should be observable as an execution failure. A valid candidate with a low quality score may instead be a successful experiment that produced an unsuitable result.

Conflating those outcomes can stop a useful search simply because several candidates were not good enough.

Publish acceptance only after the required checks

For a “find any acceptable candidate” search, a trusted evaluator can publish a binary metric only after processing the complete required evaluation set:

from polyaxon import tracking

def record_acceptance(
    completed_cases,
    expected_cases,
    quality_score,
    policy_violations,
):
    if expected_cases <= 0 or completed_cases != expected_cases:
        raise ValueError("Evaluation coverage is incomplete")

    accepted = quality_score >= 0.90 and policy_violations == 0
    tracking.log_metrics(release_ready=int(accepted))
    return accepted

Call this function inside your configured tracking run with actual measurements. The threshold is an illustrative application policy, not a universal quality standard.

The corresponding matrix rule is:

earlyStopping:
  - kind: metric_early_stopping
    metric: release_ready
    value: 1
    optimization: maximize

Place this fragment under matrix, alongside its params and concurrency fields. Because no pruning policy is specified, reaching the target ends the search group. It is appropriate when the objective is to find an acceptable candidate, not to rank every candidate exhaustively.

Do not publish the acceptance metric after an easy prefix of the dataset. Incomplete evaluation can otherwise terminate the search on misleading evidence.

Prune only comparable intermediate results

Median, difference, and truncation policies are useful when intermediate measurements are comparable. For agent evaluations, that requires consistent case ordering or equivalent evaluation batches and a clear treatment of missing results.

A candidate that has processed ten easy cases should not be compared casually with one that has processed a hundred difficult cases. If you cannot establish a meaningful intermediate comparison, prefer complete-candidate results or an application-owned stopping decision.

Keep the policy and evaluation revision in the experiment record. Run comparisons should distinguish completed, pruned, failed, and stopped candidates.

Preserve useful work before termination

Write progress and per-case evidence incrementally through the artifact workflow. A stopped candidate may not execute its ordinary final-report path.

Bound external calls and make side effects idempotent where needed. A scheduler stop does not undo a provider request or external write already performed.

Use early stopping to save work that the experiment no longer needs. Preserve enough evidence to explain why the search ended and which conclusions remain valid.