Schedule agent evaluations and backfill missed days

An agent regression suite should run often enough to catch changes in models, data, and external tools. When a scheduled run is missed, the team also needs a deliberate way to evaluate the missing period.

Polyaxon schedules and date-range matrices cover these two execution patterns. Your evaluation application defines the data window and quality checks; the platform organizes when the work runs and how its progress is tracked.

Define what a nightly evaluation means

Choose whether the nightly run uses a fixed regression set, the previous completed day's production samples, or both. These answer different questions.

The fixed set reveals changes in behavior under stable inputs. A recent sample reveals how the application handles the current workload. Keep their results separate so changing traffic does not look like a model regression.

Record the dataset, candidate, evaluator, and resolved window with every run. If the application selects the previous completed day, define that calculation in reviewed code with an explicit timezone and late-arrival policy.

Schedule the reviewed component

This operation assumes your team has registered YOUR_ORG/nightly-agent-evaluation:v1. Its implementation resolves the completed data window and declares the string inputs shown.

version: 1.1
kind: operation
name: nightly-agent-evaluation
hubRef: YOUR_ORG/nightly-agent-evaluation:v1

params:
  candidate_revision:
    value: agent-v12
  evaluator_revision:
    value: acceptance-v4

cache:
  disable: true

schedule:
  kind: cron
  cron: "0 2 * * *"
  dependsOnPast: true
  maxRuns: 30

The cron expression selects 02:00 in the scheduler's configured time basis; verify that configuration rather than assuming it matches a user's laptop timezone.

dependsOnPast: true makes the schedule wait for the preceding execution to finish before scheduling another. It does not prove the preceding result met the quality threshold. maxRuns bounds this example to a limited rollout.

See cron schedules for the supported fields. Disable operation caching when the purpose is to obtain a fresh measurement.

Backfill explicit historical windows

Use a date-range matrix for missed periods rather than changing the nightly component to silently process an arbitrary backlog:

version: 1.1
kind: operation
name: agent-evaluation-backfill
hubRef: YOUR_ORG/daily-agent-evaluation:v1

params:
  candidate_revision:
    value: agent-v12
  evaluator_revision:
    value: acceptance-v4

cache:
  disable: true

matrix:
  kind: grid
  concurrency: 2
  params:
    day:
      kind: daterange
      value: ["2026-04-01", "2026-04-08", 1]

The registered daily component must declare day as a date input, plus the two string revision inputs. Confirm the generated windows before a substantial backfill and use the documented backfill pattern.

The component must load the intended historical data. Supplying a date parameter does not reconstruct a dataset that was never retained.

Make duplicate and late executions safe

Define a result key from the data window, dataset revision, candidate revision, and evaluator revision. A repeated execution can then be identified without overwriting unrelated evidence.

If the report is published to an external system, use that identity for idempotency. Platform retries do not automatically prevent duplicate notifications or writes.

Decide how late-arriving records are handled. You might freeze the original snapshot and create a new dataset revision for a correction. Do not silently compare two reports bearing the same window label but different underlying records.

Report coverage as well as quality

Track expected cases, completed cases, invalid inputs, and execution failures alongside quality metrics. A high score from a partially processed window is not a complete nightly result.

Persist the per-case report and window manifest through artifact tracking. Use the timeline to distinguish queue delays from execution problems.

Backfills can compete with current evaluations. Bound concurrency and, where available, use queues to allocate capacity without starving the normal workload.

Recurring evaluation then becomes an auditable process: each run has a defined window, missed work can be addressed explicitly, and a result always states how much of the intended workload it actually covered.