Design measurable AI agents before you build

Teams often instrument an agent after the first failure, then discover that the evidence they need was never produced. The system recorded a model response but not the retrieved source, attempted tool action, approval, resulting state, or version that made the decision.

Measurability starts as an application-design artifact. Before choosing a framework, define the observable outcome, unacceptable actions, events that explain the path, and evaluator that will decide whether a run passed. The implementation can then produce the evidence its release and operation require.

Start with a task contract

Define one user-visible task rather than “the agent” as a whole. A refund-support task, research task, and coding task have different outcomes, authority, and budgets.

A compact contract can be stored beside the evaluation dataset:

task: issue-approved-refund
contractVersion: refund-contract-v3
eligibleWhen:
  - authenticated customer owns the order
success:
  observedState: refund.status == completed
  response: explains the amount and destination
requiredEvents:
  - order.read
  - policy.read
  - approval.checked
  - refund.attempted
  - refund.completed
prohibitedEvents:
  - refund.completed_without_approval
budgets:
  maxToolCalls: 8
  maxDurationSeconds: 45
segments:
  - refund reason
  - payment method
  - policy version

The values are illustrative. Choose budgets from user expectations and operating evidence. The essential fields are the externally verified final state, required and prohibited behavior, measurement population, and contract version.

Distinguish refusal, escalation, incomplete execution, and task failure. A correct escalation can be a successful outcome for an ineligible or ambiguous case. A polished response paired with an unauthorized refund is a failure regardless of its language quality.

Design the event vocabulary

List the states and boundaries an investigation must cross:

Use stable task and trace identifiers, event names, clocks, and parent relationships across services. Attempted and completed actions require separate events. Otherwise a denied update and a successful update can look identical in a count of tool calls.

Capture the minimum sensitive content needed for the purpose. Prefer source references, hashes, structured classifications, and redacted fields when raw prompts or records are unnecessary. Define retention and access with the schema, not as cleanup after traces accumulate.

The OpenTelemetry generative AI semantic conventions can provide common fields for model and agent operations. Add application-specific task, policy, and state-change semantics rather than forcing every outcome into model-call attributes.

Choose evaluators from the failure modes

Every critical requirement needs an evaluator and a source of truth:

Use deterministic checks for schemas, permissions, budgets, and state whenever possible. Use model-based judges for semantic properties that genuinely require them, and calibrate those judges against expert-reviewed cases. Preserve judge disagreement and explanation; do not silently convert an uncertain score into ground truth.

Test evaluators too. Construct known passes and failures, measure false passes and false failures, and verify behavior when required evidence is missing. An evaluator that defaults missing tool records to “safe” creates confidence from an instrumentation outage.

Create a resettable test environment

Agent tests need controlled initial state. Provide synthetic or protected test records, deterministic clocks where relevant, idempotent tools, and a way to inspect and reset external state between cases. Record the environment version with every result.

Exercise the boundaries, not only ideal requests:

• Missing, stale, conflicting, or unauthorized data.
• Tool timeouts, partial success, malformed results, and retries.
• Duplicate requests and interrupted sessions.
• Approval denied, expired, or granted for a different action.
• Handoffs with missing identity or excessive authority.
• Direct and indirect prompt-injection attempts.
• Long contexts, high concurrency, and exhausted budgets.

Verify final state after the agent stops. For a coding agent, inspect the files and commands; for a support agent, inspect the customer record and messages; for a research agent, verify cited source revisions. The agent's description of its own work is another output, not the system of record.

Build the scorecard before the dashboard

Define each indicator with its population, numerator or calculation, data source, delay, segments, exclusions, and owner. Keep task success, policy violations, latency, and cost separate so one cannot compensate for another.

Useful task-level measures include:

• Eligible tasks reaching a verified success or correct escalation.
• Unauthorized attempted and completed actions.
• Required steps missing from otherwise successful trajectories.
• End-to-end latency and time to first useful response.
• Model calls, tool calls, retries, tokens, and cost per successful task.
• Human correction, escalation, and unresolved rates.

Segment by task class, consequence, model, prompt, tool, data revision, release, and route. A global average can improve while a low-volume high-risk case regresses. The guide to SLOs for AI applications and agents explains populations, good events, delayed outcomes, and error budgets.

Establish the pre-release loop

Start with a small expert-reviewed seed set. Run the first implementation, inspect failures, and update the contract only when the product requirement has actually changed. Fixing code and weakening the expected behavior are different decisions and should have different records.

For every behavior-changing release:

1. Freeze the task contract, dataset, environment, application, and evaluator versions.
2. Run the same cases against the current and proposed versions.
3. Compare paired results and segment-level regressions.
4. Review new failures and evaluator disagreements.
5. Apply hard gates for critical policy invariants.
6. Publish the results and explicit approval decision.
7. Release to a bounded population with rollback signals.

Polyaxon can run evaluation jobs and simulators as containerized operations, compare inputs and metrics, and preserve reports as artifacts. Use a DAG when dataset preparation, execution, aggregation, and approval evidence have dependencies. Pin the component, environment image, dataset, and evaluator so results remain interpretable.

Connect production behavior to the next test

After release, monitor the same contract with production traces, deterministic outcome checks, sampled semantic evaluation, and human feedback. Keep provisional results separate from delayed final outcomes. An initially successful ticket may be reopened later; a refund may settle after the interaction ends.

When a task fails, move from the affected measure to the trace, versioned inputs, and external state. The agent tracing guide shows how to connect retrieval, model, tool, and handoff spans. Preserve the incident example, sanitize it, assign an expected result, and add it to the regression dataset.

Monitor the measurement system itself: missing events, unmatched task IDs, evaluator errors, changes in sampling, delayed labels, and exclusion rates. A flat quality graph during a telemetry outage is not evidence of stable behavior.

The finished design should let an engineer implement the agent and its evidence together. It should also let a reviewer state what success means, which actions are unacceptable, how the system proves both, and what the team will do when a new failure appears.