Beyond agent traces: State, artifacts, and recovery

A trace shows that an agent generated a report, called a publishing tool, and received a timeout. That explains the request sequence. To decide what happens next, an operator also needs the report revision, the task's committed state, and evidence from the publishing service about whether it accepted the update.

Inspecting a failure and continuing the affected work are different problems. Closing that gap requires connecting observability to application state and controlled execution.

The useful operator experience starts with a trace and leads to the records needed to make a recovery decision. Each record has a different purpose and may require different storage and retention.

Give each record a clear job

Use a small set of connected records:

An output can exist before the application commits it as complete. An external action can succeed before its caller records a receipt. The interface should preserve those distinctions instead of collapsing everything into one green or red status.

For the runtime boundaries behind these records, see designing the runtime layer for AI agents.

Correlate the task across multiple executions

Record a stable logical task ID alongside the framework thread ID, Polyaxon run UUID, execution attempt, and trace identifiers. Link each artifact and checkpoint to the attempt that produced it.

A long-lived task can span several traces. OpenTelemetry's trace model provides span links for causal relationships, including asynchronous work that begins after the originating operation. Use those links where appropriate and retain the task ID for a complete application-level history.

Make the mappings navigable in both directions. From a failed span, an operator should find its input revision and task state. From a completed task, the operator should find the attempts, artifacts, and evaluations behind the accepted result.

Use task identifiers in records suited to individual lookups, such as traces and run metadata. Aggregate metrics by bounded dimensions such as release, task class, and outcome so the monitoring system does not need a separate time series for every task.

Retain evidence that supports the next decision

For a generated report, preserve the source revision, relevant extraction, prompt and model identifiers, accepted output, and evaluator version. Reference larger or protected payloads through controlled artifact storage.

Record enough to distinguish reuse from regeneration. If an operator compares two reports, they should be able to tell whether both used the same source evidence, whether retrieval ran again, and whether a different model result changed the execution path.

Telemetry and recovery state also need different availability guarantees. OpenTelemetry's sampling guidance describes controlling the volume of collected traces. A task's ability to recover must not depend on whether its diagnostic trace was retained. Store the authoritative state required for continuation under the runtime's own persistence contract.

Coordinate retention across linked records. If an artifact has expired or access has been revoked, preserve an explicit unavailable status. Do not reconstruct missing evidence from a summary and present it as the original input.

Turn an incident into a controlled recovery

For the publishing timeout, inspect the task state and the destination before choosing an action. The report might be committed, the publication might be confirmed remotely, or both may still be uncertain.

Present the operator with actions tied to current state:

• Reconcile an external outcome using the destination's records.
• Retry unfinished work when its inputs and side-effect contract permit it.
• Resume a waiting task with the required decision recorded.
• Start a new branch when changing the input, prompt, code, or provider.
• Cancel further execution while accounting for outstanding actions.

These actions need authorization and state checks when invoked. A dashboard can become stale between inspection and execution. Require the expected checkpoint revision so a recovery request cannot overwrite progress made by another worker.

Keep the original failure record and append the recovery decision, actor, reason, and resulting attempt. The durable execution guide covers the state and side-effect rules that make these controls safe.

Compare a corrected branch with its actual baseline

Suppose the publishing failure exposed an unrelated error in the generated report. Changing the retrieval input and rerunning the analysis creates a new candidate with different evidence.

Record which outputs were reused, which steps executed again, and which dependencies changed. Reevaluate downstream results when an upstream input changes. Any approval should identify the artifact revision that the reviewer actually accepted.

Separate the recovery question from the quality question: did the continuation avoid repeated effects, and did the new result satisfy the task? A successful retry can publish a poor report; a better report can still be published twice by a broken recovery path.

Compare both candidates using the same evaluation criteria where possible, and retain the evaluator version. The production traces to regression tests guide explains how to convert the incident into a repeatable case for future releases.

Connect the evidence through Polyaxon

Polyaxon's artifact tracking supports saved assets and reference-only lineage metadata. Choose the method according to where the evidence lives. Logging a reference records its relationship to the run; the application remains responsible for saving the referenced file.

Attach manifests, reports, and evaluation outputs to execution runs. For artifacts in an external store, record their immutable version and accessible location with the lineage reference. Use the metadata and artifact query interfaces to locate relevant runs and retrieve the retained evidence.

Keep the mapping from logical task to execution attempts in application metadata. Polyaxon run records describe workload execution; the agent framework or state service supplies the checkpoint semantics for continuation. Link recovery attempts back to their source evidence so infrastructure history and application history can be inspected together.

For teams building an incident interface, the first useful view is a task page that combines current state, linked traces, artifact revisions, and confirmed actions. Add recovery controls only when the corresponding runtime transition and authorization checks exist.

Measure whether incidents become resolvable

Track how often an investigator can retrieve the needed inputs, identify the last committed state, and determine the outcome of an external action. Record unresolved cases explicitly, including missing artifacts, unknown versions, and incomplete action receipts.

Measure time to a justified recovery decision alongside time to detect a failure. Review whether the resulting attempt completed correctly and whether a regression case now covers the incident. That establishes whether the collected evidence helps the team operate and improve the agent.