Durable execution for AI agents

A document-review agent has extracted the relevant clauses and prepared its findings. Before it publishes the report, its worker disappears. Starting a fresh worker is straightforward. Deciding which findings can be reused, which action remains unfinished, and whether the report was already published requires a recovery protocol.

Durable execution preserves enough progress for a task to continue after its executing process is lost. Applying that idea in production means defining precisely what counts as committed work, what remains uncertain, and what the application is allowed to repeat.

The agent runtime architecture guide describes who owns these responsibilities. This guide focuses on the state transitions they must implement.

Define the recovery boundary

A useful checkpoint captures a decision the application can trust after a restart. For document review, that might be a validated extraction from a particular source revision, followed by a separately committed analysis of that extraction.

Define the boundary using three questions: what inputs identify the work, what output proves completion, and what unfinished effects could remain if the worker disappears now?

Choose boundaries that avoid repeating expensive work without forcing every token or temporary variable into storage. A smaller checkpoint interval reduces the work exposed to failure but adds serialization, storage, and coordination overhead.

Commit outputs before acknowledging progress

Suppose extraction writes a file and then updates the task record to say it is complete. If the file exists only in the worker's temporary directory, the task record can outlive the result it references.

A practical pattern is to write an immutable output, validate its completeness, and commit a reference to that exact version in the state store. Advance the task using the expected previous state revision so a stale worker cannot commit over a newer attempt.

When the artifact and state stores cannot share a transaction, define the intermediate cases. A completed upload with no committed reference may leave an orphan that can be cleaned later. A committed reference to an unavailable artifact must stop recovery until the result is restored or the step is explicitly repeated. Consumers should follow committed references rather than guessing from filenames.

Choose storage durability to match the failure model. A local file can survive a process restart and still disappear with its node. Recovery across nodes or clusters also requires that the replacement workload can access the same committed state.

Distinguish retry, resume, and a new branch

Name the operation precisely in both the API and the operator interface:

• A retry attempts unfinished work again under its recorded inputs and retry policy.
• A resume advances the same logical task from its committed state.
• A branch creates a new execution from selected earlier evidence with changed inputs or code.

These are application-level meanings; individual frameworks may use different terms. Check what their recovery API actually executes.

Some workflow engines reconstruct execution from persisted history, while others load an application checkpoint. Integration code must respect the chosen engine's execution contract, including how it records external results and handles code changes.

For a corrected extraction, record a new branch linked to the original task. Rerun the dependent analysis and review steps using the new extraction. Reusing the old approval would connect a decision to content that the reviewer never saw.

Reconcile external effects independently

Publishing the report introduces a gap between two systems: the destination can accept it before the agent commits its receipt. A worker failure in that interval leaves an uncertain outcome.

Use the destination's idempotency contract where available. Persist the action key and arguments before dispatch, and reuse that key only for the same intended action. The receiving service must recognize duplicate requests and return the previously established outcome rather than applying the action again.

If the destination cannot deduplicate, reconcile through a reliable lookup or operator review. A local completed flag cannot independently guarantee that an external action occurred exactly once. Keep uncertain outcomes visible until there is enough evidence to resolve them.

Preserve approval and cancellation state

Store an approval request as task state with a specific artifact revision, authorized reviewer, deadline, and status. The waiting task can release its execution capacity once that state is committed. A later event can assign a worker to continue.

Resolve approval, expiration, and cancellation through guarded transitions. If cancellation wins, a late approval should not restart the task. If the proposed output changes, require a decision about the new revision. Recheck access to any destination when the action runs.

Give cancellation its own completion criteria. Stopping a worker prevents further local computation, but an external request already in flight may still complete. Record and reconcile that request so the user can distinguish cancellation accepted from all effects accounted for.

Configure infrastructure recovery in Polyaxon

Use Polyaxon termination settings to bound workload attempts and execution time. Inside the application, restore the logical task ID and committed checkpoint before continuing. A workload retry starts code; that code and its framework decide what application work remains.

Use a persistent application state backend for checkpoint coordination and configured artifact storage for reports and other retained outputs. Polyaxon's output collection can preserve run artifacts, but the checkpoint protocol must establish when a result is durably available before it advances the task.

Record the task ID, state revision, attempt, and parent execution with the run metadata and artifacts. This makes a replacement run inspectable without assuming it has the same infrastructure identifier as its predecessor. DAGs can coordinate preparation, execution, and evaluation around that application recovery path.

Qualify the recovery behavior

Build failure cases around the commit boundaries: worker loss before an upload, after the upload but before state commitment, and after an external service accepts an action. Include duplicate callbacks, incompatible code versions, expired approvals, and inaccessible state storage.

For each case, inspect the final task and destination state. Count repeated computation, additional model calls, unresolved actions, and time until a usable result. A successful replacement pod alone does not establish successful recovery.

Keep the original failure, the recovery decision, and the final outcome connected through the execution evidence. That record establishes whether the task continued correctly and which part of the protocol needs to change when it did not.