Recover agent actions after a lost tool response

Suppose a deployment tool accepts an agent's request to activate a model, but its response is lost. Before the worker restarts, a person changes the requested model and asks to cancel the old work.

The replacement worker has two questions to answer: what happened at the destination, and what is it authorized to do now? A retry policy that answers only the first can execute an obsolete action or mark the wrong task revision complete.

We will follow that case through recovery, then define how to compare alternative policies. The task controller and deployment service below are application-owned. Polyaxon runs the evaluation attempts and records their evidence.

Recover the action that was actually sent

Task release-42, revision 17, authorizes activating model m7 on staging/fraud. Before sending the request, the controller records action a73, the exact model digest, destination, operation, and approval reference. Each worker attempt has its own execution ID; they share the logical action ID.

The destination applies m7 and records a receipt for a73. The caller times out before recording that receipt locally. A timeout establishes that the caller lacks a result. It does not establish that the request failed to apply.

On recovery, look up a73 at the destination and verify the receipt's operation, arguments, and destination. Do not replace it with an action for whatever model happens to be current when the worker restarts.

Automatic retries need an explicit service contract. HTTP semantics distinguish safely repeatable requests from requests whose effects must be established before retrying. A service's idempotency contract must define how it handles duplicate keys, changed arguments, and late requests; the AWS Builders' Library explains those design choices.

For this case, assume the deployment service deduplicates the same action ID and arguments for a documented interval and exposes a durable operation result. A matching terminal result remains evidence of the outcome even after the retry window expires. When no outcome can be established and no valid retry contract remains, keep the action unresolved until another reliable source or operator can account for it. A failed lookup is not evidence that nothing happened.

Keep old effects separate from new authorization

While a73 is unresolved locally, the user changes the requested model to m8. The task advances to revision 18 and records a request to cancel the old work.

The replacement now finds the receipt proving that m7 applied. It should record that historical effect, preserve the cancellation request, and leave m8 awaiting approval. The receipt for revision 17 does not complete revision 18.

Bind approval to the intended operation, target, artifact digest, and relevant plan revision. A changed model or destination requires authorization for that changed action. Store approval status and expiry in trusted application state, where the executor can check them without consulting the model's conversation.

Serialize dispatch eligibility with task changes and cancellation. For example, the dispatcher and cancellation handler can lock the same task record and conditionally claim an action against its current revision. Only a successful claim permits dispatch.

That claim is a meaningful boundary: cancellation before it prevents a new dispatch; cancellation after it may race with an already authorized request. If cancellation must revoke an in-flight write, the destination needs a revocation or fencing mechanism enforced when the effect commits. Stopping a worker alone cannot provide that guarantee.

Until the old action is accounted for, this protocol holds newer writes to the same target. That avoids allowing an unresolved older request to overwrite a newer deployment. Undoing an applied update is a separate action with its own authorization.

Make recovery decisions explicit

The following function selects a recovery decision after the controller has verified the destination record's identity. absent means a successful lookup found no operation; unknown includes an unavailable or inconclusive lookup.

from typing import Literal

Effect = Literal["applied", "rejected", "pending", "absent", "unknown"]


def recovery_decision(
    effect: Effect,
    same_task_revision: bool,
    authorization_valid: bool,
    deduplication_valid: bool,
) -> str:
    if effect == "applied":
        return "record_observed_effect"
    if effect == "rejected":
        return "record_rejection"
    if effect in {"pending", "unknown"}:
        return "wait_or_escalate"
    if effect == "absent":
        if same_task_revision and authorization_valid and deduplication_valid:
            return "retry_same_action"
        return "hold_dispatch"
    raise ValueError("Unknown destination outcome")

This is decision logic, not a dispatcher. Before acting on retry_same_action, atomically recheck the current task and authorization, then claim the attempt. Reuse the original action ID and arguments within the destination's deduplication contract. A late original request can still arrive after an absent lookup; deduplication is what makes that race tolerable. The service's retention and request-validity rules must cover that late arrival, not just the moment the retry was sent.

Persist observed effects in the action ledger using a conditional update on that action's revision. If another worker already recorded the same receipt, treat the repeated observation as such. If the update conflicts, reload. Update the current task separately under its own revision guard so an old receipt cannot overwrite a newer request or cancellation.

Put a deadline and an owner on unresolved actions. Waiting without a bounded escalation path turns uncertainty into abandoned work. Durable agent execution covers the broader checkpoint and state-store contract.

Compare policies against final state

Three useful baselines are stopping after a timeout, retrying with a stable action ID, and reconciling the effect before rechecking authorization. The second can recover ordinary lost responses under a valid deduplication contract. It still needs task-version and authorization checks when intent changes.

Inject faults at the same protocol boundaries for each policy, such as after destination acceptance, rather than after an arbitrary elapsed delay. Keep the scenario revision and recovery deadline fixed. A compact evaluation set should include:

These are evaluation requirements, not reported results. For each case, retain the task revisions, approval state, action identity, destination ledger, and attempted transitions. The evaluator should derive correctness from those records, rather than from the recovery worker's assertion that it succeeded.

Report task completion, unauthorized dispatches, duplicate effects, unresolved actions, and time until the effect is accounted for. Count every admitted case, including setup and recovery failures. When the destination evidence is unavailable, keep that uncertainty visible instead of reporting zero duplicate effects.

Retain the comparison in Polyaxon

Use a separate Polyaxon run for each scenario and policy, with fixed component/image and scenario revisions. A grid study can enumerate those inputs for your evaluation component:

matrix:
  kind: grid
  concurrency: 2
  params:
    policy:
      kind: choice
      value: [stop_on_timeout, retry_same_key, reconcile_and_reauthorize]
    scenario:
      kind: choice
      value: [lost_response, changed_task, cancel_inflight, duplicate_receipt, uncertain_lookup]

This fragment requires your own component declaring policy and scenario, plus a mock destination and failure injector. Keep policy comparison separate from real deployments; the example does not require a live model service. The pure decision function above is one part of that evaluator, not a complete simulation.

Inside an initialized tracking run, save the evaluator's sanitized report and the measurements it actually established:

import json
from pathlib import Path

from polyaxon import tracking


def record_recovery_case(report: dict, measurements: dict) -> None:
    tracking.log_inputs(
        scenario=report["scenario"],
        policy_revision=report["policy_revision"],
        task_id=report["task_id"],
        action_id=report["action_id"],
    )
    output_dir = Path(tracking.get_outputs_path())
    output_dir.mkdir(parents=True, exist_ok=True)
    (output_dir / "recovery.json").write_text(
        json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8"
    )
    tracking.log_metrics(**measurements)
    tracking.log_outputs(recovery_outcome=report["outcome"])

Include the case revision and execution-attempt links in report; pass only observed numeric measurements. Run comparison and the saved artifacts make policy differences inspectable. The report must distinguish an executor finishing normally from a logical task completing correctly.