Designing the runtime layer for AI agents

A support agent reads a case, drafts a response, gets approval, and posts an update. The update succeeds, but the connection drops before the agent receives confirmation. Its container then restarts. Should it post again, ask for approval again, or continue to the next step?

The answer depends on the runtime around the agent: the system that records progress, manages execution attempts, and decides how work continues after an interruption. Model quality helps the agent choose an action. Runtime design determines whether the application can complete that action reliably.

A practical runtime model separates four responsibilities: the model, the harness that drives its tool loop, the runtime that manages execution over time, and the platform that governs the application. One product may cover several of these areas, but the ownership and interfaces still need to be explicit.

Give the task an identity that survives its workers

A user task can outlive several containers, tool requests, and deployment versions. Give it a stable identifier, then record individual execution attempts beneath it. A replacement worker should be able to discover the task's last committed state without relying on the previous worker's memory or local files.

For the support workflow, maintain a record such as:

Keep the logical task ID separate from the Polyaxon run UUID and any framework conversation ID. Link them explicitly. This allows an operator to connect one customer task to its execution history even when a fresh infrastructure run handles recovery.

Assign one active owner to each state transition. If an old worker can reconnect after its replacement starts, both may try to advance the task. Use the state store's concurrency controls, such as conditional writes or leases with fencing tokens, so a stale worker cannot overwrite newer progress.

Define the contract between coordination and execution

The coordinator decides which step is eligible and records its outcome. The execution target performs the work: a service handles a model request, a job processes a document batch, or a tool endpoint updates the support case.

Pass an explicit envelope between them. Include the task and step identities, input and code revisions, deadline, attempt number, and a reference to the permitted credentials. Return a typed result with an output reference or a failure category. Keep large artifacts in storage and secrets out of the envelope itself.

A coordinator may be restarted too. Its ability to recover comes from committed state and a defined recovery protocol. Keeping it in a separate process helps isolate failures, but that separation alone does not preserve progress.

When an execution target disappears, the coordinator needs to distinguish work that never started, work that failed, and work whose outcome is uncertain. The durable execution guide develops the checkpoint and side-effect rules behind those decisions.

Support framework differences explicitly

Standardize the execution envelope, identity mapping, resource policy, and operator-facing statuses across teams. Allow framework adapters to own their checkpoint format and continuation API.

Agent frameworks differ in how they separate thread checkpoints from information shared across tasks. An adapter needs to configure a durable backend, retain the framework's task identity, and invoke the correct continuation behavior. In-memory state alone cannot support recovery after the process disappears.

Running two frameworks on the same infrastructure does not make their saved state interchangeable. Keep the application and state-schema versions with each unfinished task. A new worker must either understand that version or follow an explicit migration path.

Approvals should cross this interface as recorded decisions about a specific action revision. The framework determines how that decision resumes its execution. The platform determines who may submit it and retains the decision record. The execution service enforces the authority required when the action actually runs.

Give each retry and budget an owner

Retries can occur inside a model SDK, a tool client, the agent framework, and the workload controller. If each independently restarts a large unit of work, a short service outage can produce repeated model calls and duplicated computation.

Choose the smallest recoverable unit and bound its attempts. Coordinate the limits across layers so a task's total deadline and cost budget still apply after a worker replacement.

Workload restart settings are part of this design. The Kubernetes Job documentation notes that applications must handle replacement pods and that even a single-completion job can sometimes start the same program twice. Treat repeated execution as a condition the application can encounter.

Connect the runtime to Polyaxon execution

Polyaxon supplies building blocks for running the agent application and its supporting workloads. Make the connection to application state explicit:

• Use containerized jobs for bounded work and services for long-running endpoints. Select CPU, memory, GPU, storage, and networking requirements for the work each performs.
• Use DAG operations for explicit dependencies such as preparation, batch agent execution, evaluation, and report collection. Keep dynamic model-and-tool loops in the application or framework that owns them.
• Configure termination policies to bound workload retries and execution time. The application startup path must restore its checkpoint and interpret unfinished actions.
• Retain reports and outputs through configured artifact connections. Choose storage that survives the failure you need to recover from, and use a state backend with the consistency your checkpoint protocol requires.
• Where enabled in the commercial offering, use queues for workload priority, concurrency, quotas, and routing to execution environments.

Artifact collection and checkpoint commitment have different timing requirements. Writing a file locally and waiting for a later upload leaves a recovery gap if the worker disappears first. Confirm that critical state reaches durable storage before acknowledging the transition it represents.

Polyaxon's sandbox interface gives an application process and filesystem access inside a running service container. Its permissions, mounts, and network access follow that service's configuration. Define those execution controls alongside the agent's tool permissions, and persist required results before replacing the service.

This division gives platform teams shared execution infrastructure while allowing each agent framework to implement its state semantics. The control-plane guide covers the versioning, evaluation, and release decisions around that runtime.

Make the architecture inspectable

Start with one representative task and follow it across acceptance, dispatch, execution, persistence, and completion. At each boundary, identify the owner, stored record, deadline, and recovery action. Confirm that a replacement worker can find the same task and that an operator can distinguish a queued task from a stalled one.

Connect those records to traces and artifacts. Then use the guide to operating long-running agents to choose worker lifetimes and capacity controls from the actual workload.

The architecture is useful when the team can explain what happens to an unfinished task after its worker disappears: which state survives, who owns the next attempt, and how the application determines whether its actions completed.