Operating long-running AI agents
Choose jobs, services, queues, and capacity controls for agent workloads using task duration, dependency limits, resume latency, and execution cost.
A live support assistant, a document-analysis batch, and a coding agent awaiting review may use the same model provider. Their operating requirements differ: the assistant needs a prompt response, the batch needs enough throughput to finish by its deadline, and the coding task needs its state retained while a person reviews the result.
Agent infrastructure should follow those requirements. Expensive model calls, variable execution paths, and human waits are useful dimensions to measure, but they do not imply that every agent needs low concurrency, can tolerate slow resumption, or requires a separate infrastructure stack.
An agent can be exposed as a service while its longer tasks execute asynchronously. The design question is how each part uses capacity and how the task survives changes to that capacity.
Separate request latency from task duration
Define what the user expects from the first response. Some tasks must finish within the interaction. Others can acknowledge acceptance, return a task identifier, and expose progress until a result is ready.
For asynchronous work, commit acceptance before acknowledging it. Return enough information for the caller to query status, cancel, or reconnect after losing its own connection. Deduplicate repeated submissions according to an explicit request identity so refreshing a page does not create another task.
Measure time to acceptance, time to first useful output, and time to accepted completion separately. An application that responds immediately but leaves work queued indefinitely has a different problem from one that begins quickly and spends most of its time in model calls.
Choose worker lifetime by workload
Use deployment units that match the work and its startup cost:
| Workload | Useful execution shape | Main tradeoff |
|---|---|---|
| Interactive agent requests | Warm service workers | Reserved capacity supports response latency |
| Independent document or evaluation batches | Bounded jobs | Startup overhead versus independent scheduling |
| Local model inference | Dedicated serving capacity | Model loading, memory residency, and batching |
| Tasks awaiting a human decision | Persisted wait state with later dispatch | Resume latency and reliable event delivery |
| Command execution with temporary files | A scoped execution environment | Isolation policy, startup time, and retained results |
These shapes can coexist. A warm service may accept requests and dispatch expensive work to jobs. A shared inference service may serve many tasks without loading the model in every worker.
Persistent workers can multiplex many waiting tasks when the runtime supports it; a waiting task does not necessarily occupy a dedicated process. Equally, releasing a task worker does not eliminate the cost of the runtime, database, or event delivery system. Measure the whole deployment before choosing a worker model.
Schedule against the limiting dependency
Agent throughput is often constrained by something other than local CPU: provider request or token limits, tool-service concurrency, database connections, or GPU memory for local inference.
Increasing worker count helps only when the dependency can absorb the additional demand. Track outstanding calls and queue age by task class and destination. Use backpressure before repeated rate-limit errors become the normal way to control traffic.
Separate infrastructure quotas from application budgets. A CPU limit constrains local execution; it does not limit spending at a hosted model provider. Enforce task-level model-call, token, tool-call, and elapsed-time budgets in the application, and preserve the consumed budget when the task resumes.
If one task can delegate work, include its children in the parent budget. Track whether cancellation reaches those children and whether they have already launched external actions. Otherwise one canceled request can continue consuming resources through its descendants.
Keep dynamic decisions compatible with recovery
A task may choose a tool, repeat a search, or stop early based on a model result. The runtime must retain the decision and know how it affects later work.
Dynamic decisions can run within durable workflow systems when non-deterministic work is isolated and its outcome is recorded. A model call can produce a variable result while the workflow makes subsequent decisions from that committed outcome. The relevant constraint is how execution is recorded and reconstructed.
Evaluate engines against the application you intend to run: supported waits, history size, payload limits, version compatibility, cancellation, and operational overhead. Include the team's ability to operate the state backend and repair failed tasks. The runtime architecture guide provides the responsibility map for that assessment.
Deploy without abandoning unfinished tasks
A rollout changes more than the code handling new requests. Existing tasks may still refer to old prompt versions, tool schemas, or checkpoint formats.
Define how workers drain and how unfinished tasks find compatible code. Keep a deadline for draining, a recovery route for interrupted work, and a record of the version that accepted each task. If state needs migration, make that migration visible and retain its source revision.
Measure startup and resume time under the conditions the application will encounter: cold container images, remote artifact reads, provider limits, and reduced cluster capacity. A background report may tolerate those delays; an interactive assistant may need warm capacity and a tighter admission policy.
Apply the execution model in Polyaxon
Polyaxon can run bounded jobs and long-running services with explicit resource requirements and managed connections. Use the same container and configuration references across a task's qualification and production execution where practical.
In the commercial offering, queues provide priorities, concurrency, quotas, and routing to namespaces or clusters. These controls regulate workload dispatch. The application still needs provider-aware rate limits and task budgets for external model and tool calls.
Apply termination policies for bounded attempts and timeouts. Pair replacement workloads with the durable execution protocol so they recover the intended task state. Preserve results in the configured artifact store and keep interactive scratch files separate from the outputs required for recovery.
Use DAGs for supporting work such as dataset preparation, batch evaluations, and report generation. The existing pipeline orchestration guide explains dependencies and resource placement for those operations.
Measure completed work and occupied capacity together
Review queue delay, startup time, active processing, external-call waits, human waits, and recovery separately. Pair resource-hours and provider charges with accepted task outcomes. Retain failed and canceled attempts in the cost total even when the final run succeeds.
Track aged tasks as well as errors. A task waiting for an approval callback or an unavailable worker can remain incomplete while every service health check passes. Give each wait state an owner, expiration, and next action.
Use agent SLOs to decide which delays matter to users. That lets the team choose warm capacity, job dispatch, and scheduling controls from measurable service needs.