An AI agent deployment checklist for the first production release

The first production release of an agent needs a definition of ready that an application owner, platform engineer, and operator can all inspect. A demonstration shows that a task can succeed. Release qualification establishes the conditions under which the team is willing to accept real work.

For a Polyaxon team, the checklist can point to concrete records: a qualification run, its input configuration, a case-level report, and an approved release operation. Component versions make the qualification procedure reusable, while run lineage connects its execution to the artifacts and dependent operations.

Use it for a bounded first release. A read-only research assistant and an agent that changes customer records need different acceptance criteria, even if they use the same framework.

Write the release contract

Suppose an agent investigates failed data imports. It reads logs and documentation, proposes a diagnosis, and drafts a remediation plan. The initial release cannot execute that plan.

Record the permitted inputs, outputs, tools, data sources, users, and maximum task duration. Include unsupported cases: missing logs, inaccessible customer data, and incidents that require an infrastructure change. Define the escalation destination and the evidence sent with it.

Keep this contract beside the release configuration. A later change that adds a write-capable tool changes the release's authority and requires new qualification, even when the user-facing interface stays the same.

Collect evidence for each launch decision

Use a checklist whose entries point to observable results:

Each item needs an owner and a disposition: accepted, failed, or explicitly deferred within a narrower release scope. An unexamined item should remain visible.

Separate qualification from permission to release

Use a qualification component that checks the release contract, runs your evaluator, saves the report, and exits unsuccessfully when release-blocking checks fail. A completed scoring process is not necessarily a passing candidate: the qualification program must translate failed criteria into a failed operation.

The following DAG template assumes your team has registered two application-specific components: your-team/qualify-agent:1.0 and your-team/release-agent:1.0. They are illustrative names, not components bundled with Polyaxon. Both accept release_revision; the qualification component also accepts dataset_revision. Replace them with your own registered definitions and use a revision that resolves to a fixed image and configuration.

version: 1.1
kind: component
name: qualify-and-release-agent
run:
  kind: dag
  operations:
    - name: qualify
      hubRef: your-team/qualify-agent:1.0
      params:
        release_revision: {value: agent-release-17}
        dataset_revision: {value: import-incidents-v4}
      termination:
        timeout: 1800
        maxRetries: 0
    - name: release
      hubRef: your-team/release-agent:1.0
      dependencies: [qualify]
      trigger: all_succeeded
      isApproved: false
      params:
        release_revision: {value: agent-release-17}
      termination:
        timeout: 600
        maxRetries: 0

DAG dependencies and triggers require qualification to succeed before the release step can proceed. Manual approval holds the release operation for an explicit decision through the UI, CLI, or API. The timeouts and retry counts are example workload settings from the termination reference, not recommended defaults for every agent.

The reviewer should inspect the qualification report before approving, and the release component should verify that the approved evidence belongs to the exact candidate being deployed. Avoid a mutable latest reference between evaluation and release. Operation approval controls launching this step; it does not implement end-user authorization or approve every future agent action.

Qualify the client experience

For asynchronous work, the client needs a stable task reference and a status API or event stream. A connection closing should not force the user to guess whether the investigation is still running.

Define the response to duplicate submissions. The application might return the existing task for a repeated request key, or intentionally create a new investigation when the input changes. Document that behavior and bind the key to the requesting identity.

Exercise malformed inputs, duplicate requests, expired credentials, a canceled task, and delayed completion notifications. Confirm that client-visible statuses describe business progress rather than merely the state of one worker container.

Evaluate with the release's actual constraints

Run the candidate on ordinary cases, ambiguous cases, and cases where the right behavior is to ask for help. Include long logs, misleading documentation, and evidence that contradicts the initial incident report.

Use the tool set and permissions intended for production. An evaluation with broad development credentials can conceal a missing access path or permit behavior the released application cannot perform.

Keep quality criteria separate from operational criteria. A correct diagnosis delivered after the incident has been resolved may fail the turnaround requirement. A fast, well-formed diagnosis may still cite the wrong import attempt. The agent evaluation guide covers task and trajectory assessment in more detail.

Rehearse one failure at each dependency boundary

Make a connector unavailable, expire a credential, interrupt a worker, and make an artifact write fail. Verify the task's resulting state and what the operator sees. For each failure, define whether the application retries, waits, escalates, or stops.

Save the expected and observed outcomes in the qualification report. Use the run monitoring view to inspect workload status and resource evidence, then correlate those records with the application task ID. A Kubernetes workload becoming healthy again does not prove that the original investigation resumed correctly.

Keep retries within an overall task deadline. If the model client, tool adapter, and workload controller can all retry, record how their limits interact. For write-capable releases, include uncertain external outcomes and duplicate-effect prevention in the rehearsal.

The durable execution guide explains checkpoint and side-effect semantics. Here, the release decision is whether those semantics have been demonstrated for the task you intend to launch.

Make overload behavior intentional

Specify how much work can be accepted, queued, and executed concurrently. Give clients an understandable response when capacity is unavailable. Separate limits for worker resources from limits imposed by model and tool providers.

Measure the time from task acceptance to completion, including queue wait. Increasing worker count will not help if every worker is blocked by the same provider quota. Set a maximum age for queued investigations so obsolete requests do not consume capacity after a long outage.

Keep qualification attached to the release

Keep the release revision and dataset revision visible in the run comparison table. Use tracking to log your evaluator's pass rates and operational measurements, and use artifact lineage to find the manifest and qualification report. The production engineering workflow includes an SDK example for retaining these results.

Store the release contract, case-level results, and launch decision as controlled artifacts. The qualification code and deployment process must implement the pass criteria and promotion rules; a successful infrastructure run alone does not approve the application.

Begin with a recoverable rollout

Choose a limited user group and task scope, identify the person monitoring the launch, and define conditions that stop further expansion. A shadow evaluation should use isolated or disabled write tools so comparison traffic cannot accidentally perform real actions.

Before launch, practice reverting the application configuration. Decide how old and new versions handle unfinished tasks and incompatible state. Reverting code cannot undo actions already completed in external systems.

The first release is ready when its owners can explain its allowed behavior, produce evidence for its acceptance criteria, and demonstrate how they will respond when those criteria stop holding.