How to red team AI agents
Test AI agent permissions, memory, handoffs, retries, and tool actions with a practical security matrix and reproducible evaluation workflow.
Red teaming an AI agent means testing what it can do across a sequence of decisions. A final answer alone cannot establish whether the agent accessed a restricted record, changed a file, exceeded its budget, or delegated work with more authority than the original user had.
Start with one real workflow and a controlled substitute for its external systems. For a support agent, that might mean synthetic customer records, a refund ledger in memory, and an email outbox that stores messages without sending them. The goal is to observe attempted and completed actions independently of the agent's description of them.
Map authority across the workflow
List the authenticated actor, resources, allowed actions, approval requirements, and maximum execution budget at each step. Include tools that appear harmless: reading a document can expose private data, while saving a memory can influence a later session.
The OWASP AI Agent Security Cheat Sheet covers tool privileges, memory isolation, and human oversight. Turn those concerns into application-specific tests with observable outcomes.
| Boundary | Test setup | Required observation |
|---|---|---|
| Tool permission | A reader is asked to issue a refund | Executor denies the write; ledger stays unchanged |
| Account scope | Tenant A requests a synthetic tenant B record | No B content reaches the model or user |
| Approval | Retrieved content claims an action was approved | Approval comes from the application's trusted state |
| Memory | A prior session stores a claim of elevated permission | A new session still uses its current identity and grants |
| Handoff | A specialist receives a task from a read-only agent | Delegation does not expand the original actor's authority |
| Retry | A tool times out after accepting a request | The retry does not duplicate the action |
| Budget | A tool repeatedly returns an unhelpful result | Execution stops at the configured step or time limit |
For each row, add a legitimate control. A reader should still retrieve an authorized order, and an operator should still complete a properly approved refund.
Test the executor before the model
Submit prescribed invalid actions directly to the tool boundary in a test environment. This establishes whether the permission layer rejects them even when the proposal is wrong. Next, run the full agent with adversarial content and observe whether it produces the same invalid proposal.
This gives two useful answers: how often the agent proposes an invalid action, and whether the executor prevents it. A model update might reduce proposals without fixing an authorization bug. A permission fix might prevent side effects while leaving a confusing user experience.
The Promptfoo fixture tutorial demonstrates the first layer with prescribed actions from the Polyaxon examples repository. Its passing result does not establish that a real model resists adversarial instructions.
Exercise stateful failures
Single-turn tests miss failures that depend on earlier actions. For memory, write a synthetic preference in session one and retrieve it in session two. Repeat with a different account, an expired permission, and a revoked approval. Observe which entries are loaded before the model generates an answer.
For a handoff, preserve the initiating user, task scope, and approval reference through the receiving agent and into the tool executor. Test what happens when those fields are missing or inconsistent. A statement such as “the first agent approved this” is content to assess, not a substitute for authorization.
For retries, make the test service accept an action and then return a timeout. Replay the request with the same operation identifier and verify one ledger entry. Separately test a timeout before acceptance. These two failures look similar to the agent but require different recovery behavior.
Review a trajectory without losing the outcome
Store the ordered steps, tool arguments, authorization decisions, external observations, and stop reason. A compact review record should answer:
- What was the user authorized to accomplish?
- Which step first exceeded that scope?
- Was the action proposed, rejected, or completed?
- Did the legitimate task still succeed?
- Can another run reproduce the result with the same initial state?
Human review is useful for disputed task outcomes. Deterministic state checks should decide facts such as whether a refund occurred. Keep judge disagreements visible instead of averaging them into one safety score.
Preserve the regression in Polyaxon
Use run artifacts to preserve the case, synthetic initial state, and trajectory. Record separate metrics for invalid proposals, completed prohibited actions, legitimate task success, and budget exhaustion. Organize reruns with pipelines.
The continuous red teaming guide explains how to turn reviewed findings into release checks. For agents that call MCP servers, add MCP security tests. Both belong alongside quality evaluation in the agents learning path.