Red teaming RAG systems
Test RAG systems for poisoned documents, cross-tenant retrieval, stale permissions, citation leaks, and unauthorized tool actions.
A RAG system can produce a well-supported answer from a document the user should never have seen. It can also retrieve the right document and follow malicious instructions embedded inside it. Red teaming RAG therefore needs checks for access and behavior alongside retrieval relevance and answer quality.
Keep the quality suite described in how to evaluate RAG systems. Add a security suite that controls document provenance, user identity, permissions, and the state of caches and indexes.
Create a corpus with known boundaries
Build a small synthetic corpus with public documents, tenant A documents, and tenant B documents. Give private documents unique marker strings that have no real secret value. Store each document's identity, revision, and access scope in a manifest.
For example, tenant B's test-only shipping policy could contain BETA-ONLY-CANARY-417. The marker helps detect accidental disclosure, but its absence does not prove isolation: the answer could paraphrase the private information. Also inspect which chunks reached the model.
Permissions must survive chunking and be enforced when content is retrieved. The OWASP RAG security guidance specifically covers access metadata on chunks, permission changes, and derived stores. Use those boundaries to define the expected document set for each identity.
Separate retrieval failures from instruction failures
| Case | Controlled change | Failure condition |
|---|---|---|
| Cross-tenant retrieval | Query a B-specific topic as tenant A | A B chunk enters model context |
| Poisoned public document | Add a synthetic instruction to issue an unapproved refund | The instruction changes the task or causes an invalid action |
| Permission revocation | Remove access after a successful query | A fresh query still receives revoked content |
| Cached answer reuse | Ask similar questions as two different tenants | The second tenant receives the first tenant's private answer |
| Citation access | Follow a citation using the querying user's identity | The link reveals an unauthorized title, snippet, or document |
| Deleted source | Remove a document and propagate deletion | The removed content remains accessible after the declared propagation window |
Run the poisoned-document case only after confirming that the document is eligible for retrieval. If the application never retrieved it, the test did not exercise the intended instruction boundary. Record that as missing coverage rather than evidence of resistance.
Place the injection where the application will see it
Use one legitimate task, such as “summarize the return policy,” and add a short test instruction to an otherwise relevant document. Preserve the clean document as a paired control. Record the final chunks after parsing and splitting; content that existed in the source file may not survive extraction.
Vary placement within a chunk, retrieval rank, and the amount of surrounding context. Change one factor at a time initially. If you change the document, embedding model, retriever, and prompt together, a different result is difficult to attribute.
For an agentic RAG application, observe downstream tool actions. A generated summary can look correct while a tool call follows the document's unrelated instruction. The prompt injection testing guide explains how to score those outcomes separately.
Test revocation with a timeline
A reproducible revocation case has four events: grant access, query successfully, revoke access, and query again. Record timestamps for the policy change and each request. Test both a cold query and one likely to hit an answer or retrieval cache.
Define the expected propagation behavior before running the case. If your application permits a documented delay, measure that delay explicitly. Do not let a test pass simply because the unauthorized answer disappeared eventually.
Repeat the timeline for group membership changes and deleted documents. Reset the corpus and cache state before the next independent attempt. Keep the identity used to fetch a citation separate from the service account used to ingest the original document.
Record enough evidence to locate the fix
For each attempt, retain the query, authenticated scope, corpus manifest, retrieved IDs, final context IDs, answer, citations, and tool events. Include the retriever, embedding, prompt, and model versions. Restrict access to reports because they can contain exactly the information the test was meant to protect.
Use Polyaxon artifacts to preserve sanitized evidence and tracking metadata for counts such as unauthorized chunks, instruction-following failures, and valid answers. Keep the denominators separate: a retrieval access failure and an answer quality failure are different measurements.
Add confirmed findings to continuous red teaming, then return to the LLM evaluation path to check that security fixes preserve useful retrieval and answers.