Continuous AI red teaming in CI/CD
Turn AI security findings into repeatable CI checks with versioned cases, complete result manifests, explicit release gates, and retained evidence.
Continuous AI red teaming combines ongoing discovery with repeatable checks for known failures. Discovery finds new ways an application can cross a boundary. A release suite verifies that reviewed cases remain fixed when a model, prompt, tool, retrieval system, or permission changes.
Use separate datasets for those jobs. A changing discovery set helps you learn; a fixed regression set helps you compare. Moving newly generated cases directly into a release gate without review can turn ambiguous labels or broken fixtures into deployment blockers.
Choose checks by the change being released
| Change | Focused checks | Shared checks |
|---|---|---|
| Prompt or model | Instruction handling, legitimate task completion | Known consequential failures |
| Tool implementation | Arguments, authorization, side effects, retries | End-to-end agent tasks |
| Retrieval pipeline | Tenant scope, poisoned passages, cache identity | Answer usefulness and citations |
| Memory or handoffs | Identity propagation, stale permissions, persistence | Execution budgets and actions |
| Runtime or credentials | Storage, network, service account access | Tool permissions under the new environment |
A small suite can run on each relevant change. A larger scheduled discovery job can explore additional inputs and repeated attempts. Reserve a reviewed holdout set for release decisions so repeatedly tuning against the same examples does not become the entire evaluation strategy.
Version the complete test contract
Keep stable case IDs and a dataset revision. Each case needs initial state, input, expected outcome, severity, and an evaluator definition. Record the application's revision and image digest, model identifier, generation settings, and dependency versions alongside the run.
The result manifest should list expected case IDs and repetition counts before execution starts. Afterward, compare it with observed results. Missing rows, duplicate attempts, malformed output, and setup failures must remain visible. A report containing three passes is not a successful six-case suite.
When behavior is stochastic, choose an attempt budget before comparing candidates. Use the same budget and stopping rule for the baseline. Store transport retries separately from deliberate additional model samples.
Make the release decision explicit
The following is an example policy for a synthetic support application. Its thresholds are a design choice, not an industry standard:
{
"policy_version": "support-release-v1",
"required_cases": 6,
"attempts_per_case": 1,
"maximum_completed_unauthorized_actions": 0,
"minimum_legitimate_control_passes": 2,
"maximum_execution_errors": 0,
"require_complete_evidence": true
}Apply the completeness check before computing quality or security rates. Then evaluate hard boundaries, followed by legitimate task behavior. Do not average a prohibited action into a broad quality score where many easy successes could hide it.
Use separate outcomes for passed, failed, and inconclusive. If a required tool fixture or evaluator is unavailable, the release check is inconclusive and should not authorize promotion. Route disputed judgments to review with the original evidence and the rubric version.
Preserve reports when a check fails
Evaluation runners commonly signal failures through a nonzero exit code. Capture that status, retain the report, and then return failure to the caller. Uploading artifacts only after a successful command loses the evidence you most need.
The Promptfoo CLI distinguishes failed evaluations from other execution errors. The Polyaxon tutorial includes a wrapper that records the status and exports available reports before the job exits. Confirm the behavior against the pinned runner version in your own image.
Also test the gate itself. Introduce a controlled defect in an in-memory fixture and verify that the suite fails. Remove a result row and verify that completeness fails. These exercises establish that a green job is connected to the policy you intended to enforce.
Keep discovery findings reviewable
A discovery finding should include the affected boundary, a minimal reproduction, the original trajectory, severity, and an owner. Check whether the action actually occurred and whether the expected policy was applicable. Sanitize production-derived examples before adding them to a shared corpus.
After remediation, run the original case, nearby variations, and paired legitimate controls. Promote reviewed cases into the fixed suite with a new dataset revision. The trace-to-regression guide covers case minimization and dataset maintenance in more detail.
Orchestrate the work with Polyaxon
Use pipelines to separate setup, execution, scoring, and promotion. Scheduled runs can run broader suites, while run artifacts preserve reports and manifests.
Make promotion depend on your explicit gate result; recording a metric does not by itself enforce a release policy. Give evaluation jobs test credentials, and keep production promotion credentials in the separate authorized step.
Continue with red teaming metrics or the full AI red teaming path.