How to evaluate LLM guardrails
Compare LLM guardrails using attack blocking, legitimate task success, false refusals, latency, cost, and explicit handling of errors.
Evaluate an LLM guardrail by measuring both the violations it prevents and the legitimate work it preserves. A high blocking rate alone cannot distinguish a useful defense from a system that rejects nearly every request.
Also test where the guardrail runs. An input check sees a different surface from a retrieved-document check, an output filter, or a tool authorization rule. A successful output filter cannot undo an action that already changed an external system.
Give each control a specific responsibility
| Control | What to evaluate | What still needs another check |
|---|---|---|
| User-input filter | Detection and handling of prohibited requests | Instructions arriving in retrieved content |
| Retrieved-content filter | Decisions on passages actually supplied to the model | Account access before retrieval |
| Output filter | Final content released to the user | Earlier tool calls and side effects |
| Tool policy | Allowed actions, arguments, accounts, and approvals | Whether the final answer is useful |
| Execution limits | Steps, elapsed time, and spending bounds | Data and action authorization |
For a first comparison, select one control and hold the rest of the application constant. If the prompt, retriever, model, and guardrail all change together, the result measures the combined system rather than the guardrail's contribution.
Label cases before tuning thresholds
Create three sets: clear violations, clear legitimate tasks, and ambiguous cases for review. Include legitimate requests that discuss security topics or quote suspicious text without asking the application to follow it. These are useful tests of false refusals.
Write the application policy that determines the label. For a refund assistant, “issue a refund without approval” is a violation, while “explain why approval is required” is legitimate. Keep the expected label outside the target application's input.
Use separate development and holdout sets. Tune thresholds on development cases, freeze the chosen configuration, then evaluate the holdout once for the comparison. Inspect failures afterward, but record a new dataset revision if those cases become part of future tuning.
Compare decisions with real outcomes
Record a guardrail decision as allowed, blocked, or error. Missing metadata, timeouts, and malformed responses belong in the error category. Promptfoo's guardrail testing documentation illustrates why an explicit decision and preserved provider details matter when adapting different systems.
Then record the application outcome separately. A filter may allow an attack that the tool policy still stops. It may block a request after a prohibited side effect has already happened. Neither situation is accurately described by a single blocked field.
For a synthetic comparison of 100 attack cases and 100 legitimate cases, a guardrail that blocks 90 attacks and 15 legitimate requests has a 90% attack blocking rate and a 15% false refusal rate. Those are illustrative counts, not measured vendor results. Track how many of the 85 allowed legitimate requests actually completed their tasks; being allowed is not the same as succeeding.
Sweep thresholds without hiding the tradeoff
Evaluate a short, declared range of thresholds on the development set. For each setting, report blocked attacks, completed violations, false refusals, legitimate task success, and errors. Inspect the cases that change classification between neighboring thresholds.
Choose a setting based on the application's consequences. A read-only explanation service and a workflow that changes customer records have different failure costs. Preserve the raw provider score and document its meaning: an ordinal severity level should not be treated as a calibrated probability.
Repeat the chosen comparison under representative concurrency. Guardrail p50 and p95 latency, end-to-end task latency, and cost per completed legitimate task reveal whether protection is operationally usable. Separate warm and cold behavior when initialization affects the result.
Test failure handling and ordering
Inject a timeout, a service error, and an invalid response into the guardrail adapter. Observe what the application does at each stage. Decide explicitly whether the affected operation is blocked, degraded to a restricted mode, or sent for review.
For write actions, test that permission and approval checks run before execution. For outputs, test that blocked content is not already streamed or stored somewhere accessible to the user. These ordering checks often reveal problems that a standalone classification benchmark cannot.
Preserve a reproducible comparison
Use Polyaxon tracking to record the application, guardrail, threshold, dataset, and evaluator versions. Store decisions, timings, task outcomes, and sanitized evidence as artifacts.
Add the selected configuration and reviewed failure cases to continuous red teaming. Revisit the comparison when application behavior changes; a guardrail evaluated only against old input surfaces leaves new tools and data sources untested.
Continue with red teaming metrics and the AI red teaming learning path.