Run Promptfoo evaluations on Kubernetes with Polyaxon

You can run Promptfoo in a containerized Polyaxon job, store its exported report as a run artifact, and use the evaluation result to determine whether the job succeeds. This tutorial provides a small executable example of that workflow.

The example uses six deterministic tool-boundary cases: two legitimate controls and four prohibited actions. It does not call a model, spend API credits, or measure prompt injection resistance. Its purpose is to verify the runner, assertions, report handling, and failure gate before you connect a real application.

Get the example from GitHub

The runnable project lives in the AI security evaluations directory of polyaxon-examples. Clone the repository and enter that directory:

git clone https://github.com/polyaxon/polyaxon-examples.git
cd polyaxon-examples/ai-security-evals

The example contains:

The suite pins Promptfoo 0.122.2. Local execution requires Node.js 22.22.0 or newer and Python 3.9 or newer; the container uses Node 24 and installs Polyaxon 2.16.4. Check Promptfoo's runtime requirements before changing the pinned release.

For the cluster steps, you also need a configured Polyaxon deployment and CLI, a project and eligible queue, an artifact store, and a container registry reachable by your cluster. The local fixture needs no GPU.

Understand what the fixture checks

The provider prescribes an action for each case and evaluates it against synthetic trusted state. The cases cover reading one's own order, completing an independently approved refund, cross-tenant access, approval claimed in a document or memory, and a read-only handoff.

The assertions inspect the fixture's action ledger. For example, a denied refund must leave the refund list empty. A message claiming that the manager approved an action does not change the fixture's approval state.

Promptfoo's custom JavaScript provider interface lets an adapter return output for evaluation. Here that output is a JSON record of the decision and observed actions. The fixture deliberately does not infer an action from the supplied text, so it tests the execution boundary rather than the model's behavior.

Run the passing and failing cases locally

From the example directory:

npm ci
python3 run.py --local-output ./reports-safe

Use a fresh output directory each time; the wrapper rejects an existing report to prevent stale evidence from satisfying a new run.

Then enable the fixture's intentional defect, which permits every prescribed action in memory:

FIXTURE_MODE=unsafe python3 run.py --local-output ./reports-unsafe

The local check for this article used Promptfoo 0.122.2, Node 25.1.0, one attempt per case, concurrency two, no result cache, and no model provider:

These are fixture results, not an AI security benchmark. The failing run is expected and confirms that the gate catches the controlled defect.

Preserve results before returning failure

The wrapper invokes promptfoo eval with JSON export, caching disabled, and history writes disabled. It compares observed case IDs against the six expected IDs before accepting a pass. A missing, duplicate, or malformed result cannot silently satisfy that completeness check.

It writes three files: results.json, manifest.json, and status.json. The manifest records fixture mode and hashes of the cases, provider, and configuration. The status preserves Promptfoo's exit code and the wrapper's gate decision. The Promptfoo CLI reference documents export options and exit behavior.

Without --local-output, the wrapper initializes Polyaxon tracking, writes beneath the run's outputs path, registers available reports with artifact logging, and logs counts. It returns failure only after that reporting step. If the process cannot produce a valid report, the available status and manifest still explain the failed run. Artifact upload failures also prevent a successful job.

Telemetry and update checks are disabled in the wrapper using Promptfoo's documented environment settings. This fixture makes no provider calls. Adding a remote target or model judge changes its network and data requirements.

Build and submit the job

Replace the registry path with one you control. Build for your cluster's architecture, push the image, and use an immutable digest for repeatable cluster runs:

docker build -t registry.example.com/ai/security-evals:v1 .
docker push registry.example.com/ai/security-evals:v1

The supplied component accepts the image as an input and requests CPU and memory only. Submit it with your project and queue:

polyaxon run -p YOUR_PROJECT -f polyaxonfile.yaml \
  -P image=registry.example.com/ai/security-evals@sha256:YOUR_IMAGE_DIGEST \
  -q YOUR_AGENT/YOUR_QUEUE

Configure registry credentials and workload identity through your deployment's existing policies. The component quick start and job reference explain the workload structure.

In the run, inspect the report artifacts and gate_passed metric. Preserve the image digest with the run input. The local checks above do not verify your registry, cluster admission policies, or artifact-store connectivity; validate those with the first submitted fixture job.

Connect an actual application

Replace the fixture provider with an adapter that calls your authorized test deployment. Keep authenticated test identity and approvals in the test service; do not grant permissions based on a field supplied by the model. Return independently observed tool actions and task outcomes for assertions.

The existing suite's exact completeness check assumes one provider and one attempt per case. Extend the result identity to include provider and repetition before adding either. Add request timeouts, explicit transport errors, and bounded concurrency appropriate to the endpoint.

Use the prompt injection guide to build real application cases and continuous red teaming to define a release policy. For larger suites, continue with batch LLM evaluations in the LLM evaluation path.