Evaluation articles

Separate evaluator changes from application improvements

Compare old and new application outputs under both evaluator versions, inspect changed decisions, and retain the four comparisons in Polyaxon.

Keep failed candidates visible in pipeline reports

Reconcile evaluation results against an expected candidate manifest, preserve failed and missing outcomes, and separate pipeline reporting from release approval.

Version the dataset behind every evaluation

Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.

Prevent data leakage in ML and LLM evaluation datasets

Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.

Connect LLM production traces to evaluation runs

Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.

Retry failed evaluation shards with Indexed Jobs

Give evaluation shards stable indexes and independent retry budgets with Kubernetes Indexed Jobs, while keeping incomplete results visible.

Run Promptfoo evaluations on Kubernetes with Polyaxon

Package a Promptfoo suite as a Polyaxon job, export evaluation reports, track artifacts, and verify that failed checks fail the workload.

Beyond agent traces: State, artifacts, and recovery

Connect agent traces to checkpoint state, versioned artifacts, action receipts, and evaluations so failures lead to controlled recovery and reproducible improvements.

Design measurable AI agents before you build

Define agent outcomes, prohibited actions, events, evaluators, budgets, segments, and feedback loops before implementation begins.

Run batch LLM evaluations on Kubernetes

Design reliable batch LLM evaluations with stable shards, bounded concurrency, GPU-aware placement, resumable results, and complete aggregation.

Build a shared library of agent evaluators

Package application-owned agent evaluators as versioned Polyaxon components with typed contracts, reusable execution profiles, and consistent result evidence.

Continuous AI red teaming in CI/CD

Turn AI security findings into repeatable CI checks with versioned cases, complete result manifests, explicit release gates, and retained evidence.

Build guardrails for AI-generated code

Control what coding agents can read, change, execute, merge, and deploy with layered checks from context selection through production.

How to evaluate LLM guardrails

Compare LLM guardrails using attack blocking, legitimate task success, false refusals, latency, cost, and explicit handling of errors.

AI red teaming metrics: measuring failures and coverage

Measure AI red teaming with explicit attack success rates, attempt budgets, coverage, false refusals, severity, and evaluator uncertainty.

How to evaluate LLM routers for cost, quality, and latency

Test LLM routing policies with task-level quality, cost, latency, fallbacks, route stability, and model-aware production evidence.

Red teaming RAG systems

Test RAG systems for poisoned documents, cross-tenant retrieval, stale permissions, citation leaks, and unauthorized tool actions.

How to test prompt injection in LLM applications

Build prompt injection tests for user input, retrieved documents, and tool responses, with checks for data access and actual side effects.

Verified data pipelines for reliable AI agents

Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.

Offline vs. online evaluation for generative AI

Use offline evaluation for reproducible release decisions and online evaluation for real production behavior, then connect both in one feedback loop.

Build evaluation leaderboards with Polyaxon joins

Use Polyaxon joins to select comparable evaluation runs and build application-owned leaderboards with explicit cohort, completeness, and artifact evidence.

LLM-as-a-judge: Design and validate model-based evaluators

Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.

Turn production traces into regression tests

Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.

How to evaluate RAG systems

Evaluate retrieval and generation separately and end to end with representative datasets, groundedness checks, retrieval metrics, and production feedback.