Evaluation articles

Connect LLM production traces to evaluation runs

Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.

Run Promptfoo evaluations on Kubernetes with Polyaxon

Package a Promptfoo suite as a Polyaxon job, export evaluation reports, track artifacts, and verify that failed checks fail the workload.

Beyond agent traces: State, artifacts, and recovery

Connect agent traces to checkpoint state, versioned artifacts, action receipts, and evaluations so failures lead to controlled recovery and reproducible improvements.

Design measurable AI agents before you build

Define agent outcomes, prohibited actions, events, evaluators, budgets, segments, and feedback loops before implementation begins.

Run batch LLM evaluations on Kubernetes

Design reliable batch LLM evaluations with stable shards, bounded concurrency, GPU-aware placement, resumable results, and complete aggregation.

Build a shared library of agent evaluators

Package application-owned agent evaluators as versioned Polyaxon components with typed contracts, reusable execution profiles, and consistent result evidence.

Continuous AI red teaming in CI/CD

Turn AI security findings into repeatable CI checks with versioned cases, complete result manifests, explicit release gates, and retained evidence.

Build guardrails for AI-generated code

Control what coding agents can read, change, execute, merge, and deploy with layered checks from context selection through production.

How to evaluate LLM guardrails

Compare LLM guardrails using attack blocking, legitimate task success, false refusals, latency, cost, and explicit handling of errors.

AI red teaming metrics: measuring failures and coverage

Measure AI red teaming with explicit attack success rates, attempt budgets, coverage, false refusals, severity, and evaluator uncertainty.

How to evaluate LLM routers for cost, quality, and latency

Test LLM routing policies with task-level quality, cost, latency, fallbacks, route stability, and model-aware production evidence.

Red teaming RAG systems

Test RAG systems for poisoned documents, cross-tenant retrieval, stale permissions, citation leaks, and unauthorized tool actions.

How to test prompt injection in LLM applications

Build prompt injection tests for user input, retrieved documents, and tool responses, with checks for data access and actual side effects.

Verified data pipelines for reliable AI agents

Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.

Offline vs. online evaluation for generative AI

Use offline evaluation for reproducible release decisions and online evaluation for real production behavior, then connect both in one feedback loop.

Build evaluation leaderboards with Polyaxon joins

Use Polyaxon joins to select comparable evaluation runs and build application-owned leaderboards with explicit cohort, completeness, and artifact evidence.

LLM-as-a-judge: Design and validate model-based evaluators

Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.

Turn production traces into regression tests

Convert representative production failures into sanitized, reproducible evaluation cases that protect future LLM and agent releases.

How to evaluate RAG systems

Evaluate retrieval and generation separately and end to end with representative datasets, groundedness checks, retrieval metrics, and production feedback.

What is AI red teaming?

Design scoped adversarial tests for AI applications, examine tool and retrieval boundaries, and turn findings into regression coverage.

How to evaluate AI agents

A practical framework for evaluating AI agent outcomes, trajectories, tool use, safety, latency, and cost before and after release.

Schedule agent evaluations and backfill missed days

Run recurring agent evaluations and controlled historical backfills with Polyaxon schedules, date-range matrices, explicit window identity, and fresh results.

Build an LLM pipeline to classify ML failure reports

Use Polyaxon DAGs, tracked outputs, artifacts, and run comparison to develop a reviewable classifier for ML workload failure reports.

Build and evaluate LLM function calling

Connect bounded LLM tools to Polyaxon sandboxes, validate calls in the host, retain execution receipts, and compare tool behavior in repeatable evaluation runs.