Separate evaluator changes from application improvements
Compare old and new application outputs under both evaluator versions, inspect changed decisions, and retain the four comparisons in Polyaxon.
LLMOpsEvaluationExperimentation
Latest articles
The latest guides, tutorials, and product updates.
Keep failed candidates visible in pipeline reports
Reconcile evaluation results against an expected candidate manifest, preserve failed and missing outcomes, and separate pipeline reporting from release approval.
MLOpsOrchestrationEvaluation
Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
Connect LLM production traces to evaluation runs
Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.
LLMOpsObservabilityEvaluation
Retry failed evaluation shards with Indexed Jobs
Give evaluation shards stable indexes and independent retry budgets with Kubernetes Indexed Jobs, while keeping incomplete results visible.
KubernetesEvaluationPipelines
Run Promptfoo evaluations on Kubernetes with Polyaxon
Package a Promptfoo suite as a Polyaxon job, export evaluation reports, track artifacts, and verify that failed checks fail the workload.
PromptfooEvaluationKubernetes
Beyond agent traces: State, artifacts, and recovery
Connect agent traces to checkpoint state, versioned artifacts, action receipts, and evaluations so failures lead to controlled recovery and reproducible improvements.
AI AgentsObservabilityEvaluation
Design measurable AI agents before you build
Define agent outcomes, prohibited actions, events, evaluators, budgets, segments, and feedback loops before implementation begins.
AI AgentsEvaluationObservability
Run batch LLM evaluations on Kubernetes
Design reliable batch LLM evaluations with stable shards, bounded concurrency, GPU-aware placement, resumable results, and complete aggregation.
EvaluationKubernetesLLMOps
Build a shared library of agent evaluators
Package application-owned agent evaluators as versioned Polyaxon components with typed contracts, reusable execution profiles, and consistent result evidence.
AI AgentsEvaluationAutomation
Continuous AI red teaming in CI/CD
Turn AI security findings into repeatable CI checks with versioned cases, complete result manifests, explicit release gates, and retained evidence.
Red TeamingPipelinesEvaluation
Build guardrails for AI-generated code
Control what coding agents can read, change, execute, merge, and deploy with layered checks from context selection through production.
GuardrailsSecurityEvaluation
How to evaluate LLM guardrails
Compare LLM guardrails using attack blocking, legitimate task success, false refusals, latency, cost, and explicit handling of errors.
GuardrailsRed TeamingEvaluation
AI red teaming metrics: measuring failures and coverage
Measure AI red teaming with explicit attack success rates, attempt budgets, coverage, false refusals, severity, and evaluator uncertainty.
Red TeamingEvaluationMetrics
How to evaluate LLM routers for cost, quality, and latency
Test LLM routing policies with task-level quality, cost, latency, fallbacks, route stability, and model-aware production evidence.
LLMOpsEvaluationObservability
Red teaming RAG systems
Test RAG systems for poisoned documents, cross-tenant retrieval, stale permissions, citation leaks, and unauthorized tool actions.
Red TeamingRAGEvaluation
How to test prompt injection in LLM applications
Build prompt injection tests for user input, retrieved documents, and tool responses, with checks for data access and actual side effects.
Red TeamingEvaluationSecurity
Verified data pipelines for reliable AI agents
Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.
AI AgentsDataOpsEvaluation
Offline vs. online evaluation for generative AI
Use offline evaluation for reproducible release decisions and online evaluation for real production behavior, then connect both in one feedback loop.
EvaluationLLMOps
Build evaluation leaderboards with Polyaxon joins
Use Polyaxon joins to select comparable evaluation runs and build application-owned leaderboards with explicit cohort, completeness, and artifact evidence.
AI AgentsEvaluationAutomation
LLM-as-a-judge: Design and validate model-based evaluators
Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.
EvaluationLLMOps
Turn production traces into regression tests
Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.
LLMOpsObservabilityEvaluation
How to evaluate RAG systems
Evaluate retrieval and generation separately and end to end with representative datasets, groundedness checks, retrieval metrics, and production feedback.
RAGEvaluation