What is AI red teaming?
Design scoped adversarial tests for AI applications, examine tool and retrieval boundaries, and turn findings into regression coverage.
Red TeamingEvaluationAI Agents
All articles
Page 2 of 2
How to evaluate AI agents
A practical framework for evaluating AI agent outcomes, trajectories, tool use, safety, latency, and cost before and after release.
AI AgentsEvaluation
Find retrieval regressions before changing the prompt
Trace missing evidence through retrieval, reranking, and context assembly, then replay frozen context in Polyaxon to locate a RAG regression.
LLMOpsRAGEvaluation
Schedule agent evaluations and backfill missed days
Run recurring agent evaluations and controlled historical backfills with Polyaxon schedules, date-range matrices, explicit window identity, and fresh results.
AI AgentsEvaluationAutomation
Build an LLM pipeline to classify ML failure reports
Use Polyaxon DAGs, tracked outputs, artifacts, and run comparison to develop a reviewable classifier for ML workload failure reports.
LLMOpsDataOpsEvaluation
Build and evaluate LLM function calling
Connect bounded LLM tools to Polyaxon sandboxes, validate calls in the host, retain execution receipts, and compare tool behavior in repeatable evaluation runs.
AI AgentsToolsEvaluation
Build and evaluate AI agent guardrails in Polyaxon
Use Polyaxon to version guardrail experiments, compare case-level outcomes, constrain execution, and require evidence before promoting an agent release.
AI AgentsSecurityEvaluation
Cache pipeline steps without hiding LLM regressions
Cache deterministic Polyaxon preparation steps while forcing fresh LLM evaluations, using explicit dependency identities and separate cache policies.
AI AgentsEvaluationAutomation
Evaluate zero-shot prompting for production tasks
Use Polyaxon runs, versioned components, grid searches, case artifacts, and comparison views to qualify zero-shot prompts for production ML workflows.
PromptingLLMOpsEvaluation
Stop expensive agent evaluation sweeps early
Use Polyaxon failure and metric early-stopping rules to bound agent evaluation sweeps while preserving complete evidence and avoiding premature quality decisions.
AI AgentsEvaluationAutomation
Evaluate with only the data available at decision time
Reconstruct evaluation features using event time and actual availability time, excluding late arrivals and later corrections while preserving the source versions in Polyaxon.
DataOpsEvaluationData
Tune RAG retrieval with Polyaxon experiment matrices
Compare RAG chunk sizes, retrieval depth, and model choices with Polyaxon experiment matrices, fixed evaluation inputs, and quality-aware run comparisons.
AI AgentsEvaluationAutomation
Optimize LLM performance and cost with controlled experiments
Track usage and pricing assumptions in Polyaxon, compare cost against quality in run dashboards, and control resources and concurrency during LLM experiments.
LLMOpsEvaluationGuides
Correct evaluation labels and rescore saved predictions
Keep corrected labels as a new dataset version, reuse predictions when model inputs are unchanged, and compare evaluation runs in Polyaxon.
DataOpsMLOpsLineage
Evaluate a SageMaker endpoint from Polyaxon
Call an existing SageMaker endpoint from a Polyaxon job and retain model-specific predictions and evaluation results.
MLOpsSagemakerEvaluation