Recover agent actions after a lost tool response
Reconcile uncertain tool effects, preserve approval and cancellation across task changes, and compare recovery policies using final-state evidence.
AI AgentsOrchestrationReliability
Latest articles
The latest guides, tutorials, and product updates.
Optimize a small-model harness with autoresearch on Polyaxon
Build a measured research loop around Jev or a self-hosted classifier using Polyaxon jobs, sandboxes, experiment matrices, and per-case evidence.
LLMOpsAutoresearchClassification
Autoresearch as a controlled experiment
A design note on autonomous experimentation: fixed protocols, selection bias, independent evaluation, and durable research campaigns in Polyaxon.
RoadmapAI AgentsExperimentation
Extract invoice fields with Qwen-VL on Polyaxon
Serve an open vision-language model on your GPU cluster, extract typed invoice fields, and evaluate whether the values actually match the document.
LLMOpsInferenceEvaluation
Separate evaluator changes from application improvements
Compare old and new application outputs under both evaluator versions, inspect changed decisions, and retain the four comparisons in Polyaxon.
LLMOpsEvaluationExperimentation
Keep failed candidates visible in pipeline reports
Reconcile evaluation results against an expected candidate manifest, preserve failed and missing outcomes, and separate pipeline reporting from release approval.
MLOpsOrchestrationEvaluation
Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
Connect LLM production traces to evaluation runs
Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.
LLMOpsObservabilityEvaluation
Retry failed evaluation shards with Indexed Jobs
Give evaluation shards stable indexes and independent retry budgets with Kubernetes Indexed Jobs, while keeping incomplete results visible.
KubernetesEvaluationPipelines
Run Promptfoo evaluations on Kubernetes with Polyaxon
Package a Promptfoo suite as a Polyaxon job, export evaluation reports, track artifacts, and verify that failed checks fail the workload.
PromptfooEvaluationKubernetes
Beyond agent traces: State, artifacts, and recovery
Connect agent traces to checkpoint state, versioned artifacts, action receipts, and evaluations so failures lead to controlled recovery and reproducible improvements.
AI AgentsObservabilityEvaluation
Design measurable AI agents before you build
Define agent outcomes, prohibited actions, events, evaluators, budgets, segments, and feedback loops before implementation begins.
AI AgentsEvaluationObservability
Run batch LLM evaluations on Kubernetes
Design reliable batch LLM evaluations with stable shards, bounded concurrency, GPU-aware placement, resumable results, and complete aggregation.
EvaluationKubernetesLLMOps
Build a shared library of agent evaluators
Package application-owned agent evaluators as versioned Polyaxon components with typed contracts, reusable execution profiles, and consistent result evidence.
AI AgentsEvaluationAutomation
Continuous AI red teaming in CI/CD
Turn AI security findings into repeatable CI checks with versioned cases, complete result manifests, explicit release gates, and retained evidence.
Red TeamingPipelinesEvaluation
Build guardrails for AI-generated code
Control what coding agents can read, change, execute, merge, and deploy with layered checks from context selection through production.
GuardrailsSecurityEvaluation
How to evaluate LLM guardrails
Compare LLM guardrails using attack blocking, legitimate task success, false refusals, latency, cost, and explicit handling of errors.
GuardrailsRed TeamingEvaluation
AI red teaming metrics: measuring failures and coverage
Measure AI red teaming with explicit attack success rates, attempt budgets, coverage, false refusals, severity, and evaluator uncertainty.
Red TeamingEvaluationMetrics
How to evaluate LLM routers for cost, quality, and latency
Test LLM routing policies with task-level quality, cost, latency, fallbacks, route stability, and model-aware production evidence.
LLMOpsEvaluationObservability
Red teaming RAG systems
Test RAG systems for poisoned documents, cross-tenant retrieval, stale permissions, citation leaks, and unauthorized tool actions.
Red TeamingRAGEvaluation
How to test prompt injection in LLM applications
Build prompt injection tests for user input, retrieved documents, and tool responses, with checks for data access and actual side effects.
Red TeamingEvaluationSecurity
Verified data pipelines for reliable AI agents
Design agent data paths that preserve source identity, versions, permissions, freshness, retrieval evidence, and verification results.
AI AgentsDataOpsEvaluation
Offline vs. online evaluation for generative AI
Use offline evaluation for reproducible release decisions and online evaluation for real production behavior, then connect both in one feedback loop.
EvaluationLLMOps