LLM-as-a-judge: Design and validate model-based evaluators
Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.
EvaluationLLMOps
LLMOps
Learn how to evaluate, deploy, observe, and control the cost of LLM applications, from initial experiments to production feedback.
All articles
Page 2 of 2
Extend your MLOps workflow to AI agent development
Package an agent evaluator as a Polyaxon component, track candidate revisions and task metrics, and compare changes using existing MLOps workflows.
AI AgentsMLOpsLLMOps
Turn production traces into regression tests
Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.
LLMOpsObservabilityEvaluation
Find retrieval regressions before changing the prompt
Trace missing evidence through retrieval, reranking, and context assembly, then replay frozen context in Polyaxon to locate a RAG regression.
LLMOpsRAGEvaluation
What is LLMOps? From prototype to production
LLMOps applies repeatable development, evaluation, deployment, and observability practices to production LLM applications and AI agents.
LLMOpsMLOps
Serve DeepSeek V4 on Polyaxon
Configure a DeepSeek V4 Pro service on B200 GPUs and check model loading, reasoning output, tool calls, and resource use.
LLMOpsInference
Observe coding agents from prompt to sandbox execution
Use Polyaxon tracking, sandbox execution receipts, artifacts, and run comparison to investigate coding-agent failures and qualify changes.
AI AgentsObservabilityLLMOps
Build an AI agent governance workflow in Polyaxon
Use Polyaxon component versions, evaluation runs, pipeline dependencies, manual approval, and scoped execution to make agent releases reviewable.
AI AgentsGovernanceLLMOps
Build an LLM pipeline to classify ML failure reports
Use Polyaxon DAGs, tracked outputs, artifacts, and run comparison to develop a reviewable classifier for ML workload failure reports.
LLMOpsDataOpsEvaluation
Compare serverless and dedicated LLM inference with Polyaxon
Use Polyaxon components, matrices, tracking, and run comparisons to evaluate serverless model APIs against dedicated inference services on quality, latency, and cost.
LLMOpsInferenceInfrastructure
Serve Qwen 3.6 on Polyaxon
Deploy Qwen3.6-27B with SGLang, then check the model's API behavior, memory use, and response quality before sending it production traffic.
LLMOpsInference
Evaluate zero-shot prompting for production tasks
Use Polyaxon runs, versioned components, grid searches, case artifacts, and comparison views to qualify zero-shot prompts for production ML workflows.
PromptingLLMOpsEvaluation
Secure the data lifecycle of production LLM applications
Configure Polyaxon workload credentials, service accounts, connections, and artifact handling while protecting LLM data across providers, caches, and evaluation.
LLMOpsSecurityGovernance
Turn LLM experiments into a reusable engineering knowledge base
Use Polyaxon run queries, comparison dashboards, artifact reports, and lineage to preserve the evidence behind LLM engineering decisions.
LLMOpsGuides
Optimize LLM performance and cost with controlled experiments
Track usage and pricing assumptions in Polyaxon, compare cost against quality in run dashboards, and control resources and concurrency during LLM experiments.
LLMOpsEvaluationGuides
LLM agents: architecture and production operations
Map LLM agent reasoning, tools, execution, evaluation, and release workflows to Polyaxon services, sandboxes, jobs, components, and DAGs.
AI AgentsLLMOpsOrchestration
An evidence-based LLMOps maturity assessment
Assess LLMOps practices using Polyaxon run evidence: versioned components, comparison reports, artifact lineage, scheduling presets, and release approvals.
LLMOpsMLOpsGovernance
Design search, retrieval, and recommendations with LLMs
Compare lexical, dense, and hybrid retrieval with a Polyaxon grid search, then inspect ranking quality, answer evidence, latency, and recommendation outcomes.
LLMOpsRAGGuides
Build knowledge-grounded LLM applications
Build and evaluate versioned RAG indexes with Polyaxon operations, resource configuration, connections, tracked manifests, and case-level reports.
RAGLLMOpsGuides