Turn production traces into regression tests
Turn a failure cluster into a reviewed regression case, preserve the causal tool response, and record baseline and candidate checks with Polyaxon.
LLMOpsObservabilityEvaluation
LLMOps
Learn how to evaluate, deploy, observe, and control the cost of LLM applications, from initial experiments to production feedback.
Start here
Essential guides, selected for this topic.
What is LLMOps? From prototype to production
LLMOps applies repeatable development, evaluation, deployment, and observability practices to production LLM applications and AI agents.
LLMOpsMLOps
LLM-as-a-judge: Design and validate model-based evaluators
Use LLMs as scalable evaluators without treating them as ground truth: design clear rubrics, calibrate against humans, and monitor bias and drift.
EvaluationLLMOps
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
Latest articles
The latest guides, tutorials, and product updates.
Reuse Hugging Face weights across Polyaxon runs
Prepare a pinned model revision in a CPU job, then reuse its files from read-only storage in Polyaxon training, notebook and inference workloads.
LLMOpsHugging FaceStorage
Optimize a small-model harness with autoresearch on Polyaxon
Build a measured research loop around Jev or a self-hosted classifier using Polyaxon jobs, sandboxes, experiment matrices, and per-case evidence.
LLMOpsAutoresearchClassification
Build multimodal search with EmbeddingGemma 2 on Polyaxon
Deploy EmbeddingGemma 2 as a Polyaxon GPU service, then use text or an image to search a mixed collection of documents and photos.
LLMOpsInferenceRAG
Extract invoice fields with Qwen-VL on Polyaxon
Serve an open vision-language model on your GPU cluster, extract typed invoice fields, and evaluate whether the values actually match the document.
LLMOpsInferenceEvaluation
Size GPU memory for LLM serving
Calculate Qwen3-4B weight and KV-cache storage, understand what the estimate excludes, and use a Polyaxon service to check the actual capacity.
LLMOpsInferenceGPU
Benchmark vLLM speculative decoding on Polyaxon
Compare ordinary decoding with prompt-lookup proposals under the same model, workload, and GPU allocation, including latency, failures, and output checks.
LLMOpsvLLMInference
Run batch inference with vLLM on Polyaxon
Generate predictions from a JSONL prompt file with vLLM, schedule the work on a Polyaxon GPU agent, and save the results as run artifacts.
LLMOpsvLLMInference
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
Train a decision classifier on Polyaxon
Fine-tune an open Qwen model for bounded support-routing decisions with Polyaxon jobs, track the adapter, and serve the reviewed version through vLLM.
LLMOpsFine TuningClassification
Deploy Laya typed decisions on Polyaxon
Serve the open Laya typed-decisions checkpoint through Polyaxon, with a CPU classifier endpoint, explicit model scope, and a review path for uncertain classifications.
LLMOpsInferenceClassification
Serve a Jev-style Qwen classifier on Polyaxon
Deploy an open Qwen model behind a typed classification API on Polyaxon, then check the decision schema and calibration before routing real work.
LLMOpsInferenceClassification
Serve DiffusionGemma on Polyaxon
Adapt Google's Gemma-on-Kubernetes deployment choices to Polyaxon with a DiffusionGemma vLLM service, a batch evaluation job, and an explicit compatibility checklist.
LLMOpsInferenceKubernetes
Separate evaluator changes from application improvements
Compare old and new application outputs under both evaluator versions, inspect changed decisions, and retain the four comparisons in Polyaxon.
LLMOpsEvaluationExperimentation
Make vLLM prefix caching work for repeated prompts
Understand exact prefix reuse in vLLM, structure repeated context, account for replica routing, and compare cold and warm workloads with Polyaxon.
LLMOpsInferencevLLM
Version the dataset behind every evaluation
Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.
DataOpsMLOpsLLMOps
Prevent data leakage in ML and LLM evaluation datasets
Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.
DataOpsMLOpsLLMOps
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Tune LLM hyperparameters within a compute budget
Design a small LoRA fine-tuning sweep, control its execution in Polyaxon, and compare validation quality, resource use, and repeatability.
Hyperparameter TuningLLMOpsGuides
Connect LLM production traces to evaluation runs
Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.
LLMOpsObservabilityEvaluation
Fine-tune Mistral 7B with LoRA on Kubernetes
Plan a Mistral 7B LoRA fine-tuning workflow on Kubernetes with versioned data, GPU scheduling, Polyaxon tracking, evaluation, and adapter packaging.
LLMOpsKubernetesGuides
Govern model access with an AI gateway
Operate an AI gateway as a policy boundary for workload identity, eligible models, routing, budgets, privacy, evidence, and controlled configuration changes.
LLMOpsGovernanceFinops
Run batch LLM evaluations on Kubernetes
Design reliable batch LLM evaluations with stable shards, bounded concurrency, GPU-aware placement, resumable results, and complete aggregation.
EvaluationKubernetesLLMOps
An AI agent deployment checklist for the first production release
Build an agent release checklist around Polyaxon qualification jobs, versioned components, artifact reports, termination settings, and manual approval.
AI AgentsLLMOpsGuides
Production LLM systems: Where to invest after the prototype
Use Polyaxon run tracking, comparison dashboards, resource monitoring, and repeatable evaluation to decide what to improve after an LLM prototype.
LLMOpsMLOpsGuides