Run batch inference with vLLM on Polyaxon
Generate predictions from a JSONL prompt file with vLLM, schedule the work on a Polyaxon GPU agent, and save the results as run artifacts.
LLMOpsvLLMInference
Latest articles
The latest guides, tutorials, and product updates.
Keep inference available during Kubernetes node drains
Set a PodDisruptionBudget around usable inference capacity, inspect blocked evictions, and account for replacement GPUs and model warmup.
KubernetesInferenceReliability
Serve multiple LoRA adapters with vLLM on Polyaxon
Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.
LLMOpsvLLMInference
Spread inference replicas across Kubernetes zones
Use topology spread constraints for inference availability, understand minDomains and GPU capacity, and preserve placement intent in Polyaxon.
KubernetesSchedulingInference
Deploy Laya typed decisions on Polyaxon
Serve the open Laya typed-decisions checkpoint through Polyaxon, with a CPU classifier endpoint, explicit model scope, and a review path for uncertain classifications.
LLMOpsInferenceClassification
Serve a Jev-style Qwen classifier on Polyaxon
Deploy an open Qwen model behind a typed classification API on Polyaxon, then check the decision schema and calibration before routing real work.
LLMOpsInferenceClassification
Serve DiffusionGemma on Polyaxon
Adapt Google's Gemma-on-Kubernetes deployment choices to Polyaxon with a DiffusionGemma vLLM service, a batch evaluation job, and an explicit compatibility checklist.
LLMOpsInferenceKubernetes
Make vLLM prefix caching work for repeated prompts
Understand exact prefix reuse in vLLM, structure repeated context, account for replica routing, and compare cold and warm workloads with Polyaxon.
LLMOpsInferencevLLM
Optimize LLM inference with repeatable benchmarks
Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.
LLMOpsInferenceGuides
Control notebook and inference service lifetimes
Reuse Polyaxon's termination specification to bound notebook and inference service lifetimes, stop idle services, and distinguish inactivity from ongoing work.
PolyaxonSchedulingNotebooks
Mount model artifacts with Kubernetes image volumes
Separate immutable model files from the serving image with Kubernetes image volumes, and choose when an OCI artifact fits better than a download or PVC.
KubernetesStorageInference
Autoscale inference services using workload metrics
Use custom metrics with Kubernetes HPA for independent inference replicas, account for model startup, and distinguish desired replicas from usable capacity.
KubernetesInferenceMonitoring
Roll out a new model when every GPU is occupied
Plan GPU inference rollouts around surge capacity, temporary reduced availability, model warmup, and request draining with a worked Deployment scenario.
KubernetesInferenceDeployment
Share CPU and memory across containers in a Pod
Use Pod-level resource budgets for cooperating containers, understand the remaining isolation boundaries, and evaluate the fit for ML services.
KubernetesResourcesInference
Self-hosted vs. managed AI inference
Choose between self-hosted and managed AI inference using data policy, model access, latency, utilization, reliability, staffing, and cost per accepted outcome.
InferenceInfrastructureFinops
Serve DeepSeek V4 on Polyaxon
Configure a DeepSeek V4 Pro service on B200 GPUs and check model loading, reasoning output, tool calls, and resource use.
LLMOpsInference
Compare serverless and dedicated LLM inference with Polyaxon
Use Polyaxon components, matrices, tracking, and run comparisons to evaluate serverless model APIs against dedicated inference services on quality, latency, and cost.
LLMOpsInferenceInfrastructure
Serve Qwen 3.6 on Polyaxon
Deploy Qwen3.6-27B with SGLang, then check the model's API behavior, memory use, and response quality before sending it production traffic.
LLMOpsInference
When CPU-accelerated AI inference makes sense
Evaluate CPU inference using model fit, precision, latency, throughput, memory, Kubernetes placement, and end-to-end cost instead of assuming every model needs a GPU.
InferenceKubernetesPerformance