Inference

Practical guides, tutorials, and updates from the Polyaxon team.

Latest articles

The latest guides, tutorials, and product updates.

One replica model for inference, training, and RL

Polyaxon's roadmap for named replica groups, heterogeneous services, coordinated training, and RL workloads across Kubernetes scheduling backends.

RoadmapPolyaxonKubernetes

Deploy Qwen3 Reranker on Polyaxon

Deploy Qwen3-Reranker-0.6B as a Polyaxon GPU service, rerank a text-search shortlist, and compare relevance against added latency.

LLMOpsInferenceRAG

Spread inference replicas across Kubernetes zones

Use topology spread constraints for inference availability, understand minDomains and GPU capacity, and preserve placement intent in Polyaxon.

KubernetesSchedulingInference

Build multimodal search with EmbeddingGemma 2 on Polyaxon

Deploy EmbeddingGemma 2 as a Polyaxon GPU service, then use text or an image to search a mixed collection of documents and photos.

LLMOpsInferenceRAG

Extract invoice fields with Qwen-VL on Polyaxon

Serve an open vision-language model on your GPU cluster, extract typed invoice fields, and evaluate whether the values actually match the document.

LLMOpsInferenceEvaluation

Size GPU memory for LLM serving

Calculate Qwen3-4B weight and KV-cache storage, understand what the estimate excludes, and use a Polyaxon service to check the actual capacity.

LLMOpsInferenceGPU

Benchmark vLLM speculative decoding on Polyaxon

Compare ordinary decoding with prompt-lookup proposals under the same model, workload, and GPU allocation, including latency, failures, and output checks.

LLMOpsvLLMInference

Run batch inference with vLLM on Polyaxon

Generate predictions from a JSONL prompt file with vLLM, schedule the work on a Polyaxon GPU agent, and save the results as run artifacts.

LLMOpsvLLMInference

Keep inference available during Kubernetes node drains

Set a PodDisruptionBudget around usable inference capacity, inspect blocked evictions, and account for replacement GPUs and model warmup.

KubernetesInferenceReliability

Serve multiple LoRA adapters with vLLM on Polyaxon

Deploy two task-specific LoRA adapters over one Qwen base model, select them by name, and measure quality and shared GPU capacity with Polyaxon.

LLMOpsvLLMInference

Deploy Laya typed decisions on Polyaxon

Serve the open Laya typed-decisions checkpoint through Polyaxon, with a CPU classifier endpoint, explicit model scope, and a review path for uncertain classifications.

LLMOpsInferenceClassification

Serve a Jev-style Qwen classifier on Polyaxon

Deploy an open Qwen model behind a typed classification API on Polyaxon, then check the decision schema and calibration before routing real work.

LLMOpsInferenceClassification

Serve DiffusionGemma on Polyaxon

Adapt Google's Gemma-on-Kubernetes deployment choices to Polyaxon with a DiffusionGemma vLLM service, a batch evaluation job, and an explicit compatibility checklist.

LLMOpsInferenceKubernetes

Make vLLM prefix caching work for repeated prompts

Understand exact prefix reuse in vLLM, structure repeated context, account for replica routing, and compare cold and warm workloads with Polyaxon.

LLMOpsInferencevLLM

Optimize LLM inference with repeatable benchmarks

Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.

LLMOpsInferenceGuides

Control notebook and inference service lifetimes

Reuse Polyaxon's termination specification to bound notebook and inference service lifetimes, stop idle services, and distinguish inactivity from ongoing work.

PolyaxonSchedulingNotebooks

Mount model artifacts with Kubernetes image volumes

Separate immutable model files from the serving image with Kubernetes image volumes, and choose when an OCI artifact fits better than a download or PVC.

KubernetesStorageInference

Autoscale inference services using workload metrics

Use custom metrics with Kubernetes HPA for independent inference replicas, account for model startup, and distinguish desired replicas from usable capacity.

KubernetesInferenceMonitoring

Roll out a new model when every GPU is occupied

Plan GPU inference rollouts around surge capacity, temporary reduced availability, model warmup, and request draining with a worked Deployment scenario.

KubernetesInferenceDeployment

Share CPU and memory across containers in a Pod

Use Pod-level resource budgets for cooperating containers, understand the remaining isolation boundaries, and evaluate the fit for ML services.

KubernetesResourcesInference

Self-hosted vs. managed AI inference

Choose between self-hosted and managed AI inference using data policy, model access, latency, utilization, reliability, staffing, and cost per accepted outcome.

InferenceInfrastructureFinops

Serve DeepSeek V4 on Polyaxon

Configure a DeepSeek V4 Pro service on B200 GPUs and check model loading, reasoning output, tool calls, and resource use.

LLMOpsInference

Compare serverless and dedicated LLM inference with Polyaxon

Use Polyaxon components, matrices, tracking, and run comparisons to evaluate serverless model APIs against dedicated inference services on quality, latency, and cost.

LLMOpsInferenceInfrastructure

Serve Qwen 3.6 on Polyaxon

Deploy Qwen3.6-27B with SGLang, then check the model's API behavior, memory use, and response quality before sending it production traffic.

LLMOpsInference