Blog

Make vLLM prefix caching work for repeated prompts

Understand exact prefix reuse in vLLM, structure repeated context, account for replica routing, and compare cold and warm workloads with Polyaxon.

Resize CPU and memory without replacing Kubernetes Pods

Use Kubernetes in-place resource resizing for running workloads, inspect whether changes took effect, and understand the limits for Polyaxon services.

Native gang scheduling reaches beta in Kubernetes 1.37

Explore native gang scheduling in Kubernetes 1.37, its potential for Polyaxon training and sandboxes, and how it compares with KAI, Kueue, and Volcano.

Keep failed candidates visible in pipeline reports

Reconcile evaluation results against an expected candidate manifest, preserve failed and missing outcomes, and separate pipeline reporting from release approval.

Work inside a team from Python and the CLI

Scope Polyaxon Python clients and CLI commands to a team, review runs across its projects, and keep explicit project targets and saved defaults clear.

Design a search space before launching the sweep

Use Polyaxon matrices to choose parameter scales, separate meaningful combinations, and control the size of a hyperparameter sweep.

Version the dataset behind every evaluation

Use Polyaxon data references, artifact logging, and registered versions to retain the dataset and split behind each evaluation.

Keep distributed training workers close together

Understand native topology-aware workload scheduling, combine rack locality with gang placement, and assess the tradeoff between waiting and communication.

Resume interrupted training without losing progress

Build recoverable PyTorch training jobs with complete checkpoints, durable storage, and explicit restoration. Practice recovery locally and with Polyaxon.

Work interactively with Polyaxon sandboxes

Choose between sandbox shells and exec, understand PTY lifetime and output replay, and manage terminal attachment explicitly through the Python SDK.

Did you find better hyperparameters—or a lucky seed?

Confirm hyperparameter finalists across matched training seeds, compare variation and paired differences, and reserve compute for a defensible final choice.

Connect your local tools to Polyaxon with SSH

Connect native terminals and IDEs to a Polyaxon service, forward application ports, and use tmux explicitly when you want to resume a shell.

Prevent data leakage in ML and LLM evaluation datasets

Choose evaluation boundaries, keep related examples together, audit duplicate overlap, and retain reproducible split manifests for ML and LLM experiments.

Keep long-running workloads authenticated through token rotation

Use projected ServiceAccount tokens and refresh-aware clients so long-running training, notebooks, and services can keep authenticating without static credentials.

Reconnect to your Polyaxon shell

Learn how ordinary ops shells differ from tmux sessions, then detach and reconnect to the same Polyaxon shell from the CLI or UI.

Enforce ML workload rules with CEL admission policies

Use Kubernetes ValidatingAdmissionPolicy to check workload ownership and image references, with scoped warning and enforcement stages for ML namespaces.

Optimize LLM inference with repeatable benchmarks

Compare inference optimizations against a fixed workload, latency limits, and quality checks, with benchmark configurations and results tracked in Polyaxon.

Combine run filters with AND and OR

Use OR to combine metric thresholds, negated conditions, and independent groups of filters in Polyaxon queries.

Control notebook and inference service lifetimes

Reuse Polyaxon's termination specification to bound notebook and inference service lifetimes, stop idle services, and distinguish inactivity from ongoing work.

Automate multiple runs with async Python clients

Monitor several Polyaxon runs concurrently with async Python clients, retrieve recent logs, and manage concurrency, timeouts, and client cleanup.

Tune LLM hyperparameters within a compute budget

Design a small LoRA fine-tuning sweep, control its execution in Polyaxon, and compare validation quality, resource use, and repeatability.

Control Kubernetes Job retries with pod failure policies

Stop retrying permanent errors, preserve the retry budget for marked disruptions, and inspect Kubernetes Job failure decisions with a concrete example.

Write shorter Polyaxonfiles

Use direct parameter values and let input and output defaults imply optionality, with a look ahead at simpler workload definitions planned for Polyaxon v3.

Connect LLM production traces to evaluation runs

Keep release, dataset, evaluator, and sampling context connected as you investigate LLM failures and compare fixes with Polyaxon.