Reuse torch.compile caches across Polyaxon runs
Export compiler artifacts from one GPU run, import them into private caches in later runs, and measure whether reuse reduces startup cost.
A new GPU training job can use the same image and model code as the previous job and still spend time compiling its first steps. Cached container layers and model weights do not preserve the compiler artifacts created by torch.compile.
PyTorch can export those artifacts after a model has compiled and executed, then import them in a later process. In Polyaxon, a warm-up job can publish the cache snapshot to persistent storage, and subsequent jobs can read it while keeping their writable compiler caches local.
Preserve compiler artifacts between processes
PyTorch's cache-bundle APIs collect compiler artifacts with torch.compiler.save_cache_artifacts() and populate caches in another process with torch.compiler.load_cache_artifacts(). Reuse depends on compatible software, hardware, graphs, and compiler settings. A new process may still capture the graph before finding reusable backend artifacts.
This walkthrough uses a single-process CUDA training job that already runs with the Inductor backend. The API examples follow PyTorch 2.15. Keep your tested image pinned by digest, its matching Triton/CUDA environment, GPU target, model code, input shapes/dtypes, and compile options fixed for the initial comparison.
Conceptual cache-miss and cache-hit paths, not measured durations. The second run can compile new entries when its graph or configuration differs.
Each run will use a private writable cache. A current upstream report describes corruption involving shared cache writers across container PID namespaces. Publishing a snapshot and importing it into local caches also gives the warm-up and consumer workloads separate storage permissions.
Configure a snapshot writer and readers
Use an existing PVC named pytorch-compile-cache in the workload namespace. If consumers run concurrently on different nodes, its storage must support those mounts—for example, a ReadWriteMany-capable volume. A read-only mount does not change a volume's access-mode or topology constraints. See the PVC integration for setup.
Add these connections to the compute agent's deployment configuration, or the Polyaxon CE deployment configuration:
connections:
- name: compile-cache-write
kind: volume_claim
schema:
volumeClaim: pytorch-compile-cache
mountPath: /mnt/compile-cache
readOnly: false
- name: compile-cache-read
kind: volume_claim
schema:
volumeClaim: pytorch-compile-cache
mountPath: /mnt/compile-cache
readOnly: trueGive the writer connection to the warm-up workload and the reader connection to consumers. Scope access to the trusted workloads producing and using these compiler artifacts. Use a new snapshot directory for each published environment; do not overwrite one that consumers already use.
Add the cache fields to your training job
In the examples, train.yaml is your existing working GPU Polyaxonfile. Add the cache connection alongside its existing connections and merge the environment entries, retaining the image, training command, queue, and resources. The simplified format requires Polyaxon 2.18 or later:
connections: [compile-cache-read]
env:
TORCHINDUCTOR_CACHE_DIR: "/tmp/compile-cache/{{ globals.uuid }}/inductor"
TRITON_CACHE_DIR: "/tmp/compile-cache/{{ globals.uuid }}/triton"
TORCHINDUCTOR_FX_GRAPH_CACHE: "1"
TORCHINDUCTOR_AUTOGRAD_CACHE: "1"
COMPILE_CACHE_MODE: load
COMPILE_CACHE_BUNDLE: /mnt/compile-cache/training-v3-a100/snapshot-01/cache.binThe /tmp directories belong to the run's container, rather than the shared volume. globals.uuid gives each run a distinct path. The FX graph and AOTAutograd settings enable the corresponding disk caches; AOTAutograd caching requires the FX graph cache. PyTorch cache configuration.
Replace training-v3-a100/snapshot-01 with your own image/GPU-target label and unused snapshot identifier. That label organizes storage; PyTorch's cache keys determine actual compatibility. COMPILE_CACHE_MODE and COMPILE_CACHE_BUNDLE are application variables used by the hooks below.
Export after representative work, import before compilation
Add this hook before torch.compile and the first compiled execution in your training entry point. It loads only in consumer mode:
import os
import time
from pathlib import Path
import torch
bundle_load_seconds = 0.0
if os.environ.get("COMPILE_CACHE_MODE", "off") == "load":
started = time.perf_counter()
cache_info = torch.compiler.load_cache_artifacts(
Path(os.environ["COMPILE_CACHE_BUNDLE"]).read_bytes()
)
bundle_load_seconds = time.perf_counter() - started
print(f"cache_bundle_loaded={cache_info is not None}", flush=True)The loader populates compiler caches. A non-None result does not establish that your next graph will hit them; inspect the compilation diagnostics as well.
For the producer, run representative forward and backward steps, then add this hook afterward:
import os
from pathlib import Path
import torch
if os.environ.get("COMPILE_CACHE_MODE", "off") == "save":
artifacts = torch.compiler.save_cache_artifacts()
if artifacts is None:
raise RuntimeError("No compiler artifacts were exported")
artifact_bytes, _ = artifacts
destination = Path(os.environ["COMPILE_CACHE_BUNDLE"])
destination.parent.mkdir(parents=True, exist_ok=True)
with destination.open("xb") as stream:
stream.write(artifact_bytes)The exporter returns artifact bytes and cache information. Exclusive creation prevents this hook from overwriting an existing snapshot. Start readers only after the producer succeeds; a file left by a failed producer is not a published snapshot. Warm the input variants you intend to reuse, including backward work for training.
Compare three fresh runs
First, swap compile-cache-read for compile-cache-write in train.yaml and set COMPILE_CACHE_MODE: save. Keep any other connections. Choose an unused snapshot directory, then submit the warm-up run:
polyaxon run -p acme/training -f train.yaml \
--cache=false --name compile-cache-producerReplace acme/training with your project. Wait for successful completion and confirm the exported bundle before continuing. Then change the connection back to compile-cache-read and the mode to load, keeping the same bundle path:
polyaxon run -p acme/training -f train.yaml \
--cache=false --name compile-cache-consumerFor a fresh-process control, submit the same consumer configuration with import disabled:
polyaxon run -p acme/training -f train.yaml \
--cache=false --name compile-cache-control \
--env COMPILE_CACHE_MODE=offEach run has its own local cache directory. The --cache=false option disables Polyaxon's operation-result cache, ensuring the job actually executes; it does not disable PyTorch's compiler caches. Keep the same workload and compute settings across the comparison.
Measure the first compiled training step after normal model/device setup, before any compiled warm-up. This helper accepts your existing train_step callable, which should perform the same forward, backward, and optimizer work in every run. Use the same initial model state and batch sequence; an already-loaded batch helps isolate compilation from input loading:
import time
from statistics import median
import torch
def measure_steps(train_step, later_steps=10):
def elapsed_step():
torch.cuda.synchronize()
started = time.perf_counter()
train_step()
torch.cuda.synchronize()
return time.perf_counter() - started
first = elapsed_step()
later = [elapsed_step() for _ in range(later_steps)]
return {
"first_step_seconds": first,
"later_step_median_seconds": median(later),
}
timings = measure_steps(train_step)
timings["bundle_load_seconds"] = bundle_load_seconds
print(timings, flush=True)torch.cuda.synchronize() waits for pending work on the current CUDA device, so the elapsed time includes completed GPU work. This is a startup diagnostic: it includes whatever your callback does and synchronizes every measured step. For training throughput, use a representative profiling and measurement workflow.
Record the values with your existing Polyaxon tracking, such as tracking.log_outputs(**timings) after tracking initialization, and compare the runs. Repeat the consumer and control as fresh jobs, alternating their order to expose timing variation. Include bundle loading plus the first step when judging the consumer's startup cost. Record container/image pulling and model loading separately if end-to-end startup is the objective.
Check hits and explain misses
For a separate diagnostic run, temporarily add TORCH_LOGS: "+inductor,recompiles" to the environment. Review the backend's cache-hit/miss diagnostics. recompiles explains guard-triggered recompilation; it is not proof of a disk-cache hit. Remove verbose diagnostics for timing comparisons. PyTorch compiler observability.
If reuse is absent, check the pinned PyTorch/Triton environment and GPU target, input shapes/dtypes, compile options, and whether the relevant graphs were exercised by the producer. Some graphs can bypass caching. A successfully imported bundle or a faster second run alone is insufficient evidence of a particular cache hit.
Compiler artifacts do not preserve model weights, optimizer state, or dataset position. Keep training checkpoints and model-weight caches under their own lifecycle. Publish a new compiler snapshot when the environment or representative workload changes, and keep it only when the measured startup benefit exceeds the cost of importing it.