Polyaxon v3 is coming →

Reuse torch.compile caches across Polyaxon runs

Export compiler artifacts from one GPU run, import them into private caches in later runs, and measure whether reuse reduces startup cost.

November 19, 2026by Polyaxon

A new GPU training job can use the same image and model code as the previous job and still spend time compiling its first steps. Cached container layers and model weights do not preserve the compiler artifacts created by torch.compile.

PyTorch can export those artifacts after a model has compiled and executed, then import them in a later process. In Polyaxon, a warm-up job can publish the cache snapshot to persistent storage, and subsequent jobs can read it while keeping their writable compiler caches local.

PyTorch wordmark and a smaller Kubernetes logo separated by a slash on a light background

Preserve compiler artifacts between processes

PyTorch's cache-bundle APIs collect compiler artifacts with torch.compiler.save_cache_artifacts() and populate caches in another process with torch.compiler.load_cache_artifacts(). Reuse depends on compatible software, hardware, graphs, and compiler settings. A new process may still capture the graph before finding reusable backend artifacts.

This walkthrough uses a single-process CUDA training job that already runs with the Inductor backend. The API examples follow PyTorch 2.15. Keep your tested image pinned by digest, its matching Triton/CUDA environment, GPU target, model code, input shapes/dtypes, and compile options fixed for the initial comparison.

A cold GPU run captures a graph, compiles and executes it, then exports a cache snapshot to persistent storage. A new run imports that snapshot into its own private writable cache, captures the graph, and reuses compatible artifacts when executing.

Conceptual cache-miss and cache-hit paths, not measured durations. The second run can compile new entries when its graph or configuration differs.

Each run will use a private writable cache. A current upstream report describes corruption involving shared cache writers across container PID namespaces. Publishing a snapshot and importing it into local caches also gives the warm-up and consumer workloads separate storage permissions.

Configure a snapshot writer and readers

Use an existing PVC named pytorch-compile-cache in the workload namespace. If consumers run concurrently on different nodes, its storage must support those mounts—for example, a ReadWriteMany-capable volume. A read-only mount does not change a volume's access-mode or topology constraints. See the PVC integration for setup.

Add these connections to the compute agent's deployment configuration, or the Polyaxon CE deployment configuration:

connections:
  - name: compile-cache-write
    kind: volume_claim
    schema:
      volumeClaim: pytorch-compile-cache
      mountPath: /mnt/compile-cache
      readOnly: false
  - name: compile-cache-read
    kind: volume_claim
    schema:
      volumeClaim: pytorch-compile-cache
      mountPath: /mnt/compile-cache
      readOnly: true

Give the writer connection to the warm-up workload and the reader connection to consumers. Scope access to the trusted workloads producing and using these compiler artifacts. Use a new snapshot directory for each published environment; do not overwrite one that consumers already use.

Add the cache fields to your training job

In the examples, train.yaml is your existing working GPU Polyaxonfile. Add the cache connection alongside its existing connections and merge the environment entries, retaining the image, training command, queue, and resources. The simplified format requires Polyaxon 2.18 or later:

connections: [compile-cache-read]
env:
  TORCHINDUCTOR_CACHE_DIR: "/tmp/compile-cache/{{ globals.uuid }}/inductor"
  TRITON_CACHE_DIR: "/tmp/compile-cache/{{ globals.uuid }}/triton"
  TORCHINDUCTOR_FX_GRAPH_CACHE: "1"
  TORCHINDUCTOR_AUTOGRAD_CACHE: "1"
  COMPILE_CACHE_MODE: load
  COMPILE_CACHE_BUNDLE: /mnt/compile-cache/training-v3-a100/snapshot-01/cache.bin

The /tmp directories belong to the run's container, rather than the shared volume. globals.uuid gives each run a distinct path. The FX graph and AOTAutograd settings enable the corresponding disk caches; AOTAutograd caching requires the FX graph cache. PyTorch cache configuration.

Replace training-v3-a100/snapshot-01 with your own image/GPU-target label and unused snapshot identifier. That label organizes storage; PyTorch's cache keys determine actual compatibility. COMPILE_CACHE_MODE and COMPILE_CACHE_BUNDLE are application variables used by the hooks below.

Export after representative work, import before compilation

Add this hook before torch.compile and the first compiled execution in your training entry point. It loads only in consumer mode:

import os
import time
from pathlib import Path

import torch

bundle_load_seconds = 0.0
if os.environ.get("COMPILE_CACHE_MODE", "off") == "load":
    started = time.perf_counter()
    cache_info = torch.compiler.load_cache_artifacts(
        Path(os.environ["COMPILE_CACHE_BUNDLE"]).read_bytes()
    )
    bundle_load_seconds = time.perf_counter() - started
    print(f"cache_bundle_loaded={cache_info is not None}", flush=True)

The loader populates compiler caches. A non-None result does not establish that your next graph will hit them; inspect the compilation diagnostics as well.

For the producer, run representative forward and backward steps, then add this hook afterward:

import os
from pathlib import Path

import torch

if os.environ.get("COMPILE_CACHE_MODE", "off") == "save":
    artifacts = torch.compiler.save_cache_artifacts()
    if artifacts is None:
        raise RuntimeError("No compiler artifacts were exported")
    artifact_bytes, _ = artifacts
    destination = Path(os.environ["COMPILE_CACHE_BUNDLE"])
    destination.parent.mkdir(parents=True, exist_ok=True)
    with destination.open("xb") as stream:
        stream.write(artifact_bytes)

The exporter returns artifact bytes and cache information. Exclusive creation prevents this hook from overwriting an existing snapshot. Start readers only after the producer succeeds; a file left by a failed producer is not a published snapshot. Warm the input variants you intend to reuse, including backward work for training.

Compare three fresh runs

First, swap compile-cache-read for compile-cache-write in train.yaml and set COMPILE_CACHE_MODE: save. Keep any other connections. Choose an unused snapshot directory, then submit the warm-up run:

polyaxon run -p acme/training -f train.yaml \
  --cache=false --name compile-cache-producer

Replace acme/training with your project. Wait for successful completion and confirm the exported bundle before continuing. Then change the connection back to compile-cache-read and the mode to load, keeping the same bundle path:

polyaxon run -p acme/training -f train.yaml \
  --cache=false --name compile-cache-consumer

For a fresh-process control, submit the same consumer configuration with import disabled:

polyaxon run -p acme/training -f train.yaml \
  --cache=false --name compile-cache-control \
  --env COMPILE_CACHE_MODE=off

Each run has its own local cache directory. The --cache=false option disables Polyaxon's operation-result cache, ensuring the job actually executes; it does not disable PyTorch's compiler caches. Keep the same workload and compute settings across the comparison.

Measure the first compiled training step after normal model/device setup, before any compiled warm-up. This helper accepts your existing train_step callable, which should perform the same forward, backward, and optimizer work in every run. Use the same initial model state and batch sequence; an already-loaded batch helps isolate compilation from input loading:

import time
from statistics import median

import torch

def measure_steps(train_step, later_steps=10):
    def elapsed_step():
        torch.cuda.synchronize()
        started = time.perf_counter()
        train_step()
        torch.cuda.synchronize()
        return time.perf_counter() - started

    first = elapsed_step()
    later = [elapsed_step() for _ in range(later_steps)]
    return {
        "first_step_seconds": first,
        "later_step_median_seconds": median(later),
    }

timings = measure_steps(train_step)
timings["bundle_load_seconds"] = bundle_load_seconds
print(timings, flush=True)

torch.cuda.synchronize() waits for pending work on the current CUDA device, so the elapsed time includes completed GPU work. This is a startup diagnostic: it includes whatever your callback does and synchronizes every measured step. For training throughput, use a representative profiling and measurement workflow.

Record the values with your existing Polyaxon tracking, such as tracking.log_outputs(**timings) after tracking initialization, and compare the runs. Repeat the consumer and control as fresh jobs, alternating their order to expose timing variation. Include bundle loading plus the first step when judging the consumer's startup cost. Record container/image pulling and model loading separately if end-to-end startup is the objective.

Check hits and explain misses

For a separate diagnostic run, temporarily add TORCH_LOGS: "+inductor,recompiles" to the environment. Review the backend's cache-hit/miss diagnostics. recompiles explains guard-triggered recompilation; it is not proof of a disk-cache hit. Remove verbose diagnostics for timing comparisons. PyTorch compiler observability.

If reuse is absent, check the pinned PyTorch/Triton environment and GPU target, input shapes/dtypes, compile options, and whether the relevant graphs were exercised by the producer. Some graphs can bypass caching. A successfully imported bundle or a faster second run alone is insufficient evidence of a particular cache hit.

Compiler artifacts do not preserve model weights, optimizer state, or dataset position. Keep training checkpoints and model-weight caches under their own lifecycle. Publish a new compiler snapshot when the environment or representative workload changes, and keep it only when the measured startup benefit exceeds the cost of importing it.