Reuse Hugging Face weights across Polyaxon runs

A replacement inference Pod should not need to download the same model weights again. Neither should every trial in a training sweep. Put the Hugging Face model cache on persistent storage, prepare the revision once, and mount those files in subsequent runs.

This walkthrough uses a CPU job to download a pinned revision of Qwen3-0.6B. Training jobs, notebooks and inference services can then read it from the same volume. The cache saves repeated downloads; each process still loads the model into its own CPU or GPU memory.

Give the cache persistent storage

Use a Polyaxon compute agent with CPU and GPU queues, and a storage class that supports the intended mounts. For concurrent consumers on different nodes, this example uses ReadWriteMany storage. ReadWriteOnce permits multiple Pods on one node; setting a mount read-only does not give it multi-node access. Check your storage driver's capabilities and topology. Kubernetes volume access modes.

Save this as model-cache-pvc.yaml, replacing the storage class with an existing RWX-capable class:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache-pvc
spec:
  accessModes: [ReadWriteMany]
  storageClassName: YOUR_RWX_STORAGE_CLASS
  resources:
    requests:
      storage: 20Gi

Create it in the namespace where the agent schedules these workloads:

kubectl --context YOUR_CLUSTER -n YOUR_WORKLOAD_NAMESPACE \
  apply -f model-cache-pvc.yaml

The capacity is illustrative; allow space for all retained revisions. Configure filesystem permissions so the preparation job can write and consumers can read. The PVC must be available in the actual workload namespace. A connection with the same name on another agent does not replicate the files to that cluster.

Register writer and reader connections

Merge these volume-claim connections into the compute agent's deployment configuration, then apply it through your normal agent update process. For Polyaxon CE, use its deployment configuration.

connections:
  - name: model-cache-write
    kind: volume_claim
    schema:
      volumeClaim: model-cache-pvc
      mountPath: /mnt/model-cache
      readOnly: false
  - name: model-cache-read
    kind: volume_claim
    schema:
      volumeClaim: model-cache-pvc
      mountPath: /mnt/model-cache
      readOnly: true

These names resolve to the same storage with different mount permissions. Select one connection per workload. Restrict access to the writable connection to the users preparing the cache; naming it “write” does not enforce that restriction by itself.

Share HF_HUB_CACHE, while keeping HF_HOME in a job-local writable directory. HF_HOME can contain the saved Hub token and other caches. This separation lets consumers share model files without sharing the downloader's credential directory. Hugging Face environment settings.

Prepare the revision on CPU

Save this simplified Polyaxonfile as prepare-model-cache.yaml. It requires Polyaxon 2.18 or later and uses huggingface_hub 2.3.0 in a Python 3.12 container:

name: prepare-qwen-cache
connections: [model-cache-write]
container:
  image: python:3.12-slim
resources:
  cpu: 2
  memory: 2Gi
env:
  HF_HOME: /tmp/huggingface
  HF_HUB_CACHE: /mnt/model-cache/hub
  MODEL_ID: Qwen/Qwen3-0.6B
  MODEL_REVISION: c1899de289a04d12100db370d81485cdf75e47ca
cmd: |
  python -m pip install --no-cache-dir huggingface_hub==2.3.0
  python - <<'PY'
  import os
  from huggingface_hub import snapshot_download

  path = snapshot_download(
      repo_id=os.environ["MODEL_ID"],
      revision=os.environ["MODEL_REVISION"],
  )
  print(f"MODEL_PATH={path}", flush=True)
  PY

Submit it to a CPU queue on the agent containing these connections:

polyaxon run -p YOUR_PROJECT --queue YOUR_AGENT/YOUR_CPU_QUEUE \
  -f prepare-model-cache.yaml --logs

Wait for the job to succeed, then copy the MODEL_PATH printed in its logs. The full commit fixes the model revision; the snapshot download fetches the repository files, including weights, configuration and tokenizer. This example deliberately avoids file filters that could omit something the consumer needs.

Qwen3-0.6B is public and ungated. For a private or gated model, supply HF_TOKEN to the preparation job through a secret connection using an account authorized to download it. Cached files remain readable to anyone with filesystem access, so cache readership must match your intended model access.

Load the prepared files in another run

Merge these fields into an existing training job, notebook or service, keeping its image, command and resource settings. Replace the placeholder with the exact path from the preparation job:

connections: [model-cache-read]
env:
  HF_HOME: /tmp/huggingface
  HF_HUB_CACHE: /mnt/model-cache/hub
  HF_HUB_OFFLINE: "1"
  MODEL_PATH: PASTE_THE_PREPARED_SNAPSHOT_PATH

Retain any other required connections and environment entries. Route the consumer to a queue that can mount the same PVC. Both connection definitions use the same mount path, so the printed snapshot path remains valid there.

For Transformers, replace your model-loading lines with the following. The existing image needs PyTorch and a compatible Transformers installation; Qwen3 support requires Transformers 4.51.0 or later. Keep the image's compatible dependencies rather than upgrading its Hub package just to match the downloader.

import os
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = os.environ["MODEL_PATH"]
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
model = AutoModelForCausalLM.from_pretrained(model_path, local_files_only=True)

Keep your existing device placement and precision settings. These calls read a local model directory; they do not allocate a GPU through Polyaxon or move the model onto one. An existing vLLM batch job can likewise use the local path as its model argument, with an image that supports the chosen architecture.

HF_HUB_OFFLINE disables Hub HTTP calls, not all application networking. Missing files should be fixed by the preparation job; the read-only consumer cannot download replacements into this cache. Keep checkpoints, generated output and runtime compilation caches in writable locations outside the model cache.

Retain the cache as a whole

Keep the full Hub cache mounted. Snapshot files can be symlinks to cached blobs, and current Hub versions can share blobs across repositories. Copying only a snapshot directory may leave broken references. Hugging Face cache layout.

Prepare a new pinned revision before switching consumers to it. Coordinate cache pruning with active runs and revisions you still need; stopping the preparation job does not delete the PVC. The cache also remains separate from Polyaxon's default artifact store, where you retain run outputs.

Compare model-download time and complete service-startup time separately. A warm cache still needs storage reads, model initialization and device transfer. Keep the model revision and runtime image with the Polyaxon run so later comparisons use the same inputs. Our container-caching guide explains the separate build-layer and node-image caches.