Kubernetes persistent volumes for ML workloads

A Pod filesystem disappears with the Pod. That is useful for reproducible, disposable execution, but some ML workloads need data to survive restarts or move between Pods: notebook workspaces, intermediate datasets, checkpoints, indexes, and model-server state.

Kubernetes persistent storage separates the workload lifecycle from the storage lifecycle. The abstraction is powerful, but durability still depends on the underlying storage system, reclaim policy, topology, and backup design.

Separate volumes, claims, and classes

The Kubernetes PersistentVolume documentation defines three related objects:

A Pod mounts a PVC, not an arbitrary PV. Kubernetes binds the claim to a compatible volume, or a StorageClass dynamically provisions one. A PV is cluster-scoped, but a bound claim belongs to one namespace and cannot be casually reused by Pods in another namespace.

Start with a claim

Most application teams should request storage through a PVC and let the platform's StorageClass provision it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-workspace
  namespace: ml-team
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
  storageClassName: fast-block

The class name is environment-specific. Platform owners should publish supported classes with clear performance, topology, encryption, expansion, snapshot, and cost characteristics.

Mount the claim into a workload:

apiVersion: v1
kind: Pod
metadata:
  name: training-example
  namespace: ml-team
spec:
  containers:
    - name: trainer
      image: registry.example.com/trainer@sha256:REPLACE_WITH_DIGEST
      volumeMounts:
        - name: workspace
          mountPath: /workspace
  volumes:
    - name: workspace
      persistentVolumeClaim:
        claimName: training-workspace

Replace placeholders with approved values and manage the manifest through the system that owns the workload.

Choose access modes by attachment behavior

Common access modes include:

• ReadWriteOnce: read-write attachment from a single node; several Pods on that node may still access it.
• ReadOnlyMany: read-only access from multiple nodes.
• ReadWriteMany: read-write access from multiple nodes.
• ReadWriteOncePod: read-write access restricted to one Pod where the CSI driver and cluster support it.

Access modes participate in matching and attachment. They do not replace application-level coordination. Two writers can corrupt a format that was never designed for concurrency even when the storage backend supports multi-node access.

Distributed training should not assume that a shared filesystem is the only exchange mechanism. Measure metadata operations, throughput, and failure behavior at the worker count and file pattern the framework actually uses.

Account for topology

Some volumes are tied to a zone or node. A Pod with a bound volume must schedule where that storage is accessible. This can conflict with GPU availability, taints, affinity, and queue placement.

StorageClasses using WaitForFirstConsumer delay provisioning or binding until Kubernetes considers the Pod's scheduling constraints. That can prevent a volume from being created in a zone with no eligible accelerator, but only if the storage and scheduler configuration support the intended topology.

When a Pod is Pending, inspect both scheduling and volume events:

kubectl --context production --namespace ml-team describe pod training-example
kubectl --context production --namespace ml-team describe pvc training-workspace

Set the reclaim policy deliberately

When a claim is deleted, the PV's reclaim policy determines what happens next. Delete typically removes the underlying dynamically provisioned asset; Retain preserves it for an administrator to handle.

Neither policy is a backup. A deleted file, corrupted checkpoint, compromised credential, or regional failure may affect the live volume and its replicas together. Define snapshots and backups separately, keep copies in an appropriate failure domain, and test restoration into a new claim.

Before deleting a PVC, identify the owner, dependent workloads, reclaim policy, backup state, and recovery procedure. Storage deletion is materially different from deleting a disposable Pod.

Match storage to ML data

Use persistent volumes where filesystem semantics are required. Use object or artifact storage where immutable objects, broad sharing, retention, or cross-cluster access are a better fit.

Polyaxon supports mounting Kubernetes volumes and configured artifact connections. Keep the roles distinct: a mounted filesystem supports execution, while the artifact system preserves versioned outputs and lineage.

The right persistent-volume design states what must survive, who owns deletion, where the storage can attach, how performance is measured, and how the data is restored. A PVC makes storage consumable; it does not answer those operational questions for you.