Protect notebook volumes with ReadWriteOncePod

Two notebook Pods can accidentally mount the same workspace and overwrite each other's files. A ReadWriteOnce PVC does not prevent this when both Pods run on the same node. ReadWriteOncePod restricts the claim to one Pod across the cluster. Kubernetes documents the access-mode distinction here.

For a Polyaxon notebook with a private persistent workspace, that makes the intended ownership explicit: one running notebook gets the volume; another service using that claim must wait.

Create a claim for one notebook

Use Kubernetes 1.29 or later, where ReadWriteOncePod is stable. It requires a supporting CSI driver and sufficiently recent CSI sidecars: csi-provisioner 3.0.0+, csi-attacher 3.3.0+ and csi-resizer 1.3.0+. Check your storage deployment before choosing the access mode. The official prerequisites cover these requirements.

Save this as notebook-workspace.yaml. Replace workspace-csi with your CSI StorageClass and ml-team with the namespace where the notebook Pod will run.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: notebook-workspace
  namespace: ml-team
spec:
  accessModes: [ReadWriteOncePod]
  storageClassName: workspace-csi
  resources:
    requests:
      storage: 10Gi

Create the fresh claim:

kubectl apply -f notebook-workspace.yaml

This recipe creates a new workspace. For an existing volume, plan a separate migration with your storage administrator rather than changing the bound claim's access mode in place.

Mount it in your Polyaxon notebook

Add these volume and mount fields to your existing notebook component. Keep its image, startup command and service ports; this fragment only supplies persistent storage.

run:
  kind: service
  volumes:
    - name: workspace
      persistentVolumeClaim:
        claimName: notebook-workspace
  container:
    volumeMounts:
      - name: workspace
        mountPath: /workspace

Choose the mount path your notebook image uses, and ensure its user can write to the volume. See the Polyaxon service specification for these fields and the Jupyter integration for notebook startup configuration.

Schedule the service on the compute agent connected to the cluster containing the claim. A Polyaxon project or team space is not a Kubernetes namespace: the PVC must exist in the service Pod's actual namespace.

Understand a waiting replacement

If another notebook already holds the claim, the replacement cannot use it concurrently, even on the same node. Inspect the waiting Pod and claim:

kubectl -n ml-team describe pod YOUR_NOTEBOOK_POD
kubectl -n ml-team describe pvc notebook-workspace

Check events for the conflicting volume user, storage provisioning or attachment problems. A fresh claim using a WaitForFirstConsumer StorageClass can also remain Pending until a consumer is scheduled; that alone is not evidence of an access conflict. Our persistent-volume guide explains binding and placement.

To hand the workspace to a replacement notebook, stop the old Polyaxon service and wait for its Pod to terminate and the volume to become available. Then start the replacement with the same claim. Stop the owning service through Polyaxon; deleting its Pod alone can allow its controller to recreate it.

Choose the boundary your workspace needs

Use a separate claim for each independently running notebook. If several notebooks need shared storage at the same time, choose a suitable ReadWriteMany backend and define how they coordinate writes.

RWOP limits Pod access, not the number of processes writing inside that Pod. It also does not back up the files. Keep workspace snapshots or another backup policy for recovery.