Restore notebook workspaces from volume snapshots

A notebook workspace contains code, local datasets, and files that have not yet been promoted to durable artifacts. Before a risky dependency change, the team wants a recoverable copy. Copying the notebook file alone may omit the rest of the workspace; restarting the Pod does not restore earlier disk contents.

Kubernetes volume snapshots can capture storage state through a supporting CSI driver and restore it into a new PersistentVolumeClaim. Snapshot and restore support is stable, but the cluster still needs the snapshot CRDs, controller, driver support, and a suitable VolumeSnapshotClass. Snapshot documentation.

A workspace snapshot preserves files on that volume. It does not preserve a Python kernel's memory, running processes, external databases, or credentials needed to reopen the workspace.

Choose a recoverable boundary

Save notebooks and stop or quiesce writers through the owning service or controller before taking the snapshot. Deleting one Pod while a controller immediately recreates it is not a reliable way to stop writes.

For a simple workspace, close the notebook session and finish file writes. If the volume contains a local database or an application with several related files, follow that application's consistency procedure. A storage snapshot is not automatically an application-consistent checkpoint.

Record what the snapshot should contain: the Git revision, environment or image digest, important file checksums, and the intended recovery point. Keep that record outside the volume being protected. A snapshot name alone cannot tell a future operator whether it predates the change that broke the environment.

Request the snapshot

The example assumes an existing workspace PVC in ml-team. Replace workspace-snapshots with a platform-provided VolumeSnapshotClass matching the source volume's CSI driver. Your account needs the appropriate snapshot and PVC permissions.

Save this as workspace-snapshot.yaml:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: workspace-before-upgrade
  namespace: ml-team
spec:
  volumeSnapshotClassName: workspace-snapshots
  source:
    persistentVolumeClaimName: workspace

kubectl apply -f workspace-snapshot.yaml
kubectl get volumesnapshot workspace-before-upgrade -n ml-team -o yaml

Wait for status.readyToUse: true and inspect any reported error. Record the bound snapshot content and restoreSize. Readiness means the storage snapshot is usable for restore; it does not validate the notebooks or prove that all application writes were consistent.

Before relying on retention, inspect the class's deletion policy. Delete and Retain have different consequences for the underlying snapshot when the Kubernetes object is removed. VolumeSnapshotClass documentation.

Restore into a separate claim

Create a new PVC rather than trying to overwrite the original workspace. The following example assumes the snapshot can be restored into 10Gi; increase that value to at least the reported restore size and satisfy the driver's requirements. Replace workspace-csi with a compatible StorageClass.

Save it as workspace-restore.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: workspace-restored
  namespace: ml-team
spec:
  storageClassName: workspace-csi
  accessModes: [ReadWriteOnce]
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: workspace-before-upgrade
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

kubectl apply -f workspace-restore.yaml
kubectl get pvc workspace-restored -n ml-team -o yaml

The snapshot and restoring PVC use the same namespace in this example. The persistent-volume documentation describes snapshot data sources.

If the StorageClass uses WaitForFirstConsumer, the new claim may remain Pending until a Pod references it. This allows provisioning to account for the consumer's placement. Check the binding mode before diagnosing that wait as a failed restore.

Inspect the restored files before switching

Use a temporary inspection Pod in ml-team, adapted to any required node placement and security context. This complete Pod mounts the restored claim read-only and lists its top-level entries; it does not execute code from the workspace.

apiVersion: v1
kind: Pod
metadata:
  name: inspect-restored-workspace
  namespace: ml-team
spec:
  restartPolicy: Never
  containers:
    - name: inspect
      image: python:3.12-slim
      command:
        - python
        - -c
        - |
          from pathlib import Path
          print("\n".join(sorted(p.name for p in Path('/workspace').iterdir())))
      resources:
        requests: {cpu: "100m", memory: "64Mi"}
        limits: {cpu: "500m", memory: "128Mi"}
      volumeMounts:
        - name: workspace
          mountPath: /workspace
          readOnly: true
  volumes:
    - name: workspace
      persistentVolumeClaim:
        claimName: workspace-restored
        readOnly: true

Save this as inspect-restored-workspace.yaml, apply it, and inspect its logs after completion. Use your approved image digest for maintained recovery tooling. A directory listing is only the first check: compare critical files with the saved manifest, then reopen a separate notebook against the restored claim and verify the expected environment and data.

The examples are source-reviewed, not a recorded restore test. Your recovery exercise should measure time to a usable workspace and document what was not restored.

Reconnect the workspace in Polyaxon

Polyaxon artifact connections give a restored workspace a reusable name and mount configuration. After the CSI restore creates the new PVC, expose it through a separate connection and launch a notebook or sandbox against that connection. Keep the original workspace available while the user checks files, permissions, and the notebook environment in the recovered session.

This is useful when recovering from an accidental deletion or inspecting a workspace before an environment upgrade. A scheduling preset can carry the resource and connection settings for the recovery session. Record the source snapshot and restored PVC with the operation so the accepted recovery has a clear origin. The storage system performs the snapshot and restore; Polyaxon supplies the notebook or sandbox workflow around the restored volume.

Sandbox file transfer and durable artifact logging also help preserve selected files and outputs. For notebooks subject to idle culling, keep work that must survive service termination on persistent storage. A scratch file is not protected by a PVC snapshot unless it resides on the snapshotted volume, and neither the snapshot nor a new session restores a running kernel's memory.

Remove the inspection Pod when finished. Delete a disposable restored PVC only after confirming no needed changes exist there and reviewing its reclaim policy. Keep the original workspace and recovery snapshot until the restore has been accepted; remove snapshots through the storage retention procedure rather than treating namespace cleanup as a backup policy.