Plan GCP Persistent Disk for ML workloads

Persistent disks are useful for boot volumes, notebook state, caches, databases, and workloads that need a block device attached to a Google Compute Engine VM. They are not automatically the best home for every dataset or model artifact.

For ML systems, choose storage by access pattern and lifecycle first. Then calculate capacity, performance, availability, backup, and cost together. Optimizing only the price per provisioned GiB can create a disk that is inexpensive but too slow, difficult to share, or never deleted.

Separate durable data from working data

Classify each path a workload uses:

Object storage is often a better system of record for datasets and artifacts because it is independent of one VM or zone. Persistent Disk can stage a working set close to compute or supply block storage to a stateful component. Local SSD can serve disposable, performance-sensitive scratch when the workload can recreate it.

This separation prevents a temporary training cache from receiving expensive backup retention and prevents a valuable model artifact from existing only on a worker disk.

Size for IOPS and throughput

Measure block size, read/write mix, queue depth, random IOPS, sequential throughput, latency, and daily data volume on a representative workload. Training that streams large shards has a different profile from a metadata database performing small random operations.

Google documents that Persistent Disk performance depends on factors including disk type, provisioned capacity, VM type and vCPU count, I/O size, and concurrency in its performance guide. Verify the limits for the selected VM and region rather than assuming the disk alone determines performance.

Run a warm and cold benchmark. A cache can make repeated epochs look fast while the initial dataset load remains the bottleneck. Also test concurrent workers; aggregate demand can expose a throughput limit that one process never reaches.

Do not provision a larger disk only to obtain performance without recording the resulting unused capacity. That tradeoff may still be correct, but it belongs in the cost model.

Choose availability intentionally

Match failure tolerance to the stored data. A zonal disk follows a zonal failure domain. More resilient configurations carry different costs and constraints. Snapshots protect against deletion or corruption scenarios but do not make every application-consistent without coordination.

Define recovery point and recovery time objectives for stateful services. Automate snapshot schedules, keep copies in the required location, and periodically restore into a separate environment. A successful snapshot job is not proof that the application can recover.

For replaceable working data, recovery may simply recreate a disk and refill it from object storage. Record the expected refill time and network cost so that this decision is operational rather than aspirational.

Model the full disk cost

Google's disk and image pricing separates disk capacity, snapshots, and some replication or transfer scenarios. Use the current regional price sheet during planning; avoid preserving a numeric rate in architecture documentation.

Include:

• provisioned capacity, including unused space;
• any separately provisioned performance;
• snapshots and retained snapshot chains;
• cross-location snapshot or restore traffic;
• unattached and stopped-instance disks;
• data transfer used to populate or evacuate the disk;
• operational cost of backups and restores.

Calculate cost per completed run, per training hour, or per served model—not only total storage spend. A faster disk can cost more per month while reducing expensive accelerator idle time enough to lower the cost per result.

Use Persistent Volumes carefully on Kubernetes

In Kubernetes, express the disk through a StorageClass, PersistentVolumeClaim, and the appropriate CSI driver. Confirm binding mode, topology, access mode, expansion behavior, reclaim policy, snapshot support, and what happens when a pod is rescheduled to another zone.

Use separate claims for state with different lifecycles. Do not place unrelated notebooks or tenants on one volume merely because it is convenient. Restrict mount access and ensure that deleting a workload does not unexpectedly retain sensitive data or destroy a required volume.

Monitor claim capacity, filesystem use, IOPS, throughput, latency, throttling, attachment failures, and time spent waiting for a volume. Join storage symptoms with node, pod, and workload identifiers.

Automate cleanup without losing lineage

Label disks and snapshots with environment, owner, project, purpose, and expiry. Detect unattached volumes, forgotten notebook disks, oversized claims, stale snapshots, and temporary disks that outlive their runs. Send uncertain cases to an owner before deletion.

Use Polyaxon artifacts for durable run outputs and tracking for the parameters and lineage that produced them. Let persistent disks provide the block-storage behavior a workload needs, then copy declared outputs to durable artifact storage before ephemeral compute and working volumes are retired.