Polyaxon v3 is coming →

Kubernetes storage classes overview

Understand Kubernetes StorageClasses, dynamic provisioning, reclaim policies, volume binding, and persistent volume claims.

June 6, 2024by Polyaxon

StorageClasses are policy, not just YAML. They decide how persistent volumes are provisioned, expanded, bound, and reclaimed. If that policy is wrong, the workload may start fine and fail later when state matters.

For ML teams, storage choices show up everywhere: datasets, checkpoints, model artifacts, cache directories, notebooks, metadata stores, and services. Dynamic provisioning helps, but only when the defaults match the workload.

An introduction to Kubernetes storage

Before diving into the specifics of Kubernetes storage classes, you must understand the basics of persistent volumes and persistent volume claims.

A persistent volume (PV) is a Kubernetes object that represents a piece of storage, either locally or on the cloud. Pods use the PV of a cluster to store their data. A persistent volume claim (PVC) is a Kubernetes object representing a claim on a PV by forming a one-to-one mapping with your persistent volume and specifying what's required for the persistent volume to be used by your pod.

With static provisioning, you create the backing storage yourself. For example, on Google Cloud you could create a disk in a zone used by your GKE cluster:

gcloud compute disks create gcp-disk --size=10GB --zone=us-east1-b

Then represent that disk with a PV. This example requires the Compute Engine Persistent Disk CSI driver and permissions to access the disk. Replace PROJECT_ID with the disk's Google Cloud project:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-disk
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: ""
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: pd.csi.storage.gke.io
    volumeHandle: projects/PROJECT_ID/zones/us-east1-b/disks/gcp-disk
    fsType: ext4
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - us-east1-b

The PV describes the disk's capacity, ext4 filesystem, and zone. Its node affinity keeps consumers in that zone. A matching PVC can explicitly request volumeName: pv-disk and storageClassName: "" to use this statically provisioned volume. See GKE's pre-existing disk guide for disk, filesystem, and access-mode requirements.

The next step is creating a PVC that claims the storage. The same PVC will be used in your pod definition, which will allow your pods to use the disk. The above procedure, in which you need to provision a disk for your PV, is referred to as static storage provisioning, because you create the disks manually.

However, it's not convenient or practical to manually create a disk and PV every time you need storage. Kubernetes solves this problem by providing a way to make your storage provisioning more dynamic with dynamic storage provisioning, where PVs and disks are created on the fly when PVCs or pods request storage.

What is dynamic storage provisioning?

Dynamic provisioning starts with a PVC. The claim names a StorageClass, or uses the cluster's default class when one applies. The class identifies a provisioner that creates the backing storage and PV to satisfy the claim. Kubernetes does not infer a storage tier by watching a container's disk usage.

After the PVC binds to the PV, a Pod can reference the claim in its volumes and mount it in a container. Provisioning, binding, and mounting are separate steps; the class's binding mode affects when the first two happen.

What is a storage class?

StorageClass objects were introduced in Kubernetes 1.4 and have since become essential to Kubernetes storage.

A storage class in Kubernetes defines different storage types, which allows the user to request a specific type of storage for their workloads. Storage classes also allow the cluster administrator to control which type of storage is used for specific workloads by specifying a type of storage.

The StorageClass object contains information about the provider, such as Amazon EBS or Google Compute Engine Persistent Disk, as well as which supported capabilities, such as replication or encryption, the storage should have. Kubernetes will then use information from the storage class when it creates new persistent volumes.

Why use Kubernetes storage classes?

Kubernetes storage classes enable an administrator to create and manage multiple storage configurations and bind them to individual applications or workloads. This provides greater flexibility and control over managing storage resources in a Kubernetes cluster, as you don't need to configure and create different types of storage with various specifications every time a storage request is made.

The dynamic provisioning and class isolation of storage can be used in many different scenarios, including the following common use cases.

Different quality of service levels

Different classes can correspond to varying levels of quality of service, which can mean some classes are faster or have more storage than others.

For example, a class might be designed for files that are accessed frequently, or that are accessed only occasionally. A premium solid-state drive (SSD) can be provisioned for frequently accessed files, while a lower-cost but slower hard drive (HDD) can be provisioned for occasionally accessed files.

For example, AKS offers Azure Files CSI classes for standard and premium file shares. Azure Files provides shared file storage; Azure Disk provides block storage. Inspect kubectl get storageclass to see the classes installed in your cluster.

Azure Files premium

Premium Azure Files uses SSD-backed storage for workloads that need higher file-share performance. Compare the selected tier's capacity, IOPS, throughput, and pricing with your workload's needs.

Inspect the built-in CSI class with:

kubectl describe storageclass azurefile-csi-premium

Look for the file.csi.azure.com provisioner, the storage SKU, and allowVolumeExpansion. Expansion allows a larger PVC request; it does not automatically grow the share when it fills up.

Azure Files standard

Standard Azure Files is another option for less demanding shared-file workloads. Inspect its actual configuration rather than assuming that every cluster has the same defaults:

kubectl describe storageclass azurefile-csi

Microsoft's Azure Files provisioning guide describes the built-in classes and supported storage SKUs.

Different backup policies

Different storage classes can be created for different backup policies, allowing you to provision disks with different performance and prices. For example, one class might be designed for files that are in the production environment and need to be backed up regularly, while another class might be designed for files in the quality assurance environment that only need to be backed up occasionally.

The StorageClass itself does not schedule backups. Your backup controller, snapshot configuration, and retention policy must implement that requirement. Keeping a disk with Retain also does not protect against corruption or accidental writes; plan and rehearse restoration separately.

Arbitrary policies

Different classes might also be used to support other kinds of administrator-determined policies, as well. Depending on the scenario, one class might be designed to replicate files to multiple servers, while another class might be designed to retain data on a single server, such as in test environments.

Those guarantees come from the storage backend and supported driver parameters. A class name such as replicated does not enable replication by itself.

Another example can be of a cluster administrator who might want to use a specific type of storage for databases and a different type of storage for application logs.

Using Kubernetes storage classes

A StorageClass is simply a Kubernetes object, so like other objects, it's defined by a YAML manifest. However, its properties and metadata are different from those associated with other objects.

A sample for a cluster with the standard AWS EBS CSI driver installed is below. Its IAM permissions must allow volume provisioning; EKS Auto Mode uses a different provisioner. This class uses the driver's default filesystem and requests an encrypted general-purpose SSD:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Understanding components of Kubernetes storage classes

Some keys are present in all the objects, and some are exclusive to the object type. The first three keys, apiVersion, kind, and metadata, identify the object. The other fields describe its storage policy. Kubernetes field names are case-sensitive.

apiVersion

The apiVersion field indicates which version of the storage.k8s.io API is being used.

kind

The kind field indicates the type of object being created. In this case, it is StorageClass. Other kinds include Pod and Deployment.

metadata

The metadata field contains the required name and optional labels or annotations. A StorageClass is cluster-scoped, so it does not have a namespace. PVCs and the Pods that consume them are namespaced.

provisioner

The provisioner identifies the driver responsible for dynamic volume provisioning. With CSI, controller and node components coordinate creating, attaching, and mounting storage. Creating the StorageClass does not install those components.

This example uses ebs.csi.aws.com. Older manifests may name in-tree cloud provisioners such as kubernetes.io/aws-ebs; use the supported CSI driver for new configurations. Check the EBS CSI driver documentation for installation and supported parameters.

reclaimPolicy

The reclaim policy is inherited by dynamically provisioned PVs. After a claim is deleted and its volume is released, Delete permits removal of the PV and backing storage; Retain leaves storage for manual reclamation. Deleting a Pod alone does not delete its PVC.

Delete is convenient for disposable data. Retain gives an administrator a chance to recover or reassign storage, but retained disks keep incurring costs and need explicit cleanup. Neither policy is a backup strategy. The default is Delete; the EBS example deliberately uses Retain.

allowVolumeExpansion

Sometimes, you may find that your application needs more storage than was initially provisioned. With allowVolumeExpansion: true, you can request more capacity by increasing spec.resources.requests.storage on the PVC. The driver and filesystem must support expansion. This is an explicit request, not automatic capacity management, and Kubernetes volume expansion does not support shrinking the volume.

volumeBindingMode

volumeBindingMode controls when a PVC is bound and when dynamic provisioning happens.

There are two volume binding modes that can be used:

  • Immediate: provisioning and binding begin when the claim is created, without considering a consuming Pod's scheduling constraints. This is the default mode.
  • WaitForFirstConsumer: provisioning and binding wait for a consuming Pod so the scheduler can consider its placement constraints. This helps keep zonal storage reachable from the chosen node. Support depends on the CSI driver. A pending PVC before its first consumer is expected.

With WaitForFirstConsumer, use normal scheduling constraints rather than setting the Pod's nodeName, which bypasses the scheduler. See the StorageClass reference.

mountOptions

The mount options parameter allows an administrator to specify a list of options for mounting a volume. These options can be used to debug mounting issues or fine-tune a volume's performance.

They must be supported by the driver and filesystem. Kubernetes does not validate their contents; an invalid option can prevent a volume from mounting. Omit them when the driver's defaults suffice.

Allowed topologies

Sometimes you need to restrict provisioning to particular zones. For the standard EBS CSI driver, the following class permits two zones in us-east-1. Your cluster needs eligible nodes in those zones, and Pod scheduling constraints must be compatible:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zonal-ebs
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.ebs.csi.aws.com/zone
        values:
          - us-east-1a
          - us-east-1b

Parameters

The parameters map passes driver-specific options such as disk type, encryption, or filesystem configuration. There is no universal parameter that enables replication across every backend.

The EBS example requests a general-purpose SSD (gp3). Other supported types have different IOPS, throughput, capacity, and pricing constraints.

When dealing with another provisioner, check its documentation before copying parameters from a different driver.

Create storage classes

Now that you understand the components and parameters of storage classes, let's look at how to create a pod that uses a storage class to provision volume.

You'll need an AKS cluster with the Azure Disk CSI driver enabled, permission to create a cluster-wide StorageClass, and cloud permissions and quota to provision a disk. The example creates billable storage. If you're using another provider, select an installed CSI driver and its supported parameters. The AKS disk guide describes the Azure-specific requirements.

Tutorial

The following manifest defines the storage class that you'll use:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-storage-demo
provisioner: disk.csi.azure.com
parameters:
  skuName: StandardSSD_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

This class requests locally redundant Standard SSD storage. Save it as sc-definition.yml, then create the class and a dedicated namespace for the example:

kubectl apply -f sc-definition.yml
kubectl create namespace storage-demo

The StorageClass is cluster-wide; the namespace will contain the claim and Pod.

List storage classes

In order to list all the storage classes in your cluster, you can use the kubectl get sc command. In this command, sc is an acronym for StorageClass:

kubectl get sc
kubectl describe storageclass azure-storage-demo

Create a persistent volume claim

To use the storage class you have created, you need to have a PVC. To create one, save the below manifest in a YAML file, and then apply it with kubectl apply -f <file_name>.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
  namespace: storage-demo
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: azure-storage-demo
  resources:
    requests:
      storage: 1Gi

Save this as pvc-definition.yml. The claim requests at least 1 GiB through the azure-storage-demo class; the provisioned disk follows the backend's supported size increments.

kubectl apply -f pvc-definition.yml

List PVCs and PVs

Inspect the claim and its events:

kubectl get pvc -n storage-demo
kubectl describe pvc myclaim -n storage-demo

With WaitForFirstConsumer, the claim can remain Pending until the Pod in the next step is scheduled. This is a binding policy, separate from the Delete reclaim policy. After provisioning, the PV is visible with:

kubectl get pv

Create a pod

As a final step, you'll create a pod that will use your persistent volume when it requires persistent storage for operational data. You can do this by applying the following manifest:

apiVersion: v1
kind: Pod
metadata:
  name: mypod
  namespace: storage-demo
spec:
  containers:
    - name: frontend
      image: nginx:1.30.4-alpine
      volumeMounts:
        - mountPath: /usr/share/nginx/html/data
          name: web
  volumes:
    - name: web
      persistentVolumeClaim:
        claimName: myclaim

To apply, save it, then use the command below, updating pod-definition.yml to match your file name:

kubectl apply -f pod-definition.yml
kubectl get pod,pvc -n storage-demo
kubectl describe pod mypod -n storage-demo

Nginx now has a persistent data directory beneath its document root. If the claim stays pending, read both Pod and PVC events: distinguish an unschedulable consumer from missing driver permissions, quota exhaustion, or incompatible topology before changing the class.

Clean up

For this disposable example, stop the consumer before deleting its claim. Deleting the PVC uses the Delete reclaim policy and can delete the backing disk and its data. Preserve any data you need before running these commands:

kubectl delete pod mypod -n storage-demo --wait=true
kubectl delete pvc myclaim -n storage-demo
kubectl delete storageclass azure-storage-demo
kubectl delete namespace storage-demo

Deleting a StorageClass does not itself delete existing PVCs or their backing volumes. Check the PV and cloud disk after cleanup; failed cloud operations or a Retain policy can leave storage that still requires an administrator's attention.

Conclusion

StorageClasses decide how persistent storage is created and reclaimed. Those defaults become production behavior, so they deserve the same review as compute and network policy.

For ML teams, storage policy affects datasets, checkpoints, artifacts, model services, and metadata. Polyaxon can track and organize outputs, but the underlying storage class still determines durability and performance.