Persistent volumes on managed Kubernetes: what to use for what

Compute in a cluster is disposable, which is the whole point. Storage is not, which is why storage decisions outlive the cluster that made them — and why a wrong one is expensive to undo six months later.

This is the practical map: what the primitives are, which one fits which workload, and the three settings that cause most of the damage.

The primitives, briefly

A PersistentVolumeClaim is what your workload asks for: this much space, this access mode, this class. A StorageClass decides who fulfils it and how. A PersistentVolume is the resulting piece of storage, provisioned by a CSI driver talking to the platform.

In practice you write PVCs and pick a StorageClass. Everything else happens for you.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgdata
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: nvme
  resources:
    requests:
      storage: 100Gi

Three kinds of storage, three jobs

Block storage is a disk. One writer, low latency, filesystem semantics your database expects. On our managed Kubernetes this is the NVMe class, attached over the network and following the pod as it reschedules.

File storage is a shared filesystem that several pods can mount at once. Convenient, and consistently slower and more fragile than teams expect under concurrent write load.

Object storage is not a volume at all — it is an API your application calls. It is also the correct answer far more often than the other two.

What to use for what

• Databases (Postgres, MySQL, MongoDB, Elasticsearch) — block, one volume per replica, via a StatefulSet. Never a shared filesystem.
• User uploads and generated media — object storage. This is the single highest-value change in most migrations: it removes the shared-volume requirement entirely.
• Build caches and scratch space — emptyDir. It dies with the pod, which is what you want. Add medium: Memory if it needs to be fast and small.
• Configuration and secrets — ConfigMap and Secret, not a volume you populate by hand.
• Logs — nothing. Write to stdout and let the log shipper deal with it. A PVC full of application logs is a recurring incident.
• Genuinely shared state with no alternative — file storage, reluctantly, and only after you have confirmed the workload cannot use object storage.

Access modes are about nodes, not pods

This is the detail that bites. ReadWriteOnce means the volume is mounted by one node — several pods on that same node can share it, and a pod scheduled elsewhere cannot start. ReadWriteMany means many nodes at once, and requires file storage that supports it. ReadOnlyMany is many readers, no writers.

The symptom of getting this wrong is a rollout that hangs forever: the new pod waits for a volume the old pod still holds on another node, and neither moves. With RWO volumes use strategy: Recreate rather than RollingUpdate, and accept the few seconds of downtime — it is a database, it was never going to roll cleanly.

Stateful workloads want StatefulSets

A Deployment with a PVC gives every replica the same volume, which is wrong for anything that writes. A StatefulSet gives each replica its own, with a stable name that survives rescheduling.

apiVersion: apps/v1
kind: StatefulSet
spec:
  serviceName: pg
  replicas: 3
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: nvme
        resources:
          requests:
            storage: 100Gi

You get data-pg-0, data-pg-1, data-pg-2. Note that deleting the StatefulSet does not delete those claims — which is usually a relief, occasionally a surprise on the invoice.

The reclaim policy that deletes production data

A StorageClass has a reclaimPolicy. With Delete, removing the PVC destroys the underlying volume and everything on it. With Retain, the volume survives for you to reattach or clean up manually.

Dynamic provisioning defaults to Delete, and it is correct for scratch and staging. For anything whose loss would ruin a week, use a class with Retain:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nvme-retain
provisioner: csi.antyxsoft.cloud
parameters:
  type: nvme
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

The cost of Retain is orphaned volumes you have to tidy up. The cost of Delete is a kubectl delete -f in the wrong namespace taking the database with it.

Expansion is one-way

With allowVolumeExpansion: true you can grow a volume by editing the PVC, and most drivers do it online. You cannot shrink it. Size for the next six months rather than the next three years, and monitor free space — a full volume takes the database down in a way that looks like a crash rather than a capacity problem.

kubectl patch pvc pgdata -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

A snapshot is not a backup

CSI snapshots are fast, cheap and excellent for the case where you are about to do something regrettable to a database. They also live in the same platform as the volume and share its failure modes and its credentials.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pgdata-pre-upgrade
spec:
  volumeSnapshotClassName: csi-nvme
  source:
    persistentVolumeClaimName: pgdata

Take snapshots before upgrades and migrations. Keep the real backup — logical dumps or streaming replication, shipped to object storage with immutability and separate credentials — as the thing you would actually rely on after a bad day.

Two performance notes

Network-attached storage has a latency floor. It is fine for the overwhelming majority of databases and completely wrong for etcd, which is why the control plane's storage is our problem rather than yours on a managed cluster.

IOPS are typically per volume, not per pod. Ten pods sharing one volume share its limit; ten pods with their own volumes do not. If a workload is IO-bound, more volumes often beats a bigger volume.

The short version

Databases get block storage, one volume per replica, through a StatefulSet, with Retain and Recreate. Uploads and media go to object storage, not a shared filesystem. Scratch goes to emptyDir, logs go to stdout. Size generously because shrinking is not a thing, and keep a real backup somewhere the cluster's credentials cannot reach.