Kubernetes was designed with stateless applications in mind. Pods are ephemeral. Deployments replace pods. Everything is immutable infrastructure. And then you need to run a database, and suddenly “just restart the pod” doesn’t work like it does for your API server.
Persistent storage in Kubernetes has matured significantly. The primitives are solid, the CSI driver ecosystem is extensive, and there are production-grade distributed storage solutions that work well in both cloud and on-premises environments. Understanding the stack is essential for running anything stateful in Kubernetes.
The Storage Abstraction Stack
Kubernetes storage involves several layers:
Application
↓
PersistentVolumeClaim (PVC) — what the pod requests
↓
PersistentVolume (PV) — the actual storage resource
↓
StorageClass — defines how PVs are provisioned
↓
CSI Driver — communicates with the underlying storage system
↓
Storage Backend (Longhorn, Ceph, cloud EBS, NFS, etc.)
PersistentVolumeClaim
The PVC is what your pod actually references. It’s a request for storage with specific characteristics:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
namespace: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 50Gi
The pod references the PVC by name:
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: postgres-data
containers:
- name: postgres
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
Access Modes
ReadWriteOnce (RWO): Can be mounted by a single node in read/write mode. Most block storage is RWO. Appropriate for databases.
ReadOnlyMany (ROX): Can be mounted by many nodes in read-only mode. Good for static content.
ReadWriteMany (RWX): Can be mounted by many nodes in read/write mode. Requires shared storage (NFS, CephFS, etc.). Needed for applications with multiple replicas that share a volume.
ReadWriteOncePod (RWOP): Kubernetes 1.22+. Can only be mounted by a single pod (not just node). More restrictive than RWO.
Choose the most restrictive access mode that works for your use case. Most databases need RWO; only choose RWX when you genuinely need shared concurrent access.
StorageClasses
StorageClasses define the “class” of storage—what type it is, how it’s provisioned, and what capabilities it has:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-fast
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "2880"
diskSelector: "ssd" # Only use SSDs for this class
reclaimPolicy: Retain # ALWAYS use Retain in production
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
The reclaimPolicy: Retain is critical. The alternatives are Delete (the volume is deleted when the PVC is deleted) and Recycle (deprecated). In production, you always want Retain—if someone accidentally deletes a PVC, you want the underlying data to still exist until you explicitly decide to delete it.
Longhorn: The Homelab and SMB Production Choice
Longhorn is a CNCF project that provides distributed block storage built natively for Kubernetes. It runs entirely within your Kubernetes cluster—no external storage system required.
How it works: Longhorn carves space from local node disks, replicates data across multiple nodes, and presents it as PersistentVolumes. Replicas are stored on different nodes, so the failure of a node doesn’t cause data loss.
Why it’s great for homelab:
- No external storage required—uses your nodes’ local disks
- Web UI for volume management
- Snapshots and backup to S3-compatible storage
- Volume expansion online
- CNCF-graduated, actively maintained by SUSE
# Longhorn HelmRelease
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: longhorn
namespace: longhorn-system
spec:
chart:
spec:
chart: longhorn
version: ">=1.6.0 <2.0.0"
sourceRef:
kind: HelmRepository
name: longhorn
namespace: flux-system
values:
defaultSettings:
defaultReplicaCount: 3
defaultDataLocality: best-effort
backupTarget: "s3://my-backup-bucket@us-east-1/"
backupTargetCredentialSecret: longhorn-backup-credentials
ingress:
enabled: true
ingressClassName: nginx
host: longhorn.internal.example.com
Rook-Ceph: Enterprise-Grade Distributed Storage
For larger clusters or enterprise requirements, Rook is a Kubernetes operator for Ceph—the most capable open-source distributed storage system.
Ceph provides:
- Block storage (RBD) — for databases, comparable to Longhorn
- Object storage (RGW) — S3-compatible API
- Filesystem (CephFS) — ReadWriteMany support
# Ceph cluster definition
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v18.2
dataDirHostPath: /var/lib/rook
mon:
count: 3
allowMultiplePerNode: false
mgr:
count: 2
storage:
useAllNodes: true
useAllDevices: false
deviceFilter: "^sd[b-z]" # Use devices sdb through sdz
resources:
osd:
requests:
cpu: "500m"
memory: "2Gi"
limits:
memory: "4Gi"
Rook-Ceph is more complex to operate than Longhorn but more capable. Consider it if you need:
- ReadWriteMany volumes at scale
- S3-compatible object storage internal to your cluster
- Multiple petabytes of storage
- Enterprise data protection features
StatefulSets: Managing Stateful Applications
When running databases in Kubernetes, use StatefulSets instead of Deployments:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: database
spec:
serviceName: postgres
replicas: 1
selector:
matchLabels:
app: postgres
template:
spec:
containers:
- name: postgres
image: postgres:16-alpine
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgres-storage
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: postgres-storage
annotations:
# Ensure PVC is retained even if StatefulSet is deleted
helm.sh/resource-policy: keep
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: longhorn
resources:
requests:
storage: 50Gi
StatefulSets give each pod a stable network identity and stable persistent storage. Pod postgres-0 always gets PVC postgres-storage-postgres-0. This is essential for databases where each instance needs to know its identity.
Backup Strategy
Storing data in Kubernetes doesn’t excuse you from backup. Longhorn and Rook both have backup capabilities:
# Longhorn recurring backup job
apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
name: backup-daily
namespace: longhorn-system
spec:
cron: "0 2 * * *"
task: "backup"
groups:
- default
retain: 14
concurrency: 2
labels:
backup: "daily"
Backups go to S3-compatible object storage. Test your restores regularly—a backup you haven’t tested is just data you haven’t lost yet.
The PVC Retention Rule
All PVCs must be retained if a HelmRelease is deleted.
Implement this in two ways:
- StorageClass with
reclaimPolicy: Retain(as shown above) - Helm annotation on PVC templates:
annotations:
helm.sh/resource-policy: keep
Both together provides the strongest protection. A Helm uninstall won’t delete the PVC (due to the annotation), and even if the PVC is manually deleted, the PV won’t be deleted (due to the Retain policy). You’d have to explicitly delete both the PVC and then the PV to lose data.
This is the right default for production data. Always retain; explicitly delete when you’re sure you don’t need the data.
