Storage in Kubernetes Is Still the Hard Part
Compute scheduling in Kubernetes is essentially solved. Networking has mature CNI plugins. But storage? Storage is where teams still lose hours debugging why a PVC won't bind, why a volume attachment fails during node migration, or why their database performance dropped 60% after moving to Kubernetes.
The Container Storage Interface (CSI) standardized how storage providers integrate with Kubernetes, but choosing the right driver and configuring StorageClasses correctly requires understanding your workload's I/O patterns and the tradeoffs each storage backend makes.
CSI Architecture in 30 Seconds
A CSI driver consists of two components. The controller plugin runs as a Deployment (usually one replica) and handles volume creation, deletion, and snapshotting. The node plugin runs as a DaemonSet on every node and handles mounting volumes into pods.
When you create a PVC, the external-provisioner sidecar in the controller plugin calls the CSI driver's CreateVolume RPC. When a pod using that PVC gets scheduled to a node, the kubelet on that node calls the node plugin's NodeStageVolume and NodePublishVolume RPCs to attach and mount the volume.
This two-phase design means the volume exists before the pod starts, and it persists after the pod dies. But it also means volume attachment is a blocking step in pod scheduling — if the CSI driver is slow or the storage backend has capacity issues, pods sit in ContainerCreating waiting for their volume.
StorageClass Design for Multi-Workload Clusters
I typically create four StorageClasses per cluster, each optimized for a different workload pattern:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
iops: "6000"
throughput: "250"
encrypted: "true"
kmsKeyId: "arn:aws:kms:us-east-1:123:key/abc-def"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: high-iops
provisioner: ebs.csi.aws.com
parameters:
type: io2
iops: "20000"
encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: standard
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
parameters:
type: gp3
encrypted: "true"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: bulk-storage
provisioner: efs.csi.aws.com
parameters:
provisioningMode: efs-ap
fileSystemId: fs-0abc123def
directoryPerms: "700"
reclaimPolicy: DeleteWhy WaitForFirstConsumer Matters
The volumeBindingMode: WaitForFirstConsumer setting delays volume creation until a pod actually needs it. This is critical for topology-aware storage like EBS, where volumes exist in a specific availability zone. With the default Immediate binding, the volume might get created in us-east-1a, but the pod gets scheduled to a node in us-east-1b. Result: the pod can't start because the volume is in the wrong zone.
WaitForFirstConsumer creates the volume in the same zone as the node where the pod is scheduled. I've seen this single setting fix half the "PVC stuck in Pending" tickets that teams open.
EBS CSI Driver Configuration
The AWS EBS CSI driver is the most common CSI driver in EKS clusters. The default installation works, but there are configuration details that affect performance and reliability.
First, make sure you're running the driver as a managed EKS addon, not a self-managed Helm installation. The managed addon handles upgrades and is tested against each EKS version. Self-managed installations occasionally break after EKS platform version updates.
Second, configure encryption. Every StorageClass should have encrypted: "true". If your security team requires a specific KMS key (they usually do), specify it in the StorageClass parameters. Don't rely on the EBS default encryption setting — it can be changed at the AWS account level, and you don't want a storage configuration that depends on an account-level setting.
Third, set allowVolumeExpansion: true on every StorageClass. EBS volumes can be expanded online without downtime (for gp3 and io2). Without this flag, expanding a PVC requires deleting and recreating the volume. The catch: EBS limits you to one size modification every 6 hours per volume. Plan capacity changes, don't react to them.
EFS for Shared Storage
When multiple pods need to read from the same filesystem — shared configuration, static assets, ML model files — EFS is the answer on AWS. The EFS CSI driver provisions access points, each of which acts as an isolated root directory within the filesystem.
EFS performance is the thing that surprises teams. A new EFS filesystem in bursting throughput mode starts with 100 MiB/s of burst credit. Once those credits are exhausted, throughput drops to 50 KiB/s per GiB of data stored. If you've got 10 GiB of data, your sustained throughput is 500 KiB/s. That's painfully slow for any real workload.
For workloads that need consistent performance, use provisioned throughput or elastic throughput mode:
resource "aws_efs_file_system" "shared" {
encrypted = true
throughput_mode = "elastic"
performance_mode = "generalPurpose"
lifecycle_policy {
transition_to_ia = "AFTER_30_DAYS"
}
}Elastic throughput charges based on actual data transfer rather than provisioned capacity. For bursty workloads, it's typically cheaper than provisioned throughput.
Volume Snapshots for Backup
CSI volume snapshots provide a standardized way to create point-in-time copies of persistent volumes. You need three components: the snapshot controller, a VolumeSnapshotClass, and VolumeSnapshot resources.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Retain
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-snapshot-20250915
namespace: production
spec:
volumeSnapshotClassName: ebs-snapshot-class
source:
persistentVolumeClaimName: postgres-dataI run a CronJob that creates daily snapshots of all database volumes and deletes snapshots older than 30 days. The cost is minimal — EBS snapshots are incremental, so you only pay for changed blocks.
To restore from a snapshot, create a new PVC that references the snapshot as its data source:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data-restored
spec:
storageClassName: fast-ssd
dataSource:
name: postgres-snapshot-20250915
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100GiTroubleshooting Common Issues
The most common storage issue I see is PVCs stuck in Pending. The debugging path: check if the StorageClass exists (kubectl get sc), check if the CSI driver pods are running (kubectl get pods -n kube-system | grep csi), and check the events on the PVC (kubectl describe pvc). Nine times out of ten, it's a permissions issue — the CSI driver's service account doesn't have the IAM permissions to create volumes.
The second most common issue is volume attachment timeouts during node replacement. When a node dies, the volumes attached to it need to be force-detached before they can attach to a new node. The volumeAttachmentLimit on your CSI driver and the --node-timeout flag on the attach-detach controller affect how quickly this happens. Default is 6 minutes, which is an eternity when your database pod is down.
Performance Tuning for Database Workloads
Running databases on Kubernetes with persistent volumes introduces I/O considerations that stateless workloads don't face. The storage driver, the filesystem, and the volume's IOPS/throughput configuration all affect query performance.
For PostgreSQL on EBS, I've found that gp3 with provisioned IOPS (6000 IOPS) handles most OLTP workloads well. The key insight: PostgreSQL's WAL (write-ahead log) generates sequential write I/O that's sensitive to throughput, while random read I/O depends on IOPS. If your queries involve heavy index scans, increase IOPS. If you're write-heavy, increase throughput.
Monitor I/O wait time via the node exporter's node_disk_io_time_seconds_total metric. If I/O wait exceeds 10% consistently, your storage is the bottleneck - either switch to a higher-performance storage class or optimize your query patterns.
Filesystem Choice
EBS volumes default to ext4 when formatted by the CSI driver. For most workloads, ext4 is fine. But for write-heavy database workloads, XFS offers better performance because it handles concurrent write operations more efficiently thanks to its allocation group architecture. Set the filesystem type in your StorageClass parameters.
One caveat: XFS volumes can't be shrunk, only expanded. That's rarely an issue in practice (how often do you shrink a database volume?), but it's worth knowing.
Local NVMe Storage for High-Performance Workloads
For workloads that need the absolute highest I/O performance - high-frequency trading databases, real-time analytics engines, intensive caching layers - EBS won't cut it. Instance-local NVMe storage (available on i3, i4i, and d3 instance types on AWS) provides 3-10x the IOPS and significantly lower latency than EBS.
The trade-off is durability. Local NVMe data doesn't survive instance termination. You need application-level replication (like PostgreSQL streaming replication or Redis Sentinel) to handle node failures. The local-static-provisioner from the Kubernetes SIGs project manages these volumes with a dedicated StorageClass using the kubernetes.io/no-provisioner provisioner and WaitForFirstConsumer binding mode.
Combine this with StatefulSet pod anti-affinity to ensure replicas land on different nodes. If one node's local storage fails, the replica on another node takes over.
This architecture is more complex than EBS-backed storage, but for workloads where p99 read latency matters (sub-millisecond vs 1-3ms with EBS), it's the right choice. I've seen query latency drop by 60% after moving a high-traffic PostgreSQL cluster from gp3 to local NVMe with streaming replication handling durability.