Everything you have run so far in this series shares a dirty secret: when a container dies, its files die with it. A container's filesystem is a whiteboard — easy to write on, wiped clean on restart. That is fine for stateless web servers, but the moment you want a database, an upload folder, or logs that survive a crash, you need real disk. In this part you will cook with Kubernetes storage: you will claim disk without knowing anything about it, watch data survive Pod deletion, learn what access modes and reclaim policies actually mean, and provision a volume by hand to see the machinery up close.

The plain-English mental model is this: a StorageClass is the menu of disk types your cluster offers, a PersistentVolumeClaim (PVC) is your order from that menu, and a PersistentVolume (PV) is the actual disk that shows up. One more acronym: CSI, the Container Storage Interface, is the plugin standard that lets Kubernetes talk to any storage system — cloud disks, NFS, local SSDs — without Kubernetes knowing the details of each one. You will never type "CSI" in a YAML file; it just works behind the scenes.

What you need

  • A running local cluster from Part 1 (kind or minikube).
  • Comfort with Pods from Part 2 and Deployments from Part 4.
  • No files to carry over — we create everything fresh in this part.

Recipe 1: Claim disk without thinking

Problem: Your clinerds-demo Pod writes files that vanish every time the Pod restarts. You want a folder that keeps its contents no matter what happens to the Pod.

First, see what is on the menu. Every kind and minikube cluster ships with a default StorageClass:

kubectl get storageclass 
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE AGE standard (default) rancher.io/local-path Delete WaitForFirstConsumer 12m

On kind the provisioner is rancher.io/local-path, a small CSI driver that carves directories out of the node's disk. On minikube it is called standard with the k8s.io/minikube-hostpath provisioner. Either way, the (default) marker means you can order disk without naming the menu item.

Create the claim — the order:

# demo-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: demo-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Mi 
kubectl apply -f demo-pvc.yaml kubectl get pvc 
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE demo-data Bound pvc-7f3a2c1e-9b4d-4e6a-8c5f-2d1b3e9a7f0c 100Mi RWO standard 5s

Now mount it into a Pod and write something:

# demo-pod.yaml apiVersion: v1 kind: Pod metadata: name: clinerds-demo spec: containers: - name: web image: hashicorp/http-echo args: ["-text=hello from storage"] ports: - containerPort: 5678 volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: demo-data 
kubectl apply -f demo-pod.yaml kubectl wait --for=condition=ready pod/clinerds-demo --timeout=60s kubectl exec clinerds-demo -- sh -c 'echo "i was here" > /data/proof.txt'

The proof: kill the Pod, start a fresh one, and check the file.

kubectl delete pod clinerds-demo kubectl apply -f demo-pod.yaml kubectl exec clinerds-demo -- cat /data/proof.txt 
i was here

How it works: The moment you created the PVC, the default StorageClass's CSI driver dynamically provisioned a real PersistentVolume to match your order — a 100Mi directory on the node — and bound it to your claim. You never wrote a PV manifest; one appeared with a generated pvc-... name. The Pod spec wires the claim into a named volume and mounts that volume at /data inside the container. When the Pod died, only the container's whiteboard filesystem was wiped. The volume lives on the node, and the new Pod mounted the same claim to the same disk. Your data survived because it never lived inside the container at all.

Variations:

  • Mount the same PVC into a Deployment with several replicas and every replica sees the same files — with one big catch, covered in the next recipe.
  • A Pod can mount many claims. Give each volume a unique name and its own mountPath.

Recipe 2: Access modes in plain words

Problem: Your YAML keeps asking for accessModes and you have been copying ReadWriteOnce on faith. You want to know what the choices mean before one bites you.

There are three modes, and they describe how many nodes may use the volume at once, not how many Pods:

  • ReadWriteOnce (RWO) — one node at a time, read and write. The default you almost always want. Cloud disks (EBS, GCE PD) are RWO: a disk can only attach to one machine.
  • ReadOnlyMany (ROX) — many nodes, but read-only. Good for shipping a big reference dataset to many Pods.
  • ReadWriteMany (RWX) — many nodes, read and write. This needs storage that speaks a shared-filesystem protocol, like NFS or a cloud file service. Your laptop's local-path driver cannot do this — it only hands out directories on one node.

Check the modes in play on your cluster:

kubectl get pv,pvc 
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM pvc-7f3a2c1e-... 100Mi RWO Delete Bound default/demo-data NAME STATUS VOLUME CAPACITY ACCESS MODES persistentvolumeclaim/demo-data Bound pvc-7f3a... 100Mi RWO

The ACCESS MODES column abbreviates: RWO, ROX, RWX.

How it works: The access mode is a contract between the claim and the volume. A PVC asking for RWO will only bind to a PV that offers RWO. Kubernetes enforces RWO at scheduling time — if a volume is mounted on node A, a Pod using it cannot be scheduled to node B. The classic symptom of hitting this: you scale a Deployment that mounts an RWO volume to two replicas, the second Pod lands on another node, and it sits in ContainerCreating forever with a Multi-Attach error in its events. If you genuinely need many Pods on many nodes writing to one volume, use an RWX-capable StorageClass (NFS, or a managed file service in the cloud) or rethink the design — object storage is often the better answer.

Recipe 3: Reclaim policies — what happens when you let go

Problem: You are about to delete a PVC and you need to know: does the data die with the claim, or can you get it back?

Every PV has a reclaim policy with two useful values:

  • Delete — when the claim is deleted, the volume and the data on it are destroyed. This is the default for dynamically provisioned volumes, including kind's local-path driver.
  • Retain — when the claim is deleted, the volume is kept, flagged as Released, and left alone with data intact.

Demonstrate Retain. Patch your PV (use the name from kubectl get pv):

kubectl patch pv pvc-7f3a2c1e-9b4d-4e6a-8c5f-2d1b3e9a7f0c \ -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Now delete the Pod and the claim, and look at the volume:

kubectl delete pod clinerds-demo kubectl delete pvc demo-data kubectl get pv 
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM pvc-7f3a2c1e-... 100Mi RWO Retain Released default/demo-data

The status is Released, not Available. The volume still remembers its old claim and refuses to bind to anyone new — that is deliberate, so nobody accidentally grabs a disk full of someone else's data. To re-claim it, clear the old claim reference:

kubectl patch pv pvc-7f3a2c1e-9b4d-4e6a-8c5f-2d1b3e9a7f0c \ -p '{"spec":{"claimRef": null}}' kubectl get pv # now shows Available

A fresh PVC can now bind to it and proof.txt is still sitting inside.

How it works: Deleting a PVC never touches data directly — it only ends the claim. The reclaim policy is the PV's last will: Delete tells the provisioner to shred the underlying disk, Retain tells everyone to keep their hands off until a human decides. Released is a safety lock: the volume holds the name of its former claim in spec.claimRef, and until you clear that field, the PV cannot bind again. In production, set Retain on anything you would cry about losing — database volumes are the classic case — so a fat-fingered kubectl delete pvc becomes an annoyance instead of an outage.

When it goes wrong:

  • You patched the PV but the policy still says Delete: kubectl patch typed against the wrong PV name. Re-run kubectl get pv and copy the exact name.
  • After clearing claimRef, your new PVC stays Pending: the new claim probably asks for more storage or a different access mode than the PV offers. Make the request match the volume exactly.

Recipe 4: Static provisioning to see the machinery

Problem: Dynamic provisioning feels like magic, and magic is hard to debug. You want to create a PersistentVolume by hand so the whole lifecycle is visible.

Write the PV yourself. On kind, every Pod runs on a node that is itself a container, and that node has a real filesystem — so a hostPath volume works as our pretend disk:

# manual-pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: manual-pv spec: capacity: storage: 100Mi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain hostPath: path: /tmp/manual-pv type: DirectoryOrCreate

Now a claim that matches it — note there is no StorageClass anywhere:

# manual-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: manual-data spec: accessModes: - ReadWriteOnce storageClassName: "" resources: requests: storage: 100Mi 
kubectl apply -f manual-pv.yaml kubectl apply -f manual-pvc.yaml kubectl get pv,pvc 
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM manual-pv 100Mi RWO Retain Bound default/manual-data

The empty storageClassName: "" is important: it tells Kubernetes "do not dynamically provision anything, only bind me to an existing PV." Without it, the default StorageClass would jump in and make you a second volume.

How it works: This is static provisioning: an admin (you) creates the PV first, and a PVC binds to it only if capacity, access modes, and storage class line up. That binding logic is the same one dynamic provisioning uses — the only difference is who writes the PV, a human or a CSI driver. Binding is one-to-one: a bound PV serves exactly one claim until released. Now the honest warning: hostPath means "a directory on whichever node the Pod lands on." Reschedule the Pod to another node and your data is not there. Treat this recipe as an X-ray of how PVs and PVCs fit together, never as production storage.

Recipe 5: Day-2 storage chores

Problem: The volume works. Now you need the boring operational skills: seeing how full it is, growing it, and deleting things in the right order.

Check real usage from inside the Pod:

kubectl exec clinerds-demo -- df -h /data 
Filesystem Size Used Avail Use% Mounted on /dev/vda1 75G 3.2G 72G 5% /data

There is your mount, attached at /data, with the node's real filesystem behind it. (The local-path driver does not enforce the 100Mi request — the size is a reservation for scheduling, not a hard quota. Cloud disks do enforce it, because the request becomes the actual disk size.)

Clean up in the right order when you are done:

kubectl delete pod clinerds-demo kubectl delete pvc demo-data kubectl get pv # Delete policy: gone. Retain policy: Released, yours to handle.

How it works: df -h works because a mounted volume is just a directory inside the container's filesystem view — standard Linux tools see it like any other mount. The cleanup order is the gotcha that bites beginners: always delete the PVC first and let the reclaim policy deal with the PV. If you delete a bound PV first, it hangs in Terminating because the claim still points at it, and you end up force-patching finalizers at 2 a.m. Claim first, volume second — the policy does the rest.

Variations:

  • Volume expansion: if the StorageClass has allowVolumeExpansion: true (check with kubectl get sc standard -o yaml), you can grow a volume in place by editing the PVC: kubectl edit pvc demo-data, raise spec.resources.requests.storage, save, and the filesystem grows without downtime. Shrinking is never allowed.
  • The local-path provisioner ignores allowVolumeExpansion, so test expansion on a cloud cluster or minikube with a driver that supports it.

Where to go next

You can now hand any stateless app a disk that outlives it. The next natural step is the app that is all about its disk: databases and queues that need stable names, stable storage, and ordered startup. That is exactly what StatefulSets are for, and it is where we cook next in Part 9: StatefulSets. For the full reference on everything in this part — phases, binding rules, and every field on the PV and PVC objects — keep the official Persistent Volumes documentation bookmarked. The whole menu is at the series index.