Sources: CD's devops wiki 03-kubernetes/05-workloads.md (lesson 3: Pod, ReplicaSet, Deployment, DaemonSet, StatefulSet, Job, CronJob on kubeadm v1.35) and 02-full-overview.md, cross-checked against kubernetes.io on 2026-10-28. Lab names only; screens are drawings.

Three tickets from one quarter at Contoso, all with the same root cause: the right thing was running as the wrong kind of Kubernetes object.

  1. The invoice run. A CronJob at 01:00 normally takes 20 minutes. On the last day of the month it took 70. At 02:00 the next run started anyway, both runs read the same "unbilled" table, and 412 customers were billed twice.
  2. The missing logs. The log shipper was a Deployment with replicas: 3. The cluster autoscaler added two nodes in March. For 11 days nothing from those nodes reached the SIEM, and nobody noticed, because three shippers were "Running".
  3. The database that forgot. The reporting database was a Deployment with one replica and an emptyDir. A node reboot rescheduled it: new pod, new name, empty disk. Four days of reports, gone.

Kubernetes has six ways to run a container. Each one makes a different promise. The incidents happen when the promise you need is not the promise you asked for.

TL;DR

  • Problem: Deployment is the kind everyone learns first, so everything becomes a Deployment: batch jobs, per-node agents, databases. Each of those needs a different controller with a different promise.
  • Who it hurts: customers (double billing, missing reports), security (blind spots in logging), finance (24/7 pods doing 10 minutes of work a day), on-call (incidents with no error, because every pod says Running).
  • Fix: match the workload to the kind: Deployment for stateless, StatefulSet for identity and disk, DaemonSet for one-per-node, Job for run-to-completion, CronJob for a schedule, with concurrencyPolicy: Forbid.
  • Cost: three always-on "worker" pods doing nightly work cost about $70 a month each in node capacity. As a CronJob the same work costs cents. Getting the CronJob wrong cost 412 refunds.
  • Prove it: delete a pod of each kind and watch what comes back, with which name, on which node, and when.

Table of contents

  • 1. The problem
  • 2. Who gets hurt
  • 3. Cost and value
  • 4. The six kinds and the promise each one makes
  • 5. The manifests, with the knobs that matter
  • 6. Prove it
  • Gotchas
  • Checklist
  • What changed recently
  • Sources

1. The problem

A Kubernetes controller does one thing: compare what you asked for with what exists, and fix the difference. The kinds differ in what "what you asked for" means:

You need The promise The kind
N identical copies, any name, any node, rolling updates "Keep N running" Deployment
Copies with a stable name, a stable disk and an order "Keep db-0, db-1, db-2, each with its own disk" StatefulSet
Exactly one copy on every node, including new ones "One per node, forever" DaemonSet
Run until done, retry on failure, then stop "Finish N times" Job
Run on a schedule "Start a Job at these times" CronJob
Just try a container Nothing. A bare Pod is not recreated Pod

Three tickets: double invoices, missing logs, a database that forgot

The invoice run was a CronJob, which is right, but with the default concurrencyPolicy: Allow, which means "start the next run even if the last one is still going". The log shipper was a Deployment, which promises three copies, not one per node. The database was a Deployment with emptyDir, which promises a copy, not that copy with its disk.

2. Who gets hurt

Who What they lose
Customers Double charges, duplicate emails, reports that are four days short
Finance 412 refunds and the support hours around them; always-on pods paying for idle time
Security / SOC Two nodes with no log shipper for 11 days: a blind spot nobody alerted on, because kubectl get pods was green
On-call Incidents with no error and no deploy: a node reboot, an autoscaler event, a slow month-end
The team The belief that Kubernetes "lost data" or "ran things twice", when it did exactly what the manifest said

3. Cost and value

Always-on versus on-schedule (Azure Standard D2s v5, 2 vCPU 8 GiB, East US, pay-as-you-go $0.096 an hour, about $70 a month; checked 2026-10-28):

Pattern Node capacity used Monthly cost
3 "worker" pods as a Deployment, polling for nightly work 24/7, 1 vCPU each 1.5 nodes about $105
The same work as a CronJob: one Job a night, 20 minutes, 1 vCPU 20 min × 30 about $1
The database as a StatefulSet with a 128 GiB Standard SSD claim + 1 disk + $9.60 (last week's post)

What the incidents cost: 412 refunds plus support time; 11 days of a logging blind spot that an auditor will ask about; four days of reports re-generated by hand. None of them needed new software. Each needed a different kind: line and two or three fields.

What it is worth: a CronJob with Forbid cannot overlap. A DaemonSet cannot miss a node. A StatefulSet cannot come back with the wrong disk. You are buying guarantees that a human checklist cannot give.

4. The six kinds and the promise each one makes

From my lesson notes, in the order to learn them:

Pod. The smallest unit: one or more containers sharing an IP and, optionally, a volume. A pod created directly (kind: Pod, kubectl run) has no controller. Delete it and it is gone. Use it to try an image, never to run a service.

ReplicaSet. "Keep N pods matching this selector." You rarely write one; a Deployment creates it. Learn it because the label-selector rule lives here: spec.selector.matchLabels must match spec.template.metadata.labels, or the controller manages nothing.

Deployment. A ReplicaSet with history. Rolling updates (maxSurge, maxUnavailable), rollback (kubectl rollout undo), scaling. For anything stateless: web, API, workers that keep nothing on disk.

DaemonSet. One pod per node, including nodes that join later. No replicas field. Log shippers, metrics exporters (node-exporter next week), CNI agents, security agents. Control-plane nodes carry a taint, so add a toleration if the agent must run there too.

StatefulSet. Pods named name-0, name-1, ... created and deleted in order, each with its own PersistentVolumeClaim from volumeClaimTemplates, each reachable at a stable DNS name through a headless Service (clusterIP: None): db-0.db.contoso-shop.svc.cluster.local. Databases, brokers, anything where "which copy" matters. It does not make a database clustered; it gives the database the identity and disk it needs to cluster itself.

Job. Run pods until completions successes; parallelism at a time; retry up to backoffLimit (default 6); clean up after ttlSecondsAfterFinished. restartPolicy must be Never or OnFailure.

CronJob. Creates a Job on a schedule (cron syntax, with a timeZone). The three fields that caused or would have prevented ticket 1: concurrencyPolicy (Allow, Forbid, Replace), startingDeadlineSeconds, and the history limits.

Decision: what the thing does, and which kind makes that promise

5. The manifests, with the knobs that matter

Namespace contoso-shop. Only the lines that differ from the Deployment you already know.

DaemonSet for the log shipper (ticket 2):

apiVersion: apps/v1 kind: DaemonSet metadata: name: log-shipper namespace: contoso-shop spec: selector: matchLabels: { app: log-shipper } template: metadata: labels: { app: log-shipper } spec: tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: shipper image: busybox:1.36 command: ["sh", "-c", "while true; do echo shipping from $(hostname); sleep 30; done"] resources: requests: { cpu: "20m", memory: "32Mi" } limits: { cpu: "100m", memory: "64Mi" }

No replicas. Add a node, get a pod. Remove a node, lose exactly that pod.

StatefulSet for the database (ticket 3):

apiVersion: v1 kind: Service metadata: name: db namespace: contoso-shop spec: clusterIP: None # headless: one DNS name per pod  selector: { app: db } ports: [{ port: 27017 }] --- apiVersion: apps/v1 kind: StatefulSet metadata: name: db namespace: contoso-shop spec: serviceName: db replicas: 1 selector: matchLabels: { app: db } template: metadata: labels: { app: db } spec: containers: - name: mongo image: mongo:8.0 volumeMounts: - name: data mountPath: /data/db volumeClaimTemplates: - metadata: name: data spec: accessModes: [ReadWriteOnce] resources: requests: storage: 10Gi

The pod is db-0, forever. Its claim is data-db-0, forever. Reschedule it and it comes back as db-0 with data-db-0 attached. Delete the StatefulSet and the claim stays until you delete it.

CronJob for the invoice run (ticket 1):

apiVersion: batch/v1 kind: CronJob metadata: name: invoice-run namespace: contoso-shop spec: schedule: "0 1 * * *" timeZone: "Asia/Ho_Chi_Minh" concurrencyPolicy: Forbid # the fix: never overlap  startingDeadlineSeconds: 600 # if it could not start within 10 min, skip, do not catch up  successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: spec: backoffLimit: 2 ttlSecondsAfterFinished: 86400 template: spec: restartPolicy: Never containers: - name: invoice image: busybox:1.36 command: ["sh", "-c", "echo billing unbilled rows; sleep 60; echo done"]

Forbid means the 02:00 run is skipped while the 01:00 run is still going; the event log says so. Replace would kill the slow run and start a new one, which is worse for billing. Allow is the default and is what ran twice.

The 01:00 run overran: Allow starts a second run at 02:00, Forbid skips it

6. Prove it 

kubectl apply -f workloads/ # the files above kubectl get pods -n contoso-shop -o wide
  • DaemonSet: one log-shipper-* per node, including the control plane. kubectl scale node... there is no such thing; add a node (minikube node add) and watch a shipper appear on it without touching the manifest.
  • StatefulSet: kubectl delete pod db-0. The new pod is also db-0, and kubectl get pvc shows data-db-0 still Bound. Write a file before, read it after.
  • Job: kubectl create job once --image=busybox:1.36 -- sh -c 'exit 0' → Completed, pod kept for logs. With exit 1 → retries until backoffLimit, then Failed.
  • CronJob: set schedule: "*/1 * * * *" and sleep 150 for the test. With Allow, kubectl get jobs shows two or three running at once. With Forbid, one at a time and events saying the schedule was skipped.
  • Deployment vs bare Pod: kubectl run solo --image=nginx:1.27-alpine, delete it, it is gone. Delete a Deployment pod, it is back.

Before and after: what comes back when a pod of each kind is deleted

Gotchas

  • concurrencyPolicy defaults to Allow. Any CronJob that writes, bills, emails or moves money needs Forbid plus an idempotency key in the job itself. Forbid prevents overlap; it does not prevent a retry after a crash mid-run.
  • CronJob schedule is in UTC unless you set timeZone. The 01:00 you meant and the 01:00 the cluster ran are 7 hours apart in Vietnam.
  • startingDeadlineSeconds missing + a controller outage = catch-up storm. Set it, or a cluster that was down for a day may fire missed runs.
  • DaemonSets skip tainted nodes. No toleration, no agent on the control plane (or on GPU/spot pools with their own taints).
  • StatefulSet claims are not deleted with the StatefulSet. Good for data, bad for the bill; clean up data-db-* on purpose.
  • StatefulSet volumeClaimTemplates cannot be edited in place. This is the AdGuard incident from last week. Resize the PVCs directly, then recreate the StatefulSet with --cascade=orphan so the pods stay up.
  • Jobs need restartPolicy: Never or OnFailure. Always (the Deployment default) is rejected.
  • Completed pods pile up. Set ttlSecondsAfterFinished and the CronJob history limits, or the namespace fills with thousands of finished pods.
  • A Deployment with one replica is not HA. It is a restart policy with a nicer name.

Checklist

  • [ ] Every workload has a stated promise: N copies / stable identity / one per node / run once / on a schedule
  • [ ] No stateful process runs as a Deployment with emptyDir
  • [ ] Per-node agents are DaemonSets with the right tolerations; verified after the next node joins
  • [ ] Every CronJob: concurrencyPolicy: Forbid, timeZone, startingDeadlineSeconds, history limits, idempotent job
  • [ ] Jobs: backoffLimit, ttlSecondsAfterFinished, restartPolicy: Never
  • [ ] Nightly or weekly work is not an always-on pod
  • [ ] Section 6 run once per kind, with the output pasted in the runbook

What changed recently

Checked on 2026-10-28:

  • My lesson notes target Kubernetes v1.35; all six kinds and every field above are unchanged in v1.37 (released 2026-08-26). CronJob.spec.timeZone has been stable since v1.27, so there is no reason left to write UTC schedules.
  • ttlSecondsAfterFinished and the suspend field on CronJobs are stable; kubectl create job --from=cronjob/invoice-run manual-1 is still the way to run a scheduled job by hand.
  • The StatefulSet resize limitation is still real in v1.37 (the volumeClaimTemplates field is immutable), which is why the AdGuard post-mortem of 2025-12-02 is worth reading before the first resize.
  • VM price (Standard D2s v5, East US, $0.096 an hour) from a third-party calculator that reads Azure's retail price API; Azure's own page renders client-side.

Sources

Cheat sheet: six kinds, six promises, the fields that matter

Thanks for reading! I'm Cuong, and on CD Security Lab I rebuild real Microsoft security cases in a safe lab and explain them simply. Follow along:

Got a question about your own tenant? Drop it in the comments.