kubectl apply -f web.yaml returns almost immediately, but several separate components still have to act before your containers run and answer on a stable name. This guide follows one Deployment and one Service through a cluster and explains each component by the job it does along the way. By the end you will know where to look when a step stalls and what keeps working when a piece of the control plane fails.
Build a lab cluster and the manifest
kind runs each Kubernetes node as a Docker container and bootstraps it with kubeadm, so the control plane components show up as Pods you can inspect. k3d, which wraps k3s, is lighter, but k3s runs the control plane inside a single process, so those Pods do not appear. Use kind for this walkthrough, with two workers. Save this as kind.yaml:
kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker kind create cluster --config kind.yaml kubectl create namespace demo kubectl get pods -n kube-system You should see etcd, kube-apiserver, kube-controller-manager, and kube-scheduler Pods for the control-plane node, a kube-proxy Pod per node, coredns, and kindnet, which is kind's network plugin.
This is web.yaml, the manifest the rest of the article follows:
apiVersion: apps/v1 kind: Deployment metadata: name: web namespace: demo spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: - name: web image: nginx:1.27 ports: - containerPort: 80 resources: requests: cpu: 100m memory: 64Mi limits: memory: 128Mi readinessProbe: httpGet: path: / port: 80 periodSeconds: 5 livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: web namespace: demo spec: selector: app: web ports: - port: 80 targetPort: 80 The API server checks and stores the request
kubectl reads your kubeconfig, converts the YAML to JSON, and sends HTTPS requests to kube-apiserver. Add -v=6 to see them: for a new object, a GET that returns 404 and then a POST that returns 201.
kubectl apply -f web.yaml -v=6 Every other component also goes through the API server, and it is the only component that reads or writes etcd. Each request passes three gates in order.
Authentication establishes who is calling, from a client certificate, a bearer token (ServiceAccounts use these), or an OIDC token. Kubernetes keeps no user accounts of its own. A user is whatever name the credential carries.
Authorization decides whether that identity may perform this verb on this resource. On nearly every cluster this means RBAC: Roles and ClusterRoles list allowed verbs, and RoleBindings attach them to users, groups, or ServiceAccounts. Check your own access:
kubectl auth can-i create deployments -n demo Admission control runs last. Mutating admission may change the object, for example by filling in defaults. The object is then validated against its schema, and validating admission accepts or rejects it without modifying it. Pod Security Admission, ResourceQuota, and any webhooks or policies you install run at these stages. Only then is the object written to etcd.
The API server also serves watches. Controllers, the scheduler, and kubelets hold long-running watch connections and are told about changes as they happen. To check the API server and its view of etcd:
kubectl get --raw='/readyz?verbose' Every line should start with [+], including [+]etcd ok, and the output should end with readyz check passed.
etcd holds the desired state
etcd is a consistent key-value store and the cluster's source of truth. At this point it holds a Deployment and a Service describing what you want. Nothing is running yet.
etcd replicates data with the Raft consensus algorithm, and a write commits only when a majority of members (the quorum) agree. Three members tolerate one failure and five tolerate two. Four members still tolerate only one, because quorum becomes three, so an even member adds another thing that can fail without adding tolerance. That is why control planes run three or five etcd members.
Everything lives in etcd, Secrets included. Secret values are base64-encoded, and anyone can decode them with a base64 decoder, so base64 gives no confidentiality. Unless you configure encryption at rest, treat an etcd snapshot, or read access to Secrets, as access to the credentials themselves.
On a kubeadm cluster, take a snapshot from a control-plane node with etcdctl installed:
sudo etcdctl --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /var/backups/etcd-$(date +%F).db Copy the file off the node and protect it like the Secrets inside it. Managed services such as EKS, GKE, and AKS run and back up etcd for you, and you cannot reach it directly.
Controllers turn the Deployment into Pods
kube-controller-manager runs many controllers in one process. Each is a reconciliation loop: watch objects through the API server, compare the desired state in spec with the actual state, and act to close the gap. Controllers never start containers. They only create, update, or delete API objects.
The Deployment controller sees web with no ReplicaSet and creates one named web- plus a hash of the Pod template. The ReplicaSet controller sees that ReplicaSet wants three Pods and has none, so it creates three Pod objects. Those Pods exist in the API but have no node, so nothing runs yet.
kubectl get deployment,replicaset,pods -n demo Each Pod's metadata.ownerReferences points to its ReplicaSet, which points to the Deployment. Delete a Pod and the ReplicaSet controller creates a replacement. Change the image and the Deployment controller creates a second ReplicaSet, scaling it up while the old one scales down. Other controllers in the same process maintain the Service's EndpointSlices and watch node heartbeats, which matters when a node fails.
The scheduler picks a node
kube-scheduler watches for Pods with an empty spec.nodeName. For each one it works in two phases.
Filtering removes nodes that cannot run the Pod. A node drops out if its unreserved CPU or memory is smaller than the Pod's resource requests (requests, not live usage), if it has a taint the Pod does not tolerate, or if it fails the Pod's node affinity or nodeSelector. Scoring then ranks the remaining nodes, favoring things like spreading replicas apart, free resources, and preferred affinity rules.
The scheduler then binds the Pod: it tells the API server which node to use, and spec.nodeName is set. That is its only output. On this kind cluster the control-plane node carries the node-role.kubernetes.io/control-plane:NoSchedule taint, so all three Pods land on the workers:
kubectl get pods -n demo -o wide kubectl describe node kind-worker describe node shows taints, conditions, allocatable capacity, and the sum of requests already placed on the node. When no node passes filtering, the Pod stays Pending with a FailedScheduling event that counts why each node was rejected, such as Insufficient cpu.
The kubelet starts the containers
The kubelet on each node watches the API server for Pods bound to that node. When a web Pod appears, the kubelet calls the container runtime through the Container Runtime Interface (CRI), a gRPC API. containerd and CRI-O are the usual runtimes. Kubernetes 1.24 removed dockershim, so the kubelet no longer drives Docker Engine directly. Images built with Docker still run, since they are standard OCI images.
The runtime creates the Pod sandbox, calls the CNI plugin to attach networking, pulls nginx:1.27, and starts the container. The kubelet reports status back to the API server. kind nodes are containers, so you can look from the node's side:
docker exec kind-worker crictl ps The kubelet also runs the probes. A failing liveness probe makes the kubelet restart the container. A failing readiness probe leaves the container running but marks the Pod not ready, so it stops receiving Service traffic. A startup probe, if you define one, holds off the other two until the application has started.
The whole story so far is recorded as events:
kubectl get events -n demo --sort-by=.metadata.creationTimestamp Read them in order: ScalingReplicaSet from the Deployment controller, SuccessfulCreate from the ReplicaSet controller, Scheduled from default-scheduler, then Pulling, Pulled, Created, and Started from the kubelet.
Traffic reaches the Pods through a Service
Every Pod gets its own IP address and can reach every other Pod without NAT. Kubernetes defines that model, and a CNI plugin such as Calico, Cilium, or Flannel implements it. Pod IPs come from the Pod CIDR and Service IPs from a separate Service CIDR. Neither should overlap your node network or anything routed to the cluster, such as a VPN range. kubeadm's default Service CIDR is 10.96.0.0/12, and k3s uses 10.42.0.0/16 for Pods and 10.43.0.0/16 for Services. Check candidate ranges for overlap with the subnet calculator.
Pods get replaced and their IPs change, so the Service gets a stable virtual ClusterIP. EndpointSlices list the Pod IPs behind it, with their readiness:
kubectl get endpointslices -n demo -l kubernetes.io/service-name=web kube-proxy runs on every node, watches Services and EndpointSlices, and programs iptables or IPVS rules (newer releases also offer nftables) that rewrite traffic for the ClusterIP to one ready Pod IP. No process listens on a ClusterIP. Cilium can replace kube-proxy entirely using eBPF.
CoreDNS gives the Service a name, web.demo.svc.cluster.local. Pods in the same namespace can use plain web because of the search domains in their /etc/resolv.conf:
kubectl run tmp -n demo --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- http://web HTTP traffic from outside usually arrives through an Ingress, which needs an Ingress controller such as Traefik (k3s ships it), or through the Gateway API with its Gateway and HTTPRoute resources. Both route by host and path to Services.
What breaks when a component fails
Knowing the path tells you what each failure stops.
When the API server is down, running Pods keep running and Service routing keeps working, because kubelet and kube-proxy act on what they last received. The kubelet still restarts crashed containers. kubectl fails, though, and nothing is scheduled, scaled, or replaced.
When etcd loses quorum, the API server cannot commit writes, so the cluster is frozen for changes and reads may fail or time out. Workloads keep running as in the previous case. Bring members back to restore quorum. Restoring a snapshot is a last resort, because it rolls cluster state back to the snapshot time.
With the scheduler down, new Pods stay Pending with no node. With the controller manager down, deleted Pods are not replaced and Deployments stop progressing.
When a node's kubelet stops reporting (crash, network partition, or a stopped kubelet), the node lifecycle controller marks it NotReady after a grace period of under a minute and taints it. On a default cluster, admission adds a 300-second toleration for those taints to every Pod, so Pods are evicted about five minutes later and the ReplicaSet recreates them on healthy nodes. Check what your Pods actually carry, then simulate a node failure:
kubectl get pod -n demo -l app=web -o jsonpath='{.items[0].spec.tolerations}' docker stop kind-worker2 kubectl get nodes -w If CoreDNS fails, connections by IP keep working and service names stop resolving.
Where to go next
Keep these commands close when a cluster misbehaves:
-
kubectl get pods -n kube-systemfor control plane and node agents -
kubectl get --raw='/readyz?verbose'for API server and etcd health -
kubectl describe node <name>for conditions, taints, and allocated requests -
kubectl get events --sort-by=.metadata.creationTimestampfor the order things happened -
kubectl get endpointslicesfor which Pods a Service actually routes to
Clean up with kind delete cluster. If one host is enough for your workload, the Docker Compose stack guide covers a much smaller setup, and the subnetting guide helps when you plan Pod and Service ranges for a real network.
