Continuing my Kubernetes the Hard Way homelab build. Steps 01-08 are already done, this covers step 09.

Original guide: 09-bootstrapping-kubernetes-workers.md

Thoughts I had while doing this

Same pattern as the last two steps I worked on, just on the worker nodes this time: copy files over, install and configure them, then start them as services. Quick TLDR:

  1. From the jumpbox, make a few directories and copy the worker binaries, configs, and service files to each worker node
  2. On each worker, install the OS dependencies, disable swap, and create the directories the components need
  3. Set up pod networking (CNI), containerd, the kubelet, and kube-proxy
  4. Start the worker services

The worker steps below were done on node-0 first, then repeated on node-1.

Prerequisites (on the jumpbox)

From what I can tell, this part is mostly making directories and copying files over to the worker nodes. First, the per-node configs, with each node's pod subnet filled in from machines.txt:

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do SUBNET=$(grep ${HOST} machines.txt | cut -d " " -f 4) sed "s|SUBNET|$SUBNET|g" \ configs/10-bridge.conf > 10-bridge.conf sed "s|SUBNET|$SUBNET|g" \ configs/kubelet-config.yaml > kubelet-config.yaml scp 10-bridge.conf kubelet-config.yaml \ root@${HOST}:~/ done 10-bridge.conf 100% 265 525.3KB/s 00:00 kubelet-config.yaml 100% 610 1.7MB/s 00:00 10-bridge.conf 100% 265 538.1KB/s 00:00 kubelet-config.yaml 100% 610 1.6MB/s 00:00

Then the binaries, configs, and service files:

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do scp \ downloads/worker/* \ downloads/client/kubectl \ configs/99-loopback.conf \ configs/containerd-config.toml \ configs/kube-proxy-config.yaml \ units/containerd.service \ units/kubelet.service \ units/kube-proxy.service \ root@${HOST}:~/ done containerd 100% 56MB 138.3MB/s 00:00 containerd-shim-runc-v2 100% 8060KB 125.4MB/s 00:00 containerd-stress 100% 22MB 117.8MB/s 00:00 crictl 100% 38MB 129.3MB/s 00:00 ctr 100% 23MB 103.9MB/s 00:00 kubelet 100% 74MB 87.5MB/s 00:00 kube-proxy 100% 64MB 79.6MB/s 00:00 runc 100% 11MB 131.8MB/s 00:00 kubectl 100% 55MB 142.0MB/s 00:00 ... (same again for node-1)

A gotcha here: the CNI plugins copy failed 

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do scp \ downloads/cni-plugins/* \ root@${HOST}:~/cni-plugins/ done scp: /root/cni-plugins/: No such file or directory scp: /root/cni-plugins/: No such file or directory

The cni-plugins directory didn't exist on the workers yet. Fix was to create it on each worker node, then re-run the same command from the jumpbox:

root@jumpbox:~/kubernetes-the-hard-way# for HOST in node-0 node-1; do scp \ downloads/cni-plugins/* \ root@${HOST}:~/cni-plugins/ done bandwidth 100% 4546KB 137.5MB/s 00:00 bridge 100% 5163KB 133.9MB/s 00:00 dhcp 100% 12MB 153.0MB/s 00:00 host-local 100% 3965KB 124.6MB/s 00:00 loopback 100% 4018KB 131.4MB/s 00:00 portmap 100% 4603KB 139.1MB/s 00:00 ... (same again for node-1)

Provision a Kubernetes worker node

This part is just installing OS dependencies, disabling swap, and creating directories for the binaries. Everything from here on runs on the worker node itself.

Install the dependencies:

root@node-0:~# { apt-get update apt-get -y install socat conntrack ipset kmod } ... The following NEW packages will be installed: conntrack ipset libipset13 socat 0 upgraded, 4 newly installed, 0 to remove and 72 not upgraded. ...

Check that swap is off (no output means it's already disabled):

root@node-0:~# swapon --show

Create the directories and move the binaries into place:

root@node-0:~# mkdir -p \ /etc/cni/net.d \ /opt/cni/bin \ /var/lib/kubelet \ /var/lib/kube-proxy \ /var/lib/kubernetes \ /var/run/kubernetes root@node-0:~# { mv crictl kube-proxy kubelet runc \ /usr/local/bin/ mv containerd containerd-shim-runc-v2 containerd-stress /bin/ mv cni-plugins/* /opt/cni/bin/ }

Configure CNI networking

This part sets up the networking so pods can communicate with each other:

root@node-0:~# mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/ root@node-0:~# { modprobe br-netfilter echo "br-netfilter" >> /etc/modules-load.d/modules.conf } root@node-0:~# { echo "net.bridge.bridge-nf-call-iptables = 1" \ >> /etc/sysctl.d/kubernetes.conf echo "net.bridge.bridge-nf-call-ip6tables = 1" \ >> /etc/sysctl.d/kubernetes.conf sysctl -p /etc/sysctl.d/kubernetes.conf } net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1

What's CNI actually doing here?

The CNI part was more interesting than the rest of this step. It wasn't just copying files and binaries around, so I dug into what it's actually doing.

From what I understand, CNI (Container Network Interface) is the part that configures networking for pods so they can communicate with each other. The plugin binaries in /opt/cni/bin do the actual work, and the config files in /etc/cni/net.d tell the runtime which plugins to call and how. Breaking down what each piece does:

  • The bridge: 10-bridge.conf sets up a "bridge", basically a virtual switch on the node that pods plug into so they can talk to each other. Each node has its own pod subnet (the SUBNET value filled in from machines.txt in the prerequisites), and pods get their IPs from it.
  • The loopback: 99-loopback.conf gives each pod its own loopback interface.
  • The firewall part: the br-netfilter module and the two sysctl settings make traffic crossing the bridge pass through the host's iptables rules. This matters because kube-proxy relies on iptables to route traffic to Services.

Also, it seems that each bridge only connects the pods on its own node.

Configure containerd 

root@node-0:~# { mkdir -p /etc/containerd/ mv containerd-config.toml /etc/containerd/config.toml mv containerd.service /etc/systemd/system/ }

Configure the kubelet 

root@node-0:~# { mv kubelet-config.yaml /var/lib/kubelet/ mv kubelet.service /etc/systemd/system/ }

Configure the Kubernetes proxy 

root@node-0:~# { mv kube-proxy-config.yaml /var/lib/kube-proxy/ mv kube-proxy.service /etc/systemd/system/ }

Start the worker services 

root@node-0:~# { systemctl daemon-reload systemctl enable containerd kubelet kube-proxy systemctl start containerd kubelet kube-proxy } Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /etc/systemd/system/containerd.service. Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service. Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /etc/systemd/system/kube-proxy.service. root@node-0:~# systemctl is-active kubelet active

Another gotcha: the nodes never showed up

The kubelet was running, but when I checked from the controller, the cluster had no nodes:

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server "kubectl get nodes \ --kubeconfig admin.kubeconfig" No resources found

The kubelet logs on the worker showed why. The API server kept rejecting it with "forbidden" errors:

Oct 25 19:41:52 k8s-solo-w1 kubelet[706]: E1025 19:41:52.168304 706 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"k8s-solo-w1\" is forbidden: User > Oct 25 19:41:52 k8s-solo-w1 kubelet[706]: I1025 19:41:52.378476 706 kubelet_node_status.go:75] "Attempting to register node" node="k8s-solo-w1" Oct 25 19:41:52 k8s-solo-w1 kubelet[706]: E1025 19:41:52.381530 706 kubelet_node_status.go:113] "Unable to register node with API server, error getting existing node" err="nodes \"k8s-solo-w1\" is f>

The worker was still calling itself by its old VM name (k8s-solo-w1), but its kubelet certificate was issued for node-0 back in step 04. The fix was to set the hostname to the name registered in the certificate, then restart the kubelet:

hostnamectl set-hostname node-0 systemctl restart kubelet

Honestly, I thought this wasn't needed since I'd already set up /etc/hosts in step 03 with entries for node-0 and node-1. But as I understand it now, /etc/hosts only maps names to IPs. The kubelet picks its node name from the machine's own hostname, and the API server only lets a certificate for system:node:node-0 manage the node object named node-0. So the hostname itself has to match the certificate, not just resolve.

Repeated the same fix on the second worker (node-1), and both nodes came up:

root@jumpbox:~/kubernetes-the-hard-way# ssh root@server \ "kubectl get nodes \ --kubeconfig admin.kubeconfig" NAME STATUS ROLES AGE VERSION node-0 Ready <none> 8d v1.32.3 node-1 Ready <none> 4m54s v1.32.3

Summary

Step 09 got both worker nodes running containerd, the kubelet, and kube-proxy, with CNI networking configured for pod communication. Both nodes now show Ready.