Kubernetes on Existing Hardware: Why `Ready` Was Only the First Milestone

Hi guys! In my previous articles, I mostly worked with Terraform and Google Cloud. I focused on provisioning infrastructure, creating reusable modules, and understanding how to manage infrastructure through code.

This time, I wanted to explore something different.

What happens when I need to manage the machines underneath Kubernetes myself?

For this project, I started building a non-production Kubernetes platform using existing hardware. Unlike GKE or other managed services, where provides manage the underlying control plane and provides integrations for several infrastructure components, I needed to configure and manage most of those components myself.

The first milestone was getting the cluster up and running.

I ran:

kubectl get nodes 

And the nodes showed Ready.

Great. The cluster was running. But then I started thinking about what that status actually meant.

Did it mean my applications could communicate with each other? Did it mean Kubernetes was using the correct network interface on every node? Did it mean my deployments would work consistently, or that I could recover the platform if something broke?

Not necessarily.

A node reporting Ready is useful, but it doesn't prove that the entire platform works as expected.

That became one of the main things I wanted to understand while building this platform.

This is the first article in my Building the Platform series. I'll document what I built, the problems I encountered, the decisions I made, and why I chose one approach over another.

I'll start with the foundation: preparing the machines, configuring the container runtime, dealing with multiple network interfaces, and testing whether the cluster actually works.

Note: Hostnames, environment identifiers, and network details are generalized. The platform discussed in this article is non-production.

1. What I Wanted to Build

I started with a small Kubernetes cluster consisting of one control-plane node and several worker nodes connected through a private network.

Instead of provisioning new cloud infrastructure, I used existing hardware. This meant I had to consider things that I normally wouldn't spend as much time thinking about when working with managed cloud services.

For example:

• The machines had different hardware and network configurations.
• Some nodes had multiple network interfaces.
• IP addresses and routing depended on the existing network.
• The machines needed to stay online instead of suspending like ordinary laptops.
• The operating system, container runtime, and Kubernetes components all needed to work together.

I also wanted this platform to support actual applications eventually, not just run a few test Pods.

That meant I would eventually need application routing, CI/CD, GitOps, persistent storage, monitoring, logging, and recovery procedures.

However, I didn't want to build everything at once without understanding how the individual components worked.

My approach was to start with the infrastructure foundation, validate it, and then build on top of it.

The overall architecture I was working toward looked like this:

 Developers | Source Repositories | CI/CD | GitOps | Kubernetes Cluster | +-------------+-------------+ | | | Applications Platform Tools Persistent Workloads | | | Services Monitoring Database | and Logs | +-------------+-------------+ | Shared Gateway | Private LAN 

This diagram represents the intended architecture of the platform. Not every component was completed or validated at the same time, so I wouldn't treat the diagram itself as proof that everything was working.

For example, having a backup configuration is not the same as successfully restoring a database. I'll cover those distinctions as I document the different parts of the platform.

2. Preparing the Machines Before Installing Kubernetes

My first priority was making sure the machines were ready to run Kubernetes.

It's tempting to start directly with kubeadm init, install the CNI, and move on to deploying applications. But if the underlying operating system or network configuration is unreliable, those problems can show up later as Kubernetes issues.

I wanted to avoid that as much as possible.

2.1. Using Laptops as Kubernetes Nodes

One of the things I had to consider was that I was using existing machines rather than dedicated server hardware.

Unlike a typical server, a laptop might suspend when its lid is closed. Its network interface might behave differently after a reboot, and its power configuration might not be suitable for running workloads continuously.

So before focusing on Kubernetes, I needed to check a few things:

• Would the machine suspend when the lid was closed?
• Would it start automatically after power was restored?
• Would swap remain disabled after a reboot?
• Was the correct network interface configured?
• Could I access the machine remotely when needed?

These might seem like small details, but they matter when you're trying to operate a cluster.

Imagine a worker node suddenly disappearing because the machine suspended. Kubernetes would see the node become unavailable, but the actual problem would be the laptop's power configuration rather than Kubernetes itself.

I wanted to avoid situations like that.

2.2. Making Swap Configuration Persistent

One example was disabling swap.

I could run:

sudo swapoff -a 

This disables swap immediately, but it doesn't necessarily prevent swap from being enabled again after a reboot.

So I also checked the current swap configuration:

swapon --show grep -nE '^[^#].*\bswap\b' /etc/fstab 

The first command shows active swap devices. The second checks for active swap entries in /etc/fstab.

If an entry exists, I need to review it and make the appropriate change rather than assuming that disabling swap once is enough.

This is a small example, but it illustrates the difference between changing the current state of a machine and configuring it so that the intended state persists.

I applied the same general approach to other host-level requirements, including kernel modules, network configuration, and container runtime settings.

2.3. Preventing Unexpected Suspension

I also needed to account for the laptop-specific power behavior.

For machines that support it, I checked the systemd login configuration to prevent the system from suspending when the lid was closed. I also considered what would happen if power was disconnected and restored.

Some of these settings belong to the operating system, while others depend on the machine's BIOS or firmware.

For example, automatic power-on after AC restoration is generally a firmware setting, not something Kubernetes can configure.

The point was to make the underlying machines as predictable as possible before moving on to the Kubernetes installation.

3. Containerd, CRI, and CNI: Understanding What Was Actually Broken

After preparing the machines, I moved on to the container runtime.

During the initial setup, I tried to inspect the runtime using crictl, only to find that the command wasn't installed.

At first glance, this might look like a container runtime problem. However, a missing diagnostic tool doesn't automatically mean that the underlying service is broken.

I needed to distinguish between three different components:

crictl -> Diagnostic client for the CRI CRI -> Interface Kubernetes uses to communicate with the runtime containerd -> Container runtime CNI -> Configuration and plugins responsible for Pod networking 

These components are related, but they don't serve the same purpose.

If crictl isn't installed, I need to install the diagnostic tool first. If it is installed but cannot communicate with containerd, I need to investigate the runtime endpoint and socket. If the runtime works but Pod networking isn't configured yet, I need to investigate the CNI separately.

This distinction helped me avoid assuming that every problem during Kubernetes installation had the same cause.

I used the following commands to inspect the runtime:

# Check whether containerd is running systemctl is-active containerd # Check whether the containerd socket exists sudo ls -l /run/containerd/containerd.sock # Check whether crictl is installed command -v crictl || echo 'crictl not installed' 

After installing crictl from a trusted, version-appropriate source and configuring the correct endpoint, I could inspect the CRI:

sudo crictl \ --runtime-endpoint unix:///run/containerd/containerd.sock \ info 

The last command assumes that crictl has already been installed and configured appropriately.

Another thing I needed to consider was the cgroup driver.

Kubernetes and containerd need compatible cgroup configurations. On a systemd-based Linux distribution, using the systemd cgroup driver is the usual approach.

If the configurations don't match, the cluster can encounter problems that aren't necessarily obvious from simply checking whether the containerd service is running.

The official Kubernetes documentation explains this relationship in more detail:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

For me, the important part was understanding the order in which these components depend on one another.

Linux Configuration | v Containerd and CRI | v Kubelet and kubeadm | v Kubernetes Control Plane | v CNI Installation | v Pod Networking Tests 

I didn't want to jump straight to debugging Pod networking if the container runtime wasn't working correctly in the first place.

Getting the order right made the troubleshooting process much easier to reason about.

4. Multi-Interface Networking: Don't Let Address Selection Be Accidental

The networking decision was more interesting.

Some machines had multiple interfaces: wired Ethernet, Wi-Fi, and a virtual-network interface. That creates ambiguity if a component automatically selects a node address without knowing which network should carry cluster traffic.

There's a difference between a host having several valid IP addresses and Kubernetes consistently using the address intended for cluster communication.

4.1. Making the Node's InternalIP Part of the Design

I chose to make the intended node address explicit at the kubelet layer, then configured Calico to use the Kubernetes node's InternalIP.

The relationship looks like this:

Selected wired LAN address | v kubelet --node-ip | v Kubernetes NodeInternalIP | v Calico node autodetection | v Pod traffic 

I can inspect the addresses Kubernetes has registered with:

kubectl get nodes \ -o custom-columns='NAME:.metadata.name,INTERNAL_IP:.status.addresses[?(@.type=="InternalIP")].address' 

Calico's operator configuration can express the address-selection policy like this:

apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: kubernetes: NodeInternalIP 

This is just configuration a snipped. It needs to be merged with the rest of the installation settings.

Calico also supports other address-selection strategies, including interface names, interface exclusions, and CIDR-based selection. For this environment, using the Kubernetes InternalIP made the node-address configuration easier to reason about.

Reference: https://docs.tigera.io/calico/latest/networking/ipam/ip-autodetection

4.2. Why I Chose Full VXLAN for the Initial Platform

I also chose full VXLAN encapsulation instead of cross-subnet-only VXLAN for the initial network model.

That doesn't mean VXLANCrossSubnet is wrong. It's a valid mode, and avoiding unnecessary encapsulation can improve efficiency in the right network topology.

The trade-off looks like this:

For this small, non-production platform, I preferred a consistent overlay model. I also didn't want to introduce BGP integration with the underlying network just to get the initial platform working.

The configuration direction was:

# Fragment of the Calico operator Installation resource spec: calicoNetwork: bgp: Disabled ipPools: - cidr: <POD_CIDR> encapsulation: VXLAN natOutgoing: Enabled 

<POD_CIDR> is a placeholder. Choose a Pod CIDR that doesn't overlap with the networks already in use, and check the complete operator API for your installed Calico version.

Reference: https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip

5. Validation: What a Green Node Status Doesn't Prove

Once the control plane and workers had joined the cluster, I needed to separate a few different levels of validation.

5.1. Level One: The Node Has Joined the Cluster

I started with the usual checks:

kubectl get nodes -o wide kubectl get pods -A -o wide 

These commands show node status and where system Pods are running. They're useful, but Ready doesn't prove that every workload path, DNS query, or application requirement is working.

5.2. Level Two: A Workload Can Use Cluster Networking

For a disposable smoke test, I would validate service discovery separately from node readiness.

First, create a namespace and a small Deployment with two replicas:

kubectl create namespace network-smoke kubectl create deployment web-smoke \ -n network-smoke \ --image=nginx:1.27 \ --replicas=2 kubectl expose deployment web-smoke \ -n network-smoke \ --port=80 \ --target-port=80 \ --type=ClusterIP kubectl rollout status deployment/web-smoke -n network-smoke 

Then create a temporary Pod for DNS and HTTP checks:

kubectl run dns-smoke \ -n network-smoke \ --image=busybox:1.36 \ --restart=Never \ --command -- sleep 3600 kubectl wait --for=condition=Ready pod/dns-smoke \ -n network-smoke --timeout=120s kubectl exec -n network-smoke dns-smoke -- \ nslookup web-smoke.network-smoke.svc.cluster.local kubectl exec -n network-smoke dns-smoke -- \ wget -qO- http://web-smoke.network-smoke.svc.cluster.local 

After checking the results, remove the temporary resources:

kubectl delete namespace network-smoke 

These commands are an example smoke-test procedure, not a captured transcript from my cluster. They assume that the CNI and DNS are working and that the cluster can access the specified container images.

A successful test means the temporary Pod can resolve the Service and receive a response from the web application. That still doesn't prove resilience. To test scheduling and failure behavior, I would also need to check where the replicas land and what happens when a worker becomes unavailable.

5.3. How I'd Diagnose a Failed Smoke Test

The useful part of a smoke test isn't just seeing a successful response. If it fails, I need to know which layer to investigate next instead of immediately changing random configuration.

For example, a Service that doesn't respond isn't automatically a Calico problem. I'd first check the Service and its endpoints:

kubectl get service web-smoke -n network-smoke -o wide kubectl get endpointslice -n network-smoke \ -l kubernetes.io/service-name=web-smoke kubectl get pods -n network-smoke -o wide 

If there are no ready endpoints, the Service might not have anything healthy to forward traffic to. A successful DNS lookup doesn't change that.

If the EndpointSlices contain healthy endpoints, I can move on to connectivity and policy instead of changing the Deployment without evidence.

This is the approach I want to use throughout the series: establish a baseline, isolate a layer, change one thing, and repeat the same test.

5.4. Level Three: The Platform Can Support Normal Operations

This is where the rest of the platform work comes in.

I didn't need to solve all of these during the initial installation. But I needed to include them in my definition of what a usable platform would eventually look like.

6. Deliberate Omissions Are Architecture Decisions Too

There were also things I didn't want to add just to make the architecture diagram look more enterprise-ready. Hey, this is a non-production ;)

For example:

• I wouldn't say PostgreSQL is highly available just because Kubernetes can restart its Pod on another worker.
• I wouldn't say storage is highly available if the storage server and control plane share a physical failure domain.
• I wouldn't say an RTO has been achieved before completing a timed restore test.
• I wouldn't introduce every observability component before establishing the basics: metrics, logs, and actionable alerts.

This isn't an argument against highly available databases, distributed storage, or distributed tracing. Those are useful tools when the requirements justify their cost and complexity.

For a constrained non-production environment, I'd rather understand exactly what a single-instance system can recover from than describe it as highly available simply because Kubernetes can restart containers.

7. What I Learned From This First Phase

The biggest change in my approach was to stop treating deployment as proof that something works.

A diagnostic binary being missing doesn't mean the runtime is broken.

A node having an IP address doesn't mean Kubernetes is using the right one.

Installing a CNI doesn't prove that DNS and cross-Pod communication work.

And a cluster reporting Ready doesn't prove that its security boundaries, delivery process, monitoring, or recovery procedures work.

This is what made the project more interesting to me than following a standard installation guide. Each layer required me to decide what I could trust, what I needed to test, and what could wait until later.

In the next article, I'll go deeper into the networking decisions: what changes when Kubernetes nodes have multiple interfaces, how Calico selects addresses, and why the overlay mode matters.

Until then, my working rule is:

Don't move to the next layer until i understand what the current layer has actually proved.