Hola Amigos 👋

Welcome back to K8s-With-Pravesh!

After covering the Datastore of Kubernetes: etcd, we're going deep into what is popularly known as the "Autopilot" of the Kubernetes Control Plane: the Controller Manager.

We'll understand how it plays a crucial role in continuously reconciling the state of our cluster.

For those of you who are new here, I'm Pravesh, an AWS Community Builder and IBM Champion. In this series, we're dissecting all the important components of a Kubernetes cluster.

So, get your terminals ready, and let's dive right in!

What is Kubernetes Controller Manager?

The Kubernetes Controller Manager (kube-controller-manager) is a core component of the Control Plane that runs multiple built-in controllers within a single process.

Each controller has a specific responsibility, but they all follow a similar mechanism: they continuously watch the state of the cluster, and whenever the actual state differs from the desired state, they take action to bring things back in sync.

This process is popularly known as the Reconciliation Loop.

The reconciliation loop operates through three major steps:

  • Watch: Watches the API Server to get updates about the current state of the cluster.
  • Compare: Compares the actual state against the desired state defined in the YAML configuration (which is stored in etcd).
  • Act: If there is a mismatch (a "drift"), it takes action to correct it.

Let's take a simple example.

Imagine you have deployed an application with 3 replicas. Due to some reason, one of the Pods gets deleted.

Now the Controller Manager notices that the current state has only 2 Pods, whereas the desired state specifies 3.

What happens next?

The ReplicaSet Controller notices this mismatch and takes action to create another Pod.

And just like that, Kubernetes brings the cluster back to its desired state.

Pretty cool, right?

Different Types of Controllers

The Kube Controller Manager is not a single controller. It is actually a combination of multiple built-in controllers compiled together into a single binary to avoid the overhead of managing multiple independent processes.

Here are some of the most critical controllers:

  • Node Controller: Monitors node health and handles situations where a node stops responding.
  • ReplicaSet Controller: Ensures that the desired number of Pods keeps running at any given time.
  • Deployment Controller: Manages declarative updates for Deployments and ReplicaSets, including rolling out new versions and rolling back to previous ones.
  • EndpointSlice Controller: Maintains EndpointSlice objects that keep track of the Pods associated with a Service.
  • Service Account Controller: Automatically creates a default ServiceAccount for newly created namespaces.
  • Job Controller: Watches Job objects and manages the lifecycle of batch tasks, ensuring that they run to completion.

There are several other controllers as well, but these are some of the most commonly encountered ones.

What about Cloud Controller Manager?

If you're using Kubernetes on a cloud platform like AWS, GCP, or Azure, there's another interesting component called the Cloud Controller Manager.

It allows the core Kubernetes project to remain independent of cloud-provider-specific implementations while allowing cloud vendors to manage their own infrastructure integrations.

For example:

  • Cloud Load Balancers
  • Node routing
  • Underlying storage volumes

Unlike the built-in controllers running inside kube-controller-manager, the Cloud Controller Manager runs as a separate component.

Controller Manager vs Scheduler

It's very common to confuse these two components, but both have completely different responsibilities.

  • Controller Manager: Ensures that the required objects exist and that the actual state matches the desired state. For example, maintaining 3 replicas of a Deployment.
  • Scheduler: Decides where those unscheduled Pods should run. For example, assigning Pod A to Node 3 and Pod B to Node 2.

The Scheduler considers several factors during node selection, including available CPU, memory, resource requests, scheduling constraints, taints, and tolerations.

So, remember:

Controller Manager decides WHAT needs to happen, while Scheduler decides WHERE a Pod should run.

High Availability Architecture: Active-Passive

If you've watched the previous episode of this series, where we discussed the RAFT consensus algorithm for Active-Active quorum, things are going to be a little different here.

The Controller Manager uses an Active-Passive (or Leader-Standby) architecture for High Availability.

Here's how it works:

  • We can run 3 or more instances of kube-controller-manager on different Control Plane nodes for redundancy.
  • Only one instance will be designated as the Leader (Active) and will actively run the reconciliation loops.
  • The other instances remain in Standby (Passive) mode, waiting for the leader to fail so they can take over.

But how does Kubernetes decide who becomes the leader?

Well, here's where things get interesting.

Leader Election Using Kubernetes Leases

The Controller Manager uses a built-in Kubernetes resource called Lease to coordinate leader election.

Here's the process:

  • When the Control Plane starts, all instances of Controller Manager compete to acquire or update a specific Lease object in the kube-system namespace.
  • Out of all the instances, the one that successfully acquires the Lease becomes the leader.

(The winner takes it all! 😄)

  • The leader must continuously renew this Lease (by default, every 2 seconds) to prove that it is still alive.
  • If the leader node crashes or loses network connectivity, it fails to renew the Lease.
  • Once the Lease expires (the default lease duration is 15 seconds), another standby instance notices the vacancy and competes to acquire the Lease.

And just like that, a new leader is elected, and the process continues.

This is how Kubernetes maintains High Availability for the Controller Manager without allowing multiple instances to actively perform the same controller work under normal circumstances.

Performance Tuning: Handling Large Clusters

By default, the Controller Manager is configured with reasonable settings to balance CPU and memory consumption.

However, in a large cluster (probably hundreds of nodes and thousands of Pods), the default settings can sometimes cause reconciliation delays.

For example, imagine scaling a Deployment from 3 to 100 replicas, but it takes several seconds before the controllers start processing all the changes.

This is where performance tuning comes into the picture.

Kubernetes provides several flags to control the concurrency of different controllers.

Here are two examples:

  • --concurrent-deployment-syncs: Controls how many Deployment objects can be synchronized concurrently. By default, it is 5. Increasing this value can help speed up Deployment reconciliation in large clusters.

  • --concurrent-endpoint-syncs: Controls how many Service endpoint synchronization operations can run concurrently. This can be useful when dealing with frequent scaling operations in large clusters.

Of course, increasing concurrency isn't always the answer. More concurrent operations also mean more CPU, memory, and API Server load.

So, always monitor your cluster before changing these values.

Split-Brain Scenario

While Leader Election (which we discussed earlier) is excellent, network partitions can cause a scary scenario called Split-Brain.

Let's understand it with an example.

Imagine Controller Manager A is currently the leader.

Suddenly, the network splits, and Controller Manager A can no longer communicate with the API Server. As a result, it cannot renew its Lease.

Controller Manager B notices that the Lease has expired and takes over as the new leader.

However, Controller Manager A might still be alive and could potentially continue executing some operations based on its local state.

Now we have a problem: two instances potentially behaving like leaders.

How does Kubernetes protect against this?

One important mechanism is Optimistic Concurrency Control at the API Server level (something we touched upon in our previous episode).

Kubernetes resources have a resourceVersion, which helps detect conflicting updates.

If Controller Manager A tries to update an object using an outdated version, the API Server rejects that update with a conflict.

This helps prevent stale updates from overwriting newer cluster state.

It's important to understand that resource version checks are not a universal solution to every possible split-brain side effect. Leader election is the primary coordination mechanism, while optimistic concurrency provides an additional layer of protection for resource updates.

Enough Theory, Let's Get Our Hands Dirty!

Now let's open our terminals and see the Controller Manager doing its work in action.

The code and commands for today's practical demonstration are available in the GitHub repository of the K8s-with-Pravesh series.

Navigate to the part-06-CM directory, and you'll find everything you need.

1. Setting Up the Kubernetes Cluster

Before we dive in, we need a Kubernetes cluster.

For the sake of simplicity, let's use Minikube.

Make sure your Docker daemon is up and running, and spin up a Minikube cluster using the following command:

minikube start

If you don't have Minikube or Docker installed, head over to their official websites and install them according to your operating system and architecture.

2. Testing the Self-Healing Mechanism

Now let's see the self-healing feature of Kubernetes in action.

We'll create an Nginx Deployment with 3 replicas using the following command:

kubectl create deployment nginx-demo --image=nginx --replicas=3

Open a new terminal and run the following command to continuously watch the Pod lifecycle:

kubectl get pods -l app=nginx-demo -w

In another terminal, list the created Pods:

kubectl get pods

Now, copy the name of one of the Pods and delete it:

kubectl delete pod <Pod-name>

Switch back to the terminal where we're watching the Pod lifecycle.

You'll notice something interesting.

The moment the Pod gets terminated, Kubernetes notices that the desired number of replicas is no longer being maintained.

Within a few seconds, another Pod is created!

That's the Reconciliation Loop in action.

The ReplicaSet Controller noticed the mismatch and took action to bring the cluster back to its desired state.

Once the practical demo is done, make sure to delete the Deployment:

kubectl delete deployment nginx-demo

3. Inspecting Controller Manager Leader Election

Now let's explore something more interesting: Leader Election using Kubernetes Leases.

We can inspect the Lease objects using the following command:

kubectl get leases -n kube-system

For a more detailed YAML output:

kubectl get leases -n kube-system -o yaml

If you're using a multi-Control-Plane architecture, you can specifically inspect the Controller Manager Lease:

kubectl get lease kube-controller-manager -n kube-system -o yaml

Look for the holderIdentity field.

This field tells us which instance currently holds the Lease.

We can also continuously watch the Lease:

kubectl get lease kube-controller-manager -n kube-system -w

In a multi-Control-Plane setup, you can observe the Lease changing when leadership transfers to another instance.

A small catch with Minikube:

Since we're using a single-node Minikube cluster, it generally runs only one Controller Manager instance. So, we can inspect the Lease mechanism, but we cannot properly demonstrate an actual leader takeover using this setup.

For that, you'll need a multi-Control-Plane Kubernetes cluster with multiple Controller Manager instances.

Once you're done with the experiments, make sure to delete your cluster:

minikube delete

What About Custom Controllers and Operators?

After going through all of this, you might have another question:

"If the Controller Manager already has so many built-in controllers, how do custom operators like Prometheus Operator or cert-manager work?"

Great question!

The built-in controllers are compiled into the Kubernetes Controller Manager binary. We cannot simply add our own controller code to that existing process without modifying and rebuilding it.

But Kubernetes provides a way to extend its functionality using Custom Operators.

These operators run as separate workloads (usually Pods) inside the cluster.

They follow the exact same Reconciliation Loop pattern that we discussed earlier, but instead of only watching core Kubernetes resources, they can watch Custom Resource Definitions (CRDs) and manage the resources associated with them.

This is one of the most powerful features of Kubernetes because it allows developers to extend Kubernetes with their own custom automation.

Conclusion

And that's the end of our deep dive into the Kubernetes Controller Manager!

Today, we explored its functionalities, different built-in controllers, how Lease-based Leader Election works, how Kubernetes handles conflicting updates, and finally, we got our hands dirty with a small practical demonstration.

But the journey is not over yet!

We've covered etcd, the Controller Manager, and the API Server. Now it's time to explore another important component of the Control Plane.

In the next episode, we'll dive into the Kubernetes Scheduler and see what actually happens behind the scenes when Kubernetes decides where to run our Pods.

If you enjoyed this blog, make sure to follow me on my socials. I regularly share my DevOps experiments, Kubernetes learnings, and cloud projects there.

Until then, keep learning, keep experimenting!

Adios Amigos 👋

Connect With Me