Hands-On Kubernetes Autoscaling: Building a Local HPA Lab with KinD and Python

Have you ever wanted to experiment with Kubernetes Horizontal Pod Autoscaler (HPA) without spinning up costly cloud clusters on AWS, GCP, or Azure?

Testing autoscaling locally in your homelab or dev environment is one of the most effective ways to master Kubernetes workload mechanics. But when you try setting up HPA locally using KinD (Kubernetes in Docker), you quickly discover that it is not just about applying an HPA YAML file. You have to wire up metrics collection, handle container-to-host networking, configure CPU requests accurately, and navigate self-signed certificates.

In this walkthrough, I will take you through building a complete, local autoscaling lab from scratch—from writing a lightweight Python web service to triggering real-time pod scaling under simulated traffic surges.

🏛️ Architecture at a Glance

Here is how the components interact inside our local KinD cluster:

flowchart TD subgraph Traffic["Workload Ingress"] Generator["Load Generator Pod (BusyBox)"] end subgraph Cluster["KinD Cluster (Docker)"] SVC["Service: autoscaling-service (ClusterIP: 80 -> 8000)"] subgraph Workload["Deployment: autoscaling-app"] P1["Pod 1 (Base)"] P2["Pod 2 (Scaled)"] PN["Pod ... (Up to 5)"] end subgraph MetricsControl["Metrics & Autoscaling Pipeline"] Kubelet["Kubelet Summary API"] MS["Metrics Server\n(--kubelet-insecure-tls)"] HPA["HPA Controller\n(Target: 50% CPU)"] end end Generator -->|HTTP Flood| SVC SVC --> P1 SVC -.-> P2 SVC -.-> PN Kubelet -->|Scrapes CPU/Mem| MS MS -->|Custom Metrics API| HPA HPA -->|Adjust Replicas: 1 ➔ 5| Workload

The Workflow:

1. A Python HTTP app runs in a Kubernetes deployment with defined CPU requests (100m) and limits (500m).
2. An HPA controller targets 50% average CPU utilization across pods.
3. Metrics Server collects pod resource consumption from the Kubelet.
4. When we simulate traffic using a load generator pod, the CPU threshold is breached, and Kubernetes automatically provisions up to 5 replicas. Once load drops, the cluster safely scales back down to 1 replica.

🛠️ Prerequisites & Local Tooling

To follow along, make sure you have:

• Docker Desktop (or Docker Engine)
• KinD (brew install kind or via mise)
• kubectl (brew install kubectl or via mise)

(Optional) If you use mise or VS Code Dev Containers, you can manage these CLI tools declaratively with a mise.toml configuration:

[tools] docker-cli = "latest" kind = "latest" kubectl = "latest" python = "latest" 

Step 1: Writing the Python Microservice

We keep the application clean and lightweight using Python's built-in http.server. It exposes a root route for service requests and a /health endpoint for Kubernetes health probes.

app.py:

from http.server import BaseHTTPRequestHandler, HTTPServer class Handler(BaseHTTPRequestHandler): def do_GET(self): if self.path == "/health": self.send_response(200) self.end_headers() self.wfile.write(b"OK") return self.send_response(200) self.end_headers() self.wfile.write(b"Kubernetes Autoscaling App") def log_message(self, format, *args): return server = HTTPServer(("0.0.0.0", 8000), Handler) server.serve_forever() 

Next, wrap it in a slim Docker image using python:3.12-slim:

Dockerfile:

FROM python:3.12-slim WORKDIR /app COPY app.py . EXPOSE 8000 CMD ["python", "app.py"] 

Build the image locally:

docker build -t kubernetes-autoscaling:1.0 . 

Step 2: Creating the KinD Cluster (with DevContainer / Host Networking Support)

If you work inside Dev Containers or DevPod, running kubectl from inside a container to reach a KinD cluster running on the host requires special attention: 127.0.0.1 inside a container refers to the container itself, not the host machine!

To fix this, we configure certSANs in kind-config.yaml to include host.docker.internal:

kind-config.yaml:

kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 kubeadmConfigPatches: - | kind: ClusterConfiguration apiServer: certSANs: - host.docker.internal nodes: - role: control-plane 

Create the cluster:

kind create cluster --name autoscaling-demo --config kind-config.yaml 

Now load the locally built Docker image into the KinD node:

kind load docker-image kubernetes-autoscaling:1.0 --name autoscaling-demo 

💡 Tip: In local KinD setups, Kubernetes cannot pull images from your host's local Docker daemon unless you explicitly load them using kind load docker-image. Also set imagePullPolicy: IfNotPresent in your deployment manifest.

Step 3: Deploying and Patching Metrics Server

By default, Kubernetes HPA cannot autoscale without Metrics Server.

In production, Kubelets have valid certificates. In KinD, Kubelet serving certificates are self-signed, causing Metrics Server to throw certificate verification errors.

1. Deploy the official Metrics Server:

 kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml 

1. Patch Metrics Server with --kubelet-insecure-tls:

 kubectl patch -n kube-system deployment metrics-server --type='json' -p='[ {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"} ]' 

1. Verify that metrics are flowing:

 kubectl rollout status -n kube-system deployment metrics-server kubectl top nodes 

Once kubectl top nodes outputs CPU and memory metrics without errors, your metrics pipeline is operational.

Step 4: Crafting the Kubernetes Manifests

1. The Deployment (k8s/deployment.yaml)

Notice the resources.requests.cpu: "100m" specification. This is critical. HPA calculates CPU utilization percentage as:

$$\text{Utilization} = \frac{\text{Current CPU Usage}}{\text{Requested CPU}} \times 100$$

Without a defined CPU request, HPA cannot compute percentage-based targets and will report <unknown>/50%.

apiVersion: apps/v1 kind: Deployment metadata: name: autoscaling-app spec: replicas: 1 selector: matchLabels: app: autoscaling-app template: metadata: labels: app: autoscaling-app spec: containers: - name: autoscaling-app image: kubernetes-autoscaling:1.0 imagePullPolicy: IfNotPresent ports: - containerPort: 8000 resources: requests: cpu: "100m" limits: cpu: "500m" readinessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 2 periodSeconds: 5 livenessProbe: httpGet: path: /health port: 8000 initialDelaySeconds: 5 periodSeconds: 10 

2. The Service (k8s/service.yaml)

A ClusterIP service provides a steady, stable DNS entry point (autoscaling-service) regardless of how many pods scale up or down:

apiVersion: v1 kind: Service metadata: name: autoscaling-service spec: selector: app: autoscaling-app ports: - port: 80 targetPort: 8000 type: ClusterIP 

3. The HorizontalPodAutoscaler (k8s/hpa.yaml)

We configure scaling boundaries between 1 and 5 replicas, targeting 50% average CPU utilization:

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: autoscaling-app spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: autoscaling-app minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 

Deploy everything to your cluster:

kubectl apply -f k8s/ 

Check the initial state:

kubectl get hpa autoscaling-app 

You should see:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaling-app Deployment/autoscaling-app 0%/50% 1 5 1 1m 

Step 5: Putting the Autoscaler to the Test

Now comes the fun part: simulating high traffic load and watching Kubernetes respond in real time.

1. Watch HPA and Pods

Open two separate terminal windows or split panes:

Terminal 1 (Watch HPA):

kubectl get hpa autoscaling-app -w 

Terminal 2 (Watch Pods):

kubectl get pods -l app=autoscaling-app -w 

2. Trigger the Load Generator

In a third terminal, run an ephemeral BusyBox pod that hammers the service with continuous HTTP requests:

kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://autoscaling-service; done" 

3. The Scale-Up Reaction

Within 1–2 minutes, watch the CPU utilization spike well above the 50% target:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE autoscaling-app Deployment/autoscaling-app 0%/50% 1 5 1 2m autoscaling-app Deployment/autoscaling-app 78%/50% 1 5 2 3m autoscaling-app Deployment/autoscaling-app 134%/50% 1 5 4 4m autoscaling-app Deployment/autoscaling-app 55%/50% 1 5 5 5m 

Kubernetes automatically scales the deployment to maximum capacity (5 pods) to absorb the incoming flood.

4. The Scale-Down Cooldown

Terminate the load generator by pressing Ctrl + C.

Kubernetes uses a default stabilization window (typically 5 minutes) before scaling down to prevent pod "flapping" (rapid oscillation between scale-up and scale-down). Once the cooldown elapses and CPU utilization drops to 0%, HPA smoothly brings replica count back to 1.

🧠 Key Takeaways & Troubleshooting Wisdom

Setting up this lab highlights several foundational concepts every Kubernetes practitioner should master:

1. Host Networking vs Container Contexts: When running inside Docker containers or DevContainers, 127.0.0.1 refers to the container itself. Use host.docker.internal along with certSANs in your control plane config to enable seamless cross-environment communication.
2. CPU Requests are Not Optional for HPA: If you don't define resources.requests.cpu, HPA cannot calculate percentage-based utilization and will stall in an <unknown> status.
3. Kubelet TLS in Local Clusters: KinD nodes rely on self-signed certs. Metrics Server requires --kubelet-insecure-tls in local testbeds to successfully pull node metrics.
4. Readiness Probes Prevent Traffic Blackholes: During sudden scale-up, newly provisioned pods must pass their /health readiness check before receiving ingress traffic from the Service.
5. The 7-Layer Kubernetes Debugging Workflow: Whenever autoscaling behaves unexpectedly, check in order:
   • Pod Status (kubectl get pods)
   • Deployment Status (kubectl get deployment)
   • Service & Endpoints (kubectl get endpoints)
   • Metrics Server Health (kubectl top pods, nodes)
   • HPA Status & Events (kubectl describe hpa)
   • Cluster Events (kubectl get events --sort-by='.metadata.creationTimestamp')
   • Pod Logs (kubectl logs)

🏁 Conclusion

Building an autoscaling environment on your local machine gives you deep visibility into how Kubernetes orchestrates scaling events, monitors workloads, and manages service discovery—all without cloud provider costs or latency.

Try tweaking the HPA algorithm with custom metrics, memory thresholds, or scaling behaviors next!

Have questions or run into issues setting up your own KinD HPA lab? Drop a comment below and let's discuss!