Restaurant Company: Deploy, Configure, Scale & Troubleshoot

Goal

You are the DevOps Engineer responsible for deploying the Restaurant Company application into Kubernetes.

By the end of this lab, you should be able to explain and demonstrate:

EKS Cluster ↓ Worker Node ↓ Deployment ↓ ReplicaSet ↓ Pods ↓ Service ↓ Application ConfigMap + Secret ↓ Pods Resources CPU + Memory ↓ Pods Metrics Server ↓ HPA ↓ Deployment replicas

Do not simply copy commands. After every section, understand what Kubernetes created and why.

Part 1 — Verify Your Kubernetes Environment

Before creating anything:

kubectl get nodes

Your Worker Node should show:

STATUS Ready

Run:

kubectl get nodes -o wide

Then:

kubectl describe node <NODE-NAME>

Find:

Capacity: cpu: memory: pods: Allocatable: cpu: memory: pods:

Questions

Students must answer:

  1. What is a Worker Node?
  2. What is Capacity?
  3. What is Allocatable?
  4. Why can Allocatable be smaller than Capacity?
  5. How many Pods can your Node run?
  6. Is Pod capacity the same thing as CPU capacity?
  7. What does 1930m CPU mean if you see it?

Remember:

1000m = 1 CPU core 500m = 0.5 CPU 100m = 0.1 CPU

Part 2 — Create Your Working Directory 

mkdir -p ~/weekend-k8s-lab cd ~/weekend-k8s-lab

Verify:

pwd

You will create:

weekend-k8s-lab/ ├── configmap.yaml ├── secret.yaml ├── deployment.yaml ├── service.yaml └── hpa.yaml

Part 3 — ConfigMap

Create:

nano configmap.yaml

Add:

apiVersion: v1 kind: ConfigMap metadata: name: restaurant-config data: APP_ENV: "production" APP_NAME: "restaurant-company"

Apply:

kubectl apply -f configmap.yaml

Verify:

kubectl get configmap 
kubectl describe configmap restaurant-config

Understand

A ConfigMap stores non-sensitive configuration separately from the application image.

Examples:

APP_ENV APP_NAME LOG_LEVEL API_URL

Interview Question

Why shouldn't configuration always be hard-coded into the Docker image?

Part 4 — Secret

Create:

nano secret.yaml

For this classroom lab only:

apiVersion: v1 kind: Secret metadata: name: restaurant-secret type: Opaque stringData: DB_PASSWORD: "JumpToTech123"

Apply:

kubectl apply -f secret.yaml

Check:

kubectl get secret 
kubectl describe secret restaurant-secret

You may see:

Data ==== DB_PASSWORD: 13 bytes

Important

Do not commit real production passwords to GitHub.

Also remember:

Kubernetes Secret is not automatically "safe because it is a Secret." Base64 encoding is not encryption.

Part 5 — Deployment

Create:

nano deployment.yaml

Use:

apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.27 ports: - containerPort: 80 env: - name: APP_ENV valueFrom: configMapKeyRef: name: restaurant-config key: APP_ENV - name: APP_NAME valueFrom: configMapKeyRef: name: restaurant-config key: APP_NAME - name: DB_PASSWORD valueFrom: secretKeyRef: name: restaurant-secret key: DB_PASSWORD resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi" readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 10 periodSeconds: 20

Apply:

kubectl apply -f deployment.yaml

Check:

kubectl get deployment 
kubectl get pods 
kubectl get rs

Students should understand:

Deployment ↓ ReplicaSet ↓ Pods

Part 6 — Prove Self-Healing

Get Pods:

kubectl get pods

Delete ONE Deployment-managed Pod:

kubectl delete pod <POD-NAME>

Immediately:

kubectl get pods -w

Observe

You deleted one Pod.

But the desired state says:

replicas: 2

Therefore:

Desired = 2 Actual = 1 ReplicaSet ↓ creates replacement ↓ Actual = 2

Interview Question

Why did Kubernetes recreate the Pod?

Do not answer:

"Because Kubernetes always recreates Pods."

Correct idea:

The Pod is managed by a ReplicaSet created by the Deployment, and the controller reconciles actual state with desired state.

Part 7 — Verify ConfigMap and Secret Injection

Find a Pod:

kubectl get pods

Check ConfigMap values:

kubectl exec -it <POD-NAME> -- printenv APP_ENV

Expected:

production

Then:

kubectl exec -it <POD-NAME> -- printenv APP_NAME

Expected:

restaurant-company

For this classroom lab only, verify that the Secret variable exists:

kubectl exec -it <POD-NAME> -- printenv DB_PASSWORD

Do not expose production secrets this way.

Part 8 — Understand Resources

Your Deployment contains:

resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "256Mi"

Requests

Requests are especially important to scheduling.

New Pod ↓ CPU request = 100m Memory request = 128Mi ↓ Scheduler ↓ Find suitable Node

Limits

Limits constrain resource usage.

CPU limit exceeded → CPU may be throttled Memory limit exceeded → Container may be terminated / OOMKilled

Verify:

kubectl describe pod <POD-NAME>

Find:

Requests: cpu: 100m memory: 128Mi Limits: cpu: 500m memory: 256Mi

Calculation

If there are 6 Pods:

6 × 100m CPU = 600m requested CPU 6 × 128Mi = 768Mi requested Memory

Part 9 — Readiness Probe

Your Deployment contains:

readinessProbe: httpGet: path: / port: 80

Readiness asks:

Should Kubernetes send normal Service traffic to this Pod?

Now intentionally break it.

Change:

path: /

to:

path: /wrong-page

Apply:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

You may see:

READY 0/1

But:

STATUS Running

Important Interview Question

Can a Pod be Running but not Ready?

YES.

Investigate:

kubectl describe pod <POD-NAME>

Look at:

Events:

Now fix:

path: /

Apply again:

kubectl apply -f deployment.yaml

Watch:

kubectl get pods -w

Expected eventually:

1/1 Running

Part 10 — Liveness Probe

Your Deployment contains:

livenessProbe: httpGet: path: / port: 80

Remember:

Readiness → Should this workload receive traffic? Liveness → Is this workload healthy enough to keep running?

Repeated liveness failures can cause kubelet to restart the Container.

Interview Question

Explain the difference between:

Readiness Probe vs Liveness Probe

in your own words.

Part 11 — Create a Service

Create:

nano service.yaml

Add:

apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx ports: - port: 80 targetPort: 80 type: ClusterIP

Apply:

kubectl apply -f service.yaml

Check:

kubectl get svc

Then:

kubectl get endpoints nginx-service

You should see Pod IP addresses.

Architecture:

Client ↓ nginx-service ↓ selector: app: nginx ↓ matching Pods

Important

The Service does not randomly discover your application.

It uses:

selector: app: nginx

and finds Pods with:

labels: app: nginx

Part 12 — Break the Service

Now create a real incident.

Change:

selector: app: nginx

to:

selector: app: wrong-label

Apply:

kubectl apply -f service.yaml

Check:

kubectl get endpoints nginx-service

You may see no endpoints.

Now troubleshoot:

kubectl get pods --show-labels

Compare:

Service selector vs Pod labels

Fix:

selector: app: nginx

Apply:

kubectl apply -f service.yaml

Check again:

kubectl get endpoints nginx-service

Interview Scenario

Pods are Running, but the Service has no endpoints. What do you check?

Answer:

Pod labels and Service selector.

Part 13 — Metrics Server

Now check resource usage:

kubectl top nodes

Then:

kubectl top pods

Understand the difference:

REQUEST 100m CPU ≠ ACTUAL USAGE for example 5m CPU ≠ LIMIT 500m CPU

Three different concepts.

Architecture:

Pod / Container ↓ kubelet ↓ Metrics Server ↓ Kubernetes Metrics API ↓ HPA

Part 14 — Create HPA

Create:

nano hpa.yaml

Add:

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: nginx-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-deployment minReplicas: 2 maxReplicas: 8 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50

Apply:

kubectl apply -f hpa.yaml

Check:

kubectl get hpa

You may see:

TARGETS cpu: 0%/50%

Interpret:

0% = current CPU utilization 50% = target

Also:

kubectl describe hpa nginx-hpa

Part 15 — Understand HPA Before Testing It

HPA does not simply mean:

CPU reaches exactly 50% → create exactly 1 Pod

Instead, HPA evaluates the metric and calculates a desired replica count.

Also:

HPA ↓ changes desired replicas ↓ Deployment / ReplicaSet ↓ Pods are created/removed

For this CPU-utilization HPA, your CPU request matters:

requests: cpu: "100m"

Very simplified example:

CPU request = 100m actual usage 10m ≈ 10% actual usage 50m ≈ 50% actual usage 80m ≈ 80%

Part 16 — Generate Traffic

Make sure:

kubectl get svc nginx-service

Then create a load generator:

kubectl run load-generator \ --image=busybox:1.36 \ --restart=Never \ -- /bin/sh -c \ 'for i in 1 2 3 4 5 6 7 8 9 10; do while true; do wget -q -O- http://nginx-service > /dev/null; done & done; wait'

Now use multiple terminals.

Terminal 1 

kubectl get hpa -w

Terminal 2 

kubectl get pods -w

Terminal 3 

kubectl top pods

Observe:

Load Generator ↓ HTTP requests ↓ Service ↓ Nginx Pods ↓ CPU utilization ↑ ↓ Metrics Server ↓ HPA ↓ Desired replicas ↑ ↓ Deployment / ReplicaSet ↓ More Pods

Do not expect exact CPU percentages or exact scaling timing to be identical on every student's cluster.

Part 17 — Stop Traffic

Delete:

kubectl delete pod load-generator

Watch:

kubectl get hpa -w

CPU usage should decrease.

Eventually HPA can reduce replicas toward:

minReplicas = 2

Scale-down may not happen immediately.

Part 18 — Production Incident: Pending Pods

This is based on a real incident from class.

Suppose:

Node Pod Capacity = 29 Current Pods = 29

HPA wants additional replicas.

What happens?

HPA ↓ Desired replicas ↑ ↓ ReplicaSet creates new Pod objects ↓ Scheduler tries to place them ↓ Node = 29/29 Pods ↓ No available Pod slot ↓ Pending

Investigate:

kubectl get pods

Then:

kubectl describe pod <PENDING-POD>

Look at:

Events:

You might find:

FailedScheduling Too many pods

Important

This is different from:

Insufficient cpu

or:

Insufficient memory

Do not guess the root cause.

Read the Events.

Part 19 — Kubernetes Troubleshooting Challenge

For each scenario, write:

1. Symptom 2. Command used 3. Root cause 4. Fix 5. Verification

Incident A 

Pod = Pending

Start:

kubectl describe pod <POD>

Check Events.

Incident B 

Pod = Running READY = 0/1

Think:

Readiness Probe

Incident C 

Pods Running Service exists Endpoints = none

Think:

Service selector vs Pod labels

Incident D 

Container keeps restarting

Start:

kubectl describe pod <POD> kubectl logs <POD>

Consider liveness failures, application errors, OOMKilled, etc.

Incident E 

HPA shows unknown CPU metric

Check:

kubectl top pods kubectl describe hpa nginx-hpa

Also verify the workload has appropriate CPU requests for utilization-based CPU HPA.

Part 20 — Commands You Must Know

By Monday, students should be comfortable with:

kubectl get nodes kubectl describe node <NODE> kubectl get pods kubectl get pods -o wide kubectl get pods -w kubectl describe pod <POD> kubectl logs <POD> kubectl get deployment kubectl describe deployment nginx-deployment kubectl get rs kubectl get svc kubectl describe svc nginx-service kubectl get endpoints nginx-service kubectl get configmap kubectl get secret kubectl apply -f deployment.yaml kubectl apply -f service.yaml kubectl top nodes kubectl top pods kubectl get hpa kubectl get hpa -w kubectl describe hpa nginx-hpa

Final Architecture — Draw This Yourself

Every student must be able to draw and explain this without notes:

 USERS ↓ Load Balancer ↓ Service ↓ ┌───────────┼───────────┐ ↓ ↓ ↓ Pod 1 Pod 2 Pod 3 │ │ │ └──── Deployment ───────┘ ↓ ReplicaSet ConfigMap ─────────→ Pods Secret ────────────→ Pods Worker Node ├── CPU ├── Memory ├── Storage ├── Networking └── Pod Capacity Pods ↓ CPU / Memory usage ↓ kubelet ↓ Metrics Server ↓ HPA ↓ Deployment desired replicas ↓ ReplicaSet ↓ More / Fewer Pods

Weekend Interview Questions

Students should prepare answers to these without reading from notes:

  1. What is Kubernetes?
  2. What is Amazon EKS?
  3. What is the Control Plane?
  4. What is a Worker Node?
  5. What is a Pod?
  6. What is a Container?
  7. What does kubectl do?
  8. What does the Scheduler do?
  9. What is a Deployment?
  10. What is a ReplicaSet?
  11. What does replicas: 3 mean?
  12. What is desired state?
  13. What is reconciliation?
  14. What happens when you delete a Deployment-managed Pod?
  15. Why can Pod IP addresses change?
  16. What problem does a Service solve?
  17. What is ClusterIP?
  18. What is LoadBalancer?
  19. How does a Service find Pods?
  20. What are labels?
  21. What is a selector?
  22. What are endpoints?
  23. What happens when Service selector does not match Pod labels?
  24. What is a ConfigMap?
  25. What is a Secret?
  26. ConfigMap vs Secret?
  27. What are resource requests?
  28. What are resource limits?
  29. Who uses requests during scheduling?
  30. What is 100m CPU?
  31. What is 1000m CPU?
  32. What can happen when CPU reaches its limit?
  33. What can happen when Memory exceeds its limit?
  34. What does OOMKilled mean?
  35. What is a Readiness Probe?
  36. What is a Liveness Probe?
  37. Can a Pod be Running but 0/1 Ready?
  38. What is Metrics Server?
  39. What does kubectl top pods show?
  40. What is HPA?
  41. What does Horizontal scaling mean?
  42. What is minReplicas?
  43. What is maxReplicas?
  44. What does averageUtilization: 50 mean?
  45. Why are CPU requests important for CPU-utilization HPA?
  46. Does HPA directly create Pods?
  47. Does HPA create Worker Nodes?
  48. What happens if HPA wants more Pods but the Node is full?
  49. What does Pending mean?
  50. How do you find why a Pod is Pending?
  51. What does FailedScheduling mean?
  52. What does Too many pods mean?
  53. What's the difference between CPU capacity and Pod capacity?
  54. What do you check when Pods are Running but Service has no endpoints?
  55. What commands would you use if an application is down?

Required Submission

Each student should submit screenshots showing:

1. kubectl get nodes 2. kubectl get pods 3. kubectl get deployment 4. kubectl get rs 5. kubectl get svc 6. kubectl get endpoints nginx-service 7. kubectl get configmap 8. kubectl get secret 9. kubectl describe pod showing Requests/Limits 10. kubectl top nodes 11. kubectl top pods 12. kubectl get hpa 13. HPA before load 14. Load generator Running 15. HPA during CPU load 16. Additional Nginx Pods created by scaling 17. HPA after load is removed 18. One troubleshooting example and its Events