Environment: AWS EKS

Namespace: dev

Tools: Kubernetes, Helm, Prometheus, Grafana, AWS EBS CSI Driver, Gmail SMTP

Estimated duration: 2–3 hours

Difficulty: Intermediate

Learning objectives

By the end of this lab, students will be able to:

1. Deploy Prometheus and Grafana into Kubernetes.
2. Configure persistent storage for Grafana using AWS EBS.
3. Understand the difference between emptyDir and PersistentVolumeClaims.
4. Configure Gmail SMTP securely using Kubernetes Secrets.
5. Create Grafana email contact points and notification policies.
6. Create an ImagePullBackOff alert using PromQL.
7. Simulate a Kubernetes incident and receive an email notification.
8. Verify Grafana data survives a Pod restart.

Important architecture decision

Students must use their own development EKS cluster or an instructor-approved isolated lab environment.

Do not modify the existing production monitoring release.

The namespace dev isolates Kubernetes objects, but it does not provide complete security or resource isolation. Students must use their own credentials and must not share passwords.

Part 1 — Verify the Kubernetes environment

Step 1: Verify AWS identity

aws sts get-caller-identity 

This confirms which AWS account and IAM identity the CLI is using.

Step 2: Connect to EKS

Replace the cluster name and region with your assigned development cluster.

export AWS_REGION="us-east-1" export EKS_CLUSTER="YOUR-DEV-CLUSTER" aws eks update-kubeconfig \ --region "$AWS_REGION" \ --name "$EKS_CLUSTER" 

Verify:

kubectl config current-context kubectl get nodes -o wide kubectl cluster-info 

Expected result: At least one worker node is Ready.

Step 3: Create the dev namespace

kubectl create namespace dev \ --dry-run=client -o yaml | kubectl apply -f - kubectl get namespaces 

All application and monitoring resources in this lab will be deployed into dev.

Part 2 — Verify AWS EBS storage

AWS EBS provides persistent block storage for Kubernetes workloads.

Step 1: Check the EBS CSI driver

aws eks describe-addon \ --cluster-name "$EKS_CLUSTER" \ --addon-name aws-ebs-csi-driver \ --region "$AWS_REGION" \ --query 'addon.{status:status,version:addonVersion,issues:health.issues}' 

Expected:

ACTIVE 

Also check:

kubectl get pods -n kube-system | grep ebs-csi 

If the add-on is missing, stop and ask the instructor to install it with the correct IAM permissions. Do not create an EBS CSI IAM role or install the add-on independently in a shared cluster.

Step 2: Check StorageClasses

kubectl get storageclass 

Look for a class provisioned by:

ebs.csi.aws.com 

For this lab, we will create a dedicated gp3 StorageClass if it does not already exist.

cat > dev-gp3-storageclass.yaml <<'EOF' apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: dev-gp3 provisioner: ebs.csi.aws.com parameters: type: gp3 encrypted: "true" reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true EOF 

Apply:

kubectl apply -f dev-gp3-storageclass.yaml kubectl get storageclass dev-gp3 

Explanation:

• gp3 is the AWS EBS volume type.
• encrypted: true enables EBS encryption.
• Delete allows Kubernetes to delete the EBS volume when the PVC is deleted, subject to the provisioner and cloud cleanup succeeding.
• WaitForFirstConsumer helps schedule the Pod and EBS volume in the same availability zone.

StorageClasses are cluster-scoped, so this step requires instructor approval in shared environments.

Part 3 — Install Prometheus and Grafana in dev

Step 1: Install Helm

Check first:

helm version 

If Helm is not installed, use the approved installation method for your operating system.

Step 2: Add the Helm repository

helm repo add prometheus-community \ https://prometheus-community.github.io/helm-charts helm repo update 

Step 3: Create a working directory

mkdir -p ~/jumptotech-monitoring-lab cd ~/jumptotech-monitoring-lab 

Step 4: Create monitoring values

cat > monitoring-values.yaml <<'EOF' alertmanager: enabled: false grafana: persistence: enabled: true type: pvc storageClassName: dev-gp3 accessModes: - ReadWriteOnce size: 5Gi resources: requests: cpu: 100m memory: 128Mi limits: cpu: 300m memory: 512Mi prometheus: prometheusSpec: retention: 6h resources: requests: cpu: 200m memory: 512Mi limits: cpu: 500m memory: 1Gi EOF 

We are enabling persistent storage from the beginning, so students do not need to migrate an existing Grafana database.

Step 5: Install the monitoring stack

helm upgrade --install dev-monitoring \ prometheus-community/kube-prometheus-stack \ --version 92.7.0 \ --namespace dev \ -f monitoring-values.yaml \ --wait \ --timeout 15m 

Verify:

helm list -n dev kubectl get pods -n dev kubectl get deployments -n dev kubectl get pvc -n dev 

Expected:

• Grafana Pod running
• Prometheus Pod running
• Grafana PVC bound to EBS
• Helm release deployed

If the installation fails, inspect Pod events and resource capacity before retrying.

Part 4 — Verify Grafana persistent storage

Step 1: Find the Grafana Deployment

kubectl get deployment -n dev \ -l app.kubernetes.io/name=grafana 

Expected Deployment name:

dev-monitoring-grafana 

Step 2: Inspect the storage volume

kubectl get deployment dev-monitoring-grafana \ -n dev \ -o jsonpath='{.spec.template.spec.volumes}' echo 

Look for a persistentVolumeClaim storage volume.

Step 3: Verify the PVC

kubectl get pvc -n dev kubectl describe pvc -n dev 

The Grafana claim should be Bound.

Discussion question: What would happen to Grafana dashboards if the application used emptyDir instead of a PVC?

Part 5 — Access Grafana

Step 1: Retrieve the Grafana admin password

Do not paste the password into chat or a shared document.

kubectl get secret dev-monitoring-grafana \ -n dev \ -o jsonpath='{.data.admin-password}' \ | base64 --decode echo 

Step 2: Start port-forwarding

kubectl port-forward \ -n dev \ svc/dev-monitoring-grafana \ 3000:80 

Open:

http://localhost:3000 

If Kubernetes is managed from a remote EC2 machine, use an SSH tunnel to forward port 3000 to your own computer. Do not expose Grafana publicly just to complete this lab.

Login:

Username: admin Password: password retrieved from Kubernetes Secret 

Step 3: Verify Grafana health

From a second terminal, run:

curl -sS http://localhost:3000/api/health 

Expected:

{ "database": "ok" } 

Additional fields may be returned.

Part 6 — Configure Gmail SMTP

Step 1: Create a Google app password

Students must use their own Gmail account.

Open:

https://myaccount.google.com/apppasswords

Requirements:

• Google 2-Step Verification enabled
• A new app password created for the Grafana lab

Never use a normal Gmail login password.

Step 2: Create a Kubernetes Secret

Run:

read -r -p "Enter Gmail address: " GMAIL_ADDRESS read -r -s -p "Enter Gmail app password: " GMAIL_APP_PASSWORD echo kubectl create secret generic grafana-smtp \ -n dev \ --from-literal=username="$GMAIL_ADDRESS" \ --from-literal=password="$GMAIL_APP_PASSWORD" unset GMAIL_APP_PASSWORD 

Verify without revealing credentials:

kubectl get secret grafana-smtp -n dev 

Expected:

grafana-smtp Opaque 2 

Step 3: Create SMTP Helm values

cat > grafana-smtp-values.yaml <<'EOF' grafana: env: GF_SMTP_ENABLED: "true" GF_SMTP_HOST: "smtp.gmail.com:587" GF_SMTP_STARTTLS_POLICY: "MandatoryStartTLS" GF_SMTP_FROM_NAME: "JumpToTech Dev Alerts" envValueFrom: GF_SMTP_USER: secretKeyRef: name: grafana-smtp key: username GF_SMTP_PASSWORD: secretKeyRef: name: grafana-smtp key: password GF_SMTP_FROM_ADDRESS: secretKeyRef: name: grafana-smtp key: username EOF 

This uses the same Gmail address for authentication and the email sender.

Step 4: Preview the Helm deployment

helm template dev-monitoring \ prometheus-community/kube-prometheus-stack \ --version 92.7.0 \ --namespace dev \ -f monitoring-values.yaml \ -f grafana-smtp-values.yaml \ > monitoring-preview.yaml 

Inspect the Grafana SMTP environment variables and Secret references in the rendered Deployment.

Step 5: Upgrade Helm

helm upgrade dev-monitoring \ prometheus-community/kube-prometheus-stack \ --version 92.7.0 \ --namespace dev \ -f monitoring-values.yaml \ -f grafana-smtp-values.yaml \ --wait \ --timeout 15m 

Verify:

kubectl get pods -n dev \ -l app.kubernetes.io/name=grafana kubectl get deployment dev-monitoring-grafana \ -n dev \ -o jsonpath='{range .spec.template.spec.containers[?(@.name=="grafana")].env[*]}{.name}{"\n"}{end}' \ | grep GF_SMTP 

Students should see seven SMTP environment variable names.

Part 7 — Configure email notifications in Grafana

Open Grafana in the browser.

Step 1: Create a contact point

Navigate to:

Alerting → Notification configuration → Contact points

Click New contact point.

Enter:

• Name: Student-Dev-Email
• Integration: Email
• Address: your own Gmail address

Save.

Step 2: Send a test notification

Click Test → Send test notification.

Expected:

Test notification sent successfully 

Check Gmail inbox and Spam.

The test may arrive with a subject containing FIRING. This is a simulated alert and does not necessarily indicate a real incident.

Step 3: Configure the notification policy

Navigate to:

Alerting → Notification configuration → Notification policies

For an isolated student Grafana installation:

1. Open the Default policy menu.
2. Select Edit.
3. Change the default contact point to Student-Dev-Email.
4. Keep grouping and timing settings unchanged.
5. Click Update policy.

Verify:

Delivered to Student-Dev-Email 

Do not change a shared organization's default notification policy.

Part 8 — Create a real ImagePullBackOff incident

We will deliberately deploy an application with a nonexistent container image.

Use only the dev namespace.

Step 1: Create the broken Deployment

cat > broken-app.yaml <<'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: broken-restaurant namespace: dev labels: app: broken-restaurant spec: replicas: 1 selector: matchLabels: app: broken-restaurant template: metadata: labels: app: broken-restaurant spec: containers: - name: restaurant image: nginx:jumptotech-image-does-not-exist-2026 imagePullPolicy: Always ports: - containerPort: 80 EOF 

Apply:

kubectl apply -f broken-app.yaml 

Step 2: Observe the failure

kubectl get pods -n dev kubectl describe deployment broken-restaurant -n dev kubectl get events -n dev \ --sort-by=.lastTimestamp \ | tail -20 

The Pod should eventually show:

ImagePullBackOff 

Step 3: Verify the Prometheus metric

In Grafana, open Explore, select the Prometheus data source, and enter:

kube_pod_container_status_waiting_reason{ namespace="dev", reason="ImagePullBackOff" } 

Run the query in Instant mode.

Expected: a matching series with value 1 while the image-pull failure is present.

If no series appears, verify that kube-state-metrics is running and that Prometheus is scraping it before continuing.

Part 9 — Create the ImagePullBackOff alert

Navigate to:

Alerting → Alert rules → New alert rule

Step 1: Alert name

DEV - Kubernetes ImagePullBackOff 

Step 2: Prometheus query A

kube_pod_container_status_waiting_reason{ namespace="dev", reason="ImagePullBackOff" } 

Use an Instant query.

Step 3: Alert condition

Configure Grafana expressions:

• Query A: Prometheus metric
• Expression B: Reduce, Last of A
• Expression C: Threshold, B is above 0.5
• Set C as the alert condition

This evaluates each matching container series and fires when its value is 1.

Step 4: Evaluation

• Evaluation interval: 1 minute
• Pending period: 2 minutes
• Keep firing for: 0 minutes

Use a dedicated folder and evaluation group, such as:

Folder: JumpToTech-Dev Evaluation group: dev-kubernetes-alerts 

Step 5: Notification routing

Use the Grafana notification policy configured earlier:

Default policy → Student-Dev-Email 

Step 6: Alert message

Summary:

ImagePullBackOff detected in dev namespace 

Description:

Kubernetes cannot pull a container image. Check the Pod image, image registry, credentials, and Kubernetes events. 

Step 7: Save the alert

Wait for the alert to evaluate.

Expected progression:

Normal → Pending → Firing 

The email notification may arrive later because Grafana's notification policy also has grouping and delivery delays.

Part 10 — Fix the incident

Once the alert is firing, students must troubleshoot and resolve the problem.

Step 1: Inspect the Pod

kubectl get pods -n dev kubectl describe deployment broken-restaurant -n dev kubectl get events -n dev \ --sort-by=.lastTimestamp \ | tail -20 

Step 2: Fix the container image

kubectl set image deployment/broken-restaurant \ restaurant=nginx:1.27 \ -n dev 

Step 3: Verify recovery

kubectl rollout status deployment/broken-restaurant \ -n dev \ --timeout=180s kubectl get pods -n dev 

Expected:

Running 1/1 

Step 4: Verify alert recovery

Return to Grafana.

The alert should transition back toward Normal after Prometheus and Grafana observe that the image-pull problem has cleared.

Students should also check whether a resolved email notification arrives.

Part 11 — Test Grafana persistence

Students must prove that Grafana can retain data after Pod replacement.

Step 1: Create a test dashboard

In Grafana:

1. Create a new dashboard.
2. Name it JumpToTech Dev Persistence Test.
3. Add a simple Prometheus panel.
4. Save the dashboard.

Step 2: Restart Grafana

kubectl rollout restart deployment/dev-monitoring-grafana \ -n dev kubectl rollout status deployment/dev-monitoring-grafana \ -n dev \ --timeout=180s 

Step 3: Verify the PVC

kubectl get pvc -n dev kubectl get pods -n dev \ -l app.kubernetes.io/name=grafana 

Step 4: Check Grafana

Log back into Grafana and verify:

• The test dashboard still exists.
• The email contact point still exists.
• The alert rule still exists.
• The notification policy is unchanged.

Expected result: The Grafana database and its saved objects survive the Pod restart.

Part 12 — Troubleshooting

Useful troubleshooting commands:

kubectl get pods -n dev kubectl get pvc -n dev kubectl get events -n dev \ --sort-by=.lastTimestamp \ | tail -30 kubectl logs -n dev \ deployment/dev-monitoring-grafana \ -c grafana \ --tail=100 helm status dev-monitoring -n dev 

Never paste credentials or decoded Kubernetes Secret values into reports.

Part 13 — Cleanup and cost control

After the instructor approves cleanup, delete the deliberately broken application:

kubectl delete deployment broken-restaurant \ -n dev \ --ignore-not-found 

If this is a disposable, dedicated student cluster, the instructor may authorize removal of the monitoring release:

helm uninstall dev-monitoring -n dev 

Helm uninstall does not guarantee deletion of all PVCs or cloud volumes.

Before deleting any PVC, identify its exact PV and EBS volume, verify that it belongs to this lab, and confirm the data is no longer needed.

Do not delete shared namespaces, StorageClasses, EBS CSI drivers, or production volumes.

AWS EBS volumes and EKS resources may incur charges while they exist.

Homework submission requirements

Students must submit a report containing:

1. AWS EKS cluster and node verification.
2. dev namespace creation.
3. Helm monitoring installation output.
4. Grafana PVC showing Bound.
5. Grafana HTTP health result.
6. SMTP environment variable names (no passwords).
7. Grafana email contact point screenshot.
8. Successful test notification screenshot.
9. Broken Kubernetes Pod showing ImagePullBackOff.
10. Prometheus query result.
11. Grafana alert showing Firing.
12. Gmail alert notification screenshot.
13. Deployment recovery showing Running.
14. Grafana dashboard persistence verification.

Student reflection questions

Question 1: Why do we use a PVC for Grafana instead of emptyDir?

Question 2: Why is a Kubernetes Secret better than putting an SMTP password directly into Helm values?

Question 3: What is the difference between ImagePullBackOff and CrashLoopBackOff?

Question 4: What does Prometheus do, and what does Grafana do?

Question 5: Why do we configure both an alert rule and a notification policy?

Question 6: Why can an alert be Firing before an email arrives?

Question 7: Why should monitoring alerts include namespace, Pod, and container labels?

Question 8: What are the limitations of storing Grafana's SQLite database on a single EBS volume?

Final expected architecture

Kubernetes application → kube-state-metrics → Prometheus → Grafana alert evaluation → Grafana notification policy → Gmail SMTP → Student email.

Completion criteria: The student must demonstrate a real Kubernetes image-pull failure, receive an alert, fix the Deployment, and verify recovery while Grafana's configuration persists across a Pod restart.