November 2026 | ~18 min read

The Shared Cluster That Wasn't Sharing

Three incidents in one month. Same cluster. Different root cause every time. All preventable.

Week 1: Noisy neighbor. The data team kicked off a backfill job — 40 pods, no resource limits, no dedicated node pool. Within minutes, the job consumed 80% of cluster CPU. The payments API started returning 504s. On-call spent two hours tracking it down because kubectl top nodes showed everything was "healthy" — the nodes were just saturated. Revenue impact: $12,000 in failed transactions.

Week 2: Accidental scraper. A misconfigured frontend pod had a Prometheus scrape target pointing at * instead of its own namespace. It was hitting every metrics endpoint in the cluster every 15 seconds. Prometheus's memory ballooned from 4GB to 28GB, OOMKilled, and took every Grafana dashboard with it. The monitoring team didn't notice for 3 hours because… their monitoring was down.

Week 3: Wrong namespace delete. A junior ML engineer with cluster-admin ran kubectl delete configmap --all in what they thought was their dev namespace. It was payments-prod. The payments service lost its feature flags, rate limit configs, and API keys. Full recovery took 45 minutes from the last backup.

Every one of these incidents had the same root cause: 12 teams sharing one EKS cluster with zero governance. No RBAC beyond cluster-admin. No network isolation. No resource guardrails. No admission control. The cluster was a shared apartment where everyone had the master key and nobody locked their door.

I built the multi-tenant platform to fix this. Not a migration to multiple clusters — that's expensive and operationally brutal. A single cluster with proper tenant isolation. Namespace boundaries. Least-privilege RBAC. Deny-by-default network policies. Workload-class node pools. Policy-as-code admission control. Per-team observability with RBAC'd dashboards.

All Terraform. All production-ready. Here's everything.

The Numbers 

┌─────────────────────────────────────────────────────────────────────┐ │ MULTI-TENANT PLATFORM RESULTS │ ├──────────────────────────────────┬──────────────┬───────────────────┤ │ Metric │ Before │ After │ ├──────────────────────────────────┼──────────────┼───────────────────┤ │ Teams on cluster │ 12 │ 12 │ │ Namespaces │ 14 │ 36 │ │ RBAC model │ cluster-admin│ namespace-scoped │ │ Cross-namespace access │ 100% open │ deny-by-default │ │ Noisy neighbor incidents/quarter │ 7 │ 0 │ │ Onboarding time (new team) │ 2 weeks │ 45 minutes │ │ Policy violations blocked/week │ 0 │ ~340 │ │ Karpenter NodePools │ 1 │ 4 │ │ P99 scheduling latency │ 38s │ 8s │ │ Platform engineer time/week │ 20 hrs │ 5 hrs │ │ Monthly compute cost │ $6,800 │ $7,200 (+6%) │ └──────────────────────────────────┴──────────────┴───────────────────┘

Yes, cost went up 6%. That's intentional. We traded $400/month for zero noisy-neighbor incidents, 75% less platform toil, and 96% faster onboarding. The three incidents in month one cost us more than $400 in engineering time alone.

Table of Contents

  • Architecture Overview
  • Namespace Strategy and Resource Quotas
  • RBAC with Least Privilege
  • Network Policies for Tenant Isolation
  • Karpenter NodePools by Workload Class
  • Autoscaling — HPA, KEDA, and Topology Spread
  • Platform Governance with Kyverno
  • Per-Team Observability
  • Lessons Learned
  • Action Items

Architecture Overview 

┌──────────────────────────────────────────────────────────────────────────┐ │ MULTI-TENANT EKS PLATFORM │ │ │ │ 12 Teams │ │ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ │ │ │Payments│ │ ML │ │ Data │ │Frontend│ │ Auth │ │ ... │ │ │ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ │ │ │ │ │ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ ▼ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ 36 Namespaces ({team}-{env}) │ │ │ │ payments-prod payments-staging payments-dev │ │ │ │ ml-prod ml-staging ml-dev │ │ │ │ data-prod data-staging data-dev ... │ │ │ └────────────────────────┬─────────────────────────────────┘ │ │ │ │ │ ┌────────────────────────┼─────────────────────────────────┐ │ │ │ Admission Control │ Network Policies │ │ │ │ ┌──────────┐ ┌─────┴──────┐ ┌──────────────┐ │ │ │ │ │ Kyverno │ │ Cilium CNI │ │ Deny-by- │ │ │ │ │ │ 5 policies│ │ │ │ default │ │ │ │ │ └──────────┘ └────────────┘ └──────────────┘ │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌────────────────────────┼─────────────────────────────────┐ │ │ │ 4 Karpenter NodePools │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ │ │ Critical │ │ General │ │ Batch │ │ System │ │ │ │ │ │ On-demand│ │ 70% spot │ │ 100% spot│ │ On-demand│ │ │ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │ └──────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌────────────────────────┼─────────────────────────────────┐ │ │ │ Observability │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ │ │ Grafana RBAC │ │ Prometheus │ │ CloudWatch │ │ │ │ │ │ per-team │ │ │ │ │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ └──────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘

The Terraform module layout:

terraform/ ├── main.tf ├── variables.tf ├── outputs.tf ├── teams.tf # Team definitions (single source of truth) ├── namespaces.tf # Namespaces + ResourceQuotas + LimitRanges ├── rbac.tf # ClusterRoles + RoleBindings + EKS access entries ├── network-policies.tf # Cilium + deny-by-default + allow rules ├── karpenter-nodepools.tf # 4 NodePools + EC2NodeClasses ├── autoscaling.tf # HPA + KEDA + PDB templates ├── kyverno.tf # Helm release + 5 ClusterPolicies ├── observability.tf # Grafana folders + teams + dashboards └── modules/ └── team-onboarding/ # Reusable module for new team setup

The foundation is a single teams local that drives everything via for_each:

# teams.tf — Single source of truth for all team configuration locals { teams = { payments = { iam_roles = ["arn:aws:iam::123456789012:role/payments-dev"] environments = ["prod", "staging", "dev"] workload_class = "critical" quota_tier = "large" grafana_viewers = ["payments-lead@company.com", "payments-oncall@company.com"] } ml = { iam_roles = ["arn:aws:iam::123456789012:role/ml-dev"] environments = ["prod", "staging", "dev"] workload_class = "batch" quota_tier = "large" grafana_viewers = ["ml-lead@company.com"] } frontend = { iam_roles = ["arn:aws:iam::123456789012:role/frontend-dev"] environments = ["prod", "staging", "dev"] workload_class = "general" quota_tier = "medium" grafana_viewers = ["frontend-lead@company.com"] } auth = { iam_roles = ["arn:aws:iam::123456789012:role/auth-dev"] environments = ["prod", "staging", "dev"] workload_class = "critical" quota_tier = "medium" grafana_viewers = ["auth-lead@company.com"] } # ... 8 more teams follow the same pattern } # Flatten teams × environments into namespace map team_namespaces = merge([ for team, config in local.teams : { for env in config.environments : "${team}-${env}" => { team = team environment = env workload_class = config.workload_class quota_tier = config.quota_tier } } ]...) }

Every resource downstream — namespaces, RBAC bindings, network policies, Grafana folders — iterates over this map. Adding a new team is a single block in teams.tf and a terraform apply.

Namespace Strategy and Resource Quotas

Convention: {team}-{env}. Three namespaces per team. 12 teams × 3 environments = 36 namespaces.

Why three environments per team instead of shared staging/dev namespaces? Because shared environments recreate the noisy-neighbor problem at a smaller scale. Team A's staging load test shouldn't blow up Team B's staging integration tests.

# namespaces.tf resource "kubernetes_namespace" "team" { for_each = local.team_namespaces metadata { name = each.key labels = { "app.kubernetes.io/managed-by" = "terraform" "platform.company.io/team" = each.value.team "platform.company.io/env" = each.value.environment "platform.company.io/tier" = each.value.quota_tier "pod-security.kubernetes.io/enforce" = each.value.environment == "prod" ? "restricted" : "baseline" } } }

Resource Quotas by Tier

Three tiers. Prod gets the most. Dev gets enough to iterate but not enough to accidentally run a production-scale load test.

locals { quota_tiers = { large = { prod = { cpu = "8", memory = "16Gi", pods = "40", services = "10", pvcs = "10" } staging = { cpu = "4", memory = "8Gi", pods = "20", services = "5", pvcs = "5" } dev = { cpu = "2", memory = "4Gi", pods = "10", services = "3", pvcs = "3" } } medium = { prod = { cpu = "4", memory = "8Gi", pods = "20", services = "5", pvcs = "5" } staging = { cpu = "2", memory = "4Gi", pods = "10", services = "3", pvcs = "3" } dev = { cpu = "1", memory = "2Gi", pods = "5", services = "2", pvcs = "2" } } small = { prod = { cpu = "2", memory = "4Gi", pods = "10", services = "3", pvcs = "3" } staging = { cpu = "1", memory = "2Gi", pods = "5", services = "2", pvcs = "2" } dev = { cpu = "500m", memory = "1Gi", pods = "3", services = "1", pvcs = "1" } } } } resource "kubernetes_resource_quota" "team" { for_each = local.team_namespaces metadata { name = "${each.key}-quota" namespace = kubernetes_namespace.team[each.key].metadata[0].name } spec { hard = { "requests.cpu" = local.quota_tiers[each.value.quota_tier][each.value.environment].cpu "requests.memory" = local.quota_tiers[each.value.quota_tier][each.value.environment].memory "limits.cpu" = local.quota_tiers[each.value.quota_tier][each.value.environment].cpu "limits.memory" = local.quota_tiers[each.value.quota_tier][each.value.environment].memory pods = local.quota_tiers[each.value.quota_tier][each.value.environment].pods services = local.quota_tiers[each.value.quota_tier][each.value.environment].services "persistentvolumeclaims" = local.quota_tiers[each.value.quota_tier][each.value.environment].pvcs } } }

LimitRange for Default Container Limits

Without a LimitRange, pods with no resource requests bypass the quota entirely. This sets sane defaults so developers don't have to remember to set limits on every container.

resource "kubernetes_limit_range" "team" { for_each = local.team_namespaces metadata { name = "${each.key}-limits" namespace = kubernetes_namespace.team[each.key].metadata[0].name } spec { limit { type = "Container" default = { cpu = each.value.environment == "prod" ? "500m" : "250m" memory = each.value.environment == "prod" ? "512Mi" : "256Mi" } default_request = { cpu = each.value.environment == "prod" ? "100m" : "50m" memory = each.value.environment == "prod" ? "128Mi" : "64Mi" } max = { cpu = each.value.environment == "prod" ? "2" : "1" memory = each.value.environment == "prod" ? "4Gi" : "2Gi" } } } }

The result: every namespace has a hard resource ceiling and every container gets default limits. The data team's backfill job? It would now hit the data-prod quota ceiling at 8 CPU instead of consuming the entire cluster.

RBAC with Least Privilege

Before: every developer had cluster-admin. After: three roles with namespace-scoped bindings.

Three Roles

team-developer: Full CRUD within their team's namespaces. Can create deployments, services, configmaps, secrets, and debug with exec/logs. Cannot access other teams' namespaces or cluster-wide resources.

team-viewer: Read-only access to their team's namespaces. For stakeholders, QA engineers, and on-call rotations that need visibility but shouldn't modify anything.

platform-admin: Cluster-wide access for the platform team. The only role that can modify namespaces, node pools, cluster policies, and CRDs.

# ClusterRole: team-developer apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: team-developer rules: - apiGroups: ["", "apps", "batch"] resources: - pods - pods/log - pods/exec - deployments - replicasets - statefulsets - daemonsets - jobs - cronjobs - services - configmaps - secrets - persistentvolumeclaims - horizontalpodautoscalers verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["autoscaling"] resources: ["horizontalpodautoscalers"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["networking.k8s.io"] resources: ["ingresses"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["policy"] resources: ["poddisruptionbudgets"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] 
# ClusterRole: team-viewer apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: team-viewer rules: - apiGroups: ["", "apps", "batch", "autoscaling", "networking.k8s.io"] resources: ["*"] verbs: ["get", "list", "watch"]

Namespace-Scoped RoleBindings

The ClusterRoles define what. The RoleBindings define where. Each team gets developer bindings only in their own namespaces.

# rbac.tf resource "kubernetes_role_binding" "team_developer" { for_each = local.team_namespaces metadata { name = "${each.value.team}-developer" namespace = kubernetes_namespace.team[each.key].metadata[0].name } role_ref { api_group = "rbac.authorization.k8s.io" kind = "ClusterRole" name = "team-developer" } subject { kind = "Group" name = "${each.value.team}-developers" api_group = "rbac.authorization.k8s.io" } } resource "kubernetes_role_binding" "team_viewer" { for_each = local.team_namespaces metadata { name = "${each.value.team}-viewer" namespace = kubernetes_namespace.team[each.key].metadata[0].name } role_ref { api_group = "rbac.authorization.k8s.io" kind = "ClusterRole" name = "team-viewer" } subject { kind = "Group" name = "${each.value.team}-viewers" api_group = "rbac.authorization.k8s.io" } }

EKS Access Entries

EKS access entries are the modern replacement for the aws-auth ConfigMap. They're managed via the AWS API, version-controlled in Terraform, and don't have the single-point-of-failure problem where a bad ConfigMap edit locks everyone out.

resource "aws_eks_access_entry" "team" { for_each = local.teams cluster_name = aws_eks_cluster.main.name principal_arn = each.value.iam_roles[0] type = "STANDARD" kubernetes_groups = [ "${each.key}-developers" ] } resource "aws_eks_access_policy_association" "team" { for_each = local.teams cluster_name = aws_eks_cluster.main.name principal_arn = aws_eks_access_entry.team[each.key].principal_arn policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy" access_scope { type = "namespace" namespaces = [for env in each.value.environments : "${each.key}-${env}"] } }

The Migration Path

We didn't flip a switch. We ran both systems in parallel for two weeks.

  1. Week 1: Created all ClusterRoles, RoleBindings, and access entries alongside existing cluster-admin grants. Verified every team could still do their work.
  2. Week 2: Removed cluster-admin bindings one team at a time. Started with teams that had staging-only workloads. Payments (the most critical) went last.
  3. Day 15: Removed the legacy aws-auth ConfigMap entries. EKS access entries became the sole authentication source.

The junior ML engineer from the Week 3 incident? They now have team-developer scoped to ml-prod, ml-staging, and ml-dev. Running kubectl delete configmap --all in payments-prod would return Forbidden.

Network Policies for Tenant Isolation

RBAC controls who can talk to the API server. Network policies control which pods can talk to each other. Without network policies, any pod can reach any other pod in the cluster — even across namespaces. This is how the misconfigured frontend scraper from Week 2 was able to hit every Prometheus endpoint.

Cilium CNI

We replaced the default AWS VPC CNI's network policy enforcement with Cilium. AWS VPC CNI supports network policies natively since EKS 1.25, but Cilium gives us two things AWS CNI doesn't: Hubble (network flow visibility) and L7 policies (HTTP method/path filtering).

# network-policies.tf resource "helm_release" "cilium" { name = "cilium" repository = "https://helm.cilium.io/" chart = "cilium" version = "1.16.4" namespace = "kube-system" set { name = "eni.enabled" value = "true" } set { name = "ipam.mode" value = "eni" } set { name = "egressMasqueradeInterfaces" value = "eth0" } set { name = "routingMode" value = "native" } set { name = "hubble.relay.enabled" value = "true" } set { name = "hubble.ui.enabled" value = "true" } }

Default-Deny Everything

The first policy applied to every namespace: deny all ingress and egress. Then we add explicit allows.

# Policy 1: Default deny all traffic apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes: - Ingress - Egress 
resource "kubernetes_network_policy" "default_deny" { for_each = local.team_namespaces metadata { name = "default-deny-all" namespace = Source：https://dev.to/garrett_yan/building-a-multi-tenant-eks-platform-for-12-teams-rbac-network-policies-and-karpenter-with-3do9
My favorites
PreviousUsing AWS to Manage AI Spec-Driven Development ArtifactsNextGiving AI agents hands without handing over the keys: how we think about safety at TwinMCP

Related posts

I Put My Country's Entire Legal Code in Git: 300,000+ Documents Since 1810

I Put My Country's Entire Legal Code in Git: 300,000+ Documents Since 1810

HackerNoon
Gemini CLI's mkdir Failed, the Agent Kept Going, and Each move Overwrote the Last File

Gemini CLI's mkdir Failed, the Agent Kept Going, and Each move Overwrote the Last File

dev.to
I built a gesture app for macOS 26. Here's how it works and how launch week went

I built a gesture app for macOS 26. Here's how it works and how launch week went

dev.to
When Scripting Saves the Day... After Almost Deleting System32🥲

When Scripting Saves the Day... After Almost Deleting System32🥲

dev.to
Goldman Sachs restricts Claude Code and GitHub Copilot to engineers

Goldman Sachs restricts Claude Code and GitHub Copilot to engineers

dev.to
Convert PNG to JPEG or WebP in the Browser: What Happens to Transparency

Convert PNG to JPEG or WebP in the Browser: What Happens to Transparency

dev.to