Karpenter 0.37 supports Kubernetes 1.23–1.30; the cluster ran 1.32. The order of the work on 2026-11-16: Karpenter, then AL2023, then the control plane.
If Karpenter on your EKS cluster logs "failed validating kubernetes version" with "error":"karpenter version is not compatible with K8s version 1.32", the controller is a release whose supported Kubernetes range ends below the cluster's version: 0.37.x supports 1.23–1.30, logs the mismatch once per controller start, and goes on creating nodes. Nothing is down because of that line. EKS also reports the Amazon Linux 2 (AL2) nodes as an ERROR for 1.33, so we moved Karpenter, its v1beta1 manifests and the AL2 nodes before the control plane. This article gives the order we used to upgrade EKS 1.32 to 1.36 in place, with Karpenter 0.37.0 → 1.14.3 and AL2 → AL2023. The cluster had been on extended support since 2026-03-23. The article lists the eight problems met on the way, two of them caused by our own actions, and eight read-only checks with the values that mean "same problem". The cluster is a company EKS cluster of about 11 nodes (9 created by Karpenter, 2 in a managed node group), declared in one Terraform repository and applied by a GitHub Actions workflow; all times are UTC, on 2026-11-16 unless a date is given.
What we saw
Before the work
An audit of the cluster on 2026-11-14 at 16:51 listed, as problem 5 of its top five, "Kubernetes 1.32 past its support, and Karpenter left behind" (title translated). It listed:
- control plane
v1.32.13-eks-577a51e, nine nodes onv1.32.9; - upstream patches for 1.32 ended 2026-02-28, EKS standard support ended 2026-03-23, 237 days before the report's local date (2026-11-15); support type
EXTENDED; - Karpenter
0.37.0, whose supported range is Kubernetes 1.23–1.30; - one "version not supported" error line in the Karpenter log at 2026-11-14 08:53, while Karpenter had still created 7 nodes in the previous 24 hours;
- EC2NodeClass
defaultwithamiFamily: AL2and no pinned AMI version; - four NodePools, each with the single disruption budget
nodes: "10%".
The same run counted one more finding under that problem, eks.nodegroup-ami-old: the managed node group AMI was 341 days old.
The audit's recommended order: Karpenter first, then the EC2NodeClass to AL2023 with a pinned AMI version, and only then Kubernetes 1.33.
At 03:15 on 2026-11-16 our own reading agreed with the audit and added two facts. There were 11 nodes: 9 created by Karpenter on Amazon Linux 2 (kernel 5.10, containerd 1.7.29) and 2 in a managed node group already on Amazon Linux 2023.9.20251208. aws eks list-insights returned 7 insights, 6 PASSING and 1 ERROR, named "Amazon Linux 2 compatibility", for 1.33:
Amazon EKS stopped publishing EKS-optimized Amazon Linux 2 (AL2) AMIs on November 26, 2025. AL2023 and Bottlerocket based AMIs for Amazon EKS are available for all supported Kubernetes versions including 1.33 and higher. The Karpenter Helm release was chart karpenter-0.37.0, 2 replicas, webhook.enabled: false, with no karpenter-crd release. The three Karpenter CRDs (nodepools, nodeclaims, ec2nodeclasses) served and stored only v1beta1, with conversion: None and no Helm labels.
A correction to the audit's "one error line at 2026-11-14 08:53": at 03:17 on 11-16 one of the two Karpenter 0.37.0 pods had 0 ERROR lines in its last 72 hours, and we did not read the other pod. We saw the message ourselves at 03:38:35, right after 0.37.8 started:
"failed validating kubernetes version" … "error":"karpenter version is not compatible with K8s version 1.32" It is logged when the controller discovers the cluster version, which is once per start. Karpenter 0.37.8 kept working after it and the EC2NodeClass stayed Ready=True.
During the work
What ran when on 2026-11-16: Karpenter and AL2023 applies, the four control-plane and node group updates, the add-on updates, and the incidents between them.
In the order they appeared:
1. A Terraform plan that would have destroyed every NodePool (03:51). Changing the manifests from karpenter.sh/v1beta1 to karpenter.sh/v1 made the first plan of PR 3 mark all five kubectl_manifest resources (the EC2NodeClass and the four NodePools) "must be replaced":
api_version = "karpenter.sh/v1beta1" -> "karpenter.sh/v1" # forces replacement Plan: 5 to add, 0 to change, 5 to destroy We did not apply it.
2. All eight existing NodeClaims marked Drifted, by our own apply (03:54:53). One second after the apply of PR 3 ended, Karpenter logged disrupting nodeclaim(s) via replace. At 03:55:18 all eight NodeClaims that existed before the apply had Drifted=True (NodeClassDrift) and three replacements were launching, one per NodePool that had nodes. The description of PR 3 had said the nodes would not change.
3. Two nodes left on AL2 for 50 minutes (04:02–04:52). PR 5 switched the EC2NodeClass to AL2023 at 04:01, and the first AL2023 Karpenter node was up at 04:02. At 04:52:43 the state was 7 Karpenter nodes on AL2023, 2 still on AL2, and nothing in progress. The events:
DisruptionBlocked … Pod has "karpenter.sh/do-not-disrupt" annotation (Pod=prod-a/api-a-…) and the same for prod-b/admin-b-…. Three pods in the cluster carried the annotation.
4. pull QPS exceeded: 27 pods not Running and target groups with no healthy target (05:13–05:16). At 05:13:09, 14 minutes after the 1.33 control-plane update started, 9 nodes were already on v1.33.13 and 27 pods were not Running. At 05:14:15, 25 of the 26 pods not Running were on one node that was 3 m 7 s old (an m8g.large in NodePool default), among them coredns, the load balancer controller, the EBS CSI controller, metrics-server, external-secrets and application pods. The EBS CSI node pod's event:
Failed to pull image "….amazonaws.com/eks/aws-ebs-csi-driver:v1.37.0": pull QPS exceeded At 05:15:29, 11 load balancer target groups had targets but no healthy one: in 8 the only target was draining, in 3 the targets were unhealthy. The cluster's owner reported 503 Service Temporarily Unavailable from the services in the same minutes. The limit of this observation: the 503s were not measured from outside. We have the target group states at one instant and one user report, and no duration per service. By 05:16:54 all pods were Running except Redis and pods being terminated.
5. public.ecr.aws/bitnami/redis in ImagePullBackOff with NotFound (05:13). redis-master-0 was in ImagePullBackOff:
Failed to pull image "public.ecr.aws/bitnami/redis:7.2.3-debian-11-r1": rpc error: code = NotFound and the same for bitnami/redis-exporter:1.55.0-debian-11-r2.
6. EKS refused the 1.34 update (05:20:22). The apply of PR 8 failed with:
Error: updating EKS Cluster (…) version: operation error EKS: UpdateClusterVersion, https response error StatusCode: 400, … InvalidRequestException: Atleast one subnet in each AZ should have 2 free IPs. Invalid AZs: { [zone-b] }, provided subnets: { … } 7. Both replicas of one API deleted at once, by our own script (05:42). Before each later control-plane step we cordoned the nodes whose replacement was blocked by an annotated pod and deleted that pod. At 05:42 the script deleted both replicas of api-a at once, because they ran on two different drifted nodes. Both replacements were Pending, then starting, for a few minutes.
8. Prometheus OOMKilled (06:24). prometheus-…-0 was in CrashLoopBackOff: container prometheus, 5 restarts, last state OOMKilled/137, limit 1536Mi.
One more delay had nothing to do with the cluster: the CI checkout step of PR 2 failed three times with fatal: repository '…' not found between 03:41 and 03:44 while GitHub's status page said "All Systems Operational". The fourth run passed with no change on our side.
At 06:34:48 the server was v1.36.5-eks-91f89a9, all 10 nodes were on v1.36.5-eks-9adbfea and Amazon Linux 2023.12.20261112, 0 pods were outside Running/Completed, and 9 of 9 Argo CD applications were Synced and Healthy. From the first merge (03:36:38) to that check: 2 h 58 m. From the first control-plane update call to the last: 63 m 57 s.
Why
karpenter version is not compatible with K8s version: a range check that logs in 0.37 and 1.0, and returns the error in 1.1
The supported range is two constants in pkg/providers/version/version.go of aws/karpenter-provider-aws. By tag:
| Karpenter tag | MinK8sVersion | MaxK8sVersion | On a version outside the range |
|---|---|---|---|
| v0.37.8 | 1.23 | 1.30 | logs and continues (observed on 1.32) |
| v1.0.14 | 1.25 | 1.31 | logs and continues (source and observed on 1.32) |
| v1.1.10 | 1.25 | 1.31 | returns the error (source only; not run) |
| v1.14.3 | 1.26 | 1.36 | in range for 1.32–1.36 |
In v1.0.14 an out-of-range version is only logged, and the function goes on:
log.FromContext(ctx).Error(err, "failed validating kubernetes version") // ... return version, nil In v1.1.10 the same check is returned to the caller:
return fmt.Errorf("validating kubernetes version, %w", err) We observed the logging behaviour. Karpenter 1.0.14 also logged the message (03:48:56) and then launched three nodes at 03:54:58–03:55:08, one of which registered within 41 s.
The returned error in 1.1 we only read. We did not run 1.1.x or any release between 1.0.14 and 1.14.3, and did not read what the caller does with the returned error (a failed reconcile or a crash). That 1.1.x would stop launching nodes on 1.32 is our reading of the returned error; because of it we went from 1.0.14 straight to a release whose range contains the current and the target Kubernetes version.
In v1.14.3 the message is reworded: karpenter is not compatible with kubernetes version (version.go line 125). The wording karpenter version is not compatible with K8s version is the one in v0.37.x through v1.2.0. The log command in step 1 below matches both.
The Karpenter compatibility matrix (karpenter.sh/docs/upgrading/compatibility, read at 03:18) gives the minimum Karpenter per Kubernetes minor:
| Kubernetes | 1.31 | 1.32 | 1.33 | 1.34 | 1.35 | 1.36 |
|---|---|---|---|---|---|---|
| Karpenter at least | 1.0.5 | 1.2 | 1.5 | 1.6 | 1.9 | 1.13 |
The same page lists v1.14 as an LTS release (released July 2026, end of life July 2027). The matrix gives only a minimum (1.13) for Kubernetes 1.36; that 1.14.3 itself allows 1.36 we take from the source constant (MaxK8sVersion = "1.36").
Supported Kubernetes range of four Karpenter releases, from version.go at each tag. Only v1.14.3 covers the whole 1.32 → 1.36 span.
Why 0.37 cannot go to 1.14 in one step
The upgrade guide (karpenter.sh/docs/upgrading/upgrade-guide, read at 03:55) says "Do not upgrade to 1.0.0+ without following the v1 Migration Guide" and "Karpenter 1.1.0 drops the support for v1beta1 APIs". The v1 migration guide (at tag v1.0.14) gives the path: first the latest patch of the current minor with webhook.enabled=true on both the karpenter-crd chart and the karpenter chart, then 1.0.x; before 1.1.0 every stored object must be v1. Its rollback targets are 0.37.1, 0.36.3, 0.35.6, 0.34.7 and 0.33.6.
The same upgrade guide lists what later releases need: in 1.1.0 "nodeClassRef.group and nodeClassRef.kind are strictly required"; 1.7.0 needs iam:ListInstanceProfiles; 1.11.0 adds ec2:DescribePlacementGroups; 1.12.0 adds ec2:DescribeInstanceStatus; 1.8.4 has a scheduling regression ("do not upgrade to this version"); 1.14.0 ships a new capacitybuffers CRD.
Our controller IAM policy is written by hand in Terraform, not taken from the upstream CloudFormation. Compared with the upstream policy by action name, nothing was missing for v1.0.14 and v1.1.10; for v1.14.3 four actions were missing: ec2:DescribeCapacityReservations, ec2:DescribeInstanceStatus, ec2:DescribePlacementGroups, iam:ListInstanceProfiles.
kubectl_manifest must be replaced: why api_version … # forces replacement planned 5 to destroy
The EC2NodeClass and the four NodePools are kubectl_manifest resources (provider gavinbunney/kubectl) holding YAML. The provider plans a change of api_version as a replacement; we know that from the plan line only and did not read the provider's schema.
Deleting a NodePool deletes its NodeClaims and therefore its nodes, and all nine Karpenter nodes belonged to those four NodePools. This part is an inference: we did not apply that plan, so the mass deletion is what the plan and Karpenter's ownership model imply, not something we watched.
Why the nodes were Drifted (NodeClassDrift) after the EC2NodeClass change
When the conversion webhook converted the stored objects to v1 (after PR 2), it rendered the EC2NodeClass with amiSelectorTerms: [{alias: al2@latest}] and metadataOptions.httpPutResponseHopLimit: 2. In PR 3 we copied those fields explicitly into the EC2NodeClass YAML.
Which field changed the hash was not isolated. The EC2NodeClass was the only object whose spec we rewrote by hand in that apply; that this rewrite caused the drift is an inference.
"Amazon Linux 2 compatibility" ERROR with amiFamily: AL2, and why two nodes stayed on AL2
The insight text above is the reason for the first: the nine Karpenter nodes ran AL2. SSM has an AL2 EKS AMI release for Kubernetes 1.32 (1.32.9-20251209) and none for 1.33 (ParameterNotFound for /aws/service/eks/optimized-ami/1.33/amazon-linux-2/recommended/release_version). EKS never refused a 1.33 update here: by the time we called it the nodes were on AL2023 and the insight had cleared. Whether a 1.33 update with AL2 nodes would have been refused, or would have left Karpenter with no image to launch, we did not observe.
For the second, the DisruptionBlocked event message contains the reason. The Karpenter disruption docs (karpenter.sh/docs/concepts/disruption, read at 08:24) say that nodes with active karpenter.sh/do-not-disrupt pods "will be excluded from Consolidation, and conditionally excluded from Drift. If the Node's owning NodeClaim has a terminationGracePeriod configured, it will still be eligible for disruption via drift", and that pods with the annotation "may block draining indefinitely". None of our four NodePools set terminationGracePeriod, before the work or after.
pull QPS exceeded: why the pods and the load balancers failed in the same minutes
Four things ran inside one terraform apply and one Karpenter reconcile: the control-plane update (started 04:58:50), Karpenter's drift replacement of its own nodes (starting within minutes), the managed node group update (05:06:08) and the add-on updates (05:14:38). Most Deployments in this cluster have one replica. That these four overlapping operations are why 25 pods were scheduled onto one new node and why 8 target groups had only a draining target is an inference from those times; no single log line states it.
After each control-plane step Karpenter replaced its nodes without any change to the EC2NodeClass. The alias al2023@v20261112 pins the AMI release date, and the same release exists per Kubernetes version (1.32.13-20261112 for 1.32, 1.36.5-20261112 for 1.36), so the alias resolves to the AMI of the cluster's Kubernetes version.
pull QPS exceeded is the kubelet's own limit on image pulls, not a refusal by the registry. The kubelet configuration reference (kubernetes.io) gives registryPullQPS as "the limit of registry pulls per second", default 5, and registryBurst, default 10. We did not read the kubelet configuration of the node, so that it ran with those defaults is an inference.
Why public.ecr.aws/bitnami/redis returned NotFound only now (bitnamilegacy)
The same two tags exist at docker.io/bitnamilegacy/ (the Docker Hub API answered 200). Kafka in the same repository was already pulling from bitnamilegacy, with a comment that the public.ecr.aws images had been deleted. That Redis had been running from an image cached on an old node is an inference: the pod was Running before the node replacement with the same image reference, and that reference now returns NotFound.
Atleast one subnet in each AZ should have 2 free IPs: a /24 at 0 free
The cluster uses two private /24 subnets, one per zone. At 05:21 the subnet in zone-b had AvailableIpAddressCount 0 and the one in zone-a 150. In zone-b, 21 in-use network interfaces held 251 private IPs; in zone-a, 17 held 101. Eight of the 15 nodes at that moment were in zone-b.
The VPC CNI ran with WARM_ENI_TARGET=1, WARM_PREFIX_TARGET=1, ENABLE_PREFIX_DELEGATION=false, no WARM_IP_TARGET and no MINIMUM_IP_TARGET. We did not measure the spare interfaces or addresses per node. The node count was 11 at 03:15 and 15 at 05:21.
The EKS user guide ("Update existing cluster to new Kubernetes version") says: "To update the cluster, Amazon EKS requires up to five available IP addresses from the subnets that you specified when you created your cluster." The API error we received says 2 in each AZ. We quote both and have not reconciled them.
The zone-b subnet at 05:21 and after the VPC CNI setting change: addresses held by network interfaces against free addresses.
What to check in your cluster now
All steps are read-only. <cluster> is your EKS cluster name. Each step ends with the value that means "same problem as ours".
1. Is Karpenter inside its supported Kubernetes range, and which release is next?
kubectl version | grep Server kubectl -n karpenter get deploy karpenter -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' kubectl -n karpenter logs -l app.kubernetes.io/name=karpenter --all-containers --since=72h --tail=-1 | grep -cE 'is not compatible with (K8s|kubernetes) version' Same problem as ours: the server is 1.32 and the image is 0.37.x (range 1.23–1.30). A count of 1 or more from the third command means the controller has logged that it is outside its range. The log line is written once per controller start, so a count of 0 over 72 hours does not mean the version is in range. The grep matches both wordings of the message (see "Why"), across all Karpenter pods (check the label on your release first). If the image is 0.3x, the path is fixed: latest patch of your minor with the webhook on → 1.0.x → manifests to v1 → a release whose range contains both your current and your target Kubernetes version (matrix in the previous section). Do not stop on 1.1.x while the cluster is on 1.32 (source reading, not a run).
2. Are the CRDs ready for the conversion, and does Helm own them?
for c in nodepools.karpenter.sh nodeclaims.karpenter.sh ec2nodeclasses.karpenter.k8s.aws; do kubectl get crd $c -o jsonpath='{.metadata.name}{" versions="}{range .spec.versions[*]}{.name}{"(storage="}{.storage}{") "}{end}{"stored="}{.status.storedVersions}{" conversion="}{.spec.conversion.strategy}{" helm="}{.metadata.labels.app\.kubernetes\.io/managed-by}{"\n"}' done helm list -A | grep -i karpenter Same as ours before the work: only v1beta1, conversion=None, helm= empty, and no karpenter-crd release. Before you move to a release that drops v1beta1 (1.1.0 and later), stored= must be ["v1"] for all three.
3. Will Terraform delete the NodePools when you change their apiVersion?
grep -rn 'kubectl_manifest' --include='*.tf' . | head # after changing apiVersion in a branch: terraform plan -no-color | grep -E 'must be replaced|forces replacement|^Plan:' Same problem: api_version = "karpenter.sh/v1beta1" -> "karpenter.sh/v1" # forces replacement and a Plan: line with as many to destroy as you have NodePools and EC2NodeClasses. Deleting a NodePool deletes its nodes, so do not apply that plan. What a safe plan looked like for us after terraform state rm and import blocks: Plan: 5 to import, 0 to add, 5 to change, 0 to destroy.
4. Does EKS already report an ERROR insight for the next version?
aws eks list-insights --cluster-name <cluster> --query 'insights[?insightStatus.status!=`PASSING`].[name,kubernetesVersion,insightStatus.status]' --output text kubectl get nodes -o custom-columns='NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage' kubectl get ec2nodeclass -o jsonpath='{range .items[*]}{.metadata.name}{" amiFamily="}{.spec.amiFamily}{" terms="}{.spec.amiSelectorTerms}{"\n"}{end}' Same problem: Amazon Linux 2 compatibility … ERROR, nodes whose OS is Amazon Linux 2, an EC2NodeClass with amiFamily=AL2. After the node image change, aws eks start-insights-refresh --cluster-name <cluster> re-evaluates without waiting for the daily run.
5. Which pods will stop Karpenter from replacing a node?
kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.annotations["karpenter.sh/do-not-disrupt"]=="true") | [.metadata.namespace,.metadata.name,.spec.nodeName] | @tsv' kubectl get events -A --field-selector reason=DisruptionBlocked --sort-by=.lastTimestamp | tail Same problem: any line from the first command; an event from the second command whose message is Pod has "karpenter.sh/do-not-disrupt" annotation means a replacement is blocked now. Karpenter excludes a node with such a pod from drift replacement unless the NodePool sets terminationGracePeriod (Karpenter disruption docs, quoted in "Why"). Check with kubectl get nodepools -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.terminationGracePeriod}{"\n"}{end}': an empty second column means no limit. Ours: three annotated pods and no terminationGracePeriod.
6. Do the subnets have room for the node churn, and for the control-plane update itself?
aws ec2 describe-subnets --subnet-ids $(aws eks describe-cluster --name <cluster> --query 'cluster.resourcesVpcConfig.subnetIds' --output text) \ --query 'Subnets[].[AvailabilityZone,CidrBlock,AvailableIpAddressCount]' --output text kubectl -n kube-system get ds aws-node -o jsonpath='{range .spec.template.spec.containers[?(@.name=="aws-node")].env[*]}{.name}={.value}{"\n"}{end}' | grep -E 'WARM_|MINIMUM_IP|PREFIX_DELEGATION' Same problem: a subnet whose AvailableIpAddressCount is below (IPs per ENI of your instance type × the nodes a roll adds), and WARM_ENI_TARGET=1 with no WARM_IP_TARGET. The EKS error asks for 2 free addresses per zone and the AWS documentation for up to five (see "Why"). Ours: the zone-b /24 at 0 free and the zone-a /24 at 150 free at 05:21, with 11 nodes at 03:15 and 15 at 05:21.
7. Does every image the cluster runs still exist in its registry?
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{range .spec.initContainers[*]}{.image}{"\n"}{end}{end}' | sort -u > images.txt while read i; do docker buildx imagetools inspect "$i" >/dev/null 2>&1 || echo "MISSING $i"; done < images.txt Private registries need a login first; crane manifest or skopeo inspect docker://… do the same. Same problem: any MISSING line. That pod runs only because the image is in the cache of the node it is on, and it will not start on a new node.
8. What goes down when every node is replaced once?
kubectl get deploy,sts -A -o json | jq -r '.items[] | select(.spec.replicas==1) | [.kind,.metadata.namespace,.metadata.name] | @tsv' | wc -l kubectl get targetgroupbindings -A --no-headers | wc -l aws elbv2 describe-target-groups --query 'length(TargetGroups)' Same problem: a first number above 0 that includes a workload serving traffic (each is unavailable while its pod moves; ours: most Deployments and all 7 production databases), and a third number larger than the second plus the groups your ingresses created (a group with hand-registered pod IPs keeps the old IP when the pod moves). describe-target-groups counts every group in the account and region, so subtract the groups of other clusters.
The audit quoted at the top of "What we saw" was a run of kubegap, an open-source audit of Kubernetes cluster operations that I maintain. It automates the readings of steps 1 and 4: its checks platform.k8s-version-eol, platform.karpenter-version-compat, karpenter.ec2nodeclass-al2, logs.signatures.karpenter.version-unsupported and eks.nodegroup-ami-old are the five findings behind the problem reported here. It does not read the CRD versions of step 2 or run the Terraform plan of step 3. Steps 5 to 8 were done by hand in this work; whether kubegap reports them was not checked for this article.
What we changed, and what we have not
No non-production cluster existed and none was built; every step below ran in production first. That is a limit of this report; we do not recommend it.
Before the first change
- Thirteen EBS snapshots at 03:33:16, one per PersistentVolume, and the cluster objects exported to YAML. The cluster has 13 PersistentVolumes (232 Gi in total), among them 7 production databases, each a single-replica StatefulSet, with no AWS Backup plan and no DLM policy. The snapshots were deleted at 07:04 after all 13 stateful pods were Running; they were never used for a restore.
- The three existing CRDs were labelled
app.kubernetes.io/managed-by=Helmand annotatedmeta.helm.sh/release-name=karpenter-crdandmeta.helm.sh/release-namespace=karpenterat 03:34:55, so that a newkarpenter-crdHelm release could take them over.
Karpenter upgrade 0.37 to 1.0, then 1.14.3: conversion webhook and the karpenter-crd chart, in four pull requests
| PR | Change | Plan | Apply (UTC) | State after |
|---|---|---|---|---|
| PR 1 | Karpenter 0.37.0 → 0.37.8, new karpenter-crd release with the webhook settings, webhook.enabled=true | 1 to add, 1 to change, 0 to destroy | 03:36:40–03:38:42 | Both charts at 0.37.8; CRDs serve v1 and v1beta1, v1beta1 stored, conversion: Webhook; 0 Pending pods |
| PR 2 | 0.37.8 → 1.0.14, four read actions added to the controller IAM policy | 0 to add, 3 to change, 0 to destroy | 03:47:04–03:49:01 | Both charts 1.0.14; storedVersions ["v1beta1","v1"] at 03:49:28 and ["v1"] for all three CRDs at 03:55 |
| PR 3 | The five manifests to v1, through terraform state rm and import blocks | 5 to import, 0 to add, 5 to change, 0 to destroy | 03:53:49–03:54:52 | Manifests v1; all eight NodeClaims that existed before the apply Drifted (see above) |
| PR 4 | 1.0.14 → 1.14.3, webhook settings and import blocks removed | 0 to add, 2 to change, 0 to destroy | 03:57:13–03:59:07 | 2 pods Running on 1.14.3; new CRDs capacitybuffers.autoscaling.x-k8s.io and nodeoverlays.karpenter.sh; EC2NodeClass and four NodePools Ready |
Details that mattered:
- After PR 1,
kubectl get nodepools.v1.karpenter.sh defaultreturnedkarpenter.sh/v1withnodeClassRef {group, kind, name}. The node security group already allowed port 8443 from the control plane, which the conversion webhook needs. - For PR 3 we backed up the state with
terraform state pull(03:51:25; the state bucket has versioning off), ranterraform state rmon the five addresses ("Successfully removed 5 resource instance(s)"), and added fiveimportblocks with ids of the formkarpenter.sh/v1//NodePool//defaultandkarpenter.k8s.aws/v1//EC2NodeClass//default. Each resource then planned as "will be updated in-place (imported from …)". Local Terraform was 1.5.7, CI 1.5.0. - At 03:56 we removed the leftover
compatibility.karpenter.sh/v1beta1-nodeclass-referenceannotation from the four NodePools. - At 06:40 Karpenter had 0 error lines in the last 30 minutes.
Amazon Linux 2 → AL2023, with a pinned alias
PR 5 replaced amiFamily: AL2 and al2@latest with amiSelectorTerms: [alias: al2023@v20261112] (plan 0 to add, 1 to change, 0 to destroy, apply 04:00:56–04:01:59). The release version came from SSM, /aws/service/eks/optimized-ami/<k8s>/amazon-linux-2023/<arch>/standard/recommended/release_version: 1.32.13-20261112 for 1.32 and 1.36.5-20261112 for 1.36, x86_64 and arm64 alike. We pinned the alias so that an AMI release by AWS does not replace the nodes. The nodes are still replaced when we change the alias value and when the control plane version changes (see "Why").
For the two nodes whose replacement was blocked by annotated pods we ran kubectl cordon on the nodes and kubectl delete pod on the two pods at 04:53; the Deployments recreated them elsewhere. By 04:56 all 12 nodes were on AL2023. After aws eks start-insights-refresh, the refresh was COMPLETED at 04:56:52 and no insight was non-PASSING.
Upgrade EKS 1.32 to 1.36: the control plane, one minor per apply
| Step | Control-plane update start | Node group update start | CI apply run |
|---|---|---|---|
| 1.32 → 1.33 (PR 6) | 04:58:50 | 05:06:08 | 04:57:58–05:17:53 (19 m 55 s) |
| 1.33 → 1.34 (PR 8 refused, done by PR 9) | 05:24:58 | 05:31:43 | 05:24:09–05:41:33 (17 m 24 s) |
| 1.34 → 1.35 (PR 10) | 05:43:54 | 05:50:52 | 05:43:03–06:00:26 (17 m 23 s) |
| 1.35 → 1.36 (PR 11) | 06:02:47 | 06:09:56 | 06:01:58–06:24:27 (22 m 29 s) |
All four EKS VersionUpdate entries are Successful. The node group update started 7 m 18 s, 6 m 45 s, 6 m 58 s and 7 m 9 s after the control-plane update. Each plan changed the cluster version, the node group version and the add-on versions in place. PR 10 was merged at 05:43:01 and PR 11 at 06:01:56.
PR 6 also moved the four add-on versions from the module directory into a variable set in the directory the CI workflow watches. The workflow triggers only on that directory, so a pull request that changed only the add-on versions where they were written would have run nothing. The add-ons went from aws-ebs-csi-driver v1.37.0-eksbuild.1, coredns v1.11.3-eksbuild.1, kube-proxy v1.32.0-eksbuild.2 and vpc-cni v1.19.0-eksbuild.1 to the default versions per Kubernetes version from aws eks describe-addon-versions: coredns v1.12.4-eksbuild.82 (1.33, 1.34), v1.13.2-eksbuild.75 (1.35), v1.14.7-eksbuild.35 (1.36); kube-proxy v1.33.10-eksbuild.69, v1.34.6-eksbuild.69, v1.35.3-eksbuild.69, v1.36.0-eksbuild.65; vpc-cni v1.22.4-eksbuild.14 and aws-ebs-csi-driver v1.68.0-eksbuild.1 for all four.
Nodes by kubelet minor at each reading on 2026-11-16, against the control-plane version. The first reading has 11 nodes on 1.32, the last 10 nodes on 1.36.5.
The control plane was at most two minors ahead of the oldest node (06:24–06:32: one node on 1.34, control plane on 1.36); the Kubernetes version skew policy allows a kubelet up to three minors older than the API server. After the 05:42 mistake with api-a, the script that clears blocked nodes was changed to delete one pod per workload per run and to skip a workload that has a pod not Ready.
Fixes made on the way
- Redis image (PR 7, merged 05:15:30). Images moved to
docker.io/bitnamilegacy, same tags. The StatefulSet had the new image, but the stuck pod was not replaced and the Helm release stayed inpending-upgrade; afterkubectl delete pod redis-master-0at 05:20 the pod was2/2 Runningin 51 s. - Subnet addresses (05:21, then PR 9).
kubectl -n kube-system set env ds/aws-node -c aws-node WARM_IP_TARGET=3 MINIMUM_IP_TARGET=6. The next read: zone-b 127 free, zone-a 174 free; at 05:42, 181 and 156. PR 9 (merged 05:24:06) put the same two values into the add-on'sconfiguration_values, and its apply performed the 1.34 update that EKS had refused in PR 8's apply. - Prometheus memory (PR 12, merged 06:26:38). Requests to 1536Mi and the limit to 3Gi; at 06:34 the pod was
2/2 Runningwith 0 restarts. Why it ran out of memory was not investigated: no memory series was read before or after. The audit of 2026-11-14 had listed the same container as OOMKilled once, at 14:34 that day, with 178,024 active series. - Redis target group (PR 13, merged 06:37:19). An NLB target group for Redis had one target, registered by hand: the pod's old IP, now
unhealthy. We registered the current IP, then added a TargetGroupBinding; the target ishealthy. - Support type (PR 14).
cluster_upgrade_policy.support_type = "STANDARD"; the EKSUpgradePolicyUpdatewas Successful at 07:08:06, anddescribe-clusterreads 1.36 / ACTIVE / STANDARD. Standard support for 1.36 ends 2027-08-02.
The audit, run again
A fresh kubegap full collection at 08:07:59–08:16 (kubegap 0adb216, rubric 0.1.96, read-only) no longer lists the five findings named at the end of "What to check in your cluster now". Only those five findings are compared: 180 of 762 collection commands failed in the second run (21 of 652 in the first), so the totals are not comparable. The second run has one new Karpenter finding, karpenter.metrics-unreadable (warn, "0 of 2 pods /metrics"); the cause was not looked into.
Not changed
- The
karpenter.sh/do-not-disruptannotations andterminationGracePeriod. The three annotated pods still carry the annotation and no NodePool setsterminationGracePeriod. We expect the next drift to be blocked on the same pods. - Single-replica Deployments. Not changed.
- Four ALB target groups whose only targets are
unhealthy, withHealth checks failed with these codes: [404](two) and[307](two): the health check path/does not return 200 on those apps. Two of the four were alreadyunhealthyat 05:15. Not changed. - A deprecated API for 1.37. The insights now list "Deprecated APIs removed in Kubernetes v1.37":
/apis/storage.k8s.io/v1beta1/volumeattributesclasses, replaced by v1, used bycsi-resizerandargocd-application-controller, one request each, both at 05:33. Not changed. - Older Helm charts (load balancer controller v2.7.2, Argo CD v2.13.0, external-secrets v0.9.13, kube-prometheus-stack 56.0.0, KEDA 2.17.2, external-dns 0.14.0) were not upgraded. On 1.36 all were Ready and had 0 error lines in 15 minutes. That is all we checked.
- Backups. Still no AWS Backup plan and no DLM policy for the 13 volumes, and the Terraform state bucket still has versioning off.
- A statement we have not verified. We told the owner during the work that EKS cannot roll a version back. The insights list now contains "rollback compatibility" entries; we did not test or read about that feature.
Where else this happened
The Incident Library, our searchable collection of public post-incident reports on benchlet.com, has no case of an upgrade held back by Karpenter's own version range or of a control-plane update refused for free subnet addresses. The nearest public cases:
- UK Ministry of Justice, 2025-03-18: during an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down. It is the nearest case to our 05:13–05:16 overlap. Source
- Stytch, 2023-02-23: deleting EKS managed node groups also deleted Karpenter's instance profile, and every worker node shut down. Our
5 to destroyplan implied the same outcome by a different cause. Source - PortSwigger, 2024-10-24: all customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java; the recorded contributing factor is an unpinned version; our EC2NodeClass had the same condition (
al2@latest) until PR 5. Source
More: Upgrade stuck, 11 cases (two of them are not Kubernetes) and Karpenter as a component, 5 cases.
Timeline
| Time (UTC) | Event |
|---|---|
| 2026-11-14 16:51 | Audit run lists Kubernetes 1.32 and Karpenter 0.37.0 as problem 5 of its top five. |
| 2026-11-16 03:15 | Our reading: server 1.32, 11 nodes, 9 on Amazon Linux 2; "Amazon Linux 2 compatibility" ERROR for 1.33. |
| 03:33–03:35 | 13 EBS snapshots; the three Karpenter CRDs labelled for the karpenter-crd release. |
| 03:36:38–03:59:07 | PR 1 to PR 4: Karpenter 0.37.0 → 0.37.8 → 1.0.14 → manifests to v1 → 1.14.3 (times per PR in the table above). |
| 03:51 | PR 3 first plan: 5 to add, 0 to change, 5 to destroy. Not applied. |
| 03:54:53 | "disrupting nodeclaim(s) via replace"; at 03:55:18 all eight NodeClaims that existed before the apply Drifted. |
| 04:00:53–04:01:59 | PR 5: al2023@v20261112. |
| 04:52:43–04:56 | 2 nodes still on AL2, DisruptionBlocked; cordon and pod delete; all 12 nodes on AL2023. |
| 04:58:50 | Control-plane update to 1.33 starts. |
| 05:13:09–05:16:54 |





