Introduction & Quick Summary
Symptom. After upgrading a cluster's nodes to Kubernetes 1.35, new nodes never become Ready. The kubelet on the node exits almost immediately:
$ systemctl status kubelet Active: failed (Result: exit-code) $ journalctl -u kubelet -n 20 --no-pager ... kubelet: cgroup v1 is no longer supported; failing to start $ kubectl get nodes NAME STATUS ROLES AGE VERSION worker-3 NotReady <none> 4m v1.34.9 The node object is registered by an older kubelet or not at all, pods stay Pending, and kubectl describe node worker-3 shows Ready False with KubeletNotReady.
Cause. Kubernetes 1.35 removed cgroup v1 support. A node booting on cgroup v1 with a 1.35 kubelet does not join, and the failure is at the host, not in any Kubernetes object. The same release also removed the --pod-infra-container-image kubelet flag, so custom node bootstrap configurations that pass it exit on startup with an unknown flag.
Fix. Move the node to cgroup v2, or set failCgroupV1: false while you plan the move.
# kubelet configuration on the node (not a Kubernetes object) apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration failCgroupV1: false # confirm what the host booted with before touching the kubelet stat -fc %T /sys/fs/cgroup # tmpfs or cgroup2fs means cgroup v2 grep -o 'cgroup[^ ]*' /proc/cmdline Core Mechanics, Signatures & Reproducible Setup
Kubernetes 1.25 added cgroup v2 support; 1.35 removed v1. The kubelet reads the host hierarchy at startup and refuses to run when the only hierarchy is v1, because every resource limit, QoS class and eviction threshold it enforces is expressed through the v2 unified hierarchy. This is not a warning you can ignore: the process exits.
Check the whole fleet in one pass rather than one node at a time:
for n in $(kubectl get nodes -o name | cut -d/ -f2); do printf "%s " "$n" kubectl debug node/$n -it --image=busybox -- \ sh -c 'stat -fc %T /sys/fs/cgroup' 2>/dev/null | tail -1 done The supported configurations are uncomplicated: Amazon Linux 2023 and Bottlerocket default to cgroup v2, and most distributions from 2022 onward do too. The risk is a launch template, nodeadm config, or custom AMI that put the host back on v1 with systemd.unified_cgroup_hierarchy=0 on the kernel command line, or an older base image that never had v2.
Reproduce the failure in a scratch cluster by starting a node with that kernel argument, then roll the kubelet to 1.35 and watch it exit. Doing it once in a test cluster is cheaper than discovering it during a production node pool rotation.
Two related removals in the same upgrade:
# this flag no longer exists; a node that passes it never becomes Ready grep -rn "pod-infra-container-image" /etc/kubernetes /var/lib/kubelet \ /etc/systemd/system/kubelet.service.d 2>/dev/null And containerd 1.x is supported for the last time in 1.35, so the next upgrade requires containerd 2.0 or later. Find the nodes that are behind:
kubectl get --raw /metrics | grep kubelet_cri_losing_support Key Comparison Table
| Kubernetes version | cgroup v1 node | containerd 1.x | kube-proxy ipvs |
|---|---|---|---|
| 1.33 | works | supported | works |
| 1.34 | works, warning | supported | works |
| 1.35 | kubelet will not start | last supported release | deprecated, warns |
| next release | removed | removed | removal planned |
| What you see | Where it comes from | First check | |
| --- | --- | --- | |
| Node never registers | kubelet exited on cgroup v1 | journalctl -u kubelet -n 50 | |
unknown flag: --pod-infra-container-image | bootstrap user data or AMI | nodeadm config, Karpenter EC2NodeClass | |
Node joins then goes NotReady | runtime not ready or sandbox image missing | crictl info, systemctl status containerd | |
kubelet_cri_losing_support metric rises | containerd 1.x | containerd --version per node | |
| kube-proxy logs a deprecation warning | mode: ipvs | switch to nftables in a separate window |
Practical Edge Cases, Error Fixes & Deep Dive Code Examples
1. cgroup v1 hides in the kernel command line
stat -fc %T /sys/fs/cgroup returning tmpfs means v1 (v2 returns cgroup2fs), but the reason is often a single kernel argument in the launch template. Remove it rather than setting failCgroupV1: false and calling it done:
# before user_data = <<-EOF #!/bin/bash sed -i 's/^GRUB_CMDLINE_LINUX="/GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0 /' /etc/default/grub update-grub EOF # after: nothing; AL2023 already boots with cgroup v2 Rebuild the node group rather than patching running nodes: the kernel argument is read at boot, so an in-place change needs a reboot anyway, and a rolling replacement is the same cost with a cleaner state.
2. failCgroupV1: false as a documented escape hatch
The field exists so that a managed platform can keep working while hosts are migrated; Bottlerocket's 1.35 image sets it itself. Use it deliberately, with a date next to it, because it keeps you on a mode that a future release removes:
# kubeadm clusters: drop it in and restart sudo tee /var/lib/kubelet/config.yaml.d/99-cgroup.yaml >/dev/null <<'EOF' apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration failCgroupV1: false EOF sudo systemctl restart kubelet Confirm the node joins and that the runtime reports a v2 hierarchy once you have moved the host, then delete the override.
3. The removed --pod-infra-container-image flag
A kubelet started with a flag it no longer knows exits at once with a usage error, so the node never registers and the log line is easy to miss among the normal startup noise. The flag lives on the host, in bootstrap user data, spec.kubelet.flags in a nodeadm NodeConfig, Karpenter EC2NodeClass user data, and custom AMI build scripts. No cluster-side query can see it:
kubectl get nodes -o wide # for each node that will not join gcloud compute instances describe ... # or aws ssm send-command Remove the flag and let the runtime supply the sandbox image. EKS-optimized AMIs never set it; custom images often do.
4. Ingress NGINX is on its way out
Ingress NGINX received best-effort maintenance until March 2026 and is archived after that, which is a schedule problem rather than a 1.35 error. A cluster that depends on ingress-nginx annotations is a cluster with a migration to plan. Enumerate the exposure before you promise a timeline:
kubectl get ingress -A -o json | jq -r '.items[] | [.metadata.namespace, .metadata.name, ([.metadata.annotations // {} | keys[] | select(startswith("nginx.ingress.kubernetes.io"))] | length)] | join(" ")' Annotations have no Gateway API equivalent in some cases, so the count of annotations per Ingress is a reasonable first estimate of the work, not just the count of objects.
5. Version skew when you cannot move every node at once
A 1.35 API server supports kubelets back to 1.32, so the control plane can move first while node pools follow. The order matters more than the speed: API servers one at a time, then controllers, then kubelets node by node after a drain, then kube-proxy no newer than the API server.
kubectl drain worker-3 --ignore-daemonsets --delete-emptydir-data --timeout=300s # upgrade kubelet and kubeadm on the node kubectl uncordon worker-3 Do not upgrade a node pool whose hosts are still on cgroup v1 with the new kubelet: the drain succeeds and the node never comes back, which turns a routine rotation into an outage.
6. Verifying the upgrade before the maintenance window
kubectl speaks to a 1.35 API server from versions 1.34 through 1.36, so the client is not the problem. The problems are all in the manifest and host layers. Check the manifest layer with the metrics the API server already exposes:
kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis grep -rn "storage.k8s.io/v1alpha1" ./manifests ./charts A non-empty metric means a client called a deprecated API since the last API server restart. Convert manifests with kubectl convert where a replacement exists, and validate the YAML you produce with the YAML validator before applying it to a test cluster.
Common Compilation/Runtime Mistakes & Version Gotchas
- Assuming
Readymeans the node is on the new version. Checkkubectl get nodes -o widefor the kubelet version, not just the status. - Setting
failCgroupV1: falseand forgetting it. The next release removes the escape hatch entirely. - Patching a node in place instead of replacing it. Kernel arguments and container runtime versions belong to the image, not the running process.
- Upgrading kube-proxy past the API server. It is unsupported and produces confusing service failures.
- Reading a
Pendingpod as a scheduling bug. With a node that never joined there is nothing to schedule onto; look at nodes first. - Leaving ipvs mode because it still works. It warns at startup today and disappears later; nftables needs kernel 5.13 or newer, which is the real constraint.
- Testing on a single-node cluster. cgroup and runtime differences only show up on real hosts, which is the whole point.
Reference: Kubernetes v1.35 release announcement and cgroup v2 documentation.
