The Shift to cgroup v2 in Kubernetes: What You Need to Know

By Paco Xu (DaoCloud) |

In Linux, cgroups (control groups) are a kernel feature used for managing system resources. Kubernetes uses cgroups to allocate resources like CPU and memory to containers, ensuring that applications run smoothly without interfering with each other. With the release of Kubernetes v1.31, support for v1 cgroup management moved into maintenance mode. Support for v2 cgroup management has been stable since Kubernetes v1.25.

Compared with cgroup v1, cgroup v2 provides a single unified hierarchy, a more consistent interface, and a stronger foundation for resource isolation and modern resource-management features.

Deprecation of cgroup v1

FEATURE STATE: Kubernetes v1.35 [deprecated]

Kubernetes has deprecated cgroup v1. Starting with Kubernetes v1.35, failCgroupV1 defaults to true, so the kubelet does not start on a cgroup v1 node by default. Administrators can temporarily set failCgroupV1: false in the kubelet configuration file, but removal will follow the Kubernetes deprecation policy. Track that work in KEP-5573: Remove cgroup v1 support.

Before upgrading to v1.35 or later, migrate every Linux node to cgroup v2 or explicitly use the temporary override. A node that still uses cgroup v1 otherwise fails during kubelet startup.

For kubeadm-managed clusters, Kubernetes v1.35 also makes this an earlier, stricter check. The SystemVerification preflight check, provided by k8s.io/system-validators, returns an error during kubeadm init, kubeadm join, and kubeadm upgrade when it detects cgroup v1 with kubelet v1.35 or later; with an older kubelet, the check remains a warning. See kubernetes/kubernetes#134744 for details.

The top FAQs cover three main areas: why to migrate, the benefits and drawbacks, and key points to keep in mind when using cgroup v2.

Limitations of cgroup v1 and Improvements with cgroup v2

The Linux kernel documentation describes both interfaces:

Let's enumerate some known issues.

active_file memory is not considered available memory

The kubelet treats active_file memory as not reclaimable. For I/O-intensive workloads, a large page cache can therefore make the kubelet report memory pressure and evict Pods. This is a known kubelet issue (kubernetes/kubernetes#43916); migrating to cgroup v2 does not by itself change that calculation. The documented workaround is to set equal memory requests and limits for containers that perform intensive I/O, after measuring an appropriate value.

Memory QoS updates in Kubernetes v1.36

Memory QoS was introduced as an alpha feature in Kubernetes v1.22 and updated in v1.27. It remains alpha in v1.36, but now separates memory throttling from memory reservation and adds tiered memory protection:

Note:

Memory QoS is available only on Linux nodes that use cgroup v2. It relies on the cgroup v2 memory controller: memory.high provides throttling, while memory.min and memory.low provide hard and soft protection when tiered reservation is enabled. cgroup v1 cannot provide this protection model.
  • Enabling the MemoryQoS feature gate applies memory.high throttling to Burstable containers. The threshold is derived from the request, limit, and memoryThrottlingFactor (default 0.9).
  • memoryReservationPolicy: None is the default. It does not write memory.min or memory.low.
  • memoryReservationPolicy: TieredReservation maps Guaranteed Pod memory requests to memory.min (hard protection) and Burstable Pod requests to memory.low (soft protection). BestEffort Pods receive neither protection.
  • The kubelet exposes alpha metrics for the total memory.min and memory.low reservations on a node.
  • Kernel 5.9 or later is recommended. On older kernels, memory.high reclaim can trigger a known livelock; from v1.36 the kubelet logs a warning when Memory QoS is enabled on an affected kernel.

For example, to opt in to tiered protection:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  MemoryQoS: true
memoryReservationPolicy: TieredReservation
memoryThrottlingFactor: 0.9

Because Memory QoS is still alpha and disabled by default, test the configuration and account for hard-reserved memory before enabling TieredReservation in production. See the Pod QoS documentation for the current mapping of Kubernetes QoS classes to cgroup v2 controls.

Container-aware OOM handling

On cgroup v2 nodes, the kubelet defaults singleProcessOOMKill to false. It therefore sets memory.oom.group for each container cgroup so that an OOM event kills all processes in that container together, rather than leaving a partially functioning multi-process container. Set singleProcessOOMKill: true only if you need the cgroup v1-compatible behavior where the kernel may kill one process at a time. See the KubeletConfiguration reference for this setting.

This behavior is scoped to a container cgroup, not the whole Pod. Also, cgroup.kill is a separate administrative interface: writing 1 to it sends SIGKILL to every process in that cgroup and its descendants; it does not configure OOM behavior. The cgroup v2 memory controller additionally provides memory.events counters that monitoring systems and userspace OOM managers can observe.

Rootless support

In cgroup v1, delegating controllers to less privileged containers may be dangerous.

Unlike cgroup v1, cgroup v2 officially supports delegation. Most Rootless Containers implementations rely on systemd for delegating v2 controllers to non-root users.

This delegation mechanism is separate from Kubernetes Pod user namespaces, which map container users to unprivileged host users. Pod user namespace support graduated to stable in Kubernetes v1.36; check its filesystem, kernel, CRI runtime, and OCI runtime prerequisites before enabling it.

What else?

  1. eBPF stories:
    • In cgroup v1, device access controls are exposed through interface files.
    • The cgroup v2 device controller has no interface files, and is implemented on top of cgroup BPF.
    • Cilium attaches BPF cgroup programs for socket-based load balancing. Its default cgroup root is /run/cilium/cgroupv2.
  2. Pressure Stall Information (PSI) reports CPU, memory, and I/O contention at node, Pod, and container level. The KubeletPSI feature graduated to stable in Kubernetes v1.36 and is always enabled. PSI requires cgroup v2, Linux 4.20 or later, CONFIG_PSI=y, and a kernel not booted with psi=0. The kubelet exposes the data through the Summary API and /metrics/cadvisor.
  3. When migrating, update software that reads the cgroup filesystem directly. The migration guidance recommends cAdvisor v0.43.0 or later and lists compatible Java, Node.js, and automaxprocs versions.

CPU weight conversion in newer OCI runtimes

cgroup v1 uses cpu.shares, whereas cgroup v2 uses cpu.weight. Newer OCI runtimes use an improved non-linear conversion that preserves the default priority and gives small CPU requests more usable granularity. The change is implemented in the OCI runtime rather than Kubernetes: it is available in runc v1.3.2 and crun v1.23. After upgrading a runtime, monitoring or policy tools that predict exact cpu.weight values may need updates. Read New Conversion from cgroup v1 CPU Shares to v2 CPU Weight for the formula, examples, and compatibility considerations.

In-place resource updates

In-place Pod vertical scaling graduated to stable in Kubernetes v1.35. Kubernetes v1.36 then enabled in-place vertical scaling for Pod-level resources by default as a beta feature. The kubelet coordinates changes between the Pod-level and container cgroups so that increases create headroom before container limits grow, while decreases constrain containers before shrinking the Pod-level boundary. Accurate aggregate enforcement for this v1.36 feature requires cgroup v2.

Adopting cgroup version 2

Requirements

Here's what you need to use cgroup v2 with Kubernetes. First up, you need to be using a version of Kubernetes with support for v2 cgroup management; that's been stable since Kubernetes v1.25 and all supported Kubernetes releases include this support.

  • OS distribution enabled cgroup v2
  • Linux kernel version is 5.8 or later (5.9 or later is recommended when using Memory QoS)
  • Container runtime supports cgroup v2. For example:
    • containerd v1.4 or later supports cgroup v2; use containerd v2.0 or later for automatic cgroup-driver discovery
    • CRI-O v1.20 or later
  • The kubelet and the container runtime are configured to use the systemd cgroup driver

The Kubernetes v1.35 release announced the end of the containerd 1.x support window. However, the final Kubernetes v1.36 runtime documentation still describes a fallback for containerd 1.x, which does not implement the RuntimeConfig CRI RPC. On those nodes the kubelet uses its configured cgroupDriver value instead of automatic discovery. That fallback is scheduled for removal in Kubernetes v1.38. For new or upgraded v1.36 nodes, prefer a currently supported containerd 2.x release and review both the Kubernetes runtime documentation and the containerd compatibility matrix.

kernel updates around cgroup v2

When Kubernetes was first announced, in 2014, only v1 cgroup existed. Version 2 cgroup management first appeared in Linux kernel 4.5, released in 2016.

  • In Linux 4.5, the cgroup v2 io, memory, and pids controllers were supported.
  • Linux 4.15 added support for the cgroup v2 cpu controller.
  • Pressure Stall Information (PSI) support began with Linux 4.20.
  • The Kubernetes project does not recommend using cgroup v2 with a Linux kernel older than 5.2 due to lack of cgroup-level task freezer support.
  • Kubernetes documents 5.8 as the minimum kernel version for cgroup v2; the root cgroup's system-level cpu.stat file was added in Linux 5.8.
  • The memory.high livelock fix used by Memory QoS is present in Linux 5.9 and later.
  • memory.peak was added in Linux 5.19.

Use systemd as cgroup driver

Configure the kubelet's cgroup driver to match the container runtime cgroup driver.

The Container runtimes page explains that the systemd driver is recommended for kubeadm based setups instead of the kubelet's default cgroupfs driver, because kubeadm manages the kubelet as a systemd service.

A minimal example of configuring the field explicitly:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

Automatic discovery of the runtime's cgroup driver through the CRI, tracked by KEP-4033, graduated to stable in Kubernetes v1.34. It requires a runtime that implements the RuntimeConfig CRI RPC (containerd v2.0+ or CRI-O v1.28+). When available, the kubelet uses the value reported by the runtime instead of its configured cgroupDriver value.

Tools and commands for troubleshooting

Tools and commands that you should know about cgroups:

  • stat -fc %T /sys/fs/cgroup/: Check whether cgroup v2 is enabled; it returns cgroup2fs.
  • systemctl list-units 'kube*' --type=slice or --type=scope: List Kubernetes-related units that systemd currently has in memory.
  • bpftool cgroup list /sys/fs/cgroup/*: List all programs attached to the cgroup CGROUP.
  • systemd-cgls /sys/fs/cgroup/*: Recursively show control group contents.
  • systemd-cgtop: Show top control groups by their resource usage.
  • tree -L 2 -d /sys/fs/cgroup/kubepods.slice: Show Pods' related cgroups directories.

How to check if a Pod CPU or memory limit is successfully applied to the cgroup file?

  • Kubernetes Pod spec: check limits spec.containers[*].resources.limits.{cpu,memory} and requests spec.containers[*].resources.requests.{cpu,memory}
  • CRI: cpu_period, cpu_quota, cpu_shares for CPU and memory_limit_in_bytes for memory limit
  • OCI Spec: memory.limit, cpu.shares, cpu.quota, cpu.period
  • Systemd Scope Unit: CPUWeight, CPUQuotaPerSecUSec, CPUQuotaPeriodUSec, MemoryMax
  • Cgroupfs value: /sys/fs/cgroup/../cpu.weight, /sys/fs/cgroup/../cpu.max, /sys/fs/cgroup/../memory.max

Further reading

Last modified August 13, 2026 at 10:31 AM PST: add note for kubeadm check and why Memory QoS use cgroup v2 (3c3a69eb37)