Missing Container Disk I/O Stats with cgroup v1 on Kernel 6.1

calendar Nov 9, 2024 · 4 min read · Linux Container  ·
Share on: copy

As the Amazon Linux 2 (AL2) approaches its End of Life on 2025-06-30, we have started migrating our container platform from AL2 to Amazon Linux 2023 (AL2023). The migration encountered a few speed bumps. In this post, we'll look at one of them: missing container disk I/O stats.

Why are container I/O dashboards blank?

Our container platform offers disk I/O metrics for containers. However, during testing, dashboards showed "No Data" for container disk I/O. We started the root causing journey with following questions:

  1. Were metrics aggregated correctly? Yes, zeros went in, zeros came out.
  2. Were raw metrics passed to the aggregation service? Yes. Logs showed that metrics were indeed delivered.
  3. Were raw metrics zeros? Yes. DEBUG level logs confirmed raw metrics were zeros.
  4. Where do raw metrics originate? From containerd.
  5. What versions of containerd and cgroups were used? containerd 1.7 and cgroup v1.

Why does containerd report zeros for Disk I/O stats?

With the investigation now pointing to containerd, we decided to first reproduce the issue in a minimal setup.

 1# Step 1. Start an EC2 instance with AL2023, al2023-ami-2023.6.20241010.0-kernel-6.1-x86_64
 2AMI_ID=ami-07c5ecd8498c59db5
 3aws ec2 run-instances \
 4    --image-id ${AMI_ID} \
 5    --instance-type t3.medium \
 6    --subnet-id ${SUBNET_ID} \
 7    --security-group-ids ${SECURITY_GROUP} \
 8    --associate-public-ip-address 
 9
10uname -r
116.1.112-124.190.amzn2023.x86_64
12
13# Step 2. Setup containerd. The iptables is needed by the cni plugin.
14sudo yum install -y containerd nerdctl iptables
15CNI_VERSION="v1.6.0"
16wget https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
17sudo mkdir -p /opt/cni/bin
18sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-${CNI_VERSION}.tgz
19
20# Step 3. Run a container with active I/O and check stats.
21sudo systemctl start containerd
22sudo nerdctl run -it --rm --entrypoint sh alpine -c "while true; do dd if=/dev/zero of=/tmp/hello bs=16M count=8 oflag=direct; sleep 10; done;"
23
24# Step 4. Check container I/O from another container
25sudo nerdctl stats --no-stream
26CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O     PIDS
2743fe103356ce   alpine-43fe1   0.00%     640KiB / 16EiB      0.00%     1.45kB / 822B   0B / 1.48GB   2
28
29# Step 5. Stop container; switch to cgroup v1; reboot
30sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
31sudo reboot
32
33# Step 6. Start a new container with the same dd command above. This time, the Block I/O stats is empty.
34sudo nerdctl stats --no-stream
35CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
369c16e7bcdb4a   alpine-9c16e   0.00%     1.621MiB / 8EiB     0.00%     1.67kB / 682B   0B / 0B     2

Now we've confirmed containerd is reporting all zeros, the next question is: where does containerd get block I/Od data? According to cgroup1/blkio.go, containerd reads from blkio.throttle.* files within the cgroup.

 1// github.com/containerd/cgroups/v3/cgroup1/blkio.go
 2settings = []blkioStatSettings{
 3    {
 4        name:  "throttle.io_serviced",
 5        entry: &stats.Blkio.IoServicedRecursive,
 6    },
 7    {
 8        name:  "throttle.io_service_bytes",
 9        entry: &stats.Blkio.IoServiceBytesRecursive,
10    },
11}

We can confirm blkio.throttle.* are indeed zeros.

 1# Step 1. Find the PID of the container.
 2sudo ctr task ls
 3TASK                                                                PID     STATUS
 4e65fa7ec929aba737c80ae717fba12f934f790e842f520be347a8e39b7ab1b20    5219    RUNNING
 5
 6# Step 2. Find the cgroup and cat blkio.throttle.* 
 7cat /proc/5219/cgroup | grep blkio 
 812:blkio:/default/8e43a35b540c20611df8a28d344f559c7b6a92e751624ced84ec123e73b9fa8d
 9
10cd /sys/fs/cgroup/blkio/default/8e43a35b540c20611df8a28d344f559c7b6a92e751624ced84ec123e73b9fa8d
11
12cat blkio.throttle.io_serviced
13259:0 Read 0
14259:0 Write 0
15259:0 Sync 0
16259:0 Async 0
17259:0 Discard 0
18259:0 Total 0
19Total 0
20
21cat blkio.throttle.io_service_bytes
22259:0 Read 0
23259:0 Write 0
24259:0 Sync 0
25259:0 Async 0
26259:0 Discard 0
27259:0 Total 0
28Total 0

Why doesn't the kernel report blkio stats with cgroup v1?

According to a recent patch, blkio.throttle.io* counts I/O that are actually throttled, which explains the missing data. After confirming the patch fixed the disk I/O stats, we proceeded to release the new container platform. A happy ending.

Closing thoughts

The industry appears to be migrating from cgroup v1 to cgroup v2. For example, in systemd v257, support for cgroup v1 is considered obsolete and the complete removal of support for cgroup v1 is scheduled for v258. Amazon Linux 2023 has also switched to cgroup v2. As more open-source development focuses on cgroup v2, using cgroup v1 may carry increased risk. The issue we discovered in this post - where a patch wasn't backported to the kernel, is one such an example. Of course, migrating to cgroup v2 isn't straightforward, not at all for a container platform first released almost a decade ago. I am curious to learn more about cgroup v2 and let's dive deep on another day.