Missing Container Disk I/O Stats with cgroup v1 on Kernel 6.1

calendar Nov 9, 2024 · 4 min read · Linux Container  ·
Share on: copy

As the Amazon Linux 2 (AL2) approaches its End of Life on 2025-06-30, we have started migrating our container platform from AL2 to Amazon Linux 2023 (AL2023). The migration encountered a few speed bumps. In this post, we'll look at one of them: missing container disk I/O stats.

Why are container I/O dashboards blank?

Our container platform offers disk I/O metrics for containers. However, during testing, dashboards showed "No Data" for container disk I/O. We started the root causing journey with following questions:

  1. Were metrics aggregated correctly? Yes, zeros went in, zeros came out.
  2. Were raw metrics passed to the aggregation service? Yes. Logs showed that metrics were indeed delivered.
  3. Were raw metrics zeros? Yes. DEBUG level logs confirmed raw metrics were zeros.
  4. Where do raw metrics originate? From containerd.
  5. What versions of containerd and cgroups were used? containerd 1.7 and cgroup v1.

Why does containerd report zeros for Disk I/O stats?

With the investigation now pointing to containerd, we decided to first reproduce the issue in a minimal setup.

 1# Step 1. Start an EC2 instance with AL2023, al2023-ami-2023.6.20241010.0-kernel-6.1-x86_64
 2AMI_ID=ami-07c5ecd8498c59db5
 3aws ec2 run-instances \
 4    --image-id ${AMI_ID} \
 5    --instance-type t3.medium \
 6    --subnet-id ${SUBNET_ID} \
 7    --security-group-ids ${SECURITY_GROUP} \
 8    --associate-public-ip-address 
 9
10uname -r
116.1.112-124.190.amzn2023.x86_64
12
13# Step 2. Setup containerd. The iptables is needed by the cni plugin.
14sudo yum install -y containerd nerdctl iptables
15wget https://github.com/containernetworking/plugins/releases/download/v1.6.0/cni-plugins-linux-amd64-v1.6.0.tgz
16sudo mkdir -p /opt/cni/bin
17sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.6.0.tgz
18
19# Step 3. Run a container with active I/O and check stats.
20sudo systemctl start containerd
21sudo nerdctl run -it --rm --entrypoint sh alpine -c "while true; do dd if=/dev/zero of=/tmp/hello bs=16M count=8 oflag=direct; sleep 10; done;"
22
23# Step 4. Check container I/O from another container
24sudo nerdctl stats --no-stream
25CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O     PIDS
2643fe103356ce   alpine-43fe1   0.00%     640KiB / 16EiB      0.00%     1.45kB / 822B   0B / 1.48GB   2
27
28# Step 5. Stop container; switch to cgroup v1; reboot
29sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
30sudo reboot
31
32# Step 6. Start a new container with the same dd command above. This time, the Block I/O stats is empty.
33sudo nerdctl stats --no-stream
34CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
359c16e7bcdb4a   alpine-9c16e   0.00%     1.621MiB / 8EiB     0.00%     1.67kB / 682B   0B / 0B     2

Now we've confirmed containerd is reporting all zeros, the next question is: where does containerd get block I/Od data? According to cgroup1/blkio.go, containerd reads from blkio.throttle.* files within the cgroup.

 1// github.com/containerd/cgroups/v3/cgroup1/blkio.go
 2settings = []blkioStatSettings{
 3    {
 4        name:  "throttle.io_serviced",
 5        entry: &stats.Blkio.IoServicedRecursive,
 6    },
 7    {
 8        name:  "throttle.io_service_bytes",
 9        entry: &stats.Blkio.IoServiceBytesRecursive,
10    },
11}

We can confirm blkio.throttle.* are indeed zeros.

 1# Step 1. Find the PID of the container.
 2sudo ctr task ls
 3TASK                                                                PID     STATUS
 4e65fa7ec929aba737c80ae717fba12f934f790e842f520be347a8e39b7ab1b20    5219    RUNNING
 5
 6# Step 2. Find the cgroup and cat blkio.throttle.* 
 7cat /proc/5219/cgroup | grep blkio 
 812:blkio:/default/8e43a35b540c20611df8a28d344f559c7b6a92e751624ced84ec123e73b9fa8d
 9
10cd /sys/fs/cgroup/blkio/default/8e43a35b540c20611df8a28d344f559c7b6a92e751624ced84ec123e73b9fa8d
11
12cat blkio.throttle.io_serviced
13259:0 Read 0
14259:0 Write 0
15259:0 Sync 0
16259:0 Async 0
17259:0 Discard 0
18259:0 Total 0
19Total 0
20
21cat blkio.throttle.io_service_bytes
22259:0 Read 0
23259:0 Write 0
24259:0 Sync 0
25259:0 Async 0
26259:0 Discard 0
27259:0 Total 0
28Total 0

Why doesn't the kernel report blkio stats with cgroup v1?

According to a recent patch, blkio.throttle.io* counts I/O that are actually throttled, which explains the missing data. After confirming the patch fixed the disk I/O stats, we proceeded to release the new container platform. A happy ending.

Closing thoughts

The industry appears to be migrating from cgroup v1 to cgroup v2. For example, in systemd v257, support for cgroup v1 is considered obsolete and the complete removal of support for cgroup v1 is scheduled for v258. Amazon Linux 2023 has also switched to cgroup v2. As more open-source development focuses on cgroup v2, using cgroup v1 may carry increased risk. The issue we discovered in this post - where a patch wasn't backported to the kernel, is one such an example. Of course, migrating to cgroup v2 isn't straightforward, not at all for a container platform first released almost a decade ago. I am curious to learn more about cgroup v2 and let's dive deep on another day.