Debugging zero disk reads after moving to Bottlerocket

Our container platform moved from Amazon Linux 2 (AL2) to Bottlerocket. Soon after the migration started, a partner team that runs Spark jobs reported a regression. Their jobs still finished successfully, but far more of them reported zero "storage read bytes": about 23% of tasks on AL2, and about 56% on Bottlerocket.

The first suspect was a bug that drops disk-read metrics. The platform had hit exactly that kind of bug before: on kernel 6.1 with cgroup v1, container block I/O stats were all zero. This time there was no bug, and the metric was correct on both platforms. The OS upgrade changed how often a job's reads were served from memory instead of disk, and the counter only sees the reads that reach the disk.

How the number is made

1. Each container runs in its own blkio cgroup (cgroup v1 on these hosts; the version doesn't matter here).
2. Every 20 seconds, the platform agent asks containerd for the container's block I/O stats. containerd reads them from the kernel's blkio.throttle.io_service_bytes_recursive file.
3. The agent sums the Read and Write entries across all block devices and ships the totals.

1$ cat /sys/fs/cgroup/blkio/<container>/blkio.throttle.io_service_bytes_recursive
2259:1 Read 134217728
3259:1 Write 134217728
4259:0 Read 13750272
5259:0 Write 0
6...
7Total 301088768

The partner's setup is similar to Docker-in-Docker: the task's container runs its own containerd, which starts the inner container where the Spark job runs. The job's I/O lands in a child cgroup nested under the task's cgroup, so the first question was whether the inner container's I/O makes it into the reported number at all.

Reproducing the missing metric

Each reproduction moved a little closer to production:

• docker-in-docker on AL2023 and on Bottlerocket,
• containerd-in-containerd on AL2023 and on our Bottlerocket image,
• a real task on the platform, with containerd started inside the task's container and an inner container generating I/O.

In each one, the inner container wrote and then read 128 MiB with direct I/O:

1dd if=/dev/zero of=/hello bs=16M count=8 oflag=direct
2dd if=/hello of=/dev/null bs=1M iflag=direct

Every setup reported exactly 128 MiB of reads, on AL2 and on Bottlerocket, and on the real task the published metric showed the same 128 MiB. The metric would not go missing.

The production data didn't look like a missing metric either. One of the partner's Bottlerocket tasks ran for more than three days and reported 0 bytes read and about 2 GB written. Why would a bug drop only reads, and only for some tasks?

The AL2 side, which everyone treated as correct, was odd too. Read bytes from one batch of the partner's test tasks on AL2:

1Total records: 264
2Zero values:   48 (18.2%)
3Median:        80 KiB
4p80:          132 KiB
5Max:          212 KiB

Even on the "good" platform, these small word-count jobs read a few dozen filesystem blocks each, and almost one in five read nothing. Maybe the zeros on Bottlerocket weren't missing data, and the jobs simply didn't read from disk.

That called for the opposite test. Proving the platform drops reads takes a job that is guaranteed to read X bytes from disk and a metric that reports less than X. One test account went back to Bottlerocket, and the partner team ran jobs that had to hit the disk. All of them reported reads. Word-count jobs on 2 GB workers reported 1 to 2 GB of reads on both platforms, for inputs of a few KB to 600 KB. The same job on a worker with more memory reported about 15 MB. Same job, same input, and a nearly 100x difference that depended only on worker memory.

Where do reads go?

A read passes through several layers on its way to the disk, and most of them can answer it from memory:

1application (Spark job → Spark → JVM)   may cache in its own heap
2→ syscall (read, mmap page fault)
3→ VFS / page cache                      a hit ends here; the disk never sees it
4→ filesystem (ext4, xfs)
5→ block layer                           ← blkio counters are charged here
6→ device driver
7→ storage device

The blkio counters live in the block layer. They are charged when an I/O request (a bio) is submitted, to the cgroup of the task that submitted it. A read the page cache serves never becomes a bio. So "storage read bytes" means bytes this cgroup had to fetch from disk because they weren't in memory: it counts cache misses.

Both kernels behave the same way. The table comes from two t3.medium instances (4 GiB RAM): AL2 with kernel 5.10, and AL2023 with kernel 6.1 switched to cgroup v1, the same kernel line our Bottlerocket images use. Each "container" is a bare blkio and memory cgroup created by hand, which keeps containerd out of the picture. The script is in the appendix.

The page cache is shared by every cgroup on the host. Once anyone has read a file, or just written it, the next reader is charged nothing, even from a different container. Only direct I/O always reaches the disk, which is why every reproduction above "worked": direct I/O skipped the very cache that was hiding the partner's reads.

Small non-zero values are mostly the kernel's own overhead. A 4 KiB read costs a 128 KiB readahead window plus a couple of metadata blocks, and starting an interpreter pulls in a few MiB of binaries and libraries. The 80 KiB median on AL2 is about what a few cold metadata blocks and a readahead window look like.

Memory decides the rest. With enough memory, the file is read from disk once. Under a tight limit, the cgroup's page cache is reclaimed between passes and every pass goes back to disk. That explains 1 GB on small workers and 15 MB on large ones: on small workers, the JVM and Spark jars keep getting evicted and read back in.

The same code doesn't even give the same answer twice. A small Python job, run six times, each in a fresh cgroup on the 6.1 host:

1run 1  (cold cache)                       read=8556544 bytes
2run 2  (warm)                             read=0 bytes
3run 3  (warm)                             read=0 bytes
4run 4  (warm)                             read=0 bytes
5run 5  (after a neighbor streamed 6 GiB)  read=1540096 bytes
6run 6  (warm again)                       read=0 bytes

Before run 5, another cgroup streamed a file larger than RAM through the cache and pushed out some of the job's pages.

Why more zeros on Bottlerocket?

A task reports zero reads when every block it needed was already in memory. For jobs that read only a few dozen kilobytes from disk, it takes very little to tip "a few blocks" into "zero", and the upgrade changed what decides it. The move from kernel 5.10 to 6.1 changed page reclaim, readahead and filesystem metadata handling, which shifts the leftover kilobytes that separate zero from non-zero. Bottlerocket's base OS also uses less memory than our AL2 image, so the same task size leaves more room for page cache.

I didn't trace the shift down to one specific kernel change. The bytes in question were a median of 80 KiB per task, and finding them would mean bio-level tracing on another team's production jobs across two OS images. The evidence in hand was enough: a known disk read is reported exactly on both platforms, jobs that must hit the disk report reads on both, and the zeros only show up for jobs that barely touch the disk.

The partner team changed their integration test to require non-zero reads only from jobs that are guaranteed to read from disk, such as memory-constrained workers. They also confirmed that job runtimes and failure rates hadn't changed on Bottlerocket. Once the test change rolled out, their accounts moved back to Bottlerocket.

What I took away

A disk-read counter counts cache misses. Per-cgroup block I/O stats count what reached the block layer, so they don't measure how much data a program reads, and zero is a normal value for a healthy workload. An alarm on "reads == 0" partly alarms on the page cache doing its job.

Test the measurement with a known signal. I spent days trying to make the metric disappear. What settled it was the reverse: generate X bytes of real disk reads and check that the metric reports at least X. Direct I/O with dd gives a known number cheaply.

Check priors against the data. A real kernel bug in this exact metric was the reason to take the report seriously. But the data didn't fit that bug: only reads, only some tasks, and tiny values even on the "good" platform.

Understand why an alarm exists before changing it. This one was there to catch a broken metrics pipeline, which the earlier all-zero bug showed is worth catching. The partner team kept it and pointed it at jobs where zero really means broken.

Appendix: the lab script

Run it as root on a cgroup v1 host. It writes 256 MiB under /var/tmp and drops the page cache several times. To switch AL2023 to cgroup v1:

1sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" && sudo reboot

 1#!/bin/bash
 2# blkio-lab.sh: where do block-I/O reads go? cgroup v1 experiments.
 3set -u
 4B=/sys/fs/cgroup/blkio
 5M=/sys/fs/cgroup/memory
 6DATA=/var/tmp/blkio-lab
 7mkdir -p $DATA
 8
 9# run <cgroup> <cmd...>: run cmd inside a blkio+memory cgroup
10run() {
11    local cg=$1; shift
12    mkdir -p $B/$cg $M/$cg
13    sh -c "echo \$\$ > $B/$cg/cgroup.procs; echo \$\$ > $M/$cg/cgroup.procs; exec \"\$@\"" sh "$@"
14}
15reads() { awk '$2=="Read"{s+=$3} END{printf "%.1f MiB", s/1048576}' $B/$1/blkio.throttle.io_service_bytes_recursive; }
16report() { printf '%-28s read=%s\n' "${2:-$1}" "$(reads $1)"; }
17drop() { sync; echo 3 > /proc/sys/vm/drop_caches; }
18
19echo "kernel $(uname -r)"
20F=$DATA/data.bin
21[ -f $F ] || dd if=/dev/urandom of=$F bs=1M count=256 status=none
22sync
23
24echo "== cold read";             drop; run e1 cat $F > /dev/null; report e1
25echo "== warm read, new cgroup";       run e2 cat $F > /dev/null; report e2
26echo "== written file, child reads"
27drop; run e3 sh -c "cp $F $DATA/copy.bin && sync"
28run e3/child cat $DATA/copy.bin > /dev/null; report e3/child; rm -f $DATA/copy.bin
29echo "== O_DIRECT read";               run e4 dd if=$F of=/dev/null bs=256k iflag=direct status=none; report e4
30echo "== 4 KiB from a cold file";  drop; run e5 dd if=$F of=/dev/null bs=4k count=1 status=none; report e5
31echo "== cold python3";          drop; run e6 python3 -c pass; report e6
32
33echo "== read 3x under memory limits"
34for lim in 1G 64M; do
35    cg=e7-$lim; mkdir -p $B/$cg $M/$cg; echo $lim > $M/$cg/memory.limit_in_bytes
36    drop; run $cg sh -c "cat $F >/dev/null; cat $F >/dev/null; cat $F >/dev/null"
37    report $cg "$cg (limit $lim)"
38done
39
40for cg in e1 e2 e3/child e3 e4 e5 e6 e7-1G e7-64M; do
41    rmdir $B/$cg $M/$cg 2>/dev/null
42done