Leaked empty directories exhaust inodes

I fixed an issue where new containers failed to launch on a node with "no space left on device". There was free space available, and the issue happened only on a small percentage of nodes. It turned out those nodes churned containers more often than the others, and the failure came from running out of inodes, not out of disk.

One host stopped launching containers. Every new task failed with the same error from the CNI plugin:

1failed to set network cached result:
2open /var/lib/cni/results/...-eth0: no space left on device

A few minutes earlier, health checks on the tasks already running failed too:

1mkdir /var/log/agent/task/<task>/<container>/<exec-id>: no space left on device

The host had been up for two weeks and had run about 50,000 short tasks in that time. The scheduler kept placing new tasks on the host for hours before it was drained from the cluster, because CPU and memory were available for new tasks. My first thought was logs. A host that churns through 150 tasks an hour writes a lot of them, and a log rotation gap can fill a disk. But the partition holding container logs and image layers was at about 30% utilization.

Two details in the errors point to a different root cause. First, both failures were mkdir and open with O_CREAT. The CNI library writes its cache with os.WriteFile, which is open(O_WRONLY|O_CREATE|O_TRUNC) followed by write, and the error names open. Nothing reported a failed write. Processes that already had a file open kept appending to it. Second, the paths were on a different partition from the one holding container logs and the content store: a 3 GiB ext4 filesystem that holds agent state, CNI state and the agent's own scratch space.

Creating a file or directory needs a free inode on every filesystem; writing to an existing file does not. On ext4 the inodes are a fixed pool allocated at mkfs, separate from the data blocks, so a filesystem that can still append but cannot create is out of inodes, not out of blocks. Filesystems that allocate inodes from free space, like xfs and btrfs, often run out of both at once.

Leaked empty directories

The agent runs commands inside containers for health checks and for interactive exec sessions. For each exec session it creates a directory on the host for the stdio FIFOs:

1/var/log/agent/task/<task-id>/<container-name>/<exec-id>/

When the exec finishes, the cleanup removes the exec-id leaf. The task-id and container-name directories above it are never removed. The task teardown path removes the task's container volumes and other per-task resource directories, but not this tree. A task with three exec'd containers leaves four empty directories, one for the task and one per container; 50,000 tasks leave about 200,000, more than the partition's 196,608 inodes.

What an inode is

An inode is the kernel's record of a file: its type, permissions, owner, size, timestamps, link count, and where its data lives. In the kernel this is struct inode in the VFS layer, and every filesystem fills one in from whatever it stores on disk. The in-memory inode is a cache; the on-disk layout is the filesystem's own business, and that is where common filesystems differ.

ext4 allocates its inode table at mkfs time. mkfs.ext4 picks one inode for every 16 KiB of filesystem (the inode_ratio in /etc/mke2fs.conf). For example, a 3 GiB filesystem gets 196,608 inodes and a 512 MiB one gets 32,768. Each on-disk inode is 256 bytes in a fixed table inside each block group. Once the table is full you get ENOSPC no matter how many data blocks are free. You cannot add inodes to an existing ext4 filesystem except by growing it with resize2fs, which adds block groups and their tables.

xfs allocates inodes on demand, in chunks of 64, from ordinary free space. There is a cap, imaxpct, defaulting to 25% of the filesystem for small filesystems, but in practice a workload runs out of blocks long before it runs out of inodes. The Inodes column of df -i on xfs is an estimate, and it grows as the filesystem fills.

btrfs has no inode table. An inode is an item in the filesystem's b-tree, limited by metadata space and a 64-bit object id. df -i reports zero inodes total, which confuses monitoring that divides by it.

erofs is read-only. The inodes are laid out at image build time (32 or 64 bytes each), and since nothing can be created at runtime there is nothing to exhaust. It shows up on container hosts as the lower layer of an overlay. Anything a container creates in its writable layer goes through overlayfs to the upper filesystem, and that is where the inode gets spent.

tmpfs has its own limit, the nr_inodes mount option, defaulting to half the machine's RAM pages. /tmp and /dev/shm on tmpfs can run out of inodes too, and df -i shows it.

Reproducing inode exhaustion

leak.sh does what the agent does: per task, three containers, each with an exec directory created and only the leaf removed. Run it against a 512 MiB ext4 image, which has the same inode ratio as the real partition at one sixth the size.

1sudo truncate -s 512M ext4.img && sudo mkfs.ext4 -q -F ext4.img
2sudo mkdir -p /mnt/lab && sudo mount -o loop ext4.img /mnt/lab
3sudo tune2fs -l /dev/loop0 | grep -E 'Inode count|Block count'
4Inode count:              32768
5Block count:              131072
6
7sudo ./leak.sh /mnt/lab 2000
8sudo ./leak.sh /mnt/lab 4000
9sudo ./leak.sh /mnt/lab 4000

The third run dies partway:

1mkdir: cannot create directory '/mnt/lab/task/...': No space left on device
2df -i /mnt/lab
3Filesystem     Inodes IUsed IFree IUse% Mounted on
4/dev/loop0      32768 32768     0  100% /mnt/lab
5
6df -h /mnt/lab
7Filesystem      Size  Used Avail Use% Mounted on
8/dev/loop0      488M  130M  322M  29% /mnt/lab

100% of inodes, 29% of blocks. Each empty directory took one inode and one 4 KiB block (for its "." and ".." entries), and the inode ratio is 16 KiB per inode, so blocks lag inodes by about four to one. From here the behaviour matches the production host exactly. touch and mkdir fail. strace shows the same errno the CNI plugin got:

1mkdir("/mnt/lab/x", 0777) = -1 ENOSPC (No space left on device)

Appending 50 MiB to an existing file succeeds. Removing the task directories brings df -i back to a dozen used inodes and mkdir works again. And the same workload on a 512 MiB xfs image completes 10,000 tasks without complaint.

Conclusion

Monitor and alarm on inode usage, and stop scheduling new tasks onto a node whose free inodes drop below a threshold. Every common host metrics agent already exports the numbers:

• OpenTelemetry Collector hostmetrics receiver: system.filesystem.inodes.usage, enabled by default alongside system.filesystem.usage.
• Prometheus node_exporter: node_filesystem_files and node_filesystem_files_free. The node-exporter mixin ships NodeFilesystemAlmostOutOfFiles and NodeFilesystemFilesFillingUp alerts on them.
• CloudWatch agent: disk_inodes_free, disk_inodes_used, disk_inodes_total, in the same disk section as disk_used_percent.
• kubelet goes further: nodefs.inodesFree below 5% is a default hard eviction threshold, so a Kubernetes node that leaks inodes starts evicting pods and reclaiming images before it hits the wall.