Remount sysfs to make its superblock rw: mount -o remount,rw /sys

While testing Amazon EFS volumes in containers, I found that sequential reads in some runs were about 4 times slower than in others. EFS volumes are mounted with efs-utils, which turns mount -t efs into an NFSv4.1 mount through a local TLS proxy. In the slow runs, the NFS mount's read_ahead_kb was 128 instead of 15360.

That symptom has a known cause. Linux 5.4 dropped the default NFS readahead from 15 times rsize, which is 15 MiB with EFS's 1 MiB rsize, to 128 KiB (NFS: Optimise the default readahead size). efs-utils has written 15 MiB back at mount time since v1.30.2 in 2021, and v3.3.2 lets its watchdog re-check the value on every poll on any Linux 5.4 or later. Even with 3.3.2 installed, some containers still read with 128 KiB of readahead, and the watchdog log shows the write failing on a read-only file system:

1WARNING - Failed to modify read_ahead_kb: 15360 with returncode: 1, error: "b'/bin/sh: line 1: /sys/class/bdi/0:52/read_ahead_kb: Read-only file system'".

In my test setup, efs-utils runs in its own privileged container, next to unprivileged containers in the same network namespace. When the privileged container is the first one to start in the network namespace, readahead is set to 15 MiB. Otherwise it stays at 128 KiB, which is why some runs were slow and others were normal. This post walks through the root cause and the one-line fix: remount sysfs so its superblock becomes rw.

NFS readahead

Each NFS mount gets an anonymous device number such as 0:52 and its own backing device info (BDI), which sysfs exposes at /sys/class/bdi/0:52. read_ahead_kb in that directory caps the readahead window for every file on the mount. A new BDI starts at VM_READAHEAD_PAGES, which is 128 KiB (mm/backing-dev.c, include/linux/pagemap.h). Up to 5.3, NFS overrode it with rsize times 15 (fs/nfs/super.c); since 5.4 it only sets io_pages and leaves the 128 KiB default.

To find the BDI of a mount, take the device number from stat:

1d=$(stat -c %d /mnt/efs)
2cat /sys/class/bdi/$((d >> 8)):$((d & 255))/read_ahead_kb

efs-utils does the same lookup and then shells out to write 15 times rsize in KiB at mount. It writes once immediately and once more after sleep 2 (optimize_readahead_window()), and the watchdog compares and rewrites it on every poll (verify_and_update_readahead()). BDIs are global, not per namespace, so the host and every container see the same file and the same value.

One sysfs superblock per network namespace

/sys isn't on any disk. It is a view of kernel objects built by kernfs. Like any mounted filesystem, it has two layers. A superblock is the kernel object for one instance of a filesystem, and a mount attaches that instance to a path. Both layers carry a read-only flag, and /proc/self/mountinfo shows both: the per-mount options come before the " - " separator, and the superblock options come at the end of the line.

124 68 0:23 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw,seclabel
2           ^ superblock device       ^ per-mount options               ^ superblock options

A write needs both to be rw: __mnt_is_readonly() returns true if the mount has MNT_READONLY or the superblock has SB_RDONLY.

sysfs keeps one superblock per network namespace. When sysfs is mounted, it records the caller's network namespace as the mount's tag (fs/sysfs/mount.c), and kernfs_test_super() reuses an existing superblock when its root and tag match. The reason is /sys/class/net, which must list only the interfaces of that namespace.

Flags are applied only when a superblock is created: sget_fc() passes the requested flags to alloc_super() for a new superblock and returns an existing one as is. A later mount that asks for rw gets a per-mount rw flag on top of whatever superblock the first mount created. The superblock lives as long as some mount of it exists. Once the last sysfs mount in a network namespace is gone, the next mount there creates a new superblock with its own flags.

Example: sysfs-netns.sh mounts sysfs twice in a new network namespace, once ro and once rw, in both orders, and writes to a read_ahead_kb through each mount:

 1== host netns
 2/sys dev=0:23 per-mount=rw,nosuid,nodev,noexec,relatime superblock=rw,seclabel
 3
 4== new netns: mount sysfs ro at /tmp/sA, then rw at /tmp/sB
 5/sys dev=0:23 per-mount=rw,nosuid,nodev,noexec,relatime superblock=rw,seclabel
 6/tmp/sA dev=0:48 per-mount=ro,relatime superblock=ro,seclabel
 7/tmp/sB dev=0:48 per-mount=rw,relatime superblock=ro,seclabel
 8/tmp/sA/class/net: lo
 9write via /tmp/sA: Read-only file system
10write via /tmp/sB: Read-only file system
11
12== new netns: mount sysfs rw at /tmp/sA, then ro at /tmp/sB
13/sys dev=0:23 per-mount=rw,nosuid,nodev,noexec,relatime superblock=rw,seclabel
14/tmp/sA dev=0:48 per-mount=rw,relatime superblock=rw,seclabel
15/tmp/sB dev=0:48 per-mount=ro,relatime superblock=rw,seclabel
16/tmp/sA/class/net: lo
17write via /tmp/sA: OK
18write via /tmp/sB: Read-only file system

In both orders the two mounts share superblock 0:48, separate from the host's 0:23, and only a mount that is rw on both layers can write.

containerd mounts /sys ro for unprivileged containers, rw for privileged ones

containerd's default spec mounts /sys as sysfs with nosuid, noexec, nodev and ro (pkg/oci/mounts.go). For a privileged container, WithPrivileged includes WithWriteableSysfs, which changes ro to rw in that container's own spec. runc then mounts a fresh sysfs inside the container's network namespace with the flags from the spec (libcontainer/rootfs_linux.go), and the kernel rules above take over. The superblock is rw only if that container's mount is the first one in the network namespace.

The fix: remount the superblock

There are three ways to try to make /sys writable, and they reach different parts of the kernel:

The third command is the fix. It clears the read-only flag on the superblock and leaves the other containers' /sys mounts alone, so they stay per-mount ro and still can't write to sysfs.

A superblock remount needs CAP_SYS_ADMIN in the user namespace that owns the superblock (fs/namespace.c). sysfs sets fc->global (fs/sysfs/mount.c), so sget_fc() makes the initial user namespace the owner of every sysfs superblock, even one created inside a user namespace (fs/super.c). A privileged container has CAP_SYS_ADMIN in the initial user namespace and can remount. An unprivileged container can't. The remount is idempotent, so it can run on every start. If it fails, log it where someone will see it: the efs-utils warnings only go to /var/log/amazon/efs inside the container.

Conclusion

Containers that share a network namespace share one sysfs superblock, and the first one to mount it picks ro or rw for all of them. In containerd, a privileged container's rw /sys only holds if it got there first. To check, compare the superblock options at the end of each container's /sys line in mountinfo.