Simplify device path on boot with udev
While prototyping Bottlerocket, I discovered it doesn't recognize additional EBS volumes specified through Block device mappings on Xen. For example, launching the same AMI on t2.medium (Xen) and t3.medium (Nitro) with "DeviceName=/dev/xvdcz":
On Nitro, the device appears at /dev/nvme1n1 and /dev/disk/by-ebs-id/xvdcz. On Xen, it appears at /dev/xvdcz.
In the prototype, the xvdcz volume is formatted, labeled, and mounted, involving format-data-volume.service and mnt-data.mount. The format service hardcodes its dependency on /dev/disk/by-ebs-id/xvdcz.
1# format-data-volume.service
2[Unit]
3Description=Format data volume (/mnt/data)
4DefaultDependencies=no
5Before=local-fs.target
6Requires=dev-disk-by\x2debs\x2did-xvdcz.device <--- hardcode
7After=dev-disk-by\x2debs\x2did-xvdcz.device
8RefuseManualStart=true
9RefuseManualStop=true
10
11[Service]
12Type=oneshot
13RemainAfterExit=true
14StandardError=journal+console
15
16ExecStart=/sbin/mkfs -t ext4 -L DATA_STORAGE /dev/disk/by-ebs-id/xvdcz <--- hardcode
17
18[Install]
19WantedBy=local-fs.target
1# mnt-data.mount
2[Unit]
3Description=Data Directory (/mnt/data)
4DefaultDependencies=no
5Conflicts=umount.target
6Requires=format-data-volume.service
7After=format-data-volume.service
8Before=local-fs.target
9RequiresMountsFor=/mnt
10
11[Mount]
12What=LABEL=DATA_STORAGE
13Where=/mnt/data
14Type=ext4
15Options=defaults
16
17[Install]
18WantedBy=local-fs.target
It took me a while to arrive at a solution that's simple and only 6 lines. In this post, we'll look at that journey.
Approach 1: Polling path existence and formatting in a helper program
The easiest solution is to change format-data-volume.service to run a helper program like below.
1# format-data-volume.service
2[Unit]
3# Remove dependency and ordering on device that only exists on Nitro
4# Requires=dev-disk-by\x2debs\x2did-xvdcz.device
5# After=dev-disk-by\x2debs\x2did-xvdcz.device
6
7[Service]
8ExecStart=/sbin/poll-and-format-data-volume
The helper, poll-and-format-data-volume, polls both paths /dev/xvdcz and /dev/disk/by-ebs-id/xvdcz periodically (e.g., every 100ms) to find the data volume device path—either /dev/xvdcz or /dev/disk/by-ebs-id/xvdcz. Once the path is identified, the helper formats and labels the volume. This approach isn't systemd-idiomatic, requires an additional program, and the polling interval adds boot latency.
Approach 2: Use service template to receive the correct device from udev
udev is a userspace device manager that handles device events and management in userspace (as opposed to the kernel). udev receives device events from the kernel and applies rules to create device nodes, set permissions, create symlinks, and trigger actions. Since udev already has information about the device node on both Xen and Nitro, we can pass this information to format-data-volume.service without polling.
Change format-data-volume.service to a template.
1# [email protected]
2[Unit]
3Requires=%i.device
4After=%i.device
5
6[Service]
7ExecStart=/sbin/mkfs -t ext4 -L DATA_STORAGE %f
Then we can match device events and call the corresponding format-data-volume.service. Install the following udev rule.
1# Nitro
2ENV{DEVLINKS}=="*/dev/disk/by-ebs-id/xvdcz*", ENV{DEVTYPE}=="disk", TAG+="systemd", \
3 ENV{SYSTEMD_WANTS}="format-data-volume@dev-disk-by\x2debs\x2did-xvdcz.service"
4
5# Xen
6ENV{DEVNAME}=="/dev/xvdcz", ENV{DEVTYPE}=="disk", TAG+="systemd", \
7 ENV{SYSTEMD_WANTS}="[email protected]"
1Source8: symlink-by-ebs-id-on-xen.rules
2
3%install
4install -d %{buildroot}/%{_cross_udevrulesdir}
5install -p -m 0644 %{S:8} %{buildroot}/%{_cross_udevrulesdir}/99-symlink-by-ebs-id-on-xen.rules
6
7%files
8%{_cross_udevrulesdir}/99-symlink-by-ebs-id-on-xen.rules
This approach formats and labels the volume. However, it's very difficult to synchronize mnt-data.mount on the completion of format-data-volume.service because systemd doesn't support Requires= on a template service. We could create a target unit as a synchronization point, e.g., data-volume-formatted.target, containing only the instance service. However, this introduces a race condition: we don't know when the udev rule will add the instance service as part of the target. The data-volume-formatted.target could become active immediately before systemd-udev detects the device.
Approach 3: Create a new symlink device /dev/disk/data-volume
The problem comes from having two different device path names. To avoid this difference, we can create a canonical path /dev/disk/data-volume.
Install the following udev rule.
1 # Nitro
2ENV{DEVLINKS}=="*/dev/disk/by-ebs-id/xvdcz*", ENV{DEVTYPE}=="disk", TAG+="systemd", \
3SYMLINK+="disk/data-volume"
4
5# Xen
6ENV{DEVNAME}=="/dev/xvdcz", ENV{DEVTYPE}=="disk", TAG+="systemd", \
7SYMLINK+="disk/data-volume"
Then update format-data-volume.service to depend on the canonical path.
1# format-data-volume.service
2[Unit]
3Requires=dev-disk-x2ddata\x2dvolume.device
4After=dev-disk-x2ddata\x2dvolume.device
5
6[Service]
7ExecStart=/sbin/mkfs -t ext4 -L DATA_STORAGE /dev/disk/data-volume
This approach is systemd-native, clean, and requires few lines of code. But can we do better?
Approach 4: Add /dev/disk/by-ebs-id/xvdcz on Xen
The key idea of approach 3 is introducing a canonical name. Taking this one step further, there's already a canonical name: /dev/disk/by-ebs-id/xvdcz. We can simply add this canonical name on Xen! Install the following rule symlink-by-ebs-id-on-xen.rules:
1ACTION=="remove", GOTO="xen_ebs_volumes_end"
2KERNEL!="xvd*", GOTO="xen_ebs_volumes_end"
3SUBSYSTEM!="block", GOTO="xen_ebs_volumes_end"
4ENV{DEVTYPE}!="disk", GOTO="xen_ebs_volumes_end"
5
6SYMLINK+="disk/by-ebs-id/%k"
7
8LABEL="xen_ebs_volumes_end"
The extra GOTO statements are a performance improvement: they short-circuit rule evaluation. Otherwise, the rule would evaluate for all devices during all add/change/remove operations.
Conclusion
I couldn't agree more with one of my colleague's comments: "systemd is pretty frustrating in that you would think you could just create the symlink yourself and the dependencies would work, but nope—it has to come through official udev channels".