Apply sysctl before systemd: a lesson from reserving ephemeral ports too late

We run an OpenTelemetry Collector (otel-collector) as a systemd service on Bottlerocket. Its Prometheus metrics endpoint was configured on port 51000, which falls inside the default Linux ephemeral port range. On a small portion of EC2 instances, the otel-collector kept restarting, failing to bind with EADDRINUSE. Sometimes it self-healed, sometimes it kept restarting and we lost logs. This post walks through the fix, which involves a race condition among systemd services and why the sysctl has to be applied before systemd starts.

Ephemeral ports

When a process makes an outbound TCP or UDP connection without binding a source port, the kernel assigns one from the ephemeral port range. Every HTTP request, DNS lookup, or EC2 instance metadata service call consumes one.

1# Note: even though the path says ipv4, this knob covers IPv6 ports too.
2$ cat /proc/sys/net/ipv4/ip_local_port_range
332768   60999

The Linux default is 32768-60999. Any port in that range can be handed out at any time. Port 51000 sits in the middle of it. If any process makes an outbound connection and the kernel picks 51000 as the source port before the collector starts, the collector's bind() fails.

Linux supports tuning the reserved ports through /proc/sys. Ports listed in /proc/sys/net/ipv4/ip_local_reserved_ports are never handed out as ephemeral source ports. /proc/sys is the procfs interface to kernel tunables. Each file is one knob, readable and writable with ordinary file I/O:

1# Reserve port 51000 from ephemeral allocation
2echo "51000" > /proc/sys/net/ipv4/ip_local_reserved_ports

sysctl is the userspace tool for the same tree. It maps a dotted name to a path by replacing dots with slashes, so net.ipv4.ip_local_reserved_ports is /proc/sys/net/ipv4/ip_local_reserved_ports:

1# read
2$ sysctl net.ipv4.ip_local_reserved_ports
3net.ipv4.ip_local_reserved_ports =
4
5# write
6$ sysctl -w net.ipv4.ip_local_reserved_ports=51000
7net.ipv4.ip_local_reserved_ports = 51000

First fix: systemd-sysctl

The usual way to apply a sysctl at boot is systemd-sysctl, which reads /etc/sysctl.d/*.conf and writes the values into procfs. For example,

1# /etc/sysctl.d/90-reserved-ports.conf
2net.ipv4.ip_local_reserved_ports = 51000

In Bottlerocket this is expressed as a setting in a defaults.d TOML file:

1# variants/my-variant/defaults.d/90-reserved-ports.toml
2[settings.kernel.sysctl]
3"net.ipv4.ip_local_reserved_ports" = "51000"

Race condition among systemd services on Graviton instances

After shipping the fix, the EADDRINUSE restarts disappeared on x86_64 EC2 instances. The issue persisted on a small percentage of Graviton instances. The pattern was about 18 restarts spread ~3.3s apart (the otel-collector systemd restart interval is 3s), then self-recovery. The loop lasted ~60s, about one TCP TIME_WAIT duration. TIME_WAIT is a compile-time constant set to 60s, and changing it means recompiling the kernel, see Appendix: TIME_WAIT is 60 seconds and not tunable. A long-running process holding the port would never self-heal, so self-healing after 60s points to a short-lived connection from port 51000 that had already closed and was sitting in TIME_WAIT.

In the first fix, Bottlerocket applies the setting through a multi-stage pipeline:

1. Build time: all defaults.d/*.toml files are merged in lexicographic order into a single defaults.toml baked into the image.
2. Early boot (storewolf): reads defaults.toml and populates an on-disk key-value datastore. No services are triggered yet.
3. Settings commit (settings-applier): thar-be-settings looks up which services are affected by the changed keys. For settings.kernel.sysctl.*, that is systemd-sysctl.
4. Config rendering: thar-be-settings renders a Handlebars template into a standard /etc/sysctl.d/ file. The TOML key becomes the line net.ipv4.ip_local_reserved_ports = 51000.
5. Service restart: thar-be-settings restarts systemd-sysctl, which re-reads the rendered config and writes to /proc/sys/net/ipv4/ip_local_reserved_ports.

The value does not reach the kernel until step 5: systemd-sysctl.service. systemd-sysctl is a oneshot unit that reads sysctl configuration from /etc/sysctl.d/.conf, /run/sysctl.d/.conf, and /usr/lib/sysctl.d/*.conf, then writes each key to the matching file under /proc/sys/. Being a systemd unit, it participates in systemd's parallel startup graph.

systemd-sysctl.service and the networking units (systemd-networkd, the DHCP client, early EC2 IMDS metadata fetches) have no After= or Before= relationship, so systemd may schedule them in either order. Below is the boot log from an instance where the port reservation landed after another service had already used port 51000:

 1~1.4s   eth0 link up
 2~1.5s   DHCPv4 acquired
 3~1.8s   early boot services call the metadata service     <- kernel assigns 51000
 4                                                             as the source port
 5~2.2s   systemd-sysctl applies ip_local_reserved_ports    <- too late; 51000 is
 6                                                             already in use
 7~2.3s   that connection closes -> 51000 enters TIME_WAIT (~60s)
 8...
 9~6.9s   otel-collector bind(0.0.0.0:51000) -> EADDRINUSE  <- restart loop begins
10...
11~62s    TIME_WAIT expires -> next restart attempt succeeds

Reserving a port does not evict a connection already using it and does not clear a TIME_WAIT entry. It only affects future ephemeral allocations. Once the kernel has handed out 51000, writing the sysctl does nothing until that socket drains. Lose the race once and the port is unavailable for a full minute with no runtime workaround.

Why only on Graviton? I do not know. This is not the first boot-timing issue I have hit on Graviton but not on x86_64. My guess is that Graviton instances boot faster, but I never confirmed it, and it does not matter much here. x86_64 was not fixed, it was lucky. The bug is still there.

Second fix: kernel command line sysctl.*=

Linux 5.8 commit 3db978d480e2 added the ability to set sysctl parameters on the kernel command line with the sysctl.*= prefix. So we can apply the port reservation at kernel initialization, before any networking can occur, before systemd starts. On a traditional Linux system it goes in the bootloader config:

1GRUB_CMDLINE_LINUX="sysctl.net.ipv4.ip_local_reserved_ports=51000"

In Bottlerocket this goes in the variant's Cargo.toml:

1kernel-parameters = [
2    "sysctl.net.ipv4.ip_local_reserved_ports=51000",
3]

The kernel documentation says:

1sysctl.*=       [KNL]
2                Set a sysctl parameter, right before loading the init
3                process, as if the value was written to the respective
4                /proc/sys/... file.

There is no race now because the port is reserved before systemd starts. The order inside kernel_init() is sequential, see Appendix: how the kernel applies sysctl parameters for the source:

 1Kernel boot
 2  │
 3  ├── kernel_init_freeable()    <- network subsystem initialized (sysctl entries exist)
 4  │                                but no userspace process has started
 5  │
 6  ├── do_sysctl_args()          <- ip_local_reserved_ports = "51000"
 7  │                                port 51000 is now reserved
 8  │
 9  └── run_init_process()        <- PID 1 (systemd) starts
10        │
11        ├── systemd-networkd    <- interfaces up, DHCP, outbound connections
12        │                          cannot use port 51000
13        │
14        └── otel-collector      <- bind(51000) succeeds

do_sysctl_args() runs after the network subsystem is initialized, so the sysctl entry exists and is writable, and before run_init_process(), so no userspace process has started and no network traffic has happened. No process can allocate an ephemeral port before PID 1 exists.

Comparing the two fixes

Conclusion

If a service binds a port in 32768-60999, it has a latent conflict waiting for the right boot timing. For a sysctl that must be in effect before the first packet, put it on the kernel command line as sysctl.*= (Linux >= 5.8). It is applied in kernel_init() before the init process (systemd) starts, so nothing can race it.

Appendix: how the kernel applies sysctl parameters

In init/main.c, kernel_init() bootstraps the system:

 1static int __ref kernel_init(void *unused)
 2{
 3    wait_for_completion(&kthreadd_done);
 4    kernel_init_freeable();       // all initcalls run here (network subsystem initialized)
 5    async_synchronize_full();
 6
 7    system_state = SYSTEM_RUNNING;
 8    numa_default_policy();
 9    rcu_end_inkernel_boot();
10
11    do_sysctl_args();             // <--- sysctl.* cmdline params applied HERE
12
13    if (ramdisk_execute_command) {
14        ret = run_init_process(ramdisk_execute_command);  // PID 1 starts HERE
15        ...
16    }
17    // tries /sbin/init, /etc/init, etc.
18}

do_sysctl_args() lives in fs/proc/proc_sysctl.c:

 1void do_sysctl_args(void)
 2{
 3    char *command_line;
 4    struct vfsmount *proc_mnt = NULL;
 5
 6    command_line = kstrdup(saved_command_line, GFP_KERNEL);
 7    if (!command_line)
 8        panic("%s: Failed to allocate copy of command line\n", __func__);
 9
10    parse_args("Setting sysctl args", command_line,
11               NULL, 0, -1, -1, &proc_mnt, process_sysctl_arg);
12
13    if (proc_mnt)
14        kern_unmount(proc_mnt);
15    kfree(command_line);
16}

It parses the saved command line, finds all sysctl.*= parameters, mounts procfs internally, and writes each value to the matching /proc/sys/... file via kernel_write().

Appendix: TIME_WAIT is 60 seconds and not tunable

TIME_WAIT duration is a compile-time constant in include/net/tcp.h:

1#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
2                                  * state, about 60 seconds	*/

There is no /proc/sys/ entry for it. Changing it means recompiling the kernel. You can observe live TIME_WAIT sockets and their remaining timers:

1ss -tan state time-wait -o
2# Local Address:Port  Peer Address:Port  timer:(timewait,54sec,0)

Related knobs exist but none shorten the wait. tcp_tw_reuse allows reusing a TIME_WAIT socket for a new outgoing connection, tcp_max_tw_buckets caps the total count, and SO_REUSEADDR lets a listener bind over a TIME_WAIT entry. (tcp_tw_recycle was removed in kernel 4.12 as unsafe behind NAT.)