A while ago I debugged a resource leak caused by a service stuck in a restart loop. Each restart left a process behind, and after a few minutes the host had over a thousand copies of it. Nothing alerted on the loop itself. We noticed CPU and memory climbing, and only found the restart loop after reading the journal. …

Read More

Bottlerocket v11.0.0 dropped the socat package from bottlerocket-core-kit (#742). If you had a service that relied on socat to bridge a Unix Domain Socket (UDS) to a TCP port, you need a replacement. This post shows how to use systemd-socket-proxyd instead. Why socat was used soci-snapshotter exposes its metrics …

Read More

I recently debugged a resource leak where a systemd service kept restarting while leaving a process behind after each restart. The root cause isn't particularly interesting: a backward-incompatible third-party dependency upgrade. But the debugging process and lessons learned are. Thousands of zombie processes from a …

Read More

I recently debugged a mysterious latency issue: after migrating a systemd service from path-activation to socket-activation, there was a consistent ~1 second time-to-available latency. The culprit was a bad practice—starting the daemon program as a new process in socket-activation. Let's dive into the details. Starting …

Read More