Our container platform moved from Amazon Linux 2 (AL2) to Bottlerocket. Soon after the migration started, a partner team that runs Spark jobs reported a regression. Their jobs still finished successfully, but far more of them reported zero "storage read bytes": about 23% of tasks on AL2, and about 56% on …

Read More

I was getting an arm64 build of our Bottlerocket variant ready for production on Graviton EC2 instances. Load tests kept failing at a low rate. Somewhere between 0.3% and 1% of instances froze a few seconds into boot and never came back. The x86_64 build of the same variant has never done this. I never found the root …

Read More

I used NVIDIA's go-dcgm to read GPU health data and stats. It hardcodes the name of the dynamic library it loads, which looked odd at first and turns out to be good practice. go-dcgm uses cgo, and its initDCGM dlopens the literal string libdcgm.so.4 at runtime: 1func initDCGM(m mode, args ...string) (err error) { 2 …

Read More

A while ago I debugged a resource leak caused by a service stuck in a restart loop. Each restart left a process behind, and after a few minutes the host had over a thousand copies of it. Nothing alerted on the loop itself. We noticed CPU and memory climbing, and only found the restart loop after reading the journal. …

Read More

We run an OpenTelemetry Collector (otel-collector) as a systemd service on Bottlerocket. Its Prometheus metrics endpoint was configured on port 51000, which falls inside the default Linux ephemeral port range. On a small portion of EC2 instances, the otel-collector kept restarting, failing to bind with EADDRINUSE. …

Read More

I recently ran into an issue after upgrading the containerd client library to v2.3.0. The root cause was a change in one of containerd's dependencies, continuity, that started enforcing fsync when copying files. containerd/continuity is a library in containerd that provides filesystem utilities: file copying, directory …

Read More

Bottlerocket v11.0.0 dropped the socat package from bottlerocket-core-kit (#742). If you had a service that relied on socat to bridge a Unix Domain Socket (UDS) to a TCP port, you need a replacement. This post shows how to use systemd-socket-proxyd instead. Why socat was used soci-snapshotter exposes its metrics …

Read More

We run a container platform. For privacy and security reasons, we do not collect kernel logs because customer workloads use the same kernel as the host and kernel messages can contain sensitive customer data, such as command-line arguments surfaced in audit logs. However, we recently hit a blind spot: foo.service was …

Read More

While prototyping Bottlerocket, I discovered it doesn't recognize additional EBS volumes specified through Block device mappings on Xen. For example, launching the same AMI on t2.medium (Xen) and t3.medium (Nitro) with "DeviceName=/dev/xvdcz": On Nitro, the device appears at /dev/nvme1n1 and …

Read More

I recently debugged a resource leak where a systemd service kept restarting while leaving a process behind after each restart. The root cause isn't particularly interesting: a backward-incompatible third-party dependency upgrade. But the debugging process and lessons learned are. Thousands of zombie processes from a …

Read More