For a long time, some of our container hosts, most often the smallest ones, would go dark every so often. A t3.micro with 1 GiB of memory, running a customer's containers, would stop sending logs and metrics all at once and never start again. The orchestrator still showed every container as RUNNING. The only signs of …

Read More

We run customer workloads in containers, and each container gets its own VM (an EC2 instance or a microVM). Every VM needs a few OCI images regardless of the workload: the language runtime image the customer's workload is built on, a log forwarding sidecar, an internal monitoring container. The obvious way to provide …

Read More

I fixed an issue where new containers failed to launch on a node with "no space left on device". There was free space available, and the issue happened only on a small percentage of nodes. It turned out those nodes churned containers more often than the others, and the failure came from running out of inodes, …

Read More

I recently dealt with an issue where a service failed occasionally during short throttling windows. The root cause was one I had seen many times before: a process holding multiple clients for the same service. This post walks through the issue, why the bug is rare in Java but common in Go, and how to avoid it. The bug …

Read More

Our container platform moved from Amazon Linux 2 (AL2) to Bottlerocket. Soon after the migration started, a partner team that runs Spark jobs reported a regression. Their jobs still finished successfully, but far more of them reported zero "storage read bytes": about 23% of tasks on AL2, and about 56% on …

Read More

I recently reviewed code that let a dynamic set of goroutines wait for updates from a single writer. Each waiter wanted to block until the writer had published something newer than what it had already seen. The implementation used channels and was hard to follow. A condition variable (sync.Cond) would have been less …

Read More

I was getting an arm64 build of our Bottlerocket variant ready for production on Graviton EC2 instances. Load tests kept failing at a low rate. Somewhere between 0.3% and 1% of instances froze a few seconds into boot and never came back. The x86_64 build of the same variant has never done this. I never found the root …

Read More

I used NVIDIA's go-dcgm to read GPU health data and stats. It hardcodes the name of the dynamic library it loads, which looked odd at first and turns out to be good practice. go-dcgm uses cgo, and its initDCGM dlopens the literal string libdcgm.so.4 at runtime: 1func initDCGM(m mode, args ...string) (err error) { 2 …

Read More

A while ago I debugged a resource leak caused by a service stuck in a restart loop. Each restart left a process behind, and after a few minutes the host had over a thousand copies of it. Nothing alerted on the loop itself. We noticed CPU and memory climbing, and only found the restart loop after reading the journal. …

Read More

We run an OpenTelemetry Collector (otel-collector) as a systemd service on Bottlerocket. Its Prometheus metrics endpoint was configured on port 51000, which falls inside the default Linux ephemeral port range. On a small portion of EC2 instances, the otel-collector kept restarting, failing to bind with EADDRINUSE. …

Read More