For a long time, some of our container hosts, most often the smallest ones, would go dark every so often. A t3.micro with 1 GiB of memory, running a customer's containers, would stop sending logs and metrics all at once and never start again. The orchestrator still showed every container as RUNNING. The only signs of …

Read More

We run customer workloads in containers, and each container gets its own VM (an EC2 instance or a microVM). Every VM needs a few OCI images regardless of the workload: the language runtime image the customer's workload is built on, a log forwarding sidecar, an internal monitoring container. The obvious way to provide …

Read More

I fixed an issue where new containers failed to launch on a node with "no space left on device". There was free space available, and the issue happened only on a small percentage of nodes. It turned out those nodes churned containers more often than the others, and the failure came from running out of inodes, …

Read More

Our container platform moved from Amazon Linux 2 (AL2) to Bottlerocket. Soon after the migration started, a partner team that runs Spark jobs reported a regression. Their jobs still finished successfully, but far more of them reported zero "storage read bytes": about 23% of tasks on AL2, and about 56% on …

Read More

I recently debugged a mysterious latency issue: after migrating a systemd service from path-activation to socket-activation, there was a consistent ~1 second time-to-available latency. The culprit was a bad practice—starting the daemon program as a new process in socket-activation. Let's dive into the details. Starting …

Read More

A bug caused around 0.5% of container workloads to fail to start during load test. This post walks through the bug and its fix, an interesting mix of Linux namespaces, Go concurrency, and syscalls. The need to run a program in its own network namespace and mount namespace soci-snapshotter is an open-source containerd …

Read More

Early this year, we migrated containerd from v1.7 to v2.0.5. However, we quickly noticed image pulls from Amazon Elastic Container Registry (ECR) began failing for both public and private ECR repositories. For example: 1# public ECR 2FATA[0031] failed to resolve reference …

Read More

I recently worked on getting amazon-ssm-agent to run inside containers on Bottlerocket. During that process, I ran into a TLS issue connecting to amazonaws.com. The root cause turned out be interesting and we'll walk through it in this post. Running amazon-ssm-agent in a container: why and how? To enable sessions …

Read More

As Amazon Linux 2 (AL2) approaches its End of Life on June 30, 2025, we have started migrating our container platform from AL2 to Bottlerocket. The migration encountered a few speed bumps. In this post, we'll examine one of them: missing container disk I/O stats. Why are container I/O dashboards blank? Since …

Read More