A bug caused around 0.5% of container workloads to fail to start during load test. This post walks through the bug and its fix, an interesting mix of Linux namespaces, Go concurrency, and syscalls. The need to run a program in its own network namespace and mount namespace soci-snapshotter is an open-source containerd …

Read More

When I first moved building Bottlerocket AMI from an EC2 host to AWS CodeBuild, I was hit by a very slow build. On an EC2 instance, I built both the x86 and Arm versions on x86 instances, and fresh builds finished in 5 minutes. However, on CodeBuild with more vCPU and memory, the build process was painfully slower. The …

Read More

Early this year, we migrated containerd from v1.7 to v2.0.5. However, we quickly noticed image pulls from Amazon Elastic Container Registry (ECR) began failing for both public and private ECR repositories. For example: 1# public ECR 2FATA[0031] failed to resolve reference …

Read More

In Detect and fix rare cases where the primary ENI does not serve default traffic , we used IMDS "meta-data/mac" to get the primary ENI's MAC address. However, we encountered the following errors in 0.5% of EC2 ARM instance launches: 1failed to get IMDS /mac: operation error ec2imds: GetMetadata, exceeded …

Read More

There are a few programs we install in Bottlerocket that cannot be built from source. For these programs, we download the binary from a secure repository and install it using an RPM spec like this: 1# foo.spec 2Name: %{_cross_os}foo 3 4Source0: foo 5 6%install 7install -d %{buildroot}%{_cross_sbindir} 8install -D -p -m …

Read More

Bottlerocket is a Linux-based operating system optimized for hosting containers. We use Bottlerocket to run millions of containers each day. There are three key differences between Bottlerocket and common Linux distributions like Amazon Linux 2023: The rootfs is read-only. There is no package manager (e.g., yum) in …

Read More

Bottlerocket is a Linux-based operating system optimized for hosting containers. At my work, we migrated from Amazon Linux to Bottlerocket and experienced the following benefits: Developer-friendly: Easy to understand and fast to build. RPM spec and configuration TOML files are all you need. Every developer can build a …

Read More

I recently dealt with a server livelock issue caused by memory page thrashing. This post refreshes the Linux memory basics I found useful for debugging the issue. Much of the content is from Chapter 7 of Systems Performance: Enterprise and the Cloud. Virtual Memory Virtual memory is an abstraction that provides each …

Read More

During testing, we encountered a rare scenario when launching EC2 instances with multiple ENIs: the primary ENI (device index 0) does not serve default network traffic. This occurs in approximately 1 out of 10,000 launches (0.01%). For example, when configuring two ENIs on an instance—ENI-0 (deviceIndex=0) from …

Read More

Security-Enhanced Linux (SELinux) is a mandatory access control (MAC) system that enhances Linux security. "Mandatory" means access control is strictly enforced by predefined policy rules—users and processes cannot modify these rules at will, ensuring security is not left to individual discretion. SELinux is …

Read More