Tips for Building Bottlerocket AMIs
Bottlerocket is a Linux-based operating system optimized for hosting containers. At my work, we migrated from Amazon Linux to Bottlerocket and experienced the following benefits:
-
Developer-friendly: Easy to understand and fast to build. RPM spec and configuration TOML files are all you need. Every developer can build a Bottlerocket AMI on an EC2 instance in just a few minutes. For example, I can build and register a new Bottlerocket AMI from scratch in less than 4 minutes on a c5.4xlarge instance. Follow-up builds take about 3 minutes due to caching.
-
Significantly improved boot performance: The time from Linux boot to containerd server running dropped from 23 seconds to 14 seconds. A recent addition of EROFS made it start even faster while using a 46% smaller root EBS volume.
-
Enhanced security: Features a read-only root filesystem. See more details at https://bottlerocket.dev/#security-focused.
Getting started with Bottlerocket is a smooth experience thanks to good documentation. This post shares some tips I've found useful.
Clean up build cache periodically
Bottlerocket uses Docker to build RPM packages. Over time, Docker can consume significant disk space.
1dev-dsk % docker system df
2TYPE TOTAL ACTIVE SIZE RECLAIMABLE
3Images 48 3 134.5GB 128.4GB (95%)
4Containers 3 1 14B 14B (100%)
5Local Volumes 53 0 68B 68B (100%)
6Build Cache 4444 15 249.6GB 248.1GB
If your build fails, check available disk space and run docker system prune -a if space is low. Alternatively, you can expand the EBS volume size live on EC2.
List all installed packages
In Bottlerocket, there's no way to install packages on a running instance. All packages come with the Bottlerocket OS image.
You can check the list of installed packages in ./build/images/${variant}/latest/application-inventory.json. For example:
1 {
2 "Name": "bottlerocket-amazon-cloudwatch-agent",
3 "Publisher": "Bottlerocket",
4 "Version": "1.300034.0",
5 "Release": "1.1755731416.9333e95b.br1",
6 "Epoch": "0",
7 "InstalledTime": "2025-08-21T08:09:05Z",
8 "ApplicationType": "Unspecified",
9 "Architecture": "x86_64",
10 "Url": "(none)",
11 "Summary": "Amazon cloudwatch agent"
12 }
Override a kit
Bottlerocket's build system, twoliter, organizes packages into kits. Currently, there are two official kits: bottlerocket-kernel-kit and bottlerocket-core-kit. You can also create your own kit for packages not included in the official kits, such as bottlerocket-extra-kit.
A kit is essentially a multi-arch OCI image hosted in a registry. You can override the image registry using a Twoliter.override file:
1[bottlerocket.bottlerocket-core-kit]
2registry = "<account-id>.dkr.ecr.us-west-2.amazonaws.com/my-own-kit"
Trigger package rebuild on source change
For artifacts that are not in a kit, you can build them into RPM packages from sources. However, it's not common practice to include source code in the packages directory. The common practice is to put the source code in the sources directory. Take the settings-plugins as an example: packages/settings-plugins contains only Cargo.toml and settings-plugins.spec, while the Rust source code is in sources/settings-plugins.
1# packages/settings-plugins/settings-plugins.spec
2%build
3%cargo_build --manifest-path %{_builddir}/sources/Cargo.toml \
4 -p settings-plugin-aws-dev
To rebuild packages/settings-plugins when sources/settings-plugins changes, use source-groups:
1# packages/settings-plugins/Cargo.toml
2[package.metadata.build-package]
3source-groups = [
4 "settings-plugins"
5]
What changes trigger package rebuild
For most packages, changes in the spec file, Cargo.toml, and any patches listed in the spec trigger a rebuild. You can list all changes that would trigger a package rebuild from the output file. Take the settings-plugins package as an example:
1% cargo make -e BUILDSYS_VARIANT=aws-dev
2% cat target/x86_64/debug/build/settings-plugins-5540b18661a3ef21/output | grep 'rerun-if'
3cargo:rerun-if-env-changed=BUILDSYS_ARCH
4cargo:rerun-if-env-changed=BUILDSYS_EXTERNAL_KITS_DIR
5cargo:rerun-if-env-changed=BUILDSYS_OUTPUT_GENERATION_ID
6cargo:rerun-if-env-changed=BUILDSYS_PACKAGES_DIR
7cargo:rerun-if-env-changed=BUILDSYS_ROOT_DIR
8cargo:rerun-if-env-changed=BUILDSYS_STATE_DIR
9cargo:rerun-if-env-changed=TLPRIVATE_SDK_IMAGE
10cargo:rerun-if-changed=Cargo.toml
11cargo:rerun-if-changed=/home/peng/github/bottlerocket/build/external-kits/external-kit-metadata.json
12cargo:rerun-if-changed=/home/peng/github/bottlerocket/sources/settings-plugins/aws-k8s/src/lib.rs
13cargo:rerun-if-changed=/home/peng/github/bottlerocket/sources/settings-plugins/aws-k8s/Cargo.toml
14cargo:rerun-if-changed=/home/peng/github/bottlerocket/sources/settings-plugins/vmware-k8s/src/lib.rs
15...
If you need to rebuild a package with changes not listed there (for example, I have a package foo that downloads some internal binary bar from S3 and I want to trigger a rebuild after a new version of bar is downloaded), the simplest way to trigger a rebuild is touch foo.spec.
Debug RPM build
When the build log isn't sufficient to debug RPM build failures, you can enter the build environment container to see what's happening:
- Add
sleep 1000to the %build section of the package's RPM spec - Run
ps aux | grep sleepto find the build container and get its PID sudo nsenter -a -t <container_pid>cd home/builder/rpmbuild/BUILDto access the build environment
Inspect RPM package contents
Bottlerocket places RPM packages in ./build/rpms/. When you need to examine files within an RPM package:
1% RPM_FILE=/home/peng/BottlerocketVariants/build/rpms/foo/foo-0.0-0.1755731416.9333e95b.br1.x86_64.rpm
2% IMAGE=public.ecr.aws/amazonlinux/amazonlinux:2023
3% docker run --rm -it -v ${RPM_FILE}:/tmp/foo.rpm $IMAGE /bin/sh
4
5# Inside the container
6sh-5.2# rpm2cpio /tmp/foo.rpm | cpio -t
7sh-5.2# mkdir data && cd data
8sh-5.2# rpm2cpio /tmp/foo.rpm | cpio -idmv
9sh-5.2# ls -lh ./x86_64-bottlerocket-linux-gnu/sys-root/usr/sbin/foo
I often inspect the RPM to examine Bottlerocket's SELinux policy:
1% sesearch -A -s runtime_t -t data_t -c file -p relabelto policy.31
2allow trusted_s global:file { mounton quotaon relabelfrom relabelto };
Inspect disk images
When you need to debug the OS disk image:
1# using the aws-dev variant as an example.
2% IMG_FILE=/home/peng/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img.lz4
3% lz4 -d ${IMG_FILE}
4% IMG_FILE=/home/peng/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img
5
6% fdisk -l ${IMG_FILE}
7dev-dsk-pezh1-2b-02d2437c % fdisk -l ${IMG_FILE}
8Disk /home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img: 2 GiB, 2147483648 bytes, 4194304 sectors
9Units: sectors of 1 * 512 = 512 bytes
10Sector size (logical/physical): 512 bytes / 512 bytes
11I/O size (minimum/optimal): 512 bytes / 512 bytes
12Disklabel type: gpt
13Disk identifier: 67987D5B-4D60-440F-A6F7-B7DD2EFE81DD
14
15Device Start End Sectors Size Type
16/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img1 2048 10239 8192 4M BIOS boot
17/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img2 10240 20479 10240 5M EFI System
18/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img3 20480 102399 81920 40M unknown
19/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img4 102400 1986559 1884160 920M unknown
20/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img5 1986560 2007039 20480 10M unknown
21/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img6 2007040 2058239 51200 25M unknown
22/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img7 2058240 2068479 10240 5M unknown
23/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img8 2068480 2150399 81920 40M unknown
24/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img9 2150400 4034559 1884160 920M unknown
25/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img10 4034560 4055039 20480 10M unknown
26/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img11 4055040 4106239 51200 25M unknown
27/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img12 4106240 4190207 83968 41M unknown
28/home/pezh/github/bottlerocket/build/images/x86_64-aws-dev/latest/bottlerocket-aws-dev-x86_64.img13 4190208 4192255 2048 1M unknown
29
30% sudo losetup --find --partscan --show ${IMG_FILE}
31/dev/loop0
32
33% sudo mkdir /mnt/bottlerocket
34% sudo mount /dev/loop0p4 /mnt/bottlerocket
35
36% ls -lh /mnt/bottlerocket
37total 80K
38lrwxrwxrwx. 1 root root 9 Jul 19 01:20 bin -> ./usr/bin
39drwxr-xr-x. 2 root root 4.0K Jul 26 23:10 boot
40drwxr-xr-x. 2 root root 4.0K Jul 19 01:20 dev
41drwxr-xr-x. 4 root root 4.0K Jul 26 23:10 etc
42drwxr-xr-x. 2 root root 4.0K Jul 19 01:20 home
43...
44
45% ls -lh /mnt/bottlerocket/bin/login
46-rwxr-xr-x. 1 root root 30 Apr 11 18:08 /mnt/bottlerocket/bin/login
Once you're done examining the files, unmount and detach the loop device:
1sudo umount -R /mnt/bottlerocket
2sudo losetup -d /dev/loop0