Pre-load container images on the Bottlerocket read-only root filesystem

calendar Oct 25, 2026 · 13 min read · Bottlerocket container Linux  ·
Share on: copy

We run customer workloads in containers, and each container gets its own VM (an EC2 instance or a microVM). Every VM needs a few OCI images regardless of the workload: the language runtime image the customer's workload is built on, a log forwarding sidecar, an internal monitoring container. The obvious way to provide them is to pre-load: bake the images into the OS image so the container content store is ready at boot, with no image pull and no snapshot creation. On Bottlerocket this runs into a problem. The pre-loaded content store sits on the read-only, dm-verity protected root filesystem, but the content store containerd actually uses lives outside it.

Root volume, workload volume and image cache

For simplicity, assume the final OS image is an EC2 AMI. The AMI carries a root EBS volume, and the instance gets a separate workload EBS volume.

  • The root volume is /dev/xvda. It is created from the EBS snapshot in the AMI and holds Bottlerocket itself. It is partitioned into a boot partition, a root partition and a data partition. The root partition is the OS: mounted read-only at / and verified block by block with dm-verity, so a single flipped bit anywhere on it fails the boot. The data partition is the writable half of the same volume: Bottlerocket mounts it at /local and bind mounts /var, /opt and /mnt from there, so logs, host-container state, and by default containerd's own /var/lib/containerd all live on it. The image cache goes on the root partition, at /usr/share/image-cache.

  • The workload volume is /dev/xvdb. It is declared in the AMI's block device mapping. On first boot the host formats it, mounts it at /mnt/workload, and bind mounts /mnt/workload/containerd over /var/lib/containerd. Everything containerd writes at runtime goes here: pulled images, the writable layer of each running container, and so on.

  • The image cache, the pre-loaded images, is a containerd root directory captured at build time, after containerd imported the images we want on every host. It ships as a tarball inside an RPM and ends up on the root partition.

Why put the image cache on the read-only root filesystem?

Baking OCI images into an AMI is an old idea. What is specific here is where the images live and what that buys.

  • Startup latency. The images are a prerequisite for running anything, so a pull on a cold VM is on the critical path of every cold start. Pulling images takes tens of seconds, or minutes for large ones.

  • Bandwidth. One VM per workload and a new VM per cold start means the same bytes are pulled once per VM, forever. At fleet scale the registry egress for a handful of images that never change could dwarf the egress for the customer workloads' own images.

  • Private images. Most of these images are built in-house and never pushed to a registry. There is no registry to pull them from at boot, no credential to distribute to every host, no registry availability to depend on, and no way for a workload on the host to pull a version we did not ship. A registry could be built for them, but the AMI is already a distribution channel, and the dm-verity hash tree covers the images along with the rest of the OS.

  • Customer-managed encryption keys. The volume containerd writes to is where the workload's data ends up: pulled images, container writable layers, anything written to a volume mount under it. Some customers require that volume to be encrypted with a KMS key they control. An EBS volume created from a snapshot is hydrated from that snapshot. If the snapshot is encrypted with our key, EBS has to decrypt with our key and re-encrypt with the customer's, which needs cross-account grants on our key and does not scale the way blank volume creation does. An unencrypted snapshot goes through encryption-by-default at creation, with the same scaling problem. A blank volume has none of this: it is encrypted with whatever key the launch request names, at full creation speed, with no key of ours involved. So the workload volume has to be empty when it is first attached to the instance.

  • EROFS compression. Bottlerocket can build the root partition as compressed EROFS, and this variant does (the erofs-root-partition image feature). The image cache inherits that compression for free: it takes about half the space it would on the ext4 workload volume, and it also uses the page cache more efficiently.

  • One EBS snapshot in the AMI instead of two. An AMI is a registration record: a name, an architecture, boot settings, and a block device mapping. Each entry in the mapping either names an EBS snapshot, in which case EBS creates a volume from it at launch, or names no snapshot, in which case EBS creates a blank volume. The straightforward way to pre-pull images is to attach a volume to a build instance, point containerd at it, pull, and snapshot it into the AMI as a second mapping. That second snapshot costs more than it looks:

    1. Launch time. A volume created from a snapshot is lazily loaded from S3; the first read of each block is slow until the whole volume has been touched. A blank volume has nothing to load.
    2. Distribution. EBS snapshots are regional. Every AMI copy to another region or share with another account copies and shares every snapshot in the mapping, and sharing an encrypted snapshot means granting the recipient access to its key. One snapshot halves that work.
    3. Build pipeline. Producing the second snapshot means launching a build instance, attaching a volume, running containerd on it, snapshotting, and waiting. Producing the image cache means running containerd in a container for 30 seconds and tarring a directory; the RPM build and the existing root image pipeline do the rest.
    4. Volume sizing. A volume created from a snapshot cannot be smaller than the snapshot. A blank volume can be any size, so the workload volume can be sized per launch, down to 1 GiB for workloads that need little scratch space.
    5. Storage cost. Every AMI version keeps its snapshots until deregistered. Two snapshots per version, times every version still in use, times every region.

What containerd keeps on disk

containerd's root directory, /var/lib/containerd by default, has one subdirectory per plugin. Four of them matter for images:

1io.containerd.metadata.v1.bolt/meta.db                  image names, references, leases
2io.containerd.content.v1.content/blobs/sha256/<digest>  manifests, configs, compressed layers
3io.containerd.snapshotter.v1.overlayfs/metadata.db      the snapshot tree
4io.containerd.snapshotter.v1.overlayfs/snapshots/<id>/fs unpacked layers

The two databases are small and opened read-write. The blobs and snapshots are large and, once written, never modified. So I copy the databases onto the workload volume, and replace every blob and every snapshot with a symlink back to the root volume. containerd opens the symlinks like any other path and never notices.

1/var/lib/containerd  (workload volume, empty at launch)
2  io.containerd.metadata.v1.bolt/meta.db            copy
3  io.containerd.content.v1.content/blobs/sha256/
4      <digest>  -> /var/lib/image-cache/io.containerd.content.v1.content/blobs/sha256/<digest>    symlink
5  io.containerd.snapshotter.v1.overlayfs/
6      metadata.db                                   copy
7      snapshots/1 -> /var/lib/image-cache/io.containerd.snapshotter.v1.overlayfs/snapshots/1      symlink
8      snapshots/2 -> ...

The content blobs have to be linked too, not just the snapshots. containerd's garbage collector walks from image references through content to snapshots. If the content store is missing the manifests, containerd considers the snapshots unreferenced and deletes them.

Build the AMI

Everything here is in bottlerocket-image-cache, a small out-of-tree Bottlerocket variant you can build and boot yourself.

Step 1: build the cache

The cache is a containerd root directory, captured after containerd pulled what we want. For the demo, two "internal" images stand in for the images you would pre-load. The first is a Python runtime with the HTTP client and a bootstrap script pre-installed, python-runtime/Dockerfile:

1FROM public.ecr.aws/docker/library/python:3.12-alpine
2RUN pip install --no-cache-dir "requests==2.32.3"
3COPY bootstrap.py /opt/runtime/bootstrap.py
4ENV PYTHONPATH=/opt/runtime
5ENTRYPOINT ["python", "/opt/runtime/bootstrap.py"]

The second is a log router sidecar, log-router/Dockerfile. Neither is pushed to a registry; they are tagged registry.example.internal/python-runtime:3.12 and registry.example.internal/log-router:1.0, and that host does not exist.

build-image-cache.sh builds both with Docker, saves them as archives, and then starts a privileged Amazon Linux 2023 container to do the containerd part. Running containerd in a sandbox rather than on the build host matters for two reasons: the containerd version has to match the one in the Bottlerocket variant, and the host's own containerd state must not leak into the cache. populate-containerd.sh runs inside the sandbox:

 1containerd --root /work/containerd --state /run/containerd &
 2
 3for archive in /work/archives/*.tar; do
 4  ctr image import "${archive}"
 5done
 6for image in ${PUBLIC_IMAGES}; do
 7  ctr image pull "${image}"
 8done
 9
10# containerd keeps compressed layer blobs so it can push later. The host only
11# runs images, from the unpacked snapshots, so the compressed copies are dead
12# weight. Only blobs with the "uncompressed" label are layers; manifests and
13# configs stay.
14ctr content ls \
15  | awk '/containerd\.io\/uncompressed=sha256/ { print $1 }' \
16  | xargs -r ctr content rm
17
18sleep 3   # stop containerd too soon and the deletes do not reliably land
19kill "${containerd_pid}"
20
21find "${root}" -type c -delete
22rm -rf "${root}"/io.containerd.snapshotter.v1.{native,blockfile,btrfs,erofs} \
23       "${root}"/io.containerd.runtime.v2.task \
24       "${root}"/io.containerd.grpc.v1.introspection \
25       "${root}"/io.containerd.sandbox.controller.v1.shim \
26       "${root}"/tmpmounts
27
28tar -C "${root}" -cf /work/image-cache.tar .

Step 2: put the cache on the root volume

image-cache.spec is the RPM that installs the tarball at /usr/share/image-cache. Three lines in it are worth mentioning:

1%global __os_install_post /usr/bin/true
2%undefine cross_check_fips
3%global __spec_clean_pre exit 0

rpmbuild runs post-install scriptlets that strip binaries, extract debuginfo, and in Bottlerocket's case also verify FIPS compliance of every ELF file. The cache contains binaries from Alpine and Debian image layers that fail all of those, and we want the bytes exactly as containerd wrote them. So the checks are turned off, and the tarball is extracted with --no-selinux --no-xattrs because labels from the sandbox mean nothing on the host. Then:

1chmod +rwX -R %{buildroot}%{_cross_datadir}/image-cache

Image layers contain read-only files and directories with odd modes. rpmbuild has to read every file to build the package, and Twoliter passes --clean, which has to delete the build root afterwards. Both fail without this. This is also why the cache is its own package rather than a subpackage of something else: there is no clean way to disable the post-install checks for one subpackage.

Step 3: relabel the cache with an overlay

Bottlerocket runs SELinux in enforcing mode, and every file on the root partition is labeled os_t. That label means "the OS may execute this", and the policy deliberately prevents the container runtime from manipulating os_t files. containerd's own data under /var/lib is labeled local_t. My first attempt added a policy rule letting the runtime associate with os_t files. It worked, but it punched a hole in exactly the boundary the labeling exists to enforce. Relabeling is the fix. image-cache.mount is a systemd mount unit that overlay-mounts the cache onto itself with a context= option:

1[Mount]
2What=overlay
3Where=/var/lib/image-cache
4Type=overlay
5Options=lowerdir=/usr/share/image-cache:/var/lib/image-cache,context=system_u:object_r:local_t:s0

Everything seen through /var/lib/image-cache is now local_t. The files are still read-only and still dm-verity protected; only the view changed. Because the cache is only ever read, the overlay needs no upperdir or workdir, which is why lowerdir can list the mount point itself as a second layer: systemd creates the directory and the mount lands on top of it.

The unit is a .mount unit rather than a mount command in a .service, so systemd knows about the mount point and other units can order on it with RequiresMountsFor=. systemd requires the unit file to be named after the mount point with the path escaped, so the file ends up as var-lib-image\x2dcache.mount; the spec computes that with systemd-escape rather than hard-coding the escape sequence.

Step 4: seed the empty volume at boot

Five systemd units take the instance from "blank workload volume" to "containerd running with the images":

1check-workload-volume.service    blkid /dev/xvdb; if it has a filesystem, drop a sentinel
2format-workload-volume.service   mkfs.ext4 -L WORKLOAD          (skipped if sentinel)
3mnt-workload.mount               LABEL=WORKLOAD -> /mnt/workload
4prepare-workload-volume.service  image-cache-link               (skipped if sentinel)
5var-lib-containerd.mount         /mnt/workload/containerd --rbind-> /var/lib/containerd

The sentinel exists because of reboots. Without it, a reboot wiped /mnt/workload back to a fresh filesystem, and any images containerd had pulled since launch, along with any running container's writable layer, were gone. check-workload-volume.service asks blkid whether the device already has a filesystem and, if so, touches /run/workload-volume-has-filesystem; the format and prepare units carry ConditionPathExists=!/run/workload-volume-has-filesystem. The check asks the device because only the device knows whether it has been formatted.

The units never mention a kernel device name. On Nitro instances the kernel calls the second volume nvme1n1, on older ones xvdb. Bottlerocket ships udev rules that read the EBS device name from the NVMe controller and create a stable /dev/disk/by-ebs-id/xvdb symlink either way, and systemd exposes it as the device unit dev-disk-by\x2debs\x2did-xvdb.device that the units depend on. The AMI declares the volume in amispec.toml, with no snapshot-id, so EBS creates it blank:

1["/dev/xvdb".ebs]
2volume-type = "gp3"
3volume-size = 20
4delete-on-termination = true

prepare-workload-volume.service does the actual seeding:

1[Unit]
2RequiresMountsFor=/mnt/workload /var/lib/image-cache
3ConditionPathExists=!/run/workload-volume-has-filesystem
4
5[Service]
6Type=oneshot
7RemainAfterExit=true
8ExecStart=/usr/bin/mkdir -p /mnt/workload/containerd
9ExecStart=/usr/bin/image-cache-link --root /mnt/workload/containerd --cache /var/lib/image-cache

RequiresMountsFor= on both paths is what ties this to the overlay mount from step 3 and the ext4 mount of the workload volume. var-lib-containerd.mount then rbind-mounts /mnt/workload/containerd over /var/lib/containerd, ordered after the core kit's prepare-var-lib-containerd.service, which creates the mount point. containerd.service is ordered After=configured.target, well after local-fs.target, so by the time it starts, its root is the seeded directory.

image-cache-link/main.go is about 170 lines of Go, most of it error handling. Stripped of the error returns, the core is:

1func populate(root, cache string) error {
2    // databases: copy
3    copyDir(cache+"/io.containerd.metadata.v1.bolt", root+"/io.containerd.metadata.v1.bolt")
4    copyFile(cache+"/io.containerd.snapshotter.v1.overlayfs/metadata.db", root+"/io.containerd.snapshotter.v1.overlayfs/metadata.db")
5    // blobs and snapshots: one symlink per entry
6    symlinkEntries(cache+"/io.containerd.content.v1.content/blobs/sha256", root+"/io.containerd.content.v1.content/blobs/sha256")
7    symlinkEntries(cache+"/io.containerd.snapshotter.v1.overlayfs/snapshots", root+"/io.containerd.snapshotter.v1.overlayfs/snapshots")
8}

Try it out

The whole mechanism runs without an AMI. test-locally.sh unpacks the tarball into a read-only bind mount standing in for the root volume, runs image-cache-link against an empty directory standing in for the workload volume, starts containerd there, and runs all three images:

 1$ image-cache/test-locally.sh
 2...
 3+ /work/image-cache-link --root /work/containerd --cache /work/cache
 4+ ctr image ls
 5REF                                           TYPE                                       SIZE
 6public.ecr.aws/docker/library/alpine:3.20     application/vnd.oci.image.index.v1+json    3.5 MiB
 7registry.example.internal/log-router:1.0      application/vnd.oci.image.manifest.v1+json 7.7 MiB
 8registry.example.internal/python-runtime:3.12 application/vnd.oci.image.manifest.v1+json 66.0 MiB
 9+ ctr run --rm registry.example.internal/python-runtime:3.12 demo-runtime python /opt/runtime/bootstrap.py serve --port 8080
10python-runtime: python 3.12.15 on x86_64
11python-runtime: requests 2.32.3 preinstalled
12python-runtime: would exec workload: serve --port 8080
13+ ctr run --rm registry.example.internal/log-router:1.0 demo-router
14log-router: forwarding /var/log/workload
15log-router: /var/log/workload does not exist, nothing to forward
16+ ctr run --rm public.ecr.aws/docker/library/alpine:3.20 demo-alpine cat /etc/alpine-release
173.20.10

Nothing was pulled. The cache is mounted read-only, so if containerd had tried to write into a snapshot the run would have failed.

The real thing is three commands and a launch:

1cargo make build-image-cache   # Docker; writes packages/image-cache/image-cache.tar
2cargo make build-variant       # Twoliter; writes build/images/x86_64-aws-image-cache/latest/
3cargo make ami                 # registers the AMI with the blank /dev/xvdb mapping

Launch the AMI with an SSM instance profile and user data that enables the admin container:

1[settings.host-containers.admin]
2enabled = true

Then aws ssm start-session, enter-admin-container, sudo sheltie, and you are root on the host:

 1# findmnt -o TARGET,SOURCE,FSTYPE /var/lib/image-cache /mnt/workload /var/lib/containerd
 2TARGET               SOURCE                     FSTYPE
 3/var/lib/image-cache overlay                    overlay
 4/mnt/workload        /dev/nvme1n1               ext4
 5/var/lib/containerd  /dev/nvme1n1[/containerd]  ext4
 6
 7# ls -l /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots | head -3
 8lrwxrwxrwx. 1 root root 59 ... 1 -> /var/lib/image-cache/io.containerd.snapshotter.v1.overlayfs/snapshots/1
 9lrwxrwxrwx. 1 root root 59 ... 2 -> /var/lib/image-cache/io.containerd.snapshotter.v1.overlayfs/snapshots/2
10
11# ctr run --rm registry.example.internal/python-runtime:3.12 demo python /opt/runtime/bootstrap.py
12python-runtime: python 3.12.15 on x86_64
13python-runtime: requests 2.32.3 preinstalled

The instance cannot resolve registry.example.internal. The image came off the root volume, through a relabeling overlay, through a symlink on an ext4 volume that was blank thirty seconds earlier. Reboot it and check-workload-volume.service finds the filesystem, the format and the seeding are skipped, and containerd picks up where it left off.

Conclusion

Nothing here is specific to Bottlerocket except the SELinux labels and the OS image build system. The split between containerd's two small read-write databases and its large immutable blobs and snapshots makes a cross-volume image store possible on any Linux host. The lessons that cost the most time were the small ones: link the content blobs too or GC eats the snapshots; relabel with an overlay instead of loosening policy; use a .mount unit so RequiresMountsFor= works; and ask the block device, not a flag, whether this is a first boot.

I'm currently open to engineer opportunities. If you liked this post and would like to have me on your team, reach out on X, LinkedIn or GitHub.