AL2023 vs. AL2: less disk space with ext4

We started migrating from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023) a month ago. While testing workloads on AL2023 in the pre-production environment, I noticed slightly higher disk utilization compared to the same workload on AL2. In this post, I'll share my investigation.

AL2023 has less free disk space with ext4, compared to AL2

Although disk utilization metrics increased on AL2023, the "Used" space remained the same. The higher utilization is due to a decrease in the total ext4 filesystem size. The table below compares sizes of ext4 formatted EBS volumes.

For example, on a 128GiB EBS volume, the free space decreased by 0.7%, from 125.8GiB to 124.9 GiB. You can reproduce this behavior using the steps below.

 1# Pin AMIs for reproducing.
 2# AL2: ami-04907d7291cd8e06a, amzn2-ami-kernel-5.10-hvm-2.0.20241031.0-x86_64-gp2
 3# AL2023: ami-066a7fbea5161f451, al2023-ami-2023.6.20241031.0-kernel-6.1-x86_64
 4
 5INSTANCE_TYPE=t3.medium
 6aws ec2 run-instances \
 7    --image-id ${AMI_ID} \
 8    --instance-type ${INSTANCE_TYPE} \
 9    --subnet-id ${SUBNET_ID} \
10    --key-name example-1 \
11    --security-group-ids ${SECURITY_GROUP} \
12    --associate-public-ip-address 
13
14uname -r
155.10.227-219.884.amzn2.x86_64
16
17# Create a 128GiB EBS volume and attach it to the instance.
18aws ec2 create-volume --availability-zone us-west-2a --size 128 --volume-type gp3
19aws ec2 attach-volume --volume-id ${VOLUME_ID} --instance-id ${INSTANCE_ID} --device /dev/sdb
20
21# Confirm the block device size is 128GiB.
22sudo blockdev --getsize64 /dev/nvme1n1
23137438953472
24
25# Create ext4 fs and mount it. 
26sudo mkfs -t ext4 /dev/sdb
27sudo mkdir /mnt/data1 && sudo mount /dev/sdb /mnt/data1
28
29# Check total fs size: "1B-blocks"
30df --block-size=1 /mnt/data1
31Filesystem        1B-blocks  Used    Available Use% Mounted on
32/dev/nvme1n1   135084904448 24576 128196157440   1% /mnt/data1

Does this mean applications have less disk space?

It appears the filesystem uses more space for metadata on AL2023 than on AL2. Metadata accounting is tricky and filesystem behavior as free space is exhausted is complicated, so I wanted to confirm if the writable space for applications is genuinely reduced. Using the script below, I created 50 MiB files until no space was left.

 1#!/bin/sh
 2
 3# Directory to write files to
 4TARGET_DIR=$1
 5# Size of each file in MiB
 6FILE_SIZE=$2
 7
 8count=0
 9total_size_mb=0
10
11echo "Writing ${FILE_SIZE}MiB files to ${TARGET_DIR} until disk is full..."
12
13while true; do
14    filename="${TARGET_DIR}/file_${count}.dat"
15    if dd if=/dev/zero of="$filename" bs=1M count=$FILE_SIZE status=none 2>/dev/null; then
16        ((count++))
17        total_size_mb=$((count * FILE_SIZE))
18        echo -ne "\rFiles created: $count (Total: ${total_size_mb}MiB)"
19    else
20        echo -e "\n\nDisk full or error occurred!"
21        break
22    fi
23done

Here are the results for a 128 GiB EBS volume:

Since the last 50MiB file was written partially, "Used space" is larger than the "Total size of files". The difference in used space between AL2 and AL2023 is 896 MiB, matching the filesystem size difference. Thus, applications indeed have less writable disk space on AL2023 when using ext4.

Is the difference caused by mke2fs.conf?

The mkfs -t ext4 command uses mkfs.ext4 to create ext4 file systems, so the difference could stem from different configurations in AL2 and AL2023. Comparing the features using dumpe2fs:

 1# AL2
 2sudo dumpe2fs /dev/sdb | grep features
 3dumpe2fs 1.42.9 (28-Dec-2013)
 4Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
 5Journal features:         journal_64bit
 6
 7# AL2023
 8sudo dumpe2fs /dev/sdb | grep features
 9dumpe2fs 1.46.5 (30-Dec-2021)
10Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
11Journal features:         journal_64bit journal_checksum_v3

While the configurations differ, swapping mke2fs.conf between AL2023 and AL2 yielded the same free space difference. Therefore, mke2fs.conf is not the cause.

What about the mkfs version?

The mkfs utility is installed from the e2fsprogs package, which differ between AL2 (v1.42.9) and AL2023 (v1.46.5). To test its impact, I attempted to swap mkfs versions. However, the AL2 repository does not provide e2fsprogs-1.46.5, neither does the AL2023 repository provide e2fsprogs-1.42.9. So I built e2fsprogs-1.46.5 from source on AL2. Building e2fsprogs-1.42.9 on AL2023 failed due to a compile error, "undefined reference to makedev", likely caused by an incompatibility with libc on AL2023.

Using v1.46.5 to format an EBS volume on AL2 resulted in a free space of 134,145,380,352 bytes, identical to AL2023. Running the file-writing script also yielded 2,427 files, the same as AL2023. This confirms that the difference in free space is caused by the mkfs version.

The e2fsprogs package also installs tune2fs, which shows and modifies ext parameter. Comparing parameters between v1.42.9 and v1.46.5 revealed most were identical, except for an increase in "extra isize" from 28 to 32. It's unclear if this change affects free space and, if so, by how much.

1# sudo tune2fs -l /dev/sdb
2Inode count:              8388608
3Block count:              33554432
4Reserved block count:     1677721
5Inode size:	          256
6Required extra isize:     32  # For v1.42.9, this value is 28.
7Desired extra isize:      32  # For v1.42.9, this value is 28.
8Journal inode:            8

Closing thoughts

Catching this issue in pre-production was fortunate, and diving into the root cause was a good exercise. I lack expertise in ext4 to investigate further. For now, I will proceed with the AL2023 migration, keeping in mind the reduced free space when formatting EBS volume of certain sizes with ext4.