Page-cache thrashing on EC2: kill the tenant before it takes the host down

For a long time, some of our container hosts, most often the smallest ones, would go dark every so often. A t3.micro with 1 GiB of memory, running a customer's containers, would stop sending logs and metrics all at once and never start again. The orchestrator still showed every container as RUNNING. The only signs of life were in CloudWatch: a CPU spike, the EBS volumes reading as fast as they could with almost no writes, and the instance's EBS byte balance draining from 100% to zero in about half an hour.

Nothing crashed and nothing was OOM-killed, and with the telemetry gone there was nothing from inside the box to look at. The workaround was to move affected containers to bigger instances. Customers aren't happy about it for a few reasons. First, a big chunk of their containers are standalone (not in a service), so relocating containers do not apply. Second, for containers in a service, this requires them to add extra healthchecks to handle stuck instances because of platform issue. And of course, customers lose trust in a platform having unexplained behaviors.

It turns out the instances were thrashing their page cache: re-reading their own file cache, e.g., program code, libraries, from disk in a loop. That can happen on any Linux machine, but on an EC2 instance backed by EBS it turns into a worse outage than bare metals, for reasons that come down to how cloud block storage is provisioned. The outcome the platform wants instead is for the tenant causing the thrash to be killed. An OOM kill is fast, logged and visible to the customer, who can then pick a bigger memory size. A thrashing host gives nobody anything to act on. The second half of this post takes a t3.micro and tries multiple mechanisms to protect the host from page thrashing.

A clue from a milder case

The break came from an instance that only lost half of its byte balance and kept logging. Its kernel log showed the OOM killer firing a few times inside a container's cgroup, and a memory report with inactive_anon at 787 MiB out of 928 MiB.

Linux sorts memory pages into two kinds. Anonymous pages hold heaps, stacks and everything malloc returns: the JVM heap, the Go heap, Python objects. Nothing on disk backs them, so the only way to free one is to write it to swap, and without swap an anonymous page stays in RAM until its process exits. File-backed pages are the page cache: the contents of files that are mapped or were recently read. That includes every running program, libc, libjvm.so and containers' own binaries and filesystem. A clean file page can be dropped at any moment, because it can always be read back from disk. Each kind lives on an active and an inactive LRU list. When memory runs short, kswapd (or the allocating task itself, in direct reclaim) walks the inactive lists and evicts what it can.

With 787 MiB of anonymous memory on a 928 MiB machine and no swap, the kernel had about 140 MiB left for all program code and all file caching on the box. Reclaim could only take file pages. It took the inactive ones first, and then it started evicting active file pages, which is code that is executing right now. When a running program's code page is evicted, the program touches it again almost immediately and takes a major fault, a synchronous read from disk. The page comes back, pushes out another hot page, that one faults, and so on.

In the kernel's accounting the machine is never out of memory, because it can always find something clean to evict. The OOM killer never fires. The machine spends its I/O bandwidth and a good share of its CPU re-reading its own code, and the log shipper and metrics agent are just more processes whose code keeps getting evicted. That one mechanism accounts for the read storm without writes, the CPU spike, the containers still RUNNING and the telemetry going dark.

Two reproductions on a 1 GiB instance backed this up. A task of four idle Spring Boot servers inside a 512 MiB container limit drained the byte balance in about half an hour, saturated both volumes with reads, and lost its telemetry in under three minutes, with every container still RUNNING. A pure heap hog (stress --vm) that used almost no file-backed memory got OOM-killed inside its cgroup instead, and only the root volume thrashed. That is where the OS's code lives, and the hog's container volume had no page cache worth cycling.

The usual sysctl folklore made no difference: vm.swappiness at 0 and at 100, vm.vfs_cache_pressure, vm.compaction_proactiveness=0. Moving the volumes from gp2 to gp3 changed nothing either. Two things did help. Adding 1 GiB of swap stopped the thrashing from starting at all, while 256 MiB only delayed it. Freeing real memory, by disabling a non-essential daemon or a memory-hungry logging sidecar, usually turned the thrash into a clean OOM kill of one container: with a little headroom the kernel does reach "nothing left to reclaim".

Transparent Huge Pages turned out to matter too. Go 1.21 and later use THP for the heap by default, and the Go team recommends GODEBUG=disablethp=1 on machines with less than 1 GiB. Several long-running daemons on these hosts were Go binaries built with a recent toolchain, and booting with transparent_hugepage=never measurably reduced their resident memory. The JVM leaves THP off by default for the same reason.

Why EC2 makes it worse

Thrashing is old news. Anyone who ran Linux on a spinning disk remembers the death spiral. On a modern bare-metal or on-premises server with local NVMe, the same memory squeeze makes the machine slow and gives more chances for other tools to rescue the server from thrashing. On EC2 instance with EBS it essentially turns the disk into bricks once the EBS byte balance drops to 0%.

Every refault crosses the network

The cost of thrashing is roughly the refault rate times the cost of one read. On bare metal that read goes to a local NVMe drive. On EC2 it goes over the network to EBS, a replicated block store that AWS documents as delivering single-digit millisecond latency on gp3, in practice about 0.5 to 1 ms for a 4 KiB read. Then come the throughput and IOPS ceilings:

These are back-of-envelope figures from published numbers. 64 MiB is roughly the hot code of a few dozen daemons and runtimes. Per fault, EBS is about ten times slower than local flash, and its throughput ceiling is one to two orders of magnitude lower.

The disk runs on credits

Bigger instances drain their EBS byte balance too; the examples here use t3.micro because it is the most susceptible. Any instance type whose EBS bandwidth is listed as "up to" a figure has a credit bucket, and that covers most sizes up to 2xlarge or 4xlarge depending on the family. An m6i.large with 8 GiB has an 81 MB/s baseline, below what a single default gp3 volume can deliver, so a thrash at the volume's cap drains it the same way. An m6i.4xlarge with 64 GiB still bursts, from 625 to 1,250 MB/s. General-purpose EBS volumes have limits of their own:

The instance figures come from aws ec2 describe-instance-types. A bare-metal disk delivers the same throughput in hour two as in minute one. A thrashing t3.micro reads at whatever the tightest cap allows, which with a 30 GiB gp3 root volume is the volume's 125 MiB/s, more than ten times the instance's baseline. When the byte bucket empties, the instance drops to 10.9 MB/s, less than the sequential throughput of a 2005 laptop hard drive. A system that needed 125 MiB/s just to limp along now gets 9% of it.

Three things about the bucket make this nasty. The failure is delayed and then sudden: the instance coasts on burst credits for tens of minutes, long enough to pass health checks, and then falls off a cliff. The bucket refills at the baseline rate, so after the thrash stops the instance stays I/O-starved for hours. And there is one bucket per instance, shared by all its volumes. In the original incidents the root volume, holding OS code, and the data volume, holding container images, thrashed together and drew on the same instance budget. That is why gp3 didn't help.

No swap, and no console

Without swap, anonymous memory is pinned and the page cache is the only thing left to reclaim. As the experiments below show, it also leaves the kernel's own thrash detection half-blind. When a bare-metal server livelocks, someone still has the BMC, a serial console, or a person who can walk over to it. On EC2 the only view that doesn't depend on the guest is the platform metrics, so CloudWatch effectively is the BMC. Everything else on the box competes for the same evicted pages. During the experiments for this post, the first SSM Run Command used to drive the lab instance died with "ipc messaging received timeout signal", because the agent's worker was starved along with the workload.

The platform metrics also carry the clearest signal. A healthy server in steady state reads little from disk, because its working set is in the page cache. Sustained, heavy reads with near-zero writes and no batch job, backup or cold start behind them mean the machine is re-reading its own code.

Reproducing it on a t3.micro

The lab is a t3.micro (2 vCPU, 913 MiB usable) in us-west-2 running Amazon Linux 2023 with kernel 6.18, cgroup v2, no swap, a 30 GiB gp3 root volume and CloudWatch detailed monitoring. The production incidents involved JVMs and container runtimes, but the mechanism only needs two ingredients: a heap that pins memory, and code that has to be re-read whenever it is evicted. Two small C programs supply them.

anon-hog.c maps N MiB of anonymous memory, fills it, and touches one byte per page every 500 ms so it stays active. That is the heap. file-hot.c maps a file with PROT_READ|PROT_EXEC and keeps touching its pages, which stands in for program text and hot data. Its seq mode walks the file front to back with a 200 ms pause between passes. The skew 48 95 mode sends 95% of touches into the first 48 MiB, like the code paths a service actually runs, and 5% anywhere in the file. Each pass prints how long it took, so the slowdown is visible from the application's side.

An application is one of each, started together by app.sh in one systemd unit so they share a cgroup the way a container would:

1sudo systemd-run --unit=app1 --slice=thrash.slice ./app.sh 300 hot1.bin skew 48 95
2sudo systemd-run --unit=app2 --slice=thrash.slice ./app.sh 300 hot2.bin skew 48 95

Two applications with 300 MiB heaps and 128 MiB hot files: 600 MiB of pinned heap plus about 150 MiB for the OS leaves about 65 MiB of page cache for 256 MiB of hot files. In the control run with 100 MiB heaps, a pass over 128 MiB takes 2 ms and vmstat shows zero blocks read.

The baseline

With 300 MiB heaps the disk lights up within seconds:

1$ vmstat 5
2procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
3 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
4 0  2      0  62388      0  76012    0    0 128584     2 1607 2501  0  4 40 55  0

That is 128 MB/s of reads, the gp3 volume's throughput cap, and it keeps going. Each file-hot pass goes from 2 ms to 8 or 9 seconds. Here is sar -B during the sequential variant:

1pgpgin/s pgpgout/s  fault/s majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
2128104.0      2.4   1767.1      3.7   32164.4   61756.6      0.0   63154.2   98.66

This doesn't look like what most people expect from thrashing. kswapd is scanning about 62,000 pages a second and reclaiming almost all of them, a %vmeff of 98.7%. As far as the reclaimer can tell the system is healthy, with plenty of clean cache that is easy to drop. It has no way to know each page will be needed again 200 ms later. Major faults are only 3.7 a second, because sequential access lets readahead pull pages in before the process touches them, so the faults are counted as minor. A detector watching majflt would miss this entirely, while pgpgin, or VolumeReadBytes outside the box, shows it plainly. The skewed variant ran at about 1,400 major faults a second. Memory pressure stall information stayed near zero for the sequential variant and plateaued around 30 to 35% for the skewed one, while I/O pressure sat at 65 to 85%.

The CloudWatch side, from one uninterrupted sequential run left going until the bucket was empty (the instance started at 61% from earlier tests):

Reads run at roughly 10,000 times writes for an hour on a machine doing no useful work. The byte balance falls in a straight line, about 1.4% a minute. The IOPS bucket barely moves because readahead turns the faults into large sequential reads; a random-access variant did the opposite, about 3,400 read IOPS at only 14 MB/s, and drained the IOPS bucket instead. CPU sits at 6% because two I/O-bound processes on two vCPUs leave nothing to burn. In production, with dozens of processes in the cycle, CPU reached 80%.

At 10:30 the bucket hits zero and EC2 throttles the instance to its 87 Mbps baseline. Reads drop from 114 MB/s to 10.9 MB/s, the application goes from 2.3 s to 27 s per pass, and the rest of the host goes dark the way production hosts did:

Every process on the box now shares 10.9 MB/s of page-ins to get its own code back. A metrics agent that has to page in a few MiB of Go runtime to send a heartbeat misses its deadline. The thrash has burned the credits that recovery would have needed.

While credits lasted, the lab host stayed usable, unlike the production instances, which went dark almost at once. The difference is process count. Two I/O-bound processes leave plenty of slack, while a real host has dozens (container runtime, log shipper, metrics agent, management agent, cron) whose code all lives in the same 65 MiB of surviving cache, and thrashing gets worse much faster than linearly as processes join the cycle. Once the credits run out, the lab host goes dark too.

Letting systemd-oomd kill it

The kernel OOM killer is a last resort for when no progress is possible, and a thrashing machine can always make some progress. A userspace OOM killer works from a different signal, pressure: how much of the time tasks are stalled waiting on memory. systemd-oomd ships with systemd 247 and later. Meta's oomd, which it is modeled on, has a richer rule engine, and earlyoom kills on low free memory without looking at pressure or cgroups.

Stock settings do nothing

Turning it on for the slice:

1sudo systemctl enable --now systemd-oomd
2sudo mkdir -p /etc/systemd/system/thrash.slice.d
3printf '[Slice]\nManagedOOMMemoryPressure=kill\n' | sudo tee /etc/systemd/system/thrash.slice.d/oomd.conf
4sudo systemctl daemon-reload

The first line in its journal is a warning:

1systemd-oomd[3385]: No swap; memory pressure usage will be degraded

The defaults are DefaultMemoryPressureLimit=60% for DefaultMemoryPressureDurationSec=30s. Over three minutes of the skewed workload, pressure climbed to about 33% and stayed there, collected by collect.sh:

1epoch       psi_some10 psi_full10 majflt/s pgpgin_kB/s free_MB anon_MB act_file inact_file
21790674567   17.62      17.30      1620     73320        74      651      33        78
31790674588   30.88      30.59      1397     62580        67      651      37        81
41790674630   32.20      31.86      1385     61424        68      651      38        80
51790674692   32.75      32.54      1387     61888        67      651      37        82
61790674734   33.12      32.84      1373     63378        64      651      36        85

Nothing was killed, and both apps limped along at 8.6 s per pass.

A 20% limit kills in 35 seconds

Lowering the bar to 20% for 20 s:

1printf '[OOM]\nDefaultMemoryPressureLimit=20%%\nDefaultMemoryPressureDurationSec=20s\n' \
2  | sudo tee /etc/systemd/oomd.conf.d/lab.conf
3sudo systemctl restart systemd-oomd

The same workload, started at 09:39:28:

109:40:03 systemd-oomd[5161]: Killed /thrash.slice/app1.service due to memory pressure
2         for /thrash.slice being 30.35% > 20.00% for > 20s with reclaim activity
309:40:03 systemd[1]: app1.service: systemd-oomd killed 2 process(es) in this unit.
409:40:03 systemd[1]: app1.service: Failed with result 'oom-kill'.

1epoch       psi_some10 psi_full10 majflt/s pgpgin_kB/s free_MB anon_MB act_file inact_file
21790674793   29.20      28.65      1282     57538        64      651      41        83
31790674798   30.64      30.34      1328     60786        65      651      39        83
41790674803   31.42      31.21      1397     63904       304      350      53       127   <- kill
51790674808   20.40      20.17       271      8087       267      351      56       162
61790674813   13.68      13.52         3       172       265      351      56       163
71790674818    7.51       7.42         0         0       265      351      56       163
81790674843    0.55       0.55         0         0       261      351      56       167

Ten seconds after the kill, disk reads are at zero. app2, which had been taking 8.4 s per pass, is back to 4 ms and keeps running, and its 128 MiB of file pages now fit in the cache, so the active file list grows instead of churning. The kill is logged with its reason, the unit ends up failed with Result: oom-kill.

Why memory pressure reads low without swap

A stall only counts as memory pressure if the kernel decides the refaulted page belonged to the working set. It estimates that from the refault distance, the number of other pages evicted between this page leaving and coming back. The rule in mm/workingset.c is roughly:

1workingset_size = ACTIVE_FILE
2if swap is available:
3    workingset_size += ACTIVE_ANON + INACTIVE_ANON
4recent = (refault_distance <= workingset_size)
5# booked as a memory stall only if recent
6# and the page was on the active list when it was evicted

The question being asked is whether the page would have stayed resident if all the memory the working set could compete for had been available to it. Without swap, anonymous memory isn't in that competition, so the working set shrinks to the active file list, a few dozen MiB during this thrash. A page that cycled out and back through 60 MB of other evictions fails the test and is classified as streaming, like a file too big to cache. The process still waited 8 seconds for its code, but that wait is booked as I/O pressure. This is what systemd-oomd's "memory pressure usage will be degraded" warning refers to. It describes the classic two-list LRU, which is what stock AL2023 runs: /sys/kernel/mm/lru_gen/enabled reads 0x0000.

With a 512 MiB swap file and systemd-oomd back on its stock 60% for 30 s:

1sudo fallocate -l 512M /swapfile && sudo chmod 600 /swapfile
2sudo mkswap /swapfile && sudo swapon /swapfile
3sudo rm /etc/systemd/oomd.conf.d/lab.conf && sudo systemctl restart systemd-oomd

1epoch       psi_some10 psi_full10 majflt/s pgpgin_kB/s free_MB anon_MB act_file inact_file
21790675284   37.75      37.00      2105     82720        64      607      75        92
31790675304   73.23      72.49      2733     97418        59      602      80        96
41790675325   76.75      75.75      2744     98794       289      306     110       129   <- kill
51790675345   10.61      10.47         0         0       277      311     111       134
61790675365    1.43       1.41         0         0       276      311     111       134

109:48:44 systemd-oomd[6707]: Killed /thrash.slice/app2.service due to memory pressure
2         for /thrash.slice being 73.53% > 60.00% for > 30s with reclaim activity

The same thrash now registers as 73 to 77% memory pressure, and stock systemd-oomd kills within 50 seconds. Only 42 MiB of swap was used. Swap fixed it by making the kernel see the problem, not by absorbing memory.

Containing the thrash

The goal is to turn a thrash into a kill of the tenant causing it. The thrash is a memory deficit, and two of the tested settings remove it:

• systemd-oomd removes a tenant by killing it. Without swap it needs a lower limit, 20% for 20 s in the run above, because memory pressure under-reports the thrash.
• zram adds memory by compressing the idle daemons' heaps, and it makes memory pressure honest, so stock systemd-oomd kills on its own.

The others don't kill anything. They decide who pays for the deficit until the kill happens:

• memory.low keeps the host's agents out of it.
• io.max keeps the instance's EBS credits out of it.
• MGLRU pushes it onto the tenants, and hides it from memory pressure.

In these runs, zram plus stock systemd-oomd killed a tenant 74 s in and the host recovered at once. io.max plus memory.low didn't end the thrash, but kept the host reachable and its credits intact while the tenant thrashed.

A container host, sandbox host or Kubernetes node has more to work with than a userspace killer, and the mechanisms interact, so each one got a run of its own on a fresh t3.micro in October 2026 (time of post writing). The fresh AMI idled at about 250 MiB used instead of 150, so the two tenant apps got 250 MiB heaps to stay in the thrash-but-don't-OOM zone. Two probes stood in for the host's own processes. The agent, in its own agent.slice, reads a 24 MiB file once every 10 s, like a log shipper or metrics agent waking up to send a batch; an idle wake-up takes about 2 ms. A cold python3 -c pass from system.slice every 5 s stands in for everything else on the box, at 27 ms when idle. Everything is cgroup v2. cgroup v1 can't even limit swap per cgroup properly, and it has been deprecated for years.

Each run lasted three minutes, and the numbers are medians after the first 30 s. Two baselines with identical configuration differed by about three times: in one batch each refault took about 4 ms, in the other about 1.3 ms, and the volume was nowhere near its IOPS or throughput cap either time. Each mitigation is compared with the baseline from its own batch. The driver is driver.sh, and per-run medians are in summary.txt.

The kernel OOM killer didn't fire in any of these runs, even with tasks stalled on I/O 70 to 90% of the time.

Protecting the agents with memory.low

memory.low tells the reclaimer to leave a cgroup's memory alone while anything unprotected is left to reclaim, and memory.min is the hard version. Both cover file pages as well as anonymous ones, so they protect exactly what thrashing evicts, the agent's code and data. With MemoryLow=64M on the agent, its wake-ups went from 116 ms of re-reads to 0 ms, because its file never left the cache. The tenants were no worse off; 24 MiB was a rounding error in their deficit.

1# /etc/systemd/system/agent.slice.d/protect.conf
2[Slice]
3MemoryLow=64M

Protection is hierarchical. A child's effective protection is capped by its parent's, so it has to be set on the slice as well as the unit. The memory_recursiveprot mount option, which systemd uses, lets a protected slice pass its protection down to children that don't set their own. Page cache is charged to the cgroup that first faulted it in, so libc pages that systemd touched first belong to init.scope and the agent's protection doesn't cover them. Statically linked agents, which most Go agents are, are easier to protect than dynamically linked ones.

The disk reads didn't go down at all. Protection moves the pressure onto someone else, so the instance keeps burning EBS credits while the agent stops being the victim. memory.min is for things you would rather lose a tenant to the OOM killer than lose.

Capping tenant reads with io.max

The shared credit bucket is the EC2-specific part of the problem, and io.max is the lever that reaches it. Reads triggered by page faults are charged to the cgroup of the task that faulted, so a tenant's thrash is throttled by its own limit:

1# /etc/systemd/system/tenants.slice.d/io.conf
2[Slice]
3IOReadBandwidthMax=/dev/nvme0n1 5M
4IOReadIOPSMax=/dev/nvme0n1 500

With 5 MB/s and 500 IOPS on the tenant slice, host reads fell from 68 MB/s to the 5 MB/s cap. That is under the t3.micro's 10.9 MB/s EBS baseline, so the credit bucket stops draining, where 68 MB/s drained it at six times the baseline rate. Cold python3 stayed at 30 ms, and even without memory.low the agent's wake-ups dropped from 504 ms to 2 ms, because the tenants could no longer churn the cache fast enough to evict it. The tenants stalled completely, with no pass finishing in three minutes. They were never going to make progress in that much memory, and now their stall is theirs alone and shows up in their own cgroup's io.pressure.

The limits are per device, by major:minor number, and NVMe device names on Nitro instances aren't stable across instance types, so resolve them at boot. Set both bandwidth and IOPS: in the first batch a 20 MB/s cap did nothing, because the thrash happened to run at about 20 MB/s anyway. And systemd doesn't reset io.max when you remove the setting. After the drop-in was deleted and systemd reloaded, systemctl show reported no limit while the kernel still had one:

1$ cat /sys/fs/cgroup/thrash.slice/io.max
2259:0 rbps=5000000 wbps=max riops=500 wiops=max

It silently throttled the next three runs, which had to be redone. The cgroup file is the only reliable place to check what is applied. The proportional controllers, io.weight with the io.cost model or io.latency for protecting a latency-sensitive cgroup, are the better tools on dedicated hardware. On a burstable instance what matters is absolute bytes and IOPS against the bucket, and io.max maps onto that directly.

memory.high, and why limits have to add up

MemoryHigh=330M on each tenant, enough for the 250 MiB heap plus some cache, made no difference. Neither tenant came close to 330 MiB, because global reclaim had already taken their page cache. Per-cgroup limits stop a cgroup from growing; they don't reserve anything for the host. When the sum of tenant limits plus host overhead exceeds RAM, as it does on a bin-packed host, the global reclaimer decides who loses, and it has no idea which pages belong to the telemetry agent. memory.high and memory.max are the right tools once the limits fit in RAM. A tenant over its limit then thrashes inside its own budget, and with io.max that cost stays with the tenant.

Every tenant in these runs had its own cgroup, and they still pulled tens of MB/s through the one instance-wide EBS budget the host's agents also depend on. On a burstable instance one tenant's thrash is a noisy-neighbor problem for every other tenant and for the platform, and the cost is paid in credits.

Compressed swap

A swap file on disk makes systemd-oomd work, but on EBS it spends the same credits you are trying to save. zram, a compressed block device in RAM, and zswap, a compressed cache in front of a swap device, keep swapped pages off the disk:

1modprobe zram
2echo lzo-rle > /sys/block/zram0/comp_algorithm   # AL2023's zram offers lzo-rle and lzo
3echo 512M > /sys/block/zram0/disksize
4mkswap /dev/zram0 && swapon -p 100 /dev/zram0

The tenants went from 29 s per pass to 194 ms. With incompressible random heaps in the second batch they went from 7.3 s to 142 ms. That looked too good until /sys/block/zram0/mm_stat showed what zram was holding: about 80 MB of data compressed roughly 4:1. Random heaps can't compress that well, so the pages in zram were the cold, compressible anonymous memory of the daemons on the box. That 80 MB was the margin the page cache needed.

zram didn't stop the disk reads, which stayed at 22 to 27 MB/s, and the agent got slower: 116 ms to 1.2 s in batch A, with a p90 of 3.6 s in batch B. With swap available, reclaim weighs anonymous and file pages by recency, and a process that wakes every 10 s is the coldest thing on the box. Swap should come with memory.low on the agents, and with memory.swap.max to scope it: MemorySwapMax=0 on tenants that must not swap, no limit on the host's daemons. zswap goes further, with memory.zswap.max to cap compressed usage per cgroup and memory.zswap.writeback set to 0 to keep a cgroup's pages from ever reaching the swap device. AL2023 ships zram-generator, but its default configuration skips hosts with more than 800 MB of RAM ("zram0: system has too much memory (913MB), limit is 800MB, ignoring" on the t3.micro), so zram has to be configured by hand.

With zram in place, stock systemd-oomd at 60% for 30 s killed one tenant 74 seconds into the run, at 69.6% memory pressure. Reads went to zero and the survivor ran at 3 ms per pass, the same result as the disk swap file but without spending credits.

systemd-oomd only reads memory pressure, and in every thrashing run I/O pressure was the higher and steadier signal. cgroup v2 lets a program register a PSI trigger on any cgroup's io.pressure or memory.pressure and poll() for it; the kernel's PSI documentation includes a short example program. A monitor built that way should watch each tenant cgroup's io.pressure, so it kills the cgroup that is stalling, and memory.oom.group set to 1 makes the whole container die together. On Kubernetes, kubelet evicts based on memory.available, which ignores pressure, so a node-level PSI-aware killer is still worth having.

MGLRU

The multi-generational LRU (CONFIG_LRU_GEN, kernel 6.1 and later) is compiled into AL2023 but disabled. Turning it on with echo y > /sys/kernel/mm/lru_gen/enabled changed the picture more than anything else. The host became healthy, with cold python3 at 27 to 30 ms and the agent's file never evicted, without any memory.low. MGLRU is much better at telling pages touched recently from pages touched once. The tenants paid for it, at 16 to 136 s per pass or no pass at all, and the disk reads kept going. In batch B they rose to 127 MB/s, the gp3 cap, with about 100 KB read per fault against about 50 KB under the classic LRU.

Memory pressure went to between 0 and 0.7% while I/O pressure sat at 70 to 90%, so with MGLRU these stalls don't register as memory stalls the way the workingset.c logic above books them. systemd-oomd would never fire. min_ttl_ms is documented to protect the working set of the last N milliseconds and to invoke the OOM killer when it can't. At 1,000 and at 10,000 it never fired. MGLRU is worth enabling for what it does for the host, but only together with io.max to protect the credits and a killer that watches I/O pressure.

Watching from outside the box

The guest is the last place to find out about any of this, and the platform metrics are in CloudWatch at no extra cost. EBSByteBalance% and EBSIOBalance% in the AWS/EC2 namespace sit at 99% forever on a healthy instance, and thrashing is one of the few things that drains them continuously. An alarm at under 50% for two consecutive periods catches it; these are five-minute metrics even with detailed monitoring. EBSReadBytes against EBSWriteBytes, also per instance in AWS/EC2 and summed over its volumes, catches the read/write asymmetry, and VolumeReadBytes against VolumeWriteBytes in AWS/EBS does the same per volume. A metric-math expression of this shape works:

1reads  = SUM(EBSReadBytes)  over 5 min
2writes = SUM(EBSWriteBytes) over 5 min
3ALARM if reads > 20 * MAX(writes, 1 MB) AND reads > 1 GB for 3 of 3 periods

The constants need tuning per workload. Batch jobs and cold starts trip it briefly, and thrashing trips it for as long as the credits last. On gp2 volumes, add BurstBalance. Inside the box, while an agent can still report, /proc/pressure/io and each cgroup's io.pressure are the pressure signals, and workingset_refault_file in a cgroup's memory.stat counts the cache being re-read directly. Readahead can't hide it the way it hides major faults. cw.sh pulls the CloudWatch side for a lab instance.

What I would deploy

On a cgroup v2 multi-tenant host, the first two items get to the kill, and the next two keep the host alive until it happens:

1. A pressure-based killer on the tenant slice. systemd-oomd with ManagedOOMMemoryPressure=kill works at stock limits once swap is on; without swap, lower the limit to about 20%. A host running MGLRU needs a PSI-trigger monitor on tenant io.pressure instead. Set memory.oom.group so the whole container dies together, and report the kill to the tenant as an OOM so the customer knows to pick a bigger memory size.
2. Compressed swap, either zram or zswap with writeback disabled, plus MemorySwapMax=0 on tenants that must not swap. It makes memory pressure honest so the killer fires, and it reclaims the idle daemons' cold heaps without spending EBS credits.
3. memory.low on the slice that holds the agents (telemetry, log shipping, SSH, the orchestrator agent), sized from their measured footprint and set on both the slice and the units. The host keeps reporting while a tenant thrashes, so the kill and its reason are visible.
4. io.max on the tenant slice, with both bandwidth and IOPS limits, sized so a fully thrashing tenant population can't outspend the instance's EBS baseline. The credit bucket survives until the killer acts.
5. A host memory budget. Measure the idle footprint of the image on the target instance type with every agent running, add the per-container overhead that lives outside tenant cgroups times the most containers you will place, keep a page-cache floor of 100 MiB or more for code that has to stay resident, and sell only what is left. On Kubernetes that is --system-reserved and --kube-reserved with --enforce-node-allocatable. With the budget right, a tenant that outgrows its memory hits its own memory.max before the host runs short. Reclaim then takes only that tenant's page cache, which can still thrash inside the cgroup but no longer evicts the host's code, and in the earlier experiments a little headroom usually turned the thrash into a cgroup OOM kill.
6. MGLRU, but only once items 1 and 4 are in place, with the killer watching io.pressure. By itself it kept the host healthy with no tuning, but it also hid the thrash from memory pressure, so systemd-oomd never fires, and in one batch the tenants read at the volume cap, draining credits faster than the classic LRU did.
7. CloudWatch alarms on EBSByteBalance% and EBSIOBalance% falling and on EBSReadBytes far above EBSWriteBytes, for the case where everything above fails. They keep working after the instance stops talking.
8. GODEBUG=disablethp=1 or transparent_hugepage=never for Go daemons on hosts under 1 GiB, and fewer distinct runtimes and per-container sidecars per host. Containers built on the same base layers share page cache for those files through overlayfs, but two images that each ship their own libjvm.so do not.

All eight aim at the same outcome: the tenant that caused the deficit dies quickly with a logged OOM, and the host and its EBS credits come through intact.

Lab files

The workers are anon-hog.c and file-hot.c, built with gcc -O2. app.sh runs one of each as a single application, collect.sh is the 5-second collector, and cw.sh dumps one-minute CloudWatch metrics, reading the instance and volume IDs from LAB_ID and LAB_VOL. Raw telemetry from the first set of runs ships in the bundle's data directory, starting with baseline.csv and deplete.csv. The cgroup v2 runs used driver.sh, which resets state between runs and applies each mitigation, run.sh, which starts the probes and tenants and collects results, and a collector extended with I/O pressure, file refaults, swap used and the python3 probe. Their output is under data/cgroup-levers, one directory per run, described in README.txt, with medians in summary.txt.

References

• Kernel docs: PSI, Control Group v2 (memory.low, memory.min, memory.high, memory.swap.max, memory.zswap.writeback, memory.oom.group, io.max), Multi-Gen LRU, zram, zswap
• mm/workingset.c in the Linux source, especially the comment block on refault distance
• systemd-oomd(8), oomd.conf(5), systemd.resource-control(5)
• AWS docs: CloudWatch metrics for EC2 instances, Amazon EBS volume metrics, General Purpose SSD volumes
• Kubernetes: Reserve compute resources for system daemons, Node-pressure eviction
• Go runtime: Transparent Huge Pages guidance
• Meta's oomd, earlyoom