I recently dealt with a server livelock issue caused by memory page thrashing. This post refreshes the Linux memory basics I found useful for debugging the issue. Much of the content is from Chapter 7 of Systems Performance: Enterprise and the Cloud. Virtual Memory Virtual memory is an abstraction that provides each …

Read More

In recent testing, an interesting scenario appeared when launching EC2 instances with multiple ENIs: the primary ENI (device index 0) does not serve default network traffic. This occurs in approximately 1 out of 10,000 launches. In our use case, we must ensure that default traffic routes through the primary ENI. This …

Read More

Security-Enhanced Linux (SELinux) is a mandatory access control (MAC) system that enhances Linux security. "Mandatory" means access control is strictly enforced by predefined policy rules—users and processes cannot modify these rules at will, ensuring security is not left to individual discretion. SELinux is …

Read More

Go is known for its backward compatibility, simplicity, and six-month release cycle. But that can sometimes lead to code that works but isn’t as readable and modern as it could be. This post is a living document where I note modern Go idioms I’ve applied in production codebases to improve clarity and maintainability. …

Read More

Shell scripts are infamous for security issues and surprising behavior. So when possible, it's better to avoid using shell. For instance, we built a container platform using the Bottlerocket OS and we didn't even install shell. If someone needs to run shell, the shell must be run inside a container. That said, shell is …

Read More

I recently worked on getting amazon-ssm-agent to run inside containers on Bottlerocket. During that process, I ran into a TLS issue connecting to amazonaws.com. The root cause turned out be interesting and we'll walk through it in this post. Running amazon-ssm-agent in a container: why and how? To enable sessions …

Read More

A recent faulty release disrupted service for some customers. The root cause was a concurrency bug involving x/sync/errgroup and context cancellation. This post shares three practices we learned from the incident. These practices will help us catch similar issues during code review or alert us to problems in …

Read More

We've been using the journald-to-cwl to ship journal logs from EC2 instances to Cloudwatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panic on expected errors in Go is …

Read More

This week, I needed to install the Amazon SSM Agent and was surprised to find that GPG (GNU Privacy Guard) was the only way to verify the download. I had assumed that software downloads verification had largely transitioned to PKI (Public Key Infrastructure). This short post is a refresh on GPG. OpenPGP is an open …

Read More

While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat …

Read More