We've been using journald-to-cwl to ship journal logs from EC2 instances to CloudWatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panicking on expected errors in Go is …

Read More

This week, I needed to install the Amazon SSM Agent and was surprised to find that GPG (GNU Privacy Guard) was the only way to verify the download. I had assumed that software downloads verification had largely transitioned to PKI (Public Key Infrastructure). This short post is a refresh on GPG. OpenPGP is an open …

Read More

While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat …

Read More

It's a beautiful day, and it started with a simple code review: 1# tools/foo/main.go 2- fmt.Println("found it") 3+ log.Println("found it") The author explained the advantages of using a logging library over plain printf. The rationale was straightforward, so I approved the change without hesitation. …

Read More

We started migrating from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023) a month ago. While testing workloads on AL2023 in the pre-production environment, I noticed slightly higher disk usage compared to the same workload on AL2. In this post, I'll share my investigation. AL2023 Has Less Free Disk Space with ext4, …

Read More

Our Go programs recently triggered an alarm due to excessive panics. Panic is a Go runtime mechanism that halts execution. It got me thinking about different ways a Go program can die. I don't expect many - not like A Million Ways to Die in the West. In this post, we'll go through the various ways Go programs die. …

Read More

As Amazon Linux 2 (AL2) approaches its End of Life on June 30, 2025, we have started migrating our container platform from AL2 to Bottlerocket. The migration encountered a few speed bumps. In this post, we'll examine one of them: missing container disk I/O stats. Why are container I/O dashboards blank? Since …

Read More

I've been building a service for a month, and the day finally arrived when I had the artifact - an EC2 AMI. The AMI passed my "rigorous" manual tests so I launched 100 EC2 instances. Surprise! Around 28 instances failed to launch. What's going on? All failed instances were stuck in the …

Read More

Given an array of numbers A, find out whether it contains a 1-3-2 pattern. A 1-3-2 pattern is a subsequence of three numbers, A[i], A[j] and A[k] such that i < j < k and A[i] < A[k] < A[j]. For clarity, let's call the 1-3-2 pattern the Bronze-Gold-Silver pattern. If A[j] is Gold, then we should consider the …

Read More

Tony Hoare invented QuickSort in 1961. At the time of its publication, the best comparison-based sorting algorithm was merge sort. Merge sort divides an unordered array into two equally sized subarrays, sorts each subarray, and then merge the two subarrays to produce a sorted array. Merge sort is simple to understand. …

Read More