I recently shipped a bug that caused around 0.5% of container workloads to fail to start for one internal customer. It is an interesting mix of Linux namespaces, Go concurrency, and making syscalls in Go. This post walks through the bug and its fix. The need to run a program in its own network and mount namespace …

Read More

Go is known for its backward compatibility, simplicity, and six-month release cycle. But that can sometimes lead to code that works but isn’t as readable and modern as it could be. This post is a living document where I note modern Go idioms I’ve applied in production codebases to improve clarity and maintainability. …

Read More

A recent faulty release disrupted service for some customers. The root cause was a concurrency bug involving x/sync/errgroup and context cancellation. This post shares three practices we learned from the incident. These practices will help us catch similar issues during code review or alert us to problems in …

Read More

We've been using the journald-to-cwl to ship journal logs from EC2 instances to Cloudwatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panic on expected errors in Go is …

Read More

While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat …

Read More

It's a beautiful day, and it started with a simple code review: 1# tools/foo/main.go 2- fmt.Println("found it") 3+ log.Println("found it") The author explained the advantages of using a logging library over plain printf. The rationale was straightforward, so I approved the change without hesitation. …

Read More

Our Go programs recently triggered an alarm due to excessive panics. Panic is a Go runtime mechanism that halts execution. It got me thinking about different ways a Go program can die. I don't expect many - not like A Million Ways to Die in the West. In this post, we'll go through the various ways Go programs die. …

Read More

Modern CPUs operate significantly faster than memories. A 4.5 GHz x64 CPU operates 30 times faster than a 6000 MHz DDR5 memory of CAS Latency 36. Adding latencies incurred by the bus and memory coherency protocols, memory could be 100 times slower than registers. To mitigate the speed gap, CPU uses layers of caches, …

Read More

Given a non-decreasing array and a target value, we can find the target in logarithmic time using binary search. My first programming language is C++ and the C++ Standard Template Library (STL) provides two functions for this task. iterator lower_bound(first, last, value) returns the smallest index with a value larger …

Read More

Problem An event consists of multiple properties, each defined as a key-value pair, where the key is a string and the value is of a primitive type such as numbers or strings. Importantly, each event must include a mandatory 'Name' property. Given a list of events, the task is to count the number of events based on …

Read More