A recent faulty release disrupted service for some of our customers. The root cause was a bug related to handling Go concurrency using x/sync/errgroup and context cancellation. In this post, I'll share three good practices in Go that we learned from the incident. These practices would either alert us to similar issues …

Read More

We've been using the journald-to-cwl to ship journal logs from EC2 instances to Cloudwatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panic on expected errors in Go is …

Read More

While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat …

Read More

It's a beautiful day, and it began with a short code review. 1# tools/foo/main.go 2- fmt.Println("found it") 3+ log.Println("found it) The author explained advantages of using a logging library over the plain old printf. The rationale was straightforward, so I approved the change without hesitation. …

Read More

Our Go programs recently triggered an alarm due to excessive panics. Panic is a Go runtime mechanism that halts execution. It got me thinking about different ways a Go program can die. I don't expect many - not like A Million Ways to Die in the West. In this post, we'll go through the various ways Go programs die. …

Read More

Modern CPUs operate significantly faster than memories. A 4.5 GHz x64 CPU operates 30 times faster than a 6000 MHz DDR5 memory of CAS Latency 36. Adding latencies incurred by the bus and memory coherency protocols, memory could be 100 times slower than registers. To mitigate the speed gap, CPU uses layers of caches, …

Read More

Given a non-decreasing array and a target value, we can find the target in logarithmic time using binary search. My first programming language is C++ and the C++ Standard Template Library (STL) provides two functions for this task. iterator lower_bound(first, last, value) returns the smallest index with a value larger …

Read More

Problem An event consists of multiple properties, each defined as a key-value pair, where the key is a string and the value is of a primitive type such as numbers or strings. Importantly, each event must include a mandatory 'Name' property. Given a list of events, the task is to count the number of events based on …

Read More

Every Go program has a runtime. The runtime implements garbage collection, concurrency, stack management, and other critical features. We can configure the runtime by setting variables. In this post, we will look at GOMAXPROCS, a variable that configures concurrency. You may get free performance boost by setting …

Read More

We plan to load test our product before public Beta, with two goals in mind. Find out bottlenecks: figure out road maps of performance improvement and prepare for oncalls. Understand how much workload we can support with fixed resources. This shapes the pricing strategy and determines number of Beta partners to …

Read More