open-menu closeme
Engineering
github linkedin rss
  • Share one client per service

    calendar Oct 13, 2026 · 6 min read · Go Distributed System  ·
    Share on: twitter copy

    I recently dealt with an issue where a service failed occasionally during short throttling windows. The root cause was one I had seen many times before: a process holding multiple clients for the same service. This post walks through the issue, why the bug is rare in Java but common in Go, and how to avoid it. The bug …


    Read More
  • When sync.Cond is a better primitive than channels: lessons from Go's HTTP/2 client

    calendar Sep 17, 2026 · 14 min read · Go Concurrency  ·
    Share on: twitter copy

    I recently reviewed code that let a dynamic set of goroutines wait for updates from a single writer. Each waiter wanted to block until the writer had published something newer than what it had already seen. The implementation used channels and was hard to follow. A condition variable (sync.Cond) would have been less …


    Read More
  • How cgo silently disables Go's deadlock detector

    calendar May 4, 2026 · 7 min read · Go  ·
    Share on: twitter copy

    I recently ran into a Go test timeout that turned out to be a deadlock. That was surprising because Go has a built-in deadlock detector. The simplest possible deadlock is a goroutine that locks the same mutex twice. Running the following code exits with fatal error: all goroutines are asleep - deadlock! as expected. …


    Read More
  • Avoid using 2D map for transition table in Go

    calendar Feb 26, 2026 · 4 min read · Go  ·
    Share on: twitter copy

    This post is part 1 of a series of learnings from nilaway. nilaway is a static analysis tool that detects potential nil panics in Go code. It does report false positives, but it's far from naive. One limitation is that nilaway is flow-sensitive (it understands if x != nil) but not value-correlation-sensitive (it …


    Read More
  • cached-imds-client: cache static IMDS responses to avoid linklocal_allowance_exceeded on EC2

    calendar Feb 15, 2026 · 1 min read · AWS Go  ·
    Share on: twitter copy

    I recently encountered IMDS (Instance Metadata Service) request failures with this error: 1"caller": "actor/actor.go:101", 2"error": "operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, x available, y requested The root cause: the aws-sdk-go-v2 IMDS …


    Read More
  • Using WaitGroup to Track Work Items, Not Workers: A Multi-threaded BFS Example

    calendar Feb 15, 2026 · 5 min read · Go Concurrency  ·
    Share on: twitter copy

    WaitGroup and channels are two powerful primitives in Go for synchronizing goroutines. A common pattern uses a WaitGroup to wait for goroutines completion: 1wg.Add(1) 2go func() { 3 defer wg.Done() 4 for { 5 select { 6 case <- done: 7 return 8 case task <- tasks: 9 handle(task) 10 } 11 } 12}() 13wg.Wait() In this …


    Read More
  • Be careful making thread-aware syscalls in Go: lock the thread

    calendar Oct 20, 2025 · 10 min read · Go Container Linux  ·
    Share on: twitter copy

    A bug caused around 0.5% of container workloads to fail to start during load test. This post walks through the bug and its fix, an interesting mix of Linux namespaces, Go concurrency, and syscalls. The need to run a program in its own network namespace and mount namespace soci-snapshotter is an open-source containerd …


    Read More
  • Modern Go idioms

    calendar May 18, 2025 · 6 min read · Go  ·
    Share on: twitter copy

    Go is known for its backward compatibility, simplicity, and six-month release cycle. But that can sometimes lead to code that works yet isn't as modern as it could be. This post is a living document where I note modern Go idioms I've used to improve clarity and maintainability. Use GOOS and GOARCH in file names for …


    Read More
  • Sharp edges of errgroup: Lessons from an errgroup and Context mishap

    calendar Mar 23, 2025 · 8 min read · Go Concurrency  ·
    Share on: twitter copy

    A recent faulty release disrupted service for some customers. The root cause was a concurrency bug involving x/sync/errgroup and context cancellation. This post shares three practices we learned from the incident. These practices will help us catch similar issues during code review or alert us to problems in …


    Read More
  • Avoid panic on expected errors: lessons from operating journald-to-cwl

    calendar Feb 23, 2025 · 3 min read · Go  ·
    Share on: twitter copy

    We've been using journald-to-cwl to ship journal logs from EC2 instances to CloudWatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panicking on expected errors in Go is …


    Read More
    • ««
    • «
    • 1
    • 2
    • 3
    • »
    • »»

Peng Zhang

Software Engineer

Recent Posts

  • Remount sysfs to make its superblock rw: mount -o remount,rw /sys
  • Page-cache thrashing on EC2: kill the tenant before it takes the host down
  • Pre-load container images on the Bottlerocket read-only root filesystem
  • Leaked empty directories exhaust inodes
  • Share one client per service
  • Debugging zero disk reads after moving to Bottlerocket
  • When sync.Cond is a better primitive than channels: lessons from Go's HTTP/2 client
  • Mysterious boot hang on Graviton: mitigated but never root-caused

Tags

LINUX 30 GO 23 BOTTLEROCKET 11 CONTAINER 10 ALGORITHMS 8 INTERVIEW 7 CONCURRENCY 4 SYSTEMD 4 DISTRIBUTED SYSTEM 3 GUIDE 3 AWS 2 CONTAINERS 2 EC2 2 SELINUX 2
All Tags
ALGORITHMS8 AWS2 BOTTLEROCKET11 CONCURRENCY4 CONTAINER10 CONTAINERS2 CRYPTOGRAPHY1 DATABASES1 DISTRIBUTED SYSTEM3 DNS1 DOCKER1 EC22 GO23 GUIDE3 INTERVIEW7 LINUX30 NETWORK1 NFS1 SELINUX2 SHELL1 SYSTEMD4 TESTING1 WEB2
[A~Z][0~9]
Peng Zhang

Copyright 2022-  PENG ZHANG. All Rights Reserved

to-top