• Don't hardcode fsync in your library

    calendar Jul 20, 2026 · 5 min read · Containers Linux  ·
    Share on: copy

    I recently ran into an issue after upgrading the containerd client library to v2.3.0. The root cause was a change in one of containerd's dependencies, continuity, that started enforcing fsync when copying files. containerd/continuity is a library in containerd that provides filesystem utilities: file copying, directory …


    Read More

  • When enforcing OCI image spec broke customer containers

    calendar Jun 11, 2026 · 4 min read · Containers  ·
    Share on: copy

    A container platform shipped a change to detect CPU architecture mismatches at image pull time. If an image's config says arm64 but the host is amd64, fail fast instead of letting the container start and die with exec format error. The implementation is correct, tests passed and reviewers approved. However, it caused a …


    Read More

  • Why a Trailing Dot in DNS Can Break Your Customers

    calendar Jun 10, 2026 · 5 min read · DNS  ·
    Share on: copy

    If you build software that runs in someone else's environment, anything that isn't a SaaS you operate yourself, the following code can break your customers: 1if not hostname.endswith("."): 2 hostname += "." Appending a trailing dot to a domain name before you resolve it feels "more correct" …


    Read More

  • How cgo silently disables Go's deadlock detector

    calendar May 4, 2026 · 7 min read · Go  ·
    Share on: copy

    I recently ran into a Go test timeout that turned out to be a deadlock. That was surprising because Go has a built-in deadlock detector. The simplest possible deadlock is a goroutine that locks the same mutex twice. Running the following code exits with fatal error: all goroutines are asleep - deadlock! as expected. …


    Read More

  • Replacing socat with systemd-socket-proxyd

    calendar Mar 30, 2026 · 3 min read · Linux Systemd Bottlerocket  ·
    Share on: copy

    Bottlerocket v11.0.0 dropped the socat package from bottlerocket-core-kit (#742). If you had a service that relied on socat to bridge a Unix Domain Socket (UDS) to a TCP port, you need a replacement. This post shows how to use systemd-socket-proxyd instead. Why socat was used soci-snapshotter exposes its metrics …


    Read More

  • You probably want to disable cgo: Go's stdlib has pure-Go implementations

    calendar Mar 25, 2026 · 3 min read  ·
    Share on: copy

    CGO_ENABLED defaults to 1. That means a standard go build produces a binary that links against C libraries (e.g., glibc) at runtime. For many parts of the Go standard library, there is a C-backed implementation and a pure-Go implementation. CGO_ENABLED selects which one gets compiled in. Pure-Go alternatives also exist …


    Read More

  • Who killed my service: collecting kernel kill logs with OTEL

    calendar Mar 10, 2026 · 6 min read · Linux  ·
    Share on: copy

    We run a container platform. For privacy and security reasons, we do not collect kernel logs because customer workloads use the same kernel as the host and kernel messages can contain sensitive customer data, such as command-line arguments surfaced in audit logs. However, we recently hit a blind spot: foo.service was …


    Read More

  • Avoid using 2D map for transition table in Go

    calendar Feb 26, 2026 · 4 min read · Go  ·
    Share on: copy

    This post is part 1 of a series of learnings from nilaway. nilaway is a static analysis tool that detects potential nil panics in Go code. It does report false positives, but it's far from naive. One limitation is that nilaway is flow-sensitive (it understands if x != nil) but not value-correlation-sensitive (it …


    Read More

  • cached-imds-client: cache static IMDS responses to avoid linklocal_allowance_exceeded on EC2

    calendar Feb 15, 2026 · 1 min read · AWS Go  ·
    Share on: copy

    I recently encountered IMDS (Instance Metadata Service) request failures with this error: 1"caller": "actor/actor.go:101", 2"error": "operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, x available, y requested The root cause: the aws-sdk-go-v2 IMDS …


    Read More

  • Using WaitGroup to Track Work Items, Not Workers: A Multi-threaded BFS Example

    calendar Feb 15, 2026 · 5 min read · Go Concurrency  ·
    Share on: copy

    WaitGroup and channels are two powerful primitives in Go for synchronizing goroutines. A common pattern uses a WaitGroup to wait for goroutines completion: 1wg.Add(1) 2go func() { 3 defer wg.Done() 4 for { 5 select { 6 case <- done: 7 return 8 case task <- tasks: 9 handle(task) 10 } 11 } 12}() 13wg.Wait() In this …


    Read More