We run an OpenTelemetry Collector (otel-collector) as a systemd service on Bottlerocket. Its Prometheus metrics endpoint was configured on port 51000, which falls inside the default Linux ephemeral port range. On a small portion of EC2 instances, the otel-collector kept restarting, failing to bind with EADDRINUSE. …

Read More

I recently ran into an issue after upgrading the containerd client library to v2.3.0. The root cause was a change in one of containerd's dependencies, continuity, that started enforcing fsync when copying files. containerd/continuity is a library in containerd that provides filesystem utilities: file copying, directory …

Read More

A container platform shipped a change to detect CPU architecture mismatches at image pull time. If an image's config says arm64 but the host is amd64, fail fast instead of letting the container start and die with exec format error. The implementation is correct, tests passed and reviewers approved. However, it caused a …

Read More

If you build software that runs in someone else's environment, anything that isn't a SaaS you operate yourself, the following code can break your customers: 1if not hostname.endswith("."): 2 hostname += "." Appending a trailing dot to a domain name before you resolve it feels "more correct" …

Read More

I recently ran into a Go test timeout that turned out to be a deadlock. That was surprising because Go has a built-in deadlock detector. The simplest possible deadlock is a goroutine that locks the same mutex twice. Running the following code exits with fatal error: all goroutines are asleep - deadlock! as expected. …

Read More

Bottlerocket v11.0.0 dropped the socat package from bottlerocket-core-kit (#742). If you had a service that relied on socat to bridge a Unix Domain Socket (UDS) to a TCP port, you need a replacement. This post shows how to use systemd-socket-proxyd instead. Why socat was used soci-snapshotter exposes its metrics …

Read More

CGO_ENABLED defaults to 1. That means a standard go build produces a binary that links against C libraries (e.g., glibc) at runtime. For many parts of the Go standard library, there is a C-backed implementation and a pure-Go implementation. CGO_ENABLED selects which one gets compiled in. Pure-Go alternatives also exist …

Read More

We run a container platform. For privacy and security reasons, we do not collect kernel logs because customer workloads use the same kernel as the host and kernel messages can contain sensitive customer data, such as command-line arguments surfaced in audit logs. However, we recently hit a blind spot: foo.service was …

Read More

This post is part 1 of a series of learnings from nilaway. nilaway is a static analysis tool that detects potential nil panics in Go code. It does report false positives, but it's far from naive. One limitation is that nilaway is flow-sensitive (it understands if x != nil) but not value-correlation-sensitive (it …

Read More

I recently encountered IMDS (Instance Metadata Service) request failures with this error: 1"caller": "actor/actor.go:101", 2"error": "operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, x available, y requested The root cause: the aws-sdk-go-v2 IMDS …

Read More