https://peng.fyi/ Recent content on Practical Engineering Hugo -- gohugo.io en Peng Zhang Thu, 26 Feb 2026 00:00:00 +0000 https://peng.fyi/post/avoid-2d-map-for-state-transition-in-go/ Thu, 26 Feb 2026 00:00:00 +0000 https://peng.fyi/post/avoid-2d-map-for-state-transition-in-go/ This post is part 1 of a series of learnings from nilaway. nilaway is a static analysis tool that detects potential nil panics in Go code. It does report false positives, but it's far from naive. One limitation is that nilaway is flow-sensitive (it understands if x != nil) but not value-correlation-sensitive (it doesn't understand "if y == SomeConst, then x is non-nil"). From my experience, most false positives point to code smells or code that is fragile to maintain. https://peng.fyi/post/cache-imds-to-avoid-linklocal-throttle/ Sun, 15 Feb 2026 00:00:00 +0000 https://peng.fyi/post/cache-imds-to-avoid-linklocal-throttle/ I recently encountered IMDS (Instance Metadata Service) request failures with this error: 1"caller": "actor/actor.go:101", 2"error": "operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, x available, y requested The root cause: the aws-sdk-go-v2 IMDS client exhausted its retry token bucket. The AWS SDK for Go v2 implements client-side rate limiting ("retry quotas") to prevent overwhelming services. The standard strategy works as follows: Initial attempt: Consumes no tokens, adds 1 token back on success Retry attempt: Deducts 5 tokens (10 for timeouts) If tokens available: Retry proceeds If tokens exhausted: Returns QuotaExceededError On success: Adds 1 token back (max capacity: 500) When IMDS requests time out or retry frequently, they drain the token bucket quickly. https://peng.fyi/post/waitgroup-track-work-not-workers-multi-thread-bfs/ Sun, 15 Feb 2026 00:00:00 +0000 https://peng.fyi/post/waitgroup-track-work-not-workers-multi-thread-bfs/ WaitGroup and channels are two powerful primitives in Go for synchronizing goroutines. A common pattern uses a WaitGroup to wait for goroutines completion: 1wg.Add(1) 2go func() { 3 defer wg.Done() 4 for { 5 select { 6 case <- done: 7 return 8 case task <- tasks: 9 handle(task) 10 } 11 } 12}() 13wg.Wait() In this post, we'll explore an interesting use of WaitGroup and channels where the WaitGroup counts work items each goroutine handles rather than goroutines. https://peng.fyi/post/simplify-device-path-on-boot-with-udev/ Mon, 02 Feb 2026 00:00:00 +0000 https://peng.fyi/post/simplify-device-path-on-boot-with-udev/ While prototyping Bottlerocket, I discovered it doesn't recognize additional EBS volumes specified through Block device mappings on Xen. For example, launching the same AMI on t2.medium (Xen) and t3.medium (Nitro) with "DeviceName=/dev/xvdcz": On Nitro, the device appears at /dev/nvme1n1 and /dev/disk/by-ebs-id/xvdcz. On Xen, it appears at /dev/xvdcz. In the prototype, the xvdcz volume is formatted, labeled, and mounted, involving format-data-volume.service and mnt-data.mount. The format service hardcodes its dependency on /dev/disk/by-ebs-id/xvdcz. https://peng.fyi/post/think-again-before-use-killmode-process/ Fri, 12 Dec 2025 00:00:00 +0000 https://peng.fyi/post/think-again-before-use-killmode-process/ I recently debugged a resource leak where a systemd service kept restarting while leaving a process behind after each restart. The root cause isn't particularly interesting: a backward-incompatible third-party dependency upgrade. But the debugging process and lessons learned are. Thousands of zombie processes from a systemd service I have a foo.service that runs /usr/bin/start-foo, which spawns a new process to run /usr/bin/foo. A few minutes after boot, CPU and memory utilization reached 100%, growing linearly rather than spiking. https://peng.fyi/post/antipattern-in-socket-activation-spawn-new-process-for-the-daemon/ Wed, 10 Dec 2025 00:00:00 +0000 https://peng.fyi/post/antipattern-in-socket-activation-spawn-new-process-for-the-daemon/ I recently debugged a mysterious latency issue: after migrating a systemd service from path-activation to socket-activation, there was a consistent ~1 second time-to-available latency. The culprit was a bad practice—starting the daemon program as a new process in socket-activation. Let's dive into the details. Starting soci-snapshotter On-Demand Using Socket Activation soci-snapshotter is an open-source containerd snapshotter plugin that enables pulling OCI images in lazy loading mode using SOCI indexes. In our container runtime, we start soci-snapshotter only when an image has a SOCI index. https://peng.fyi/post/thread-aware-syscall-in-go-lock-the-thread/ Mon, 20 Oct 2025 00:00:00 +0000 https://peng.fyi/post/thread-aware-syscall-in-go-lock-the-thread/ A bug caused around 0.5% of container workloads to fail to start during load test. This post walks through the bug and its fix, an interesting mix of Linux namespaces, Go concurrency, and syscalls. The need to run a program in its own network namespace and mount namespace soci-snapshotter is an open-source containerd snapshotter plugin that enables pulling OCI images in lazy loading mode. It downloads the necessary layers to start the container and then downloads the rest while the container is running. https://peng.fyi/post/speed-up-building-br-image-in-codebuild/ Mon, 20 Oct 2025 00:00:00 +0000 https://peng.fyi/post/speed-up-building-br-image-in-codebuild/ When I first moved building Bottlerocket AMI from an EC2 host to AWS CodeBuild, I was hit by a very slow build. On an EC2 instance, I built both the x86 and Arm versions on x86 instances, and fresh builds finished in 5 minutes. However, on CodeBuild with more vCPU and memory, the build process was painfully slower. The x86 CodeBuild uses a compute of "145 GB memory, 72 vCPUs". The build finishes in 18 minutes, with the build-variant step taking 97% of the time. https://peng.fyi/post/mysterious-image-pull-failure-wait-for-interface-ready/ Sun, 12 Oct 2025 00:00:00 +0000 https://peng.fyi/post/mysterious-image-pull-failure-wait-for-interface-ready/ Early this year, we migrated containerd from v1.7 to v2.0.5. However, we quickly noticed image pulls from Amazon Elastic Container Registry (ECR) began failing for both public and private ECR repositories. For example: 1# public ECR 2FATA[0031] failed to resolve reference "public.ecr.aws/aws-cli/aws-cli:2.31.5@sha256:9cb6ab9c8852d7e1e63f43299dca0628e92448d1c7589f7ed40344e7f61aad59": \ 3unexpected status from HEAD request to https://public.ecr.aws/v2/aws-cli/aws-cli/blobs/sha256:9cb6ab9c8852d7e1e63f43299dca0628e92448d1c7589f7ed40344e7f61aad59: \ 4401 Unauthorized 5 6# private ECR 7FATA[0031] failed to resolve reference "<account-id>.dkr.ecr.us-west-2.amazonaws.com/xxxx@sha256:1db8db35b1afaa9d2df40f68e35cc0e4f406b5f73667b1ba2d0f73dbb15aed01": \ 8<account-id>.dkr.ecr.us-west-2.amazonaws.com/xxxx@sha256:1db8db35b1afaa9d2df40f68e35cc0e4f406b5f73667b1ba2d0f73dbb15aed01: \ 9not found The failures occurred at a low frequency of around 0. https://peng.fyi/post/retry-imds-request/ Mon, 15 Sep 2025 00:00:00 +0000 https://peng.fyi/post/retry-imds-request/ In Detect and fix rare cases where the primary ENI does not serve default traffic , we used IMDS "meta-data/mac" to get the primary ENI's MAC address. However, we encountered the following errors in 0.5% of EC2 ARM instance launches: 1failed to get IMDS /mac: operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, 2http response error StatusCode: 401, request to EC2 IMDS failed 1failed to get IMDS /mac: operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, 2request send failed, Get "http://169. https://peng.fyi/post/who-moved-my-program/ Thu, 11 Sep 2025 00:00:00 +0000 https://peng.fyi/post/who-moved-my-program/ There are a few programs we install in Bottlerocket that cannot be built from source. For these programs, we download the binary from a secure repository and install it using an RPM spec like this: 1# foo.spec 2Name: %{_cross_os}foo 3 4Source0: foo 5 6%install 7install -d %{buildroot}%{_cross_sbindir} 8install -D -p -m 0755 %{S:0} %{buildroot}%{_cross_sbindir} A teammate discovered that the foo binary downloaded from the repository differs from the /sbin/foo installed in the built Bottlerocket AMI: https://peng.fyi/post/debug-on-bottlerocket-and-introduce-bottlerocket-extra-kit/ Mon, 01 Sep 2025 00:00:00 +0000 https://peng.fyi/post/debug-on-bottlerocket-and-introduce-bottlerocket-extra-kit/ Bottlerocket is a Linux-based operating system optimized for hosting containers. We use Bottlerocket to run millions of containers each day. There are three key differences between Bottlerocket and common Linux distributions like Amazon Linux 2023: The rootfs is read-only. There is no package manager (e.g., yum) in Bottlerocket. Each package in Bottlerocket must be built into the OS variant. Enforced SELinux with Bottlerocket's own SELinux policies. This makes it difficult for developers to debug and experiment. https://peng.fyi/post/tips-build-bottlerocket-ami/ Wed, 20 Aug 2025 00:00:00 +0000 https://peng.fyi/post/tips-build-bottlerocket-ami/ Bottlerocket is a Linux-based operating system optimized for hosting containers. At my work, we migrated from Amazon Linux to Bottlerocket and experienced the following benefits: Developer-friendly: Easy to understand and fast to build. RPM spec and configuration TOML files are all you need. Every developer can build a Bottlerocket AMI on an EC2 instance in just a few minutes. For example, I can build and register a new Bottlerocket AMI from scratch in less than 4 minutes on a c5. https://peng.fyi/post/working-knowledge-of-linux-memory-concepts/ Mon, 04 Aug 2025 00:00:00 +0000 https://peng.fyi/post/working-knowledge-of-linux-memory-concepts/ I recently dealt with a server livelock issue caused by memory page thrashing. This post refreshes the Linux memory basics I found useful for debugging the issue. Much of the content is from Chapter 7 of Systems Performance: Enterprise and the Cloud. Virtual Memory Virtual memory is an abstraction that provides each process and the kernel with its own large, linear, and private address space. Virtual memory is also referred to as process virtual address space. https://peng.fyi/post/swap-eth-names-when-primary-eni-does-not-serve-default-traffic/ Sun, 27 Jul 2025 00:00:00 +0000 https://peng.fyi/post/swap-eth-names-when-primary-eni-does-not-serve-default-traffic/ During testing, we encountered a rare scenario when launching EC2 instances with multiple ENIs: the primary ENI (device index 0) does not serve default network traffic. This occurs in approximately 1 out of 10,000 launches (0.01%). For example, when configuring two ENIs on an instance—ENI-0 (deviceIndex=0) from subnet-0 and ENI-1 (deviceIndex=1) from subnet-1—Linux may recognize eth0 as being from subnet-1 instead of the expected subnet-0. In my use case, we must ensure default traffic routes through the primary ENI for security compliance. https://peng.fyi/post/selinux-part-1-concepts/ Sun, 15 Jun 2025 00:00:00 +0000 https://peng.fyi/post/selinux-part-1-concepts/ Security-Enhanced Linux (SELinux) is a mandatory access control (MAC) system that enhances Linux security. "Mandatory" means access control is strictly enforced by predefined policy rules—users and processes cannot modify these rules at will, ensuring security is not left to individual discretion. SELinux is available in major distributions, including Amazon Linux 2023 (AL2023) and Bottlerocket. This post is the part 1 of a series on SELinux. Labels and Contexts SELinux labels every file and process using a quadruple: (user:role:type:level). https://peng.fyi/post/modern-go-idioms/ Sun, 18 May 2025 00:00:00 +0000 https://peng.fyi/post/modern-go-idioms/ Go is known for its backward compatibility, simplicity, and six-month release cycle. But that can sometimes lead to code that works yet isn't as modern as it could be. This post is a living document where I note modern Go idioms I've used to improve clarity and maintainability. Use GOOS and GOARCH in file names for build constraints When targeting specific operating systems or architectures, Go lets us name files like foo_linux. https://peng.fyi/post/a-few-shell-surprises/ Tue, 22 Apr 2025 00:00:00 +0000 https://peng.fyi/post/a-few-shell-surprises/ Shell scripts are infamous for security issues and surprising behavior, so when possible, it's better to avoid using shell. For instance, we built a container platform using the Bottlerocket OS, and we didn't even install a shell. If someone needs to run a shell, it must be run inside a container. That said, shell is still handy for ad hoc scripting. In this post, I'll share a few surprising behaviors I've encountered recently. https://peng.fyi/post/empty-cert-pool-x509-certificate-signed-by-unknown-authority/ Tue, 15 Apr 2025 00:00:00 +0000 https://peng.fyi/post/empty-cert-pool-x509-certificate-signed-by-unknown-authority/ I recently worked on getting amazon-ssm-agent to run inside containers on Bottlerocket. During that process, I ran into a TLS issue connecting to amazonaws.com. The root cause turned out be interesting and we'll walk through it in this post. Running amazon-ssm-agent in a container: why and how? To enable sessions between a container and the outside world, we followed the same approach as the ECS Execute-Command proposal. The idea is to prepare a directory on the host that contains all the files required by SSM, then bind mount that directory into the container. https://peng.fyi/post/lessons-from-an-errgroup-and-context-mishap/ Sun, 23 Mar 2025 00:00:00 +0000 https://peng.fyi/post/lessons-from-an-errgroup-and-context-mishap/ A recent faulty release disrupted service for some customers. The root cause was a concurrency bug involving x/sync/errgroup and context cancellation. This post shares three practices we learned from the incident. These practices will help us catch similar issues during code review or alert us to problems in production. What does the buggy code do? I've simplified the program for this post as follows: One director manages three managers Each manager processes tasks periodically The director maintains a control session with an external system The director monitors for anomalies. https://peng.fyi/post/avoid-panic-on-expected-error/ Sun, 23 Feb 2025 00:00:00 +0000 https://peng.fyi/post/avoid-panic-on-expected-error/ We've been using journald-to-cwl to ship journal logs from EC2 instances to CloudWatch Logs. It is lightweight and reliable. However, we recently started receiving false positive alarms, which became annoying. This blog covers the changes we made and the key lesson learned: panicking on expected errors in Go is generally a bad idea. Where Do False Positive Alarms Come From? We run many Go programs in production, monitoring their logs and alarming if the total number of panics exceeds a threshold. https://peng.fyi/post/gpg-is-still-in-use/ Sun, 23 Feb 2025 00:00:00 +0000 https://peng.fyi/post/gpg-is-still-in-use/ This week, I needed to install the Amazon SSM Agent and was surprised to find that GPG (GNU Privacy Guard) was the only way to verify the download. I had assumed that software downloads verification had largely transitioned to PKI (Public Key Infrastructure). This short post is a refresh on GPG. OpenPGP is an open standard for encrypting and signing data, originally derived from PGP (Pretty Good Privacy). It defines the format and encryption protocols used for secure communication. https://peng.fyi/post/why-does-gomemlimit-take-up-significant-physical-memory-for-unused-virtual-memory/ Sun, 19 Jan 2025 00:00:00 +0000 https://peng.fyi/post/why-does-gomemlimit-take-up-significant-physical-memory-for-unused-virtual-memory/ While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat program The following Go program vm-demo.go demonstrates memory bloat by allocating a 400 MiB slice every second. It saves references to the slices without any read or write operations. https://peng.fyi/post/logs-go-to-stderr-by-default/ Sat, 30 Nov 2024 00:00:00 +0000 https://peng.fyi/post/logs-go-to-stderr-by-default/ It's a beautiful day, and it started with a simple code review: 1# tools/foo/main.go 2- fmt.Println("found it") 3+ log.Println("found it") The author explained the advantages of using a logging library over plain printf. The rationale was straightforward, so I approved the change without hesitation. However, two hours later, another code review came through—this time reverting the change because the load test pipeline had failed. Intriguing. Let's take a closer look. https://peng.fyi/post/al2023-vs-al2-less-disk-space-with-ext4/ Sun, 17 Nov 2024 00:00:00 +0000 https://peng.fyi/post/al2023-vs-al2-less-disk-space-with-ext4/ We started migrating from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023) a month ago. While testing workloads on AL2023 in the pre-production environment, I noticed slightly higher disk usage compared to the same workload on AL2. In this post, I'll share my investigation. AL2023 Has Less Free Disk Space with ext4, Compared to AL2 Although disk usage metrics increased on AL2023, the "Used" space remained the same. The higher disk usage is due to a decrease in the total ext4 filesystem size. https://peng.fyi/post/ways-go-programs-die/ Sun, 10 Nov 2024 00:00:00 +0000 https://peng.fyi/post/ways-go-programs-die/ Our Go programs recently triggered an alarm due to excessive panics. Panic is a Go runtime mechanism that halts execution. It got me thinking about different ways a Go program can die. I don't expect many - not like A Million Ways to Die in the West. In this post, we'll go through the various ways Go programs die. These fall into two categories: voluntarily choosing to die, and involuntarily being killed by the Go runtime or the operating system. https://peng.fyi/post/container-disk-io-stats-missing-on-kernel-6.1-cgroupv1/ Sat, 09 Nov 2024 00:00:00 +0000 https://peng.fyi/post/container-disk-io-stats-missing-on-kernel-6.1-cgroupv1/ As Amazon Linux 2 (AL2) approaches its End of Life on June 30, 2025, we have started migrating our container platform from AL2 to Bottlerocket. The migration encountered a few speed bumps. In this post, we'll examine one of them: missing container disk I/O stats. Why are container I/O dashboards blank? Since Bottlerocket shares the same kernel used by Amazon Linux 2023 (AL2023), I will use AL2023 for the ease of demonstration. https://peng.fyi/post/systemd-cycle-dependencies/ Wed, 16 Oct 2024 00:00:00 +0000 https://peng.fyi/post/systemd-cycle-dependencies/ I've been building a service for a month, and the day finally arrived when I had the artifact - an EC2 AMI. The AMI passed my "rigorous" manual tests so I launched 100 EC2 instances. Surprise! Around 28 instances failed to launch. What's going on? All failed instances were stuck in the "initializing" state, and the only way to connect to them was through EC2 Serial Console. There, I noticed something interesting. https://peng.fyi/post/monotonicity-subinterval-132/ Mon, 14 Oct 2024 00:00:00 +0000 https://peng.fyi/post/monotonicity-subinterval-132/ Given an array of numbers A, find out whether it contains a 1-3-2 pattern. A 1-3-2 pattern is a subsequence of three numbers, A[i], A[j] and A[k] such that i < j < k and A[i] < A[k] < A[j]. For clarity, let's call the 1-3-2 pattern the Bronze-Gold-Silver pattern. If A[j] is Gold, then we should consider the minimum number from A[0:j) to be Bronze, because it gives us the largest range for picking Silver. https://peng.fyi/post/hoare-partition/ Mon, 17 Jun 2024 00:00:00 +0000 https://peng.fyi/post/hoare-partition/ Tony Hoare invented QuickSort in 1961. At the time of its publication, the best comparison-based sorting algorithm was merge sort. Merge sort divides an unordered array into two equally sized subarrays, sorts each subarray, and then merge the two subarrays to produce a sorted array. Merge sort is simple to understand. However, quicksort is just as simple as merge sort but more elegant. In quicksort, there is no requirement for the two subarrays to be of equal size, and there is no merging step. https://peng.fyi/post/upstream-and-downstream/ Thu, 07 Mar 2024 00:00:00 +0000 https://peng.fyi/post/upstream-and-downstream/ I often find myself confused by "upstream" and "downstream", in the context of software development. They bother me so much that I avoid using them in my own writing and I have to pause whenever I see them. In this post, I'll show a simple rule that helps you remember the difference: downstream adds value to the output of upstream. Downstream adds value to the output of upstream. Let's take a break from software development for a moment and look at the oil industry. https://peng.fyi/post/false-sharing-cacheline-and-write-buffer/ Fri, 23 Feb 2024 00:00:00 +0000 https://peng.fyi/post/false-sharing-cacheline-and-write-buffer/ Modern CPUs operate significantly faster than memory. A 4.5 GHz x86_64 CPU operates 30 times faster than 6000 MHz DDR5 memory with CAS Latency 36. When accounting for latencies from the bus and memory coherency protocols, memory can be 100 times slower than registers. To mitigate this speed gap, CPUs use layers of caches organized around cache lines, typically 64 bytes each. However, programming languages operate at the byte level. This mismatch between byte-level abstractions and cache line realities can create a false sharing: a false sense of data independence. https://peng.fyi/post/binary-search-in-go-sdk/ Sun, 11 Feb 2024 00:00:00 +0000 https://peng.fyi/post/binary-search-in-go-sdk/ Given a non-decreasing array and a target value, we can find the target in logarithmic time using binary search. My first programming language was C++, and the C++ Standard Template Library (STL) provides two functions for this task: iterator lower_bound(first, last, value) returns the smallest index with a value greater than or equal to the target. Put another way, if you were to insert the target, the lower_bound is the smallest index at which to insert it. https://peng.fyi/post/steps-to-make-array-non-decreasing/ Sun, 28 Jan 2024 00:00:00 +0000 https://peng.fyi/post/steps-to-make-array-non-decreasing/ Problem Given an integer array A, in one step, remove all elements A[i] where A[i-1] > A[i]. Return the number of steps performed until A becomes a non-decreasing array. See examples at LeetCode 2289. Solution The naive approach executes steps one by one. Store integers in a Linked List. At each step, find all integers that are smaller than its left neighbor and remove them. However, the time complexity is O(n^2) because each step may remove just one integer. https://peng.fyi/post/calculate-linear-network-size-by-message-passing/ Sat, 20 Jan 2024 00:00:00 +0000 https://peng.fyi/post/calculate-linear-network-size-by-message-passing/ Problem In a connected network consisting of N nodes, each node is connected to either one or two neighbors, forming a line topology. The task is to develop a program that runs on each node, calculating the total number of nodes in the network. Each node is aware of its neighboring nodes and can exchange messages with them. It is important to note that nodes do not share memory; the only means of information exchange between two neighbors is through sending and receiving messages. https://peng.fyi/post/breakdown-analysis-nested-json/ Fri, 19 Jan 2024 00:00:00 +0000 https://peng.fyi/post/breakdown-analysis-nested-json/ Problem An event consists of multiple properties, each defined as a key-value pair, where the key is a string and the value is of a primitive type such as numbers or strings. Importantly, each event must include a mandatory 'Name' property. Given a list of events, the task is to count the number of events based on specified properties. To illustrate, let's consider an example with four events and two analyses: https://peng.fyi/post/factorial-growth-of-clickhouse-with-clause/ Fri, 12 Jan 2024 00:00:00 +0000 https://peng.fyi/post/factorial-growth-of-clickhouse-with-clause/ ClickHouse is a popular OLAP database. It speaks SQL and earns the reputation of "fast and resource efficient". But the support of SQL comes with surprises if not careful. In this blog, I show that a simple query of nested WITH clauses in ClickHouse generates factorial number of subqueries. The simple query is short, reads nothing, process nothing and returns nothing. Yet, it uses a lot of CPU and memory to just parse the query. https://peng.fyi/post/maglev-hash-alternatives-for-ring-hash/ Sun, 07 Jan 2024 00:00:00 +0000 https://peng.fyi/post/maglev-hash-alternatives-for-ring-hash/ In distributed systems, because there are too many requests to be handled by a single server reliably, requests are handled by a cluster of servers. In order to get high availability, the technique of distributing requests to servers needs to satisfy the following three requirements. Even distribution. Each backend take about M/N requests, where M is the number of requests and N is the number of servers. Low disruption. Adding or removing one server causes about M/N requests to be re-distributed. https://peng.fyi/post/monotonicity-sliding-window/ Mon, 27 Nov 2023 00:00:00 +0000 https://peng.fyi/post/monotonicity-sliding-window/ A function is monotonic if it preserves the order of its arguments, i.e., if $x \le y$, then $f(x) \le f(y)$. In this post, we examine a class of problems where the argument is an interval. By identifying monotonic functions, we can reduce the number of intervals to enumerate by an order of magnitude, from $O(n^2)$ to $O(n)$. This algorithm is often known as sliding window, because enumerating intervals is like sliding a window across the data. https://peng.fyi/post/monotonicity-stack/ Thu, 23 Nov 2023 00:00:00 +0000 https://peng.fyi/post/monotonicity-stack/ Given an array of numbers $A$, for each number $A[i]$, find the largest subarray that contains $A[i]$ and $A[i]$ is the minimum of the subarray. For example, for $A=[2, 0, 3, 5, 1, 1, 0, 2 1]$, the largest subarray for $A[4]$ is $[3,5,1,1]$. Let's represent the subarray for $A[i]$ as the left boundary $l[i]$ and right boundary $r[i]$. $$l[i] = \min_{j \le i }\lbrace \forall_{j \le k \le i} A[k] \ge A[i] \rbrace $$ $$r[i] = \max_{j \ge i }\lbrace \forall_{i \le k \le j} A[k] \ge A[i] \rbrace $$ https://peng.fyi/post/gomaxprocs-in-container/ Wed, 22 Nov 2023 00:00:00 +0000 https://peng.fyi/post/gomaxprocs-in-container/ Every Go program has a runtime. The runtime implements garbage collection, concurrency, stack management, and other critical features. We can configure the runtime by setting variables. In this post, we will look at GOMAXPROCS, a variable that configures concurrency. You may get free performance boost by setting GOMAXPROCS when running Go in containers. What is GOMAXPROCS? The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. https://peng.fyi/post/simple-customizable-loadtest/ Sun, 15 Oct 2023 00:00:00 +0000 https://peng.fyi/post/simple-customizable-loadtest/ We plan to load test our product before public Beta, with two goals in mind. Find out bottlenecks: figure out road maps of performance improvement and prepare for oncalls. Understand how much workload we can support with fixed resources. This shapes the pricing strategy and determines number of Beta partners to onboard, without burning runway. Because it is complicated to generate meaningful loads to the system, we cannot use general tools like K6. https://peng.fyi/post/kth-number-in-two-sorted-arrays/ Tue, 10 Oct 2023 00:00:00 +0000 https://peng.fyi/post/kth-number-in-two-sorted-arrays/ Given two sorted arrays of integers, $A$ and $B$, find the $K$-th smallest integer from A and B in $O(\min(\log{N}, \log{M}))$ time. $N$ and $M$ are the size of $A$ and $B$ respectively. Without loss of generality, we can assume A and B are of the same length N. Because both A and B are sorted, we can merge A and B into a sorted array C in $O(N)$ and the answer would be C[K-1]. https://peng.fyi/post/priority-map/ Mon, 09 Oct 2023 00:00:00 +0000 https://peng.fyi/post/priority-map/ I implemented a job scheduler at work recently. Each job has a unique ID and an expiration time. The job ID is immutable while the expiration time may change. The job scheduler schedules the job of the earliest expiration time. Go comes with heap.Interface and map. My first implementation uses a hashmap and a slice that implements the heap.Interface. The hashmap maps the Job and the index of the Job in the heap array. https://peng.fyi/post/gotchas-of-defer-in-go/ Sun, 10 Sep 2023 00:00:00 +0000 https://peng.fyi/post/gotchas-of-defer-in-go/ A defer statement invokes a function just before the surrounding function returns. Multiple defers within a function are executed in reverse order of their calls, Last In First Out (LIFO). Defer is commonly used to ensure resource cleanup, regardless of the function's success or failure. For instance, 1func (h *Handler) Handle(ctx context.Context) { 2 // Use defer to catch and log panics. This prevents web server crash. 3 defer func() { 4 if r := recover(); r ! https://peng.fyi/post/cors-error-with-504/ Fri, 01 Sep 2023 00:00:00 +0000 https://peng.fyi/post/cors-error-with-504/ Like many developers, I often leave browser's "Developer Tools" open for websites of interests. Last week, while playing with our staging service, I saw repeated CORS errors in the Console tab and 504 Gateway Timeouts in the Network tab. It was my first time seeing these two errors together. So I decided to look into it a bit. In this blog, I will reproduce the issue and share some good practices handling CORS. https://peng.fyi/post/context-cancel-go-web/ Thu, 29 Jun 2023 00:00:00 +0000 https://peng.fyi/post/context-cancel-go-web/ At iheartjane, we use Go web server to serve Ad requests. After some time in production, we noticed a lot of “context canceled” error logs. The following screenshot of CloudWatch Log Insights query shows the frequency of “context canceled” errors. It left us puzzled about the underlying causes of these context cancels. Should we worry about it? If yes, how should we reduce context cancels. What are “Context Canceled” in Go? https://peng.fyi/post/simplicity-by-default/ Tue, 23 May 2023 00:00:00 +0000 https://peng.fyi/post/simplicity-by-default/ I’ve observed quite a few instances where engineers, including myself, have created unnecessarily complex solutions. To cultivate a better R&D organization, it is crucial to choose simplicity by default. In this blog, we’ll look at three don’ts and three do’s that help make simplicity the default. Don’t invent requirements. Engineers solve problems, which are often made complex due to bloated or unclear requirements. Simplify by reducing requirements to a minimal set that satisfies the product. https://peng.fyi/post/six-options-generating-distributed-ids/ Thu, 20 Apr 2023 00:00:00 +0000 https://peng.fyi/post/six-options-generating-distributed-ids/ Identifying unique entities is a frequent requirement in software development. For instance, assigning a unique ID to each Ad impression enables us to link related events for billing and analysis. However, generating unique IDs becomes challenging when dealing with large distributed systems. In this survey, we explore various options and discuss their suitability. We also introduce the snowflake-id service, a distributed system for generating unique IDs based on the snowflake algorithm and utilizing the etcd. https://peng.fyi/post/creating-test-doubles-in-go-manual-or-generated/ Thu, 30 Mar 2023 00:00:00 +0000 https://peng.fyi/post/creating-test-doubles-in-go-manual-or-generated/ We set up test doubles when it is difficult or impossible to use real objects due to complexity or external dependencies. In Go, an interface is a collection of method signatures that define a set of behaviors. With interfaces, it is easy to create test doubles in Go. However, should you write test doubles on your own, use a test library that supports mock, or a tool that generates mock automatically?