Practical Engineering
open-menu closeme
Engineering
github linkedin rss
  • A Few Shell Surprises

    calendar Apr 22, 2025 · 3 min read · Linux Shell  ·
    Share on: twitter copy

    Shell scripts are infamous for security issues and surprising behavior. So when possible, it's better to avoid using shell. For instance, we built a container platform using the Bottlerocket OS and we didn't even install shell. If someone needs to run shell, the shell must be run inside a container. That said, shell is …


    Read More
  • x509: certificate signed by unknown authority? Maybe the cert pool is empty

    calendar Apr 15, 2025 · 6 min read · Linux Container SELinux Bottlerocket  ·
    Share on: twitter copy

    I recently worked on getting amazon-ssm-agent to run inside containers on Bottlerocket. During that process, I ran into a TLS issue connecting to amazonaws.com. The root cause turned out be interesting and we'll walk through it in this post. Running amazon-ssm-agent in a container: why and how? To enable sessions …


    Read More
  • GPG is still in use to verify downloads

    calendar Feb 23, 2025 · 2 min read · Linux Cryptography  ·
    Share on: twitter copy

    This week, I needed to install the Amazon SSM Agent and was surprised to find that GPG (GNU Privacy Guard) was the only way to verify the download. I had assumed that software downloads verification had largely transitioned to PKI (Public Key Infrastructure). This short post is a refresh on GPG. OpenPGP is an open …


    Read More
  • Why does GOMEMLIMIT take up significant physical memory for unused virtual memory?

    calendar Jan 19, 2025 · 4 min read · Go Linux  ·
    Share on: twitter copy

    While debugging memory bloat in a Go application recently, I found that removing the GOMEMLIMIT soft memory limit and disabling transparent huge pages partially mitigated the issue. However, I couldn't fully explain why these changes worked. So I thought why not ask the internet about it. A simplified memory bloat …


    Read More
  • AL2023 vs. AL2: less disk space with ext4?

    calendar Nov 17, 2024 · 7 min read · Linux  ·
    Share on: twitter copy

    We started migrating from Amazon Linux 2 (AL2) to Amazon Linux 2023 (AL2023) a month ago. While testing workloads on AL2023 in the pre-production environment, I noticed slightly higher disk usage compared to the same workload on AL2. In this post, I'll share my investigation. AL2023 has less free disk space with ext4, …


    Read More
  • Missing Container Disk I/O Stats with cgroup v1 on Kernel 6.1

    calendar Nov 9, 2024 · 4 min read · Linux Container  ·
    Share on: twitter copy

    As the Amazon Linux 2 (AL2) approaches its End of Life on 2025-06-30, we have started migrating our container platform from AL2 to Amazon Linux 2023 (AL2023). The migration encountered a few speed bumps. In this post, we'll look at one of them: missing container disk I/O stats. Why are container I/O dashboards blank? …


    Read More
  • Mind ordering cycles in systemd: how systemd breaks them can brick the server start up

    calendar Oct 16, 2024 · 3 min read · Linux  ·
    Share on: twitter copy

    I've been building a service for a month and the day finally arrived when I had the artifact - an EC2 AMI. The AMI passed my "rigourous" manual tests, and I felt confident on a Ruby Tuesday, so I launched 100 EC2 instances using the AMI. Surprise! around 28 instances failed to launch. What is going on? All …


    Read More

Peng Zhang

Software Engineer

Recent Posts

  • A few Go idioms
  • A Few Shell Surprises
  • x509: certificate signed by unknown authority? Maybe the cert pool is empty
  • Lessons from an errgroup and Context mishap
  • Avoid panic on expected errors: lessons from operating journald-to-cwl
  • GPG is still in use to verify downloads
  • Why does GOMEMLIMIT take up significant physical memory for unused virtual memory?
  • Logs default to stderr in Go and other languages: avoid using stderr to determine program success.

Tags

GO 16 ALGORITHMS 8 INTERVIEW 7 LINUX 7 GUIDE 3 CONTAINER 2 DISTRIBUTED-SYSTEM 2 WEB 2 BOTTLEROCKET 1 COMPUTER-ARCHITECTURE 1 CONCURRENCY 1 CRYPTOGRAPHY 1 DATABASES 1 SELINUX 1
All Tags
ALGORITHMS8 BOTTLEROCKET1 COMPUTER-ARCHITECTURE1 CONCURRENCY1 CONTAINER2 CRYPTOGRAPHY1 DATABASES1 DISTRIBUTED-SYSTEM2 GO16 GUIDE3 INTERVIEW7 LINUX7 SELINUX1 SHELL1 TESTING1 WEB2
[A~Z][0~9]
Peng Zhang

Copyright 2022-  PENG ZHANG. All Rights Reserved

to-top