Don't hardcode fsync in your library

I recently ran into an issue after upgrading the containerd client library to v2.3.0. The root cause was a change in one of containerd's dependencies, continuity, that started enforcing fsync when copying files.

containerd/continuity is a library in containerd that provides filesystem utilities: file copying, directory diffing, disk usage accounting, metadata preservation, and extended attribute handling. The main entry point for copying is fs.CopyDir, which recursively walks a source directory and copies each file via fs.CopyFile.

A recent commit a424ba1 ("Linux: Make copyFile sparse aware") to continuity added sparse file support to its Linux file copy path. The sparse-aware logic itself is sound. But bundled with it was a per-file fsync that caused a severe performance regression in some of my use cases. The use case is starting containers with anonymous volumes (the VOLUME directive in a Dockerfile) where the volume has many files. Anonymous volume content lives in the image layers. When such a volume is used as a mount in a container, the runtime copies the volume content to the bind mount path. This is not the normal copy-on-write overlayfs, it's a real copy.

This post walks through sparse files, the merit of the change (it reduces the on-disk size of copied volumes), the regression it introduced, the cost of fsync, and why library owners had better not force that cost on their users.

What Are Sparse Files?

A sparse file is a file whose logical size is larger than its actual disk usage. Regions that have never been written, called "holes", read as zeros but consume no storage blocks. The filesystem simply records "nothing here" instead of allocating blocks full of zeros.

Sparse files are a POSIX/Linux-level concept, not tied to any single filesystem. The interface for detecting holes, the SEEK_DATA and SEEK_HOLE options to lseek(2) added in Linux 3.1, is implemented by all commonly used Linux filesystems: ext4, XFS, Btrfs, ZFS, tmpfs, and others. Code that works with sparse files should still handle the case where a filesystem doesn't support the interface (continuity's copy falls back to a plain copy on EOPNOTSUPP/EINVAL), but in practice every mainstream Linux filesystem supports it.

Here are some quick commands to show sparse files.

 1# Create a 1 GiB file with only 8 KiB of actual data
 2touch ./sparse.txt
 3dd if=/dev/urandom of=./sparse.txt bs=4096 count=1 conv=notrunc
 4dd if=/dev/urandom of=./sparse.txt bs=4096 count=1 seek=262143 conv=notrunc
 5
 6# Check with stat
 7# "Blocks: 16" means 16 × 512 = 8192 bytes on disk. The file is logically 1 GiB but uses 8 KiB.
 8$ stat ./sparse.txt
 9  Size: 1073741824  Blocks: 16   ...
10
11# Check with du and ls
12$ du -h ./sparse.txt
138.0K    ./sparse.txt
14
15$ du -h --apparent-size ./sparse.txt
161.0G    ./sparse.txt

Copying Sparse Files: cp vs io.Copy

GNU cp is sparse-aware and copies sparse file as sparse file by default.

 1$ cp ./sparse.txt /tmp/cp_default.txt
 2$ du -h /tmp/cp_default.txt
 38.0K    /tmp/cp_default.txt          # holes preserved
 4
 5# GNU `cp` defaults to `--sparse=auto`, detecting zero-filled blocks and 
 6# turning them into holes in the destination. 
 7# With `--sparse=never` it materializes everything:
 8
 9$ cp --sparse=never ./sparse.txt /tmp/cp_never.txt
10$ du -h /tmp/cp_never.txt
111.0G    /tmp/cp_never.txt         # zeros written out

Go's io.Copy is not sparse-aware. It reads sequentially through the file. When it hits a hole, the kernel returns zero bytes. The destination receives those zeros as real writes and allocates blocks for them. A 1 GiB sparse file becomes 1 GiB on disk after io.Copy.

The Continuity 0.5.0 Change

continuity v0.4.5 uses Go's io.Copy. In v0.5.0, the commit a424ba1 ("Linux: Make copyFile sparse aware") replaced the naive io.Copy with a proper sparse-aware implementation:

 1func copyFile(target, source string) error {
 2    // Open source, create target, truncate to logical size
 3    tgt, err := os.Create(target)
 4    tgt.Truncate(size)
 5
 6    // Walk data regions, skipping holes
 7    for offset < size {
 8        dataStart = SEEK_DATA(offset)
 9        holeStart = SEEK_HOLE(dataStart)
10        copy_file_range(src, tgt, dataStart, holeStart - dataStart)
11        offset = holeStart
12    }
13
14    // THIS is the problem:
15    tgt.Sync()
16    return nil
17}

The sparse logic is correct. But the File.Sync at the end calls syscall.Fsync.

fsync is expensive

When you call write(), data goes into the kernel's page cache in RAM. The kernel's writeback daemon flushes dirty pages to disk in batches, typically within 30 seconds. This batched approach is efficient: the I/O scheduler can merge adjacent writes, reorder for sequential access, and pipeline multiple files in one flush.

fsync is a syscall that tells the kernel to:

1. Flush all dirty pages for this file to the storage device
2. Wait for the device to confirm the write is durable (on physical media)
3. Only then return to the caller

For a single file, this is fine. But the cost adds up quickly when you fsync many files.

I ran a benchmark, ad-hoc-continuity-0.5.0-fsync-regression, that copies 50K files (each 10 KiB). The benchmark drops the kernel's page cache between runs. This ensures each copy method starts from a cold cache, reading from disk rather than RAM. Without this, the results are misleading.

Results (50K files × 10 KiB, ext4, EC2 with EBS, cold cache)

That's a 17x regression.

Most fs.CopyDir callers don't need per-file fsync

The previous io.Copy implementation never called fsync. Nothing broke. The data source is always available for replay. The callers of fs.CopyDir (which calls copyFile for each regular file) often don't need per-file durability:

• Volume copy-up: When a Dockerfile uses VOLUME /path, the runtime copies image content into the volume at container start. If the machine crashes mid-copy, the container never started, so the runtime simply redoes the copy from the immutable image layer on next start.
• Image layer extraction: If extraction is interrupted, the partial snapshot is discarded and re-extracted from the content store.
• File copy utilities: The caller should decide about durability, not the library.

Conclusion

fsync is expensive. One call is fine. Thousands are catastrophic. Always think about whether you need per-item durability or whether a batch flush (sync/syncfs) suffices. For library authors, adding fsync should be treated as a breaking change. Do not make it the default behavior, especially when the previous version did not fsync at all. Slipping it in silently will break your users.