When sync.Cond is a better primitive than channels: lessons from Go's HTTP/2 client

I recently reviewed code that let a dynamic set of goroutines wait for updates from a single writer. Each waiter wanted to block until the writer had published something newer than what it had already seen. The implementation used channels and was hard to follow. A condition variable (sync.Cond) would have been less code and easier to read. Few people reach for sync.Cond, though. The sync.Cond documentation itself says that "for many simple use cases, users will be better off using channels", and in 2017 Bryan Mills proposed removing it from the language in Go 2. The Go 1.27.1 standard library, tests and runtime testdata aside, has five uses: two in the HTTP/2 client, one in net/http's server, one in database/sql, and one in the go command's worker pool. This post walks through the two in net/http/internal/http2, with code links pointing at that tag. It also covers where channels would have done the same job, including a benchmark that did not come out the way I expected; performance turns out not to be the reason to use sync.Cond.

Simplified, the code I reviewed looked like this:

 1type state struct {
 2    mu      sync.Mutex
 3    version int
 4    value   string
 5    waiters []chan struct{}
 6}
 7
 8// WaitNewer blocks until version > seen, then returns the current value.
 9func (s *state) WaitNewer(seen int) (int, string) {
10    s.mu.Lock()
11    defer s.mu.Unlock()
12    for s.version <= seen {
13        ch := make(chan struct{})
14        s.waiters = append(s.waiters, ch)
15        s.mu.Unlock()
16        <-ch
17        s.mu.Lock()
18    }
19    return s.version, s.value
20}
21
22func (s *state) Set(v string) {
23    s.mu.Lock()
24    defer s.mu.Unlock()
25    s.version++
26    s.value = v
27    for _, ch := range s.waiters {
28        close(ch)
29    }
30    s.waiters = nil
31}

The slice is a hand-written wait list and the loop of close calls is a hand-written broadcast. With a sync.Cond on the same mutex, the slice and the loop over it disappear, and the two functions shrink to their predicate and their state change:

 1func (s *state) WaitNewer(seen int) (int, string) {
 2    s.mu.Lock()
 3    defer s.mu.Unlock()
 4    for s.version <= seen {
 5        s.cond.Wait()
 6    }
 7    return s.version, s.value
 8}
 9
10func (s *state) Set(v string) {
11    s.mu.Lock()
12    defer s.mu.Unlock()
13    s.version++
14    s.value = v
15    s.cond.Broadcast()
16}

What sync.Cond does

A goroutine holds a mutex, looks at some shared state, finds that the condition it needs is not met, so it has to sleep until another goroutine changes the state. It cannot sleep while holding the mutex, because then nobody can change the state, and if it unlocks first and then sleeps, the change can land in the gap and the wakeup is missed. A condition variable does "unlock and go to sleep" as one atomic step, and hands the mutex back on wakeup.

A sync.Cond is a waiting room attached to a lock. The lock is the field c.L, usually a pointer to the mutex that already protects the state. It has three methods:

• Wait() atomically releases c.L, parks the goroutine, and re-acquires c.L when woken.
• Signal() wakes one parked goroutine, if any.
• Broadcast() wakes all parked goroutines, if any.

 1type queue struct {
 2    mu    sync.Mutex
 3    cond  sync.Cond // cond.L = &mu
 4    items []int
 5}
 6
 7func (q *queue) Get() int {
 8    q.mu.Lock()
 9    defer q.mu.Unlock()
10    for len(q.items) == 0 {
11        q.cond.Wait() // unlocks, sleeps, relocks
12    }
13    x := q.items[0]
14    q.items = q.items[1:]
15    return x
16}
17
18func (q *queue) Put(x int) {
19    q.mu.Lock()
20    defer q.mu.Unlock()
21    q.items = append(q.items, x)
22    q.cond.Signal() // one Get can make progress
23}

The condition itself, len(q.items) == 0, is whatever predicate you compute over the state protected by c.L. Signal and Broadcast carry no payload; all they say is "the state may have changed, look again". That leads to three rules every use has to follow:

1. Hold c.L while reading or changing the state the predicate depends on, and while calling Wait.
2. Call Wait in a loop that re-checks the predicate: for !ready() { c.Wait() }. A wakeup only means "the state may have changed"; Broadcast wakes goroutines whose own predicate is still false, and a Signal may land on a goroutine that cannot use it.
3. Change the state before you signal. Signal and Broadcast are no-ops when nobody is waiting. If the state is already updated, a late arriver sees it on the first predicate check and never waits; if the state is not yet updated, the signal is lost.

Rule 3 is also why the lock is required around Wait but merely allowed around Signal/Broadcast. Wait is implemented as t := notifyListAdd(); L.Unlock(); notifyListWait(t); L.Lock(). It takes a ticket while still holding the lock, so a Signal that fires between the unlock and the actual park is not lost: the runtime sees that ticket t has already been notified and returns immediately. The lock is what makes "check predicate, then take a ticket" atomic with respect to "change state, then signal".

Unlike a channel, a Cond cannot be used in a select statement, so there is no built-in way to wait with a timeout or a context (golang/go#16620). And the zero value of sync.Cond is not usable: Wait dereferences c.L, so you need sync.NewCond(&mu) or an assignment to c.L before the first Wait.

pipe.go: a buffered pipe with one reader

Every HTTP/2 stream in the client has a pipe that carries the response body. The connection's read loop goroutine receives DATA frames and writes them in; the goroutine calling resp.Body.Read reads them out. The struct is in pipe.go:

 1// pipe is a goroutine-safe io.Reader/io.Writer pair. It's like
 2// io.Pipe except there are no PipeReader/PipeWriter halves, and the
 3// underlying buffer is an interface. (io.Pipe is always unbuffered)
 4type pipe struct {
 5    mu       sync.Mutex
 6    c        sync.Cond     // c.L lazily initialized to &p.mu
 7    b        pipeBuffer    // nil when done reading
 8    unread   int           // bytes unread when done
 9    err      error         // read error once empty. non-nil means closed.
10    breakErr error         // immediate read error (caller doesn't see rest of b)
11    donec    chan struct{} // closed on error
12    readFn   func()        // optional code to run in Read before error
13}

The reader:

 1func (p *pipe) Read(d []byte) (n int, err error) {
 2    p.mu.Lock()
 3    defer p.mu.Unlock()
 4    if p.c.L == nil {
 5        p.c.L = &p.mu
 6    }
 7    for {
 8        if p.breakErr != nil {
 9            return 0, p.breakErr
10        }
11        if p.b != nil && p.b.Len() > 0 {
12            return p.b.Read(d)
13        }
14        if p.err != nil {
15            if p.readFn != nil {
16                p.readFn()     // e.g. copy trailers
17                p.readFn = nil // not sticky like p.err
18            }
19            p.b = nil
20            return 0, p.err
21        }
22        p.c.Wait()
23    }
24}

The writer, and the shared close path used by CloseWithError (deliver remaining data, then the error) and BreakWithError (drop remaining data, error now):

 1func (p *pipe) Write(d []byte) (n int, err error) {
 2    p.mu.Lock()
 3    defer p.mu.Unlock()
 4    if p.c.L == nil {
 5        p.c.L = &p.mu
 6    }
 7    defer p.c.Signal()
 8    if p.err != nil || p.breakErr != nil {
 9        return 0, errClosedPipeWrite
10    }
11    if p.b == nil {
12        return 0, errUninitializedPipeWrite
13    }
14    return p.b.Write(d)
15}
16
17func (p *pipe) closeWithError(dst *error, err error, fn func()) {
18    // ...
19    p.mu.Lock()
20    defer p.mu.Unlock()
21    if p.c.L == nil {
22        p.c.L = &p.mu
23    }
24    defer p.c.Signal()
25    if *dst != nil {
26        return // Already been done.
27    }
28    p.readFn = fn
29    if dst == &p.breakErr {
30        if p.b != nil {
31            p.unread += p.b.Len()
32        }
33        p.b = nil
34    }
35    *dst = err
36    p.closeDoneLocked()
37}

The predicate is a disjunction over four fields. The reader can proceed when breakErr is set, or the buffer has bytes, or err is set (and then it may have to run readFn first, which copies trailers into the response). All of that state already needs a mutex, because the read loop and the body reader touch it from different goroutines. The Cond adds exactly one thing on top of the mutex: a way to sleep until the state changes.

Every state change ends with defer p.c.Signal(), even on error paths. Because Signal is a no-op when nobody waits and the predicate is re-checked under the lock, over-signaling is harmless, so the writers do not try to be clever about when a wakeup is needed.

Signal rather than Broadcast is used here because an io.Reader has one reader at a time. If two goroutines did call Read concurrently, one of them could miss a wakeup; the HTTP/2 code does not support that, and neither does io.Reader in general.

The lazy if p.c.L == nil { p.c.L = &p.mu } at the top of every method is a workaround for Cond having no usable zero value. The pipe is embedded by value in clientStream (bufPipe pipe) so that each stream gets one without a constructor; patching c.L on entry, under p.mu, is what that costs.

The buffer absorbs bursts. If the read loop writes ten DATA frames while the body reader is busy elsewhere, the first Signal may wake it and the next nine are no-ops; when the reader comes back it drains all ten frames in as few Read calls as its buffer allows.

transport.go: one condition variable, multiple predicates

The second use is in transport.go, on the connection object ClientConn:

1type ClientConn struct {
2    // ...
3    mu   sync.Mutex
4    cond *sync.Cond // hold mu; broadcast on flow/closed changes
5    // ...
6}

One Cond, created with cc.cond = sync.NewCond(&cc.mu), and three places that Wait on it, each for a different predicate. Reduced to the predicate and the Wait, with cc.mu held:

 1// awaitOpenSlotForStreamLocked: a new request waits for a stream slot
 2// when the connection is at SETTINGS_MAX_CONCURRENT_STREAMS.
 3for cc.currentRequestCountLocked() >= int(cc.maxConcurrentStreams) {
 4    cc.pendingRequests++
 5    cc.cond.Wait()
 6    cc.pendingRequests--
 7}
 8
 9// awaitFlowControl: a goroutine writing a request body waits for
10// flow-control window before it may send the next DATA frame.
11for cs.flow.available() == 0 {
12    cc.cond.Wait()
13}
14
15// Shutdown: a helper goroutine waits for in-flight streams to drain.
16for len(cc.streams) > 0 && !cc.closed && !cancelled {
17    cc.cond.Wait()
18}

The real loops (awaitOpenSlotForStreamLocked, awaitFlowControl, Shutdown) also return early when the connection is closed, the stream is aborted, or the request's context is done. Those checks are plain ifs and a non-blocking select at the top of each iteration, so they run on every wakeup.

On the other side, there are nine call sites that wake waiters, and every one of them is Broadcast; Signal does not appear in the file. The wakers are: a stream finishing (forgetStreamID, which frees a slot and also lets the request-body writer give up), a WINDOW_UPDATE frame arriving, a SETTINGS frame changing the initial window size for every open stream, a stream being aborted, the request body being closed, the connection closing for an error, the read loop exiting, Shutdown being cancelled, and a PING ack clearing a rate limit on RST_STREAM.

The comment on the field, "broadcast on flow/closed changes", describes the design. The wakers do not know, and do not try to know, which waiters can make progress. A WINDOW_UPDATE on the connection window may unblock any number of body writers; a stream ending frees one slot but perhaps nobody is waiting for a slot while three goroutines are waiting for window. If forgetStreamID used Signal, the one wakeup might land on a body writer whose window is still zero, which would check its predicate, go back to sleep, and leave the request waiting for a slot asleep for good. With heterogeneous waiters, Broadcast plus re-check is the only correct choice, and it is cheap: Broadcast is a no-op when nobody is parked, and the goroutines it does wake fail their predicate and park again without leaving the function.

The pendingRequests counter around the first Wait is only safe because the lock is held on both sides of it. Wait releases cc.mu while the goroutine sleeps, so other code (the connection pool deciding whether this connection can take more requests, and the StreamsPending stat) reads an accurate count of blocked requests.

Cancellation is harder, because a Cond cannot be selected on. awaitFlowControl checks ctx.Done() on every iteration, but that only helps if something wakes it. The wakeup comes from RoundTrip: it runs the request in a separate goroutine and itself selects on ctx.Done(); when the context fires it calls cs.abortStream, which sets abortErr, closes cs.abort, and does cc.cond.Broadcast(). To cancel a Cond wait, some goroutine that can observe the cancellation has to mutate the shared state and broadcast; the waiter cannot do it on its own.

What the channel versions would look like

For the pipe, a channel version is easy and, as far as I can measure, just as fast. Keep the mutex and the buffer, replace the Cond with notify chan struct{} of capacity one, and have the writer do a non-blocking send:

1select {
2case p.notify <- struct{}{}:
3default: // a wakeup is already pending
4}

The reader does by hand what Wait does for it: p.mu.Unlock(); <-p.notify; p.mu.Lock(), then re-checks. Close has to close(p.notify) so that a parked reader wakes, which is fine because a pipe closes once. The capacity-one channel gives the same coalescing of bursts that Signal gives: ten writes while the reader is away leave one token in the channel, not ten.

To measure it, cond_pipe.go and chan_pipe.go implement the same pipe, shaped like the HTTP/2 one with a 4 KiB bytes.Buffer, and differ only in the notification primitive. pipe_bench_test.go runs four workloads against both; each iteration creates a pipe, spawns a writer goroutine, and drains the pipe from the benchmark goroutine. Six runs each with Go 1.27.1 on a 64-vCPU Xeon, summarized with benchstat (results.txt):

The Cond version is ahead by 1 to 7 percent in every row, and every row includes a goroutine spawn, so the gap in the notification itself is smaller than that. The Cond version also allocates one object fewer per pipe, but only because a sync.Cond embedded by value costs nothing while a channel is a heap object. An earlier version of the benchmark used sync.NewCond(new(sync.Mutex)), which is two allocations, and the channel won that column instead.

The common claim that sync.Cond is faster because it does not allocate compares it to the wrong thing. Against a capacity-one notification channel, the difference is noise. Performance is not the reason to choose a Cond for a pipe. The reason is readability: the reader's predicate spans several fields of mutex-protected state, and Wait expresses "release this mutex until the state changes" in one call, instead of an unlock-receive-lock sequence that has to be kept correct in every method.

For ClientConn, the channel version is less pleasant, and this is the case Bryan Mills was describing when he wrote that the second Cond in h2_bundle.go "is only used with Broadcast (never Signal), and can thus be replaced with a channel that is closed to broadcast readiness". Closing a channel is a broadcast, but a one-shot one. This connection broadcasts on every stream completion, every WINDOW_UPDATE, every SETTINGS; over the life of a connection that is thousands of broadcasts. The channel idiom for a repeatable broadcast is close-and-replace, under the mutex:

 1type ClientConn struct {
 2    mu      sync.Mutex
 3    changed chan struct{} // closed and replaced on every state change
 4}
 5
 6func (cc *ClientConn) broadcastLocked() {
 7    close(cc.changed)
 8    cc.changed = make(chan struct{})
 9}
10
11// waiter, holding cc.mu:
12for !predicate() {
13    ch := cc.changed
14    cc.mu.Unlock()
15    <-ch
16    cc.mu.Lock()
17}

This works. x/sync/semaphore uses the per-waiter variant, a list of waiters each holding its own channel, which is the shape of the code in the introduction. The single-channel version above allocates a channel per broadcast whether or not anyone is waiting (you can avoid that by allocating lazily when a waiter asks for the channel, at the cost of more state), and every waiter still wakes and re-checks exactly as with Broadcast, so nothing is gained in wakeup efficiency. What you do gain is select: awaitFlowControl could wait on ch, ctx.Done(), cs.abort and cs.reqCancel directly instead of relying on RoundTrip to broadcast on its behalf. Whether that is worth the extra channel plumbing is a judgment call; the HTTP/2 authors chose a Cond on the mutex they already had.

When I reach for sync.Cond

A condition variable is the better fit when all of these hold:

• The thing goroutines wait for is a predicate over state that already lives under a mutex. If there is no mutex, or the predicate is "a value has arrived", a channel is the natural fit.
• The predicate can become true many times, so a one-shot close(ch) does not model it and you would be into close-and-replace.
• Either there is exactly one waiter (use Signal), or there are several waiters with different predicates and the waker cannot tell which of them to wake (use Broadcast, and have every waiter loop on its own predicate).

And a channel is the better fit when the waiter needs select, especially for a context, when there is a genuine handoff of a value from one goroutine to another, or when the event happens once.