Share one client per service

I recently dealt with an issue where a service failed occasionally during short throttling windows. The root cause was one I had seen many times before: a process holding multiple clients for the same service. This post walks through the issue, why the bug is rare in Java but common in Go, and how to avoid it.

The bug

The process is a long-running container agent. Inside the agent, the supervisor registers the host with a control service and opens a command stream; a work runner then handles the requests that arrive on that stream. The supervisor and the runner each hold their own client to the control service. The supervisor's client is wrapped in a retry layer with backoff. The runner's client is the raw client with the AWS Go SDK's default of three attempts. When the control service throttled a burst of requests, the supervisor's calls waited it out and the runner's calls gave up.

It is idiomatic to reuse a client

A client-shaped type is a handle to state: a connection pool, a response cache, cached credentials, a retry token budget, background goroutines. A second client is a second copy of all of it, and the copy starts out empty. The second client works; it is slower on first use, holds its own connections, and has a fresh budget of whatever the first client already spent. None of that shows until the backend pushes back. Because of this, it is idiomatic in Go to reuse a client.

HTTP client in net/http:

The Client.Transport typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

Transport in net/http, right under a paragraph about its connection cache:

Transports should be reused instead of created as needed.

DB client in database/sql:

The returned DB is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.

The storage client in cloud.google.com/go repeats it almost word for word. gRPC's ClientConn, which owns name resolution, load balancing state and the subchannels, is meant to be created once per target and shared across callers.

Java puts the rule in the container, Go leaves it to you

Java teams rarely write this bug, because the container owns instance lifetime. Spring's reference manual on the default scope:

Only one shared instance of a singleton bean is managed, and all requests for beans with an ID or IDs that match that bean definition result in that one specific bean instance being returned by the Spring container. [...] The singleton scope is the default scope in Spring.

A client declared as a bean is process-wide without anyone deciding that it should be. Two components that inject it get the same object, with the same connection pool and the same caches. Guice defaults the other way; the first line of its Scopes page is "By default, Guice returns a new instance each time it supplies a value", and you opt in with @Singleton. Its guidance on when to bother reads like a description of an API client:

Singletons are most useful for: stateful objects, such as configuration or counters; objects that are expensive to construct or lookup; objects that tie up resources, such as a database connection pool.

Either way the lifetime decision is written down once, next to the type, and the container applies it everywhere the type is injected.

Go has no container in the standard library and no blessed one outside it. google/wire generates wiring code at build time and is no longer maintained. uber-go/fx is the closest thing to Spring's singleton scope: constructor results "are cached for reuse (so instances of a type are effectively singletons within an application)". Neither is mainstream. Kubernetes, Prometheus, etcd, CockroachDB, Consul, Terraform and most other well-known Go servers use none of them. The dominant pattern is to construct the shared things near main and pass them down.

When passing one instance down from main gets unwieldy, Go projects write a holder for the state that must be shared. Kubernetes client-go has a type whose name is the whole argument, SharedInformerFactory. Each caller asks it for an informer and the factory hands back the one it already built (informers/factory.go):

 1func (f *sharedInformerFactory) InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer {
 2    f.lock.Lock()
 3    defer f.lock.Unlock()
 4
 5    informerType := reflect.TypeOf(obj)
 6    informer, exists := f.informers[informerType]
 7    if exists {
 8        return informer
 9    }
10    ...
11    informer = newFunc(f.client, resyncPeriod)
12    f.informers[informerType] = informer
13    return informer
14}

Twenty controllers in one process asking for pods get one informer, one watch, and one cache. controller-runtime does the same at a larger grain: its Manager hands every controller the same client and the same cache. The rule Spring applies by default is the one these projects apply by hand.

Why I could not fix the bug by sharing the client

The two clients in the agent are not the same type, the result of a half-done migration. The raw generated client came first, and everything called it. Retry logic grew around it, scattered across call sites. A later change consolidated the retries into a thin wrapper: a type that embeds the client interface, adds backoff and error classification, and shadows only three of the interface's eleven methods. The other eight pass through. The same change moved the supervisor's client onto the wrapper, left the runner's client on the raw factory method, and the migration ended there.

Both client types satisfy the same interface, so no call site shows which one it holds. Handing the wrapper to the runner would change the retry behavior of every call the runner's subtree makes, several unrelated paths through legacy code whose behavior under the new policy I could not predict one by one. That is its own change with its own rollout. Handing the raw client to the supervisor goes the other way and removes protection that works.

What the two clients did share was the factory that builds them, from one configuration with different options, and a client type that can be handed a cache. The ECS agent client exports WithDiscoverPollEndpointCache for exactly this. Of the state a client owns, some parts want to be per instance, such as the retry policy here, and some want to be shared across the process. A cache of a response that never changes is the clearest case of the second kind.

Conclusion

For most uses, one process, one client per service. Spring applies that by default and Guice applies it on request. In Go nobody applies it for you, so treat constructing a second client as acquiring a second resource and name the state the new instance will not inherit.

When several clients are genuinely needed, decide sharing per field rather than per client, and put the shared fields on the thing that builds them. A factory with state is the right place for the state its products share.

Do not leave a migration half done. It is worse than no migration: two implementations that satisfy the same interface, and no call site that tells you which one it holds.