What is an ABI? Break one to find out

I used NVIDIA's go-dcgm to read GPU health data and stats. While using it, I initially found it strange that it hardcodes the dynamic library name. It turns out to be good practice. This post dives deeper into the topic behind it: ABI.

go-dcgm uses cgo, and its initDCGM dlopens the literal string libdcgm.so.4 at runtime:

 1func initDCGM(m mode, args ...string) (err error) {
 2    const (
 3        dcgmLib = "libdcgm.so.4"
 4    )
 5    lib := C.CString(dcgmLib)
 6    defer freeCString(lib)
 7
 8    dcgmLibHandle = C.dlopen(lib, C.RTLD_LAZY|C.RTLD_GLOBAL)
 9    if dcgmLibHandle == nil {
10        return fmt.Errorf("%s not found", dcgmLib)
11    }
12    ...
13}

DCGM's own C stub library takes the same approach, forwarding each entry point through dlsym on a handle from that same pinned name. Install a newer DCGM and, on the face of it, the program stops finding its library. Understanding why that is the safe design means understanding what an application binary interface is and what happens when one breaks.

So I wrote a shared library with a two-field struct, and broke it to learn. Each source file is linked where it first appears. Output below is from Amazon Linux 2023 with GCC 11.5 and glibc 2.34, on x86-64.

API and ABI

Look at the following C code that writes and reads a struct with two integer fields. First, the library, gpu_v1.c:

1struct gpu_info {
2    int gpu_id;       /* offset 0 */
3    int temperature;  /* offset 4 */
4};
5
6void gpu_info_get(struct gpu_info *info) {
7    info->gpu_id = 42;
8    info->temperature = 99;
9}

Then the consumer, main.c, declares the same struct and prints what it gets back:

 1struct gpu_info {
 2    int gpu_id;
 3    int temperature;
 4};
 5
 6extern void gpu_info_get(struct gpu_info *info);
 7
 8int main(void) {
 9    struct gpu_info info = {0};
10    gpu_info_get(&info);
11    printf("gpu_id=%d temperature=%d\n", info.gpu_id, info.temperature);
12    return 0;
13}

An application programming interface (API) is a contract between two pieces of source code. It is about names: the function is called gpu_info_get, it takes a pointer to struct gpu_info, and that struct has a field named temperature.

An application binary interface (ABI) is a contract between two pieces of compiled machine code. It is about bytes and registers: the struct is 8 bytes wide, gpu_id lives at byte offset 0, temperature lives at byte offset 4, and the pointer arrives in %rdi.

Build the library as a shared object, link the consumer against it, run:

1gcc -shared -fPIC -Wl,-soname,libgpu.so.1 -o libgpu.so.1.0.0 gpu_v1.c
2ln -sf libgpu.so.1.0.0 libgpu.so.1
3ln -sf libgpu.so.1     libgpu.so
4gcc -o app main.c -L. -lgpu
5LD_LIBRARY_PATH=. ./app

1gpu_id=42 temperature=99

That -Wl,-soname flag matters later, so note what it does now. The file on disk is named "libgpu.so.1.0.0", but the flag stamps it with a different name, the SONAME "libgpu.so.1", which is the name it declares itself to be:

1$ readelf -d libgpu.so.1.0.0 | grep -i soname
2 0x000000000000000e (SONAME)             Library soname: [libgpu.so.1]

The consumer records the SONAME, not the filename:

1$ readelf -d app | grep 'NEEDED.*gpu'
2 0x0000000000000001 (NEEDED)             Shared library: [libgpu.so.1]

This is why a Linux host has files like libcurl.so.4.8.0 with a libcurl.so.4 symlink pointing at it. The following is from an Amazon Linux 2023 host.

 1$ ls -l /usr/lib64/libcurl.so*
 2lrwxrwxrwx. 1 root root     16 Sep  2 21:17 /usr/lib64/libcurl.so.4 -> libcurl.so.4.8.0
 3-rwxr-xr-x. 1 root root 799088 Sep  2 21:17 /usr/lib64/libcurl.so.4.8.0
 4
 5$ readelf -d /usr/lib64/libcurl.so.4.8.0 | grep -i soname
 6 0x000000000000000e (SONAME)             Library soname: [libcurl.so.4]
 7
 8$ rpm -qf /usr/lib64/libcurl.so.4.8.0
 9libcurl-minimal-8.21.0-5.amzn2023.0.1.x86_64
10
11$ curl --version | head -1
12curl 8.21.0 (x86_64-amazon-linux-gnu) libcurl/8.21.0 OpenSSL/3.5.8 zlib/1.2.11 ...

Three different numbers describe one file. The release is 8.21.0. The filename carries 4.8.0. The SONAME, which is the only one the loader reads, is 4. The 4 has nothing to do with the release version. It is the ABI version, and it changes only when the layout changes. curl set it to 4 in release 7.16.0 in October 2006 and has not moved it since, through two major releases and every version in between.

What survives compilation

Before breaking anything, let's look at what the compiler emitted for the binary.

 1$ objdump -d --no-show-raw-insn app | sed -n '/<main>:/,/^$/p'
 20000000000401136 <main>:
 3  401136:    push   %rbp
 4  401137:    mov    %rsp,%rbp
 5  40113a:    sub    $0x10,%rsp
 6  40113e:    movq   $0x0,-0x8(%rbp)
 7  401146:    lea    -0x8(%rbp),%rax
 8  40114a:    mov    %rax,%rdi
 9  40114d:    call   401040 <gpu_info_get@plt>
10  401152:    mov    -0x4(%rbp),%edx
11  401155:    mov    -0x8(%rbp),%eax
12  401158:    mov    %eax,%esi
13  40115a:    mov    $0x402010,%edi
14  40115f:    mov    $0x0,%eax
15  401164:    call   401030 <printf@plt>

Search it for the word "temperature". It is not there. There are only -0x4(%rbp) and -0x8(%rbp), which mean 4 bytes into the struct and 0 bytes into the struct. The field names existed for the C compiler and then evaporated.

The library is the mirror image:

1$ objdump -d --no-show-raw-insn libgpu.so.1.0.0 | sed -n '/<gpu_info_get>:/,/ret/p'
200000000000010f9 <gpu_info_get>:
3    10fd:    mov    %rdi,-0x8(%rbp)
4    1101:    mov    -0x8(%rbp),%rax
5    1105:    movl   $0x2a,(%rax)
6    110b:    mov    -0x8(%rbp),%rax
7    110f:    movl   $0x63,0x4(%rax)

It writes 0x2a (42) to offset 0 and 0x63 (99) to offset 4. Neither side names a field. Both sides independently baked the same two numbers into instructions at compile time, and that agreement is the entire contract, the ABI. Nothing renegotiates it at runtime.

Break 1: swap two fields

Now ship a version 2, gpu_v2_bad.c. The author reorders the struct, maybe to group fields logically, and adds a third one:

1struct gpu_info {
2    int temperature;  /* offset 0, was gpu_id */
3    int gpu_id;       /* offset 4, was temperature */
4    int power_usage;  /* offset 8, new */
5};

From the API point of view this looks harmless. Same function name, same struct name, same field names, one addition. Anything recompiled against the new header works. So the author calls it a minor release and leaves the SONAME alone:

1gcc -shared -fPIC -Wl,-soname,libgpu.so.1 -o libgpu.so.1.1.0 gpu_v2_bad.c
2ln -sf libgpu.so.1.1.0 libgpu.so.1

The library's machine code shows what actually changed. The same two constants, now at each other's offsets, plus the new one at offset 8:

1$ objdump -d --no-show-raw-insn libgpu.so.1.1.0 | sed -n '/<gpu_info_get>:/,/ret/p'
200000000000010f9 <gpu_info_get>:
3    1105:    movl   $0x63,(%rax)
4    110f:    movl   $0x2a,0x4(%rax)
5    111a:    movl   $0xfa,0x8(%rax)

Run the old binary. It was never recompiled, which is the point: it stands in for software already deployed.

1$ LD_LIBRARY_PATH=. ./app
2gpu_id=99 temperature=42

The values are swapped. Not corrupted, not zeroed, swapped, which is worse, because both are individually plausible. GPU 99 is a fine device ID. 42 degrees is a fine temperature. Only the pairing is wrong.

Per offset:

That last row is a write past an 8-byte struct. Here it is harmless: the disassembly above shows main reserved 16 bytes with sub $0x10,%rsp, so the stray 4 bytes land on unused stack. In a function with more locals they land on whatever the compiler put next.

Now ask the tools whether anything is wrong. The exported symbol is identical in both libraries, down to the address:

1$ nm -D --defined-only libgpu.so.1.0.0 | grep gpu_info_get
200000000000010f9 T gpu_info_get
3$ nm -D --defined-only libgpu.so.1.1.0 | grep gpu_info_get
400000000000010f9 T gpu_info_get

And the dependency checker is satisfied, including with -r, which additionally resolves symbol relocations:

 1$ LD_LIBRARY_PATH=. ldd ./app | grep gpu
 2    libgpu.so.1 => ./libgpu.so.1 (0x00007f423b8a0000)
 3$ echo $?
 40
 5$ LD_LIBRARY_PATH=. ldd -r ./app
 6    linux-vdso.so.1 (0x00007fcd3cc97000)
 7    libgpu.so.1 => ./libgpu.so.1 (0x00007fcd3cc8c000)
 8    libc.so.6 => /lib64/libc.so.6 (0x00007fcd3ca00000)
 9    /lib64/ld-linux-x86-64.so.2 (0x00007fcd3cc99000)
10$ echo $?
110

Nothing on stderr, nothing in dmesg, no symbol errors, exit code 0. A file with the requested SONAME exists, and it exports the symbol the binary wants. Both of those are true. The loader has no concept of a struct, so there is nothing else for it to check. If the two integers had been a fan speed and a power cap, you would be writing a power cap into a fan speed.

Break 2: move a pointer

Let's try a struct that holds a C string, gpu_v1_ptr.c:

1struct gpu_info {
2    int gpu_id;          /* offset 0 */
3    int temperature;     /* offset 4 */
4    const char *name;    /* offset 8 */
5};

The version 2 author moves the pointer to the front, gpu_v2_ptr_bad.c. This is a real habit, since putting the 8-byte member first is the standard trick for avoiding padding:

1struct gpu_info {
2    const char *name;    /* offset 0 */
3    int gpu_id;          /* offset 8 */
4    int temperature;     /* offset 12 */
5};

Both structs are 16 bytes. Nothing overflows. The only change is where the pointer lives. SONAME unchanged again.

Against version 1 the consumer, main_ptr.c, is fine:

1$ LD_LIBRARY_PATH=. ./app_ptr
2gpu_id=42 temperature=99
3name pointer = 0x7f12996f6000
4name = Tesla T4

Swap in the new library and it crashes:

1$ ln -sf libgpuptr.so.1.1.0 libgpuptr.so.1
2$ LD_LIBRARY_PATH=. ./app_ptr
3gpu_id=1000214528 temperature=32621
4name pointer = 0x630000002a
5Segmentation fault (core dumped)
6$ echo $?
7139

The two integers are now halves of an address. The library wrote the pointer to "Tesla T4" across offsets 0 through 7. The consumer read offset 0 as gpu_id and offset 4 as temperature, splitting one 64-bit pointer down the middle and printing each half as a signed int:

1gpu_id      = 1000214528 = 0x3b9e1000   <- low 4 bytes
2temperature =      32621 = 0x00007f6d   <- high 4 bytes
3recombined  =       0x7f6d3b9e1000

Compare that to the pointer the working run printed, 0x7f12996f6000. Same shape: a 0x7f prefix and a page-aligned tail, because both are addresses inside a mapped shared library. The digits change on every run since ASLR relocates the library, so the shape is the signal, not the value. Once you have seen a couple of these, a suspiciously large integer starts looking like half of a pointer.

The pointer is two integers. Meanwhile the consumer read offsets 8 through 15 as its name pointer. The library put gpu_id at 8 and temperature at 12, so the consumer assembled a pointer out of them:

10x630000002a = (99 << 32) | 42 = (temperature << 32) | gpu_id

Then printf with %s dereferenced address 0x630000002a, which is not mapped, and the kernel killed the process. Exit 139 is 128 + 11, SIGSEGV.

So the same one-line change, a field reordering, gave bad data in one struct and a hard crash in another. Note also where the crash landed: in the consumer's printf, not in the library that shipped the change. If you ever chase a segfault in code that has been stable for years, one candidate is that the two sides of a call boundary disagree about the shape of what is being passed.

The fix: the SONAME is the ABI version

The library-side fix, gpu_v2.c, is one character. Bump the SONAME and keep version 1 installed:

1gcc -shared -fPIC -Wl,-soname,libgpu.so.2 -o libgpu.so.2.0.0 gpu_v2.c
2ln -sf libgpu.so.2.0.0 libgpu.so.2
3ln -sf libgpu.so.1.0.0 libgpu.so.1

Both libraries now sit on disk together. The old binary:

1$ LD_LIBRARY_PATH=. ./app
2gpu_id=42 temperature=99

It asks for libgpu.so.1, gets a library that agrees with it about offsets, and libgpu.so.2 is simply ignored. To use version 2, the consumer recompiles against the new header, main_v2.c, which does two things at once: regenerates the offsets in its own machine code, and updates the recorded dependency.

1$ gcc -o app_v2 main_v2.c -L. -lgpu
2$ readelf -d app_v2 | grep 'NEEDED.*gpu'
3 0x0000000000000001 (NEEDED)             Shared library: [libgpu.so.2]
4$ LD_LIBRARY_PATH=. ./app_v2
5gpu_id=42 temperature=99 power_usage=250

Two binaries, same function name, different libraries, both correct. That is SONAME versioning working. It is a good mechanism, but it depends entirely on a human remembering to bump the number.

The defense: pin the version at dlopen

Which brings back go-dcgm. Instead of linking normally, it loads the library at runtime by a fully version-qualified name. dlopen_main.c does the same with the toy library:

1/* pin the ABI version, the way go-dcgm pins libdcgm.so.4 */
2void *handle = dlopen("libgpu.so.1", RTLD_NOW);
3if (!handle) {
4    fprintf(stderr, "dlopen: %s\n", dlerror());
5    return 1;
6}
7gpu_info_get_fn fn = (gpu_info_get_fn)dlsym(handle, "gpu_info_get");

The DCGM stub makes the intent explicit. It dlopens a name held in s_libName, which is set to DCGM_LIB_SONAME, and that macro is generated at build time from the project's major version alone:

1#define DCGM_LIB_SONAME "libdcgm.so.@CMAKE_PROJECT_VERSION_MAJOR@"

Not the full version, just the major. The minor and patch digits are deliberately absent, because they are the digits that do not change layout.

With libgpu.so.1 present, app_dlopen behaves normally. Now remove version 1 so only the incompatible version 2 remains, the upgrade scenario:

 1$ rm libgpu.so.1
 2$ ls libgpu.so*
 3libgpu.so
 4libgpu.so.1.0.0
 5libgpu.so.1.1.0
 6libgpu.so.2
 7libgpu.so.2.0.0
 8$ LD_LIBRARY_PATH=. ./app_dlopen
 9dlopen: libgpu.so.1: cannot open shared object file: No such file or directory
10$ echo $?
111

That error is the feature. libgpu.so.2 is right there, it exports gpu_info_get, and the symbol would resolve fine. The program could have loaded it and printed something. Instead it refuses to start and names the version it wanted.

The pinned name converts an unbounded correctness problem into a bounded availability problem. A process that will not start is a crash loop, a failed health check, a page, all things every system already handles. Data that is quietly wrong is none of those.

Pinning by dlopen also makes the dependency soft, which is the other half of why agents do it. A normally-linked binary will not launch at all when the library is missing, because ld.so resolves NEEDED entries before main runs, the same failure mode as a cgo binary missing its .so. A dlopening binary launches, gets a null handle, and can degrade: skip GPU metrics, report the driver as unavailable, keep serving everything else. For an agent expected to run on both GPU and CPU hosts, that is the requirement, not a workaround trick.

Conclusion

The compiler bakes offsets into your binary and forgets the field names. Nothing at runtime re-derives them, so two halves of a call boundary can disagree about layout indefinitely with no way to notice.

As a library author, treat the SONAME as the ABI version and bump it whenever the layout changes. As a consumer, hardcoding a version-qualified name like libdcgm.so.4 is a correctness control, not laziness. It trades an undetectable wrong-data failure for a loud one.