What is an ABI? Break one to find out

I used NVIDIA's go-dcgm to read GPU health data and stats. It hardcodes the name of the dynamic library it loads, which looked odd at first and turns out to be good practice.

go-dcgm uses cgo, and its initDCGM dlopens the literal string libdcgm.so.4 at runtime:

 1func initDCGM(m mode, args ...string) (err error) {
 2    const (
 3        dcgmLib = "libdcgm.so.4"
 4    )
 5    lib := C.CString(dcgmLib)
 6    defer freeCString(lib)
 7
 8    dcgmLibHandle = C.dlopen(lib, C.RTLD_LAZY|C.RTLD_GLOBAL)
 9    if dcgmLibHandle == nil {
10        return fmt.Errorf("%s not found", dcgmLib)
11    }
12    ...
13}

DCGM's own C stub library takes the same approach, forwarding each entry point through dlsym on a handle from that same pinned name. Install a newer DCGM and the program stops finding its library. To see why that is the safe design, I wrote a shared library with a two-field struct and broke it. Output below is from Amazon Linux 2023 with GCC 11.5 and glibc 2.34, on x86-64.

API and ABI

The library, gpu_v1.c, fills in a struct with two integer fields:

1struct gpu_info {
2    int gpu_id;       /* offset 0 */
3    int temperature;  /* offset 4 */
4};
5
6void gpu_info_get(struct gpu_info *info) {
7    info->gpu_id = 42;
8    info->temperature = 99;
9}

Then the consumer, main.c, declares the same struct and prints what it gets back:

 1struct gpu_info {
 2    int gpu_id;
 3    int temperature;
 4};
 5
 6extern void gpu_info_get(struct gpu_info *info);
 7
 8int main(void) {
 9    struct gpu_info info = {0};
10    gpu_info_get(&info);
11    printf("gpu_id=%d temperature=%d\n", info.gpu_id, info.temperature);
12    return 0;
13}

An application programming interface (API) is a contract between two pieces of source code. It is about names: the function is called gpu_info_get, it takes a pointer to struct gpu_info, and that struct has a field named temperature.

An application binary interface (ABI) is a contract between two pieces of compiled machine code. It is about bytes and registers: the struct is 8 bytes wide, gpu_id lives at byte offset 0, temperature lives at byte offset 4, and the pointer arrives in %rdi.

Build the library as a shared object, link the consumer against it, run:

1gcc -shared -fPIC -Wl,-soname,libgpu.so.1 -o libgpu.so.1.0.0 gpu_v1.c
2ln -sf libgpu.so.1.0.0 libgpu.so.1
3ln -sf libgpu.so.1     libgpu.so
4gcc -o app main.c -L. -lgpu
5LD_LIBRARY_PATH=. ./app

1gpu_id=42 temperature=99

The file on disk is named libgpu.so.1.0.0, but -Wl,-soname stamps it with a different name, the SONAME libgpu.so.1, which is the name it declares itself to be:

1$ readelf -d libgpu.so.1.0.0 | grep -i soname
2 0x000000000000000e (SONAME)             Library soname: [libgpu.so.1]

The consumer records the SONAME, not the filename:

1$ readelf -d app | grep 'NEEDED.*gpu'
2 0x0000000000000001 (NEEDED)             Shared library: [libgpu.so.1]

This is why a Linux host has files like libcurl.so.4.8.0 with a libcurl.so.4 symlink pointing at it, here on Amazon Linux 2023:

 1$ ls -l /usr/lib64/libcurl.so*
 2lrwxrwxrwx. 1 root root     16 Sep  2 21:17 /usr/lib64/libcurl.so.4 -> libcurl.so.4.8.0
 3-rwxr-xr-x. 1 root root 799088 Sep  2 21:17 /usr/lib64/libcurl.so.4.8.0
 4
 5$ readelf -d /usr/lib64/libcurl.so.4.8.0 | grep -i soname
 6 0x000000000000000e (SONAME)             Library soname: [libcurl.so.4]
 7
 8$ rpm -qf /usr/lib64/libcurl.so.4.8.0
 9libcurl-minimal-8.21.0-5.amzn2023.0.1.x86_64
10
11$ curl --version | head -1
12curl 8.21.0 (x86_64-amazon-linux-gnu) libcurl/8.21.0 OpenSSL/3.5.8 zlib/1.2.11 ...

Three different numbers describe one file. The release is 8.21.0. The filename carries 4.8.0. The SONAME, which is the only one the loader reads, is 4. The 4 has nothing to do with the release version. It is the ABI version, and it changes only when the layout changes. curl set it to 4 in release 7.16.0 in October 2006 and has not moved it since, through two major releases and every version in between.

What survives compilation

Here is what the compiler emitted for main:

 1$ objdump -d --no-show-raw-insn app | sed -n '/<main>:/,/^$/p'
 20000000000401136 <main>:
 3  401136:    push   %rbp
 4  401137:    mov    %rsp,%rbp
 5  40113a:    sub    $0x10,%rsp
 6  40113e:    movq   $0x0,-0x8(%rbp)
 7  401146:    lea    -0x8(%rbp),%rax
 8  40114a:    mov    %rax,%rdi
 9  40114d:    call   401040 <gpu_info_get@plt>
10  401152:    mov    -0x4(%rbp),%edx
11  401155:    mov    -0x8(%rbp),%eax
12  401158:    mov    %eax,%esi
13  40115a:    mov    $0x402010,%edi
14  40115f:    mov    $0x0,%eax
15  401164:    call   401030 <printf@plt>

Search it for the word "temperature". It is not there. There are only -0x4(%rbp) and -0x8(%rbp), which mean 4 bytes into the struct and 0 bytes into the struct. The field names existed for the C compiler and then evaporated.

The library is the mirror image:

1$ objdump -d --no-show-raw-insn libgpu.so.1.0.0 | sed -n '/<gpu_info_get>:/,/ret/p'
200000000000010f9 <gpu_info_get>:
3    10fd:    mov    %rdi,-0x8(%rbp)
4    1101:    mov    -0x8(%rbp),%rax
5    1105:    movl   $0x2a,(%rax)
6    110b:    mov    -0x8(%rbp),%rax
7    110f:    movl   $0x63,0x4(%rax)

It writes 0x2a (42) to offset 0 and 0x63 (99) to offset 4. Neither side names a field. Both sides independently baked the same two numbers into instructions at compile time, and that agreement is the entire contract, the ABI.

Break 1: swap two fields

Now ship a version 2, gpu_v2_bad.c. The author reorders the struct, maybe to group fields logically, and adds a third one:

1struct gpu_info {
2    int temperature;  /* offset 0, was gpu_id */
3    int gpu_id;       /* offset 4, was temperature */
4    int power_usage;  /* offset 8, new */
5};

From the API point of view this looks harmless. Same function name, same struct name, same field names, one addition. Anything recompiled against the new header works. So the author calls it a minor release and leaves the SONAME alone:

1gcc -shared -fPIC -Wl,-soname,libgpu.so.1 -o libgpu.so.1.1.0 gpu_v2_bad.c
2ln -sf libgpu.so.1.1.0 libgpu.so.1

The library's machine code shows what actually changed. The same two constants, now at each other's offsets, plus the new one at offset 8:

1$ objdump -d --no-show-raw-insn libgpu.so.1.1.0 | sed -n '/<gpu_info_get>:/,/ret/p'
200000000000010f9 <gpu_info_get>:
3    1105:    movl   $0x63,(%rax)
4    110f:    movl   $0x2a,0x4(%rax)
5    111a:    movl   $0xfa,0x8(%rax)

Run the old binary. It was never recompiled; it stands in for software already deployed.

1$ LD_LIBRARY_PATH=. ./app
2gpu_id=99 temperature=42

The values are swapped, and each one is plausible on its own: GPU 99 is a fine device ID, and 42 degrees is a fine temperature.

Per offset:

That last row is a write past an 8-byte struct. Here it is harmless: the disassembly above shows main reserved 16 bytes with sub $0x10,%rsp, so the stray 4 bytes land on unused stack. In a function with more locals they land on whatever the compiler put next.

Now ask the tools whether anything is wrong. The exported symbol is identical in both libraries, down to the address:

1$ nm -D --defined-only libgpu.so.1.0.0 | grep gpu_info_get
200000000000010f9 T gpu_info_get
3$ nm -D --defined-only libgpu.so.1.1.0 | grep gpu_info_get
400000000000010f9 T gpu_info_get

And the dependency checker is satisfied, including with -r, which additionally resolves symbol relocations:

 1$ LD_LIBRARY_PATH=. ldd ./app | grep gpu
 2    libgpu.so.1 => ./libgpu.so.1 (0x00007f423b8a0000)
 3$ echo $?
 40
 5$ LD_LIBRARY_PATH=. ldd -r ./app
 6    linux-vdso.so.1 (0x00007fcd3cc97000)
 7    libgpu.so.1 => ./libgpu.so.1 (0x00007fcd3cc8c000)
 8    libc.so.6 => /lib64/libc.so.6 (0x00007fcd3ca00000)
 9    /lib64/ld-linux-x86-64.so.2 (0x00007fcd3cc99000)
10$ echo $?
110

Nothing on stderr, nothing in dmesg, no symbol errors, exit code 0. A file with the requested SONAME exists, and it exports the symbol the binary wants. The loader has no concept of a struct, so there is nothing else for it to check. If the two integers had been a fan speed and a power cap, you would be writing a power cap into a fan speed.

Break 2: move a pointer

Let's try a struct that holds a C string, gpu_v1_ptr.c:

1struct gpu_info {
2    int gpu_id;          /* offset 0 */
3    int temperature;     /* offset 4 */
4    const char *name;    /* offset 8 */
5};

The version 2 author moves the pointer to the front, gpu_v2_ptr_bad.c. This is a real habit, since putting the 8-byte member first is the standard trick for avoiding padding:

1struct gpu_info {
2    const char *name;    /* offset 0 */
3    int gpu_id;          /* offset 8 */
4    int temperature;     /* offset 12 */
5};

Both structs are 16 bytes. Nothing overflows. The only change is where the pointer lives. SONAME unchanged again.

Against version 1 the consumer, main_ptr.c, is fine:

1$ LD_LIBRARY_PATH=. ./app_ptr
2gpu_id=42 temperature=99
3name pointer = 0x7f12996f6000
4name = Tesla T4

Swap in the new library and it crashes:

1$ ln -sf libgpuptr.so.1.1.0 libgpuptr.so.1
2$ LD_LIBRARY_PATH=. ./app_ptr
3gpu_id=1000214528 temperature=32621
4name pointer = 0x630000002a
5Segmentation fault (core dumped)
6$ echo $?
7139

The two integers are now halves of an address. The library wrote the pointer to "Tesla T4" across offsets 0 through 7. The consumer read offset 0 as gpu_id and offset 4 as temperature, splitting one 64-bit pointer down the middle and printing each half as a signed int:

1gpu_id      = 1000214528 = 0x3b9e1000   <- low 4 bytes
2temperature =      32621 = 0x00007f6d   <- high 4 bytes
3recombined  =       0x7f6d3b9e1000

Compare that to the pointer the working run printed, 0x7f12996f6000. Same shape: a 0x7f prefix and a page-aligned tail, because both are addresses inside a mapped shared library. The digits change on every run because ASLR relocates the library.

Offsets 8 through 15 went the other way. The library put gpu_id at 8 and temperature at 12, and the consumer read those 8 bytes as its name pointer:

10x630000002a = (99 << 32) | 42 = (temperature << 32) | gpu_id

Then printf with %s dereferenced address 0x630000002a, which is not mapped, and the kernel killed the process. Exit 139 is 128 + 11, SIGSEGV.

The same one-line change, a field reordering, gave bad data in one struct and a crash in another. The crash landed in the consumer's printf, far from the library that shipped the change. When a segfault shows up in code that has been stable for years, a layout disagreement across a call boundary is one candidate.

The fix: the SONAME is the ABI version

The library-side fix, gpu_v2.c, is one character. Bump the SONAME and keep version 1 installed:

1gcc -shared -fPIC -Wl,-soname,libgpu.so.2 -o libgpu.so.2.0.0 gpu_v2.c
2ln -sf libgpu.so.2.0.0 libgpu.so.2
3ln -sf libgpu.so.1.0.0 libgpu.so.1

Both libraries now sit on disk together. The old binary:

1$ LD_LIBRARY_PATH=. ./app
2gpu_id=42 temperature=99

It asks for libgpu.so.1, gets a library that agrees with it about offsets, and ignores libgpu.so.2. To use version 2, the consumer recompiles against the new header, main_v2.c, which does two things at once: regenerates the offsets in its own machine code, and updates the recorded dependency.

1$ gcc -o app_v2 main_v2.c -L. -lgpu
2$ readelf -d app_v2 | grep 'NEEDED.*gpu'
3 0x0000000000000001 (NEEDED)             Shared library: [libgpu.so.2]
4$ LD_LIBRARY_PATH=. ./app_v2
5gpu_id=42 temperature=99 power_usage=250

Two binaries, same function name, different libraries, both correct. The mechanism depends entirely on a human remembering to bump the number.

The defense: pin the version at dlopen

go-dcgm does not rely on that. Instead of linking normally, it loads the library at runtime by a fully version-qualified name. dlopen_main.c does the same with the toy library:

1/* pin the ABI version, the way go-dcgm pins libdcgm.so.4 */
2void *handle = dlopen("libgpu.so.1", RTLD_NOW);
3if (!handle) {
4    fprintf(stderr, "dlopen: %s\n", dlerror());
5    return 1;
6}
7gpu_info_get_fn fn = (gpu_info_get_fn)dlsym(handle, "gpu_info_get");

The DCGM stub makes the intent explicit. It dlopens a name held in s_libName, which is set to DCGM_LIB_SONAME, and that macro is generated at build time from the project's major version alone:

1#define DCGM_LIB_SONAME "libdcgm.so.@CMAKE_PROJECT_VERSION_MAJOR@"

Only the major version goes into the name. The minor and patch digits, which by convention do not change layout, are left out.

With libgpu.so.1 present, app_dlopen behaves normally. Now remove version 1 so only the incompatible version 2 remains, the upgrade scenario:

 1$ rm libgpu.so.1
 2$ ls libgpu.so*
 3libgpu.so
 4libgpu.so.1.0.0
 5libgpu.so.1.1.0
 6libgpu.so.2
 7libgpu.so.2.0.0
 8$ LD_LIBRARY_PATH=. ./app_dlopen
 9dlopen: libgpu.so.1: cannot open shared object file: No such file or directory
10$ echo $?
111

That error is the feature. libgpu.so.2 is right there, it exports gpu_info_get, and the symbol would resolve fine. The program could have loaded it and printed something. Instead it refuses to start and names the version it wanted.

A process that will not start is a crash loop, a failed health check, a page, all things every system already handles. Data that is quietly wrong is none of those.

Pinning by dlopen also makes the dependency soft. A normally-linked binary will not launch at all when the library is missing, because ld.so resolves NEEDED entries before main runs, the same failure mode as a cgo binary missing its .so. A dlopening binary launches, gets a null handle, and can degrade: skip GPU metrics, report the driver as unavailable, keep serving everything else. For a monitoring agent that runs on both GPU and CPU hosts, that is a requirement.

Conclusion

The compiler bakes offsets into your binary and forgets the field names. Nothing at runtime re-derives them, so two halves of a call boundary can disagree about layout indefinitely with no way to notice.

As a library author, treat the SONAME as the ABI version and bump it whenever the layout changes. As a consumer, dlopen a version-qualified name like libdcgm.so.4, so a mismatch fails loudly at startup.