Dynamic linker resolution and glibc version conflicts inside portable AppImage payloads

Package Formats8 min

Relocation Aborts Before main on an Old Host

The dynamic linker aborts before main. A payload that launched cleanly on the build machine dies on the workstation, and the only line the user sees is a version-not-found message from ld-linux.

While debugging a continuous-integration artifact that failed to launch on enterprise workstations, we traced startup with LD_DEBUG=files,versions. The host dynamic linker aborted in the initial relocation phase—before the process ever entered main()—because the build container's toolchain had bound ordinary C functions to symbol versions that workstation libc did not export.

The compatibility floor is the glibc the payload was linked against. Leave libc.so.6 and ld-linux on the host. Point the payload at its own non-glibc libraries without handing that search path to every child the app later execs.

Field failures split along the host-versus-bundle axis.

  • Pre-main relocation abort: host ld-linux rejects an unexported GLIBC_2.xx node, including a GLIBC_2.34 node stamped into .gnu.version_r, and never reaches main.
  • Mid-process dlopen() crash: a bundled libc tries to load the host's /lib64/libnss_dns.so.2 across incompatible internal glibc structs.
  • Child-process poisoning: xdg-open inherits the mount's LD_LIBRARY_PATH and loads an older bundled libstdc++.so.6.

On an older host the exec fails immediately on the missing version. On a newer host an old binary usually starts, and the break shows up later, when bundled libraries meet the host linker, NSS modules, or a spawned helper.

The abort lands on the order of 2 to 5 milliseconds after the execve() syscall. stderr receives a line in this shape: ./AppRun: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found. That gap spans in the ballpark of 24 to 60 months when the compile happens on a rolling-release developer workstation and the install target is a long-term support enterprise distribution.

GLIBC_ Node Miss

A version node the host libc does not export ends the process in relocation. The AppImage wrapper never gets a chance to paper over it. Change the glibc the link step saw.

Search Order After the SquashFS Mount

Once the runtime has mounted the payload, PT_INTERP is still the host dynamic linker unless a packager rewrote the interpreter path. That host ld-linux then walks directories in a fixed sequence, and the AppImage's own library tree only wins where that sequence says it wins.

The ld.so(8) search order is strict: DT_RPATH only when DT_RUNPATH is absent, then LD_LIBRARY_PATH, then DT_RUNPATH, then /etc/ld.so.cache, then the default lib directories (/lib64, /usr/lib64).

Search Order After the SquashFS Mount

The runtime path beats a carefully stamped runpath. Default AppImage runtime logic prepends its mount directories to LD_LIBRARY_PATH. Because that variable is consulted before DT_RUNPATH, a $ORIGIN RUNPATH does not override the runtime path. Stamping DT_RUNPATH onto every ELF inside the AppDir with patchelf looks clean in the build log and still loses to the prepended mount path.

After the mount, $ORIGIN stays inside the payload tree. The runtime mounts the SquashFS image at /tmp/.mount_AppXXXX. A binary sitting at usr/bin/app resolves $ORIGIN/../lib straight to /tmp/.mount_AppXXXX/usr/lib, with no extra environment override required. Extract the AppImage and the same relative path still lands in the payload's usr/lib, because the binary still lives under the extracted usr/bin.

A static binary, or a payload linked against musl with a self-contained interpreter, bypasses this host glibc search order, and it also gives up hardware-accelerated graphics drivers that depend on host dlopen().

Version Nodes Stamped by the Linker

Versioned symbols are requirements written into the ELF at link time. readelf -V and objdump -T display them. Nothing in the AppImage download step renegotiates those nodes.

The direction of compatibility is one-way. A newer host glibc still exports the older GLIBC_2.xx versions, so a payload linked against an older glibc generally starts on a newer host. A payload that names a version the host libc does not export is rejected at symbol lookup, which is the pre-main abort above.

Ceiling on the Binary, Banner on the Host

Read the ceiling before the SquashFS image is built. The highest GLIBC_ string in .gnu.version_r is the floor ld stamped at the final link:

readelf -V binary | grep -o 'GLIBC_[0-9.]*' | sort -V | tail -n 1

Run that over the primary executable and every bundled shared object. An automated post-link step can parse .gnu.version_r across that set, pull the highest required GLIBC_ string, and compare it with a declarative target manifest before the image is packed.

Then check the oldest host you still claim. Execute that machine's libc.so.6 as a program. /lib/x86_64-linux-gnu/libc.so.6 prints its own banner, for example GNU C Library stable release version 2.31, in under 10 milliseconds. If the binary's ceiling sits above that banner, the payload will die in relocation on that host.

Readelf Ceiling

Treat the highest GLIBC_ token from readelf -V as the shipping floor. Compare it with the banner from the oldest target's libc.so.6, not with the glibc on the packager's laptop.

Keep the glibc Triad on the Host

The AppDir lib directory has a hard exclusion list on x86_64. These files stay on the host filesystem:

  • libc.so.6
  • ld-linux-x86-64.so.2 (and the loader soname for any other architecture you ship)
  • libpthread.so.0
  • libdl.so.2
  • librt.so.1
  • libresolv.so.2
  • libutil.so.1
  • every libnss_*.so.2 module

NSS modules for users and DNS must be loaded by the same glibc the process is already using. A private libc.so.6 cannot consume the host's libnss modules, so account lookups and name resolution fail even after the binary has started.

A dependency-copier that scrapes ldd and drops libc.so.6 plus libnss_files.so.2 into the AppDir will boot on a test VM and then fail network socket connections. strace on that process shows dlopen() of the host path /lib64/libnss_dns.so.2 against the bundled libc.so.6. The module and the C library disagree on internal glibc structs, and resolution dies there.

GCC Runtime Can Travel

Draw the packaging line at the GCC runtime, not at "every.so ldd printed." libstdc++.so.6 and libgcc_s.so.1 can live under $ORIGIN/../lib and satisfy newer GLIBCXX_3.4.xx nodes the host lacks. That pair bridges C++ ABI gaps across 3 to 7 major GCC releases: a bundle providing GLIBCXX_3.4.29 starts on a host whose libstdc++ stops at GLIBCXX_3.4.25, and the host C library stays untouched.

Host Triad Line

libstdc++.so.6 and libgcc_s.so.1 may ride in the payload. libc.so.6, ld-linux-x86-64.so.2, and libnss_*.so.2 stay exclusively on the host. Bundling the C++ runtime is never a reason to bundle libc.

Children Inherit the Runtime Library Path

The same LD_LIBRARY_PATH that lets the payload find its libstdc++, SSL, or image libraries is exported into the environment. Helpers, packaged plugins, and any program the app execs inherit it. Those children then prefer the bundle's copy over the host's.

A packaged desktop client that hands external links to the default browser shows the leak without any extra instrumentation. Inspect /proc/$PID/environ on the spawned xdg-open and the AppImage path is already there, passed through the host's /bin/sh into the browser binary. The browser loads the bundle's older libstdc++.so.6 and the click appears to do nothing.

The mismatch is concrete. A host child such as /usr/bin/xdg-open or /usr/bin/git aborts when it requires GLIBCXX_3.4.30 and the AppImage ships a libstdc++.so.6 capped at GLIBCXX_3.4.28. Worse, a bundled libc.so.6 left in that directory is loaded by the child's own interpreter. The host binary's PT_INTERP is /lib64/ld-linux-x86-64.so.2. That host linker maps the payload's libc and dies on _dl_allocate_tls_init or a symbol lookup failure inside ld.so, within 1 millisecond of execve().

Narrow the Path Before exec

Keep glibc filenames out of the directory the runtime places on LD_LIBRARY_PATH. The lookup misses those names and falls through to the host copies. Before spawning helpers, unset the variable or replace it with a path that contains only the libraries that child is supposed to see.

env -u LD_LIBRARY_PATH /usr/bin/xdg-open "$url"

Plugins that must see the bundle's libstdc++ stay inside the payload process. Host tools get a clean environment. That split is the practical control for the third failure shape: child-process poisoning.

glibc 2.34, released in August 2021, folded libpthread, libdl, librt, and libutil into libc.so.6. Empty stub archives remain so old static links still resolve. The shared objects those names used to be are no longer where new dynamic links record their requirements.

Moving a build runner between two long-term support container images, released two years apart, made binaries from identical C source stop launching on older hosts. objdump -T on the two builds showed pthread_create and dlopen no longer bound to separate libpthread.so.0 and libdl.so.2 stubs. The new link recorded them against libc itself.

Calls that had been pthread_create@GLIBC_2.2.5 and dlopen@GLIBC_2.2.5 were promoted to pthread_create@@GLIBC_2.34 and dlopen@@GLIBC_2.34. A payload linked on a distribution that already ships that glibc carries GLIBC_2.34 and newer requirements. The older host linker rejects those nodes before main, even when the source never called a new API on purpose. Leaving libpthread.so.0 out of the AppDir is still correct on both sides of that merge: pre-2.34 hosts load the real host libpthread with the host libc, and post-2.34 hosts have already absorbed it.

Link in the build chroot that holds the oldest glibc you still claim. The mounted payload then has a loader and a libc that already agree. The same source, linked after that August 2021 merge, promotes pthread_create@GLIBC_2.2.5 to pthread_create@@GLIBC_2.34 and raises the ELF link floor by 19 minor glibc versions.

Stay Updated

Be the first to know.

We respect your privacy. No spam.

Your Thoughts

Nothing here yet. Add your opinion.

Write a Comment

Your cookie choices