[build] E: Build liblzma.so alongside liblzma.a by esaurez · Pull Request #1 · esaurez/xz

esaurez · 2026-06-10T02:04:51Z

Summary

Builds liblzma.so alongside the existing liblzma.a so consumers can dlopen xz at run time instead of statically linking it.

Why

Today the Nanvix xz port ships only liblzma.a. Anything that wants xz (e.g., cpython's _lzma extension) has to statically embed it into the consumer binary. That works for cpython's monolithic python.elf but does not allow:

Sharing a single liblzma implementation across multiple .so consumers in the same process.
Building a cpython _lzma.so that resolves xz via DT_NEEDED liblzma.so (the model already used by _ssl.so → libssl.so and _ctypes.so → libffi.so on Nanvix).

Producing liblzma.so here unblocks the cpython migration of _lzma from --whole-archive .a in python.elf to DT_NEEDED .so.

What changed

xz uses autotools and the upstream --enable-shared path goes through libtool, which does not know about i686-nanvix. Rather than teaching libtool a new platform target, this PR keeps --disable-shared --enable-static and links liblzma.so manually from the .a after make install completes, the same approach already taken by nanvix/libffi#235.

Concretely, .nanvix/z.py:

_configure_env_overrides: appends -fPIC to CFLAGS so the same .o files compile into both .a and .so.
build (in-Docker shell script):
- Strips \r from the vendored autotools scripts before running ./configure. Required on Windows hosts where git's core.autocrlf=true converts shell scripts to CRLF and dash inside the container chokes on the trailing \r (no-op on Unix hosts where the files already have LF endings).
- Adds a gcc -shared invocation after make install DESTDIR=... that links liblzma.so from src/liblzma/.libs/liblzma.a via -Wl,--whole-archive with DT_SONAME=liblzma.so and -nostdlib (libc / libm UND, bind at dlopen). Writes the result into <install-stage>/sysroot/lib/liblzma.so.
_stage_artefacts: extends the lib_dir / build_dir copy-out to cover liblzma.so in addition to liblzma.a.
_stage_release_outputs: copies liblzma.so into lib_out() so ./z release packages it.

Architecture

liblzma.so          (PIC-recompiled from liblzma.a)
  └── UND libc / libm symbols (memcpy, memset, malloc, ...)
      → resolved against the host executable's .dynsym at dlopen time

liblzma.a (unchanged shape; same .o files now compiled with -fPIC)

liblzma.so has no transitive .so dependencies (liblzma is self-contained; libc/libm symbols are UND), so the loader does not need to walk any DT_NEEDED chain to load it.

Dependencies

Runtime dep (already merged upstream):

nanvix/nanvix#2473 — dlfcn init-array + DT_RUNPATH support. Required for any consumer to dlopen("liblzma.so") at run time.

Validation

Build runs end-to-end inside the toolchain-gcc Docker image on a Windows host:

$ ./z build
... (compiles liblzma .c files with -fPIC) ...
make install DESTDIR=/mnt/workspace/build/_install
i686-nanvix-gcc -shared -fPIC -nostdlib \
    -Wl,-soname,liblzma.so -Wl,-z,noexecstack \
    -Wl,--whole-archive src/liblzma/.libs/liblzma.a -Wl,--no-whole-archive \
    -o /mnt/workspace/build/_install/sysroot/lib/liblzma.so
... (upstream tests build) ...
info: Staged artefacts under D:\src\xz-dev\build
info: Staged release outputs under D:\src\xz-dev\.nanvix\out\release
success: Build complete

Structural verification of the produced liblzma.so:

$ i686-nanvix-readelf -d build/liblzma.so | grep -E 'SONAME|NEEDED'
 0x0000000e (SONAME)                     Library soname: [liblzma.so]

$ i686-nanvix-nm -D build/liblzma.so | grep ' T lzma_' | head -5
00007c40 T lzma_alone_decoder
00004ee0 T lzma_alone_encoder
00007fe0 T lzma_auto_decoder
000055f0 T lzma_block_buffer_bound
00008070 T lzma_block_buffer_decode

SONAME correctly set, no spurious DT_NEEDED, full lzma_* public API exported, libc symbols left UND for runtime binding. The existing liblzma.a and the upstream test_*.elf test binaries continue to build unchanged.

Completes the external-library unbundling started in the parent PR by moving the remaining four stdlib extensions (_bz2, _lzma, zlib, _sqlite3) from --whole-archive of the .a into python.elf to DT_NEEDED of standalone .so files under sysroot/lib/. After this PR, python.elf bundles no external Nanvix-ported libraries at all -- every .so wrapper (_ssl, _hashlib, _ctypes, _sqlite3, zlib, _bz2, _lzma) consumes its underlying library through DT_NEEDED. Changes: - Makefile.nanvix: - LIBS drops "-Wl,--whole-archive $(NANVIX_WHOLE_ARCHIVE_LIBS) -Wl, --no-whole-archive" (and removes the NANVIX_WHOLE_ARCHIVE_LIBS variable definition). python.elf no longer carries libsqlite3.a / libz.a / libbz2.a / liblzma.a. - Per-module *_LIBS env vars restored to their -L/-l form (LIBSQLITE3_LIBS, ZLIB_LIBS, BZIP2_LIBS, LIBLZMA_LIBS). With the .so files present in $(SYSROOT_PATH)/lib, ld resolves each one against its .so during the per-module link and emits a DT_NEEDED entry rather than embedding the symbols. - .nanvix/runtime_sos.py: REQUIRED_RUNTIME_SOS extended with libbz2.so, liblzma.so, libz.so, libsqlite3.so so .nanvix/test.py and .nanvix/package.py stage them into sysroot/lib/ at run time. - .nanvix/z.py: _DEP_EXPECTED_LIBS for bzip2, libffi, zlib, sqlite, xz extended with the .so siblings so `./z setup` validates that the new shared-library port builds installed correctly into buildroot/lib. Runtime dependencies (Wave 6 port-repo PRs that produce the .so files this PR consumes): - esaurez/bzip2#2 -- libbz2.so build target - esaurez/zlib#1 -- libz.so build target - esaurez/xz#1 -- liblzma.so build target - esaurez/sqlite#1 -- libsqlite3.so build target These four PRs must land and their port-lib releases must be pinned in cpython's nanvix.toml before this PR is mergeable upstream. Also inherits the runtime deps already required by the parent PR: nanvix/nanvix#2473 (dlopen init-array + DT_RUNPATH) and the upstream diamond DT_NEEDED loader fix + libposix pthread_once fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Completes the external-library unbundling started in the parent PR by moving the remaining four stdlib extensions (_bz2, _lzma, zlib, _sqlite3) from being statically linked into python.elf to runtime- loaded .so files that emit DT_NEEDED for libbz2.so / liblzma.so / libz.so / libsqlite3.so. After this PR, python.elf bundles no external Nanvix-ported libraries at all -- every .so wrapper (_ssl, _hashlib, _ctypes, _sqlite3, zlib, _bz2, _lzma) consumes its underlying library through DT_NEEDED. This matches upstream cpython's default behavior on every Linux distro (system-library detection enabled). Changes ------- - .nanvix/setup_local.py: add 4 *shared* entries (_bz2, _lzma, zlib, _sqlite3). The corresponding LIBSQLITE3_LIBS / ZLIB_LIBS / BZIP2_LIBS / LIBLZMA_LIBS already use -L/-l form (no change needed there); cpython's normal MODULE_*_LDFLAGS machinery wires them into each .so's link line as DT_NEEDED. - Makefile.nanvix: drop -lsqlite3 -lz -lbz2 -llzma from python.elf's LIBS (those symbols come from the standalone .so files now). LIBS segment 2 becomes empty. - .nanvix/runtime_sos.py: REQUIRED_RUNTIME_SOS gains libbz2.so / liblzma.so / libz.so / libsqlite3.so so .nanvix/test.py and .nanvix/package.py stage them into sysroot/lib/ at run time. - .nanvix/z.py: _DEP_EXPECTED_LIBS for bzip2 / zlib / sqlite / xz extended with the .so siblings so `./z setup` validates that the new shared-library port builds installed correctly into buildroot/lib. - .nanvix/test.py: extend CPYTHON_TEST_EXTERNAL_DEPS smoke test with the 4 new modules. Runtime dependencies -------------------- - nanvix/nanvix#2472 -- libm visibility fix (inherited from PR-A). - nanvix/nanvix#2473 -- dlfcn init-array + DT_RUNPATH support (inherited from PR-B). The 4 new .so files this PR consumes come from the Wave 6 port-repo PRs that must land first: - esaurez/bzip2#2 -- libbz2.so build target - esaurez/zlib#1 -- libz.so build target - esaurez/xz#1 -- liblzma.so build target - esaurez/sqlite#1 -- libsqlite3.so build target These four port-lib releases must be pinned in cpython's nanvix.toml before this PR is mergeable upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

esaurez · 2026-06-17T01:37:18Z

+            "-o -name 'missing' -o -name 'compile' -o -name 'ar-lib' "
+            "-o -name 'ltmain.sh' \\) -exec sed -i 's/\\r$//' {} + "
+            "2>/dev/null || true"
+        )


This is confusing, this should only happen if your local setup is incorrect and makes changes to files incorrectly, this probably shouldn't be part of this PR

esaurez · 2026-06-17T01:37:48Z

                configure_cmd,
                f"make -j{nproc}",
                f"make install DESTDIR={shlex.quote(str(stage_container))}",
+                # Build liblzma.so from the (PIC) liblzma.a libtool produced.


is this the right way to produce this?

Adds a liblzma.so link step inside the existing build script that links liblzma.so from the (now PIC) static archive via -Wl,--whole-archive so every liblzma entry point becomes part of the .so's .dynsym. Sets DT_SONAME=liblzma.so so consumers that link against it emit a proper DT_NEEDED entry. liblzma is self-contained -- no transitive .so deps -- so the .so records no DT_NEEDED of its own. libc / libm symbols (memcpy, memset, malloc, ...) are left UND via -nostdlib and bind at dlopen time against the host executable's .dynsym, matching the model already used by other Nanvix shared libraries such as libffi.so / libssl.so. xz's autotools libtool does not know about i686-nanvix, so the upstream --enable-shared path is not viable. The configure opts keep --disable-shared --enable-static; the .so is linked manually from the .a after `make install` completes. Changes to .nanvix/z.py: - _configure_env_overrides: append -fPIC to CFLAGS so the same .o files compile into both .a and .so. - build: add a SHAREDLIB link step after `make install DESTDIR=...` that writes liblzma.so directly into the install staging tree alongside liblzma.a. - _stage_artefacts: extend the lib_dir / build_dir copy-out to cover liblzma.so in addition to liblzma.a. - _stage_release_outputs: copy liblzma.so into lib_out() so `./z release` packages it. Adds .gitattributes forcing LF line endings on the vendored Unix shell / autotools scripts (configure, config.*, install-sh, build-aux/*, *.sh, ...). These are already committed as LF; the attribute keeps them LF on checkout regardless of a contributor's core.autocrlf setting, so dash inside the Linux toolchain container does not choke on CRLF. This replaces an earlier runtime CR-stripping workaround that was carried in the build script. Runtime dependency: the shared-library build becomes useful once the loader changes in nanvix/nanvix#2473 ([syscall] E: Run dlopen ctors/dtors and DT_RUNPATH) ship. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

esaurez mentioned this pull request Jun 10, 2026

[nanvix] E: Build bz2/lzma/zlib/sqlite3 as DT_NEEDED .so chains esaurez/cpython#20

Open

esaurez force-pushed the feat/build-shared-library branch from bdee559 to 805b7c2 Compare June 17, 2026 01:15

esaurez commented Jun 17, 2026

View reviewed changes

esaurez force-pushed the feat/build-shared-library branch from 805b7c2 to 76b6b97 Compare June 17, 2026 01:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build] E: Build liblzma.so alongside liblzma.a#1

[build] E: Build liblzma.so alongside liblzma.a#1
esaurez wants to merge 1 commit into
nanvix/v5.2.5from
feat/build-shared-library

esaurez commented Jun 10, 2026 •

edited

Loading

Uh oh!

esaurez Jun 17, 2026

Uh oh!

esaurez Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

esaurez commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

What changed

Architecture

Dependencies

Validation

Uh oh!

esaurez Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

esaurez Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

esaurez commented Jun 10, 2026 •

edited

Loading