[syscall] B: Fix diamond DT_NEEDED handling

[esaurez]

Summary

Fixes two latent bugs in the dynamic loader that prevent it from handling diamond DT_NEEDED graphs — the canonical multi-edge shape that production shared-library chains take:

libdiamond.so ─┬─ DT_NEEDED libleft.so  ──┐
               └─ DT_NEEDED libright.so ──┴── DT_NEEDED libbase.so

Both bugs only surface once DT_NEEDED chains are nontrivial. The existing dlfcn-needed-c posix-test exercises a single linear edge, so it never tripped either path. The new esaurez/posix-tests dlfcn-diamond-c suite covers this case.

Bug 1 — dlopen deadlocks on ancestor mutex

load_all_dependencies_recursive holds the parent library's mutex while iterating the registry's already-loaded entries. Each entry gets .lock()-ed to compare names. The previous code skipped only the immediate new_dlhandle self — not ancestors held by outer recursive frames. Loading libright.so (depth 1) and then recursing into its deps (depth 2) tried to lock libdiamond.so in the inner scan, which deadlocked because libdiamond.so was held by the outer frame.

Fix: thread a BTreeSet<DlHandle> of ancestor handles down the recursion and skip every entry whose handle is in that set. The ancestor is by definition not a candidate for "already loaded by a sibling" so the skip is functionally safe.

Bug 2 — dlopen races on transitive loads (diamond duplication)

Once the deadlock is gone, a second issue surfaces: the pre-loop retain only sees libraries loaded before the current frame started. In a diamond, libbase.so is loaded by libleft.so's recursion during libdiamond.so's frame, after libdiamond.so's retain has already run. Without a re-check, the outer loop would then open libbase.so a second time, producing two distinct in-memory copies with their own globals (or tripping the existing unreachable!() when the VFS reuses the file descriptor).

Fix: before each DynamicLibrary::open call, re-scan the registry for a matching name and bind the existing copy if found.

Bug 3 — dlclose over-strict refcount assertion

The BFS unload loop extracts each library from the registry when Arc::strong_count == 2 (registry + current pop). For a diamond, libbase.so has three Arc references at the time it is first popped (registry + libleft.dependencies + libright.dependencies), so extract_if correctly returns 0 entries — but the assertion demanded exactly 1, panicking the kernel. A later iteration of the loop pops libright/libleft, drops their dependencies (releasing the extra libbase ref), pushes libbase again, and the second pop succeeds at the now-correct refcount.

Fix: replace the strict assert_eq!(== 1) with an early continue when extract_if returns zero entries, keeping the (now relaxed) "at most one" assertion to catch any future runtime invariant violation.

What changes

File	Change
src/libs/syscall/src/dlfcn/syscall/dlopen.rs	Thread ancestors: &mut BTreeSet<DlHandle> through load_all_dependencies_recursive; skip ancestor handles in the retain closure and the pre-open re-check; add a pre-open registry re-check for transitive diamond loads.
src/libs/syscall/src/dlfcn/syscall/dlclose.rs	Skip rather than panic when extract_if returns 0 entries; relax the assertion message accordingly.

Net diff: +88 / −6 lines, no API changes.

Validation

• z build all FEATURES=networking LOG_LEVEL=error RELEASE=yes MEMORY_SIZE=256 PASS
• z build format-check, rust-lint-check, spellcheck PASS
• New dlfcn-diamond-c suite — 3/3 PASS on standalone VM:

  === dlfcn diamond DT_NEEDED tests ===
    PASS: dlopen(libright.so) [depth-2 chain]
    PASS: dlopen(libdiamond.so)
    PASS: re-dlopen(libdiamond.so) returns same handle
  3 passed, 0 failed
  

• All previously passing dlfcn tests (dlfcn-c, dlfcn-init-runpath-c, dlfcn-pie-c) still pass.
• Full posix-tests suite passes.

Companion PR

esaurez/posix-tests feat/dlfcn-diamond-needed — the test suite this PR fixes. Without this PR, the diamond test hangs in the loader and times out at 120s.

Why it matters

This bug blocks the entire .a → .so migration. Every real-world .so chain — including the upcoming _lxml_etree.cpython-312.so → libxslt.so → libxml2.so chain on top of esaurez/nanvix#27 — eventually produces a diamond when multiple consumers share a common lower-level library. Without this fix, dlopen of any such .so hangs the guest indefinitely.

Stacking

Stacked on top of esaurez/nanvix#27. Merge order: nanvix#27 first, then this PR.

Downstream consumers

This loader change unblocks the CPython .a → .so migration and its supporting port-libraries:

• esaurez/libxml2#1 — libxml2.so (bottom of the chain).
• esaurez/libxslt#1 — libxslt.so + libexslt.so with DT_NEEDED libxml2.so.
• esaurez/lxml#1 — liblxml_etree.so with full chain.
• esaurez/cpython#11 — Phase 4 dlopen-based lxml integration.

[esaurez]

Superseded by upstream PR nanvix#2478, which carries the same diff (rebased, deep-reviewed, with esaurez/* references stripped). Closing this fork PR; tracking continues upstream.