Skip to content

Commit 33e549e

Browse files
authored
feat: add musl libc-test conformance suite and rename posix-* to os-test-* (#56)
Rename existing POSIX conformance infrastructure to os-test-* to reflect the actual test suite name (Sortix os-test). Add musl libc-test as a second conformance suite that tests kernel-level behavior (file locking, stat edge cases, socket operations) rather than just libc function correctness. - Rename posix-conformance → os-test-conformance across all files - Rename posix-exclusion-schema → conformance-exclusion-schema (shared) - Add import, build, test, validate, and report infrastructure for libc-test - 75 libc-test programs compile for WASM, 69 pass (92.0%) - 6 exclusions: 4 wasm-limitation (dlopen/TLS), 1 wasi-gap (statvfs), 1 implementation-gap (strptime)
1 parent 32120d4 commit 33e549e

23 files changed

+1933
-62
lines changed
Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
name: libc-test Conformance
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- "native/wasmvm/**"
9+
- "packages/wasmvm/**"
10+
- "scripts/validate-libc-test-exclusions.ts"
11+
- "scripts/generate-libc-test-report.ts"
12+
- "scripts/import-libc-test.ts"
13+
- "scripts/conformance-exclusion-schema.ts"
14+
- ".github/workflows/libc-test-conformance.yml"
15+
pull_request:
16+
branches:
17+
- main
18+
paths:
19+
- "native/wasmvm/**"
20+
- "packages/wasmvm/**"
21+
- "scripts/validate-libc-test-exclusions.ts"
22+
- "scripts/generate-libc-test-report.ts"
23+
- "scripts/import-libc-test.ts"
24+
- "scripts/conformance-exclusion-schema.ts"
25+
- ".github/workflows/libc-test-conformance.yml"
26+
27+
jobs:
28+
libc-test-conformance:
29+
name: libc-test Conformance (musl kernel behavior)
30+
runs-on: ubuntu-latest
31+
steps:
32+
- name: Checkout repository
33+
uses: actions/checkout@v4
34+
35+
# --- Rust / WASM build ---
36+
- name: Set up Rust toolchain
37+
uses: dtolnay/rust-toolchain@nightly
38+
with:
39+
toolchain: nightly-2026-03-01
40+
targets: wasm32-wasip1
41+
components: rust-src
42+
43+
- name: Install wasm-opt (binaryen)
44+
run: sudo apt-get update && sudo apt-get install -y binaryen
45+
46+
- name: Cache WASM build artifacts
47+
uses: actions/cache@v4
48+
with:
49+
path: |
50+
native/wasmvm/target
51+
native/wasmvm/vendor
52+
key: wasm-${{ runner.os }}-${{ hashFiles('native/wasmvm/Cargo.lock', 'native/wasmvm/rust-toolchain.toml') }}
53+
54+
- name: Build WASM binaries
55+
run: cd native/wasmvm && make wasm
56+
57+
# --- C toolchain (wasi-sdk + patched sysroot) ---
58+
- name: Cache wasi-sdk
59+
id: cache-wasi-sdk
60+
uses: actions/cache@v4
61+
with:
62+
path: native/wasmvm/c/vendor/wasi-sdk
63+
key: wasi-sdk-25-${{ runner.os }}-${{ runner.arch }}
64+
65+
- name: Download wasi-sdk
66+
if: steps.cache-wasi-sdk.outputs.cache-hit != 'true'
67+
run: make -C native/wasmvm/c wasi-sdk
68+
69+
- name: Cache patched wasi-libc sysroot
70+
id: cache-sysroot
71+
uses: actions/cache@v4
72+
with:
73+
path: |
74+
native/wasmvm/c/sysroot
75+
native/wasmvm/c/vendor/wasi-libc
76+
key: wasi-libc-sysroot-${{ runner.os }}-${{ hashFiles('native/wasmvm/patches/wasi-libc/*.patch', 'native/wasmvm/scripts/patch-wasi-libc.sh') }}
77+
78+
- name: Build patched wasi-libc sysroot
79+
if: steps.cache-sysroot.outputs.cache-hit != 'true'
80+
run: make -C native/wasmvm/c sysroot
81+
82+
# --- Build libc-test (WASM + native) ---
83+
- name: Build libc-test binaries (WASM + native)
84+
run: make -C native/wasmvm/c libc-test libc-test-native
85+
86+
# --- Node.js / TypeScript ---
87+
- name: Set up pnpm
88+
uses: pnpm/action-setup@v4
89+
with:
90+
version: 8.15.6
91+
92+
- name: Set up Node.js
93+
uses: actions/setup-node@v4
94+
with:
95+
node-version: 22
96+
cache: pnpm
97+
cache-dependency-path: pnpm-lock.yaml
98+
99+
- name: Install dependencies
100+
run: pnpm install --frozen-lockfile
101+
102+
# --- Run conformance tests ---
103+
- name: Run libc-test conformance tests
104+
run: pnpm vitest run packages/wasmvm/test/libc-test-conformance.test.ts
105+
106+
- name: Validate exclusion list
107+
run: pnpm tsx scripts/validate-libc-test-exclusions.ts
108+
109+
# --- Generate report ---
110+
- name: Generate conformance report MDX
111+
if: always()
112+
run: pnpm tsx scripts/generate-libc-test-report.ts
113+
114+
# --- Upload artifacts ---
115+
- name: Upload conformance report
116+
if: always()
117+
uses: actions/upload-artifact@v4
118+
with:
119+
name: libc-test-conformance-report
120+
path: |
121+
libc-test-conformance-report.json
122+
docs/libc-test-conformance-report.mdx
Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: POSIX Conformance
1+
name: os-test Conformance
22

33
on:
44
push:
@@ -7,26 +7,26 @@ on:
77
paths:
88
- "native/wasmvm/**"
99
- "packages/wasmvm/**"
10-
- "scripts/validate-posix-exclusions.ts"
11-
- "scripts/generate-posix-report.ts"
10+
- "scripts/validate-os-test-exclusions.ts"
11+
- "scripts/generate-os-test-report.ts"
1212
- "scripts/import-os-test.ts"
13-
- "scripts/posix-exclusion-schema.ts"
14-
- ".github/workflows/posix-conformance.yml"
13+
- "scripts/conformance-exclusion-schema.ts"
14+
- ".github/workflows/os-test-conformance.yml"
1515
pull_request:
1616
branches:
1717
- main
1818
paths:
1919
- "native/wasmvm/**"
2020
- "packages/wasmvm/**"
21-
- "scripts/validate-posix-exclusions.ts"
22-
- "scripts/generate-posix-report.ts"
21+
- "scripts/validate-os-test-exclusions.ts"
22+
- "scripts/generate-os-test-report.ts"
2323
- "scripts/import-os-test.ts"
24-
- "scripts/posix-exclusion-schema.ts"
25-
- ".github/workflows/posix-conformance.yml"
24+
- "scripts/conformance-exclusion-schema.ts"
25+
- ".github/workflows/os-test-conformance.yml"
2626

2727
jobs:
28-
posix-conformance:
29-
name: POSIX Conformance (os-test)
28+
os-test-conformance:
29+
name: os-test Conformance (POSIX.1-2024)
3030
runs-on: ubuntu-latest
3131
steps:
3232
- name: Checkout repository
@@ -100,23 +100,23 @@ jobs:
100100
run: pnpm install --frozen-lockfile
101101

102102
# --- Run conformance tests ---
103-
- name: Run POSIX conformance tests
104-
run: pnpm vitest run packages/wasmvm/test/posix-conformance.test.ts
103+
- name: Run os-test conformance tests
104+
run: pnpm vitest run packages/wasmvm/test/os-test-conformance.test.ts
105105

106106
- name: Validate exclusion list
107-
run: pnpm tsx scripts/validate-posix-exclusions.ts
107+
run: pnpm tsx scripts/validate-os-test-exclusions.ts
108108

109109
# --- Generate report ---
110110
- name: Generate conformance report MDX
111111
if: always()
112-
run: pnpm tsx scripts/generate-posix-report.ts
112+
run: pnpm tsx scripts/generate-os-test-report.ts
113113

114114
# --- Upload artifacts ---
115115
- name: Upload conformance report
116116
if: always()
117117
uses: actions/upload-artifact@v4
118118
with:
119-
name: posix-conformance-report
119+
name: os-test-conformance-report
120120
path: |
121-
posix-conformance-report.json
122-
docs/posix-conformance-report.mdx
121+
os-test-conformance-report.json
122+
docs/os-test-conformance-report.mdx

CLAUDE.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
### POSIX Conformance Test Integrity
3636

3737
- **no test-only workarounds** — if a C override fixes broken libc behavior (fcntl, realloc, strfmon, etc.), it MUST go in the patched sysroot (`native/wasmvm/patches/wasi-libc/`) so all WASM programs get the fix; never link overrides only into test binaries — that inflates conformance numbers while real users still hit the bug
38-
- **never replace upstream test source files** — if an os-test `.c` file fails due to a platform difference (e.g. `sizeof(long)`), exclude it via `posix-exclusions.json` with the real reason; do not swap in a rewritten version that changes what the test validates
38+
- **never replace upstream test source files** — if an os-test `.c` file fails due to a platform difference (e.g. `sizeof(long)`), exclude it via `os-test-exclusions.json` with the real reason; do not swap in a rewritten version that changes what the test validates
3939
- **kernel behavior belongs in the kernel, not the test runner** — if a test requires runtime state (POSIX directories like `/tmp`, `/usr`, device nodes, etc.), implement it in the kernel/device-layer so all users get it; the test runner should not create kernel state that real users won't have
4040
- **no suite-specific VFS special-casing** — the test runner must not branch on suite name to inject different filesystem state; if a test needs files to exist, either the kernel should provide them or the test should be excluded
4141
- **categorize exclusions honestly** — if a failure is fixable with a patch or build flag, it's `implementation-gap`, not `wasm-limitation`; reserve `wasm-limitation` for things genuinely impossible in wasm32-wasip1 (no 80-bit long double, no fork, no mmap)
@@ -90,7 +90,7 @@
9090
## GitHub Issues
9191

9292
- when fixing a bug or implementation gap tracked by a GitHub issue, close the issue in the same PR using `gh issue close <number> --comment "Fixed in <commit-hash>"`
93-
- when removing a test from `posix-exclusions.json` because the fix landed, close the linked issue
93+
- when removing a test from `os-test-exclusions.json` or `libc-test-exclusions.json` because the fix landed, close the linked issue
9494
- do not leave resolved issues open — verify with `gh issue view <number>` if unsure
9595

9696
## Tool Integration Policy

docs-internal/posix-gaps-audit.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# POSIX Implementation Gaps Audit
2+
3+
> Adversarial review of the WasmVM POSIX implementation. Goal: identify what breaks when real software runs unmodified.
4+
>
5+
> os-test conformance: 3347/3350 (99.9%) — but os-test only covers C library functions, not kernel/runtime behavior.
6+
>
7+
> Each claim was verified by independent adversarial agents reading the actual source code.
8+
9+
## WILL_BREAK — Real software fails
10+
11+
All 11 claims verified TRUE against source code.
12+
13+
| # | Issue | What breaks | Where | Verified |
14+
|---|-------|-------------|-------|----------|
15+
| 1 | **No server sockets** (bind/listen/accept) | nginx, Express, Redis, Postgres, any daemon | wasi-ext has no bind/listen/accept imports; 0008-sockets.patch is client-only | TRUE |
16+
| 2 | **No Unix domain sockets** (AF_UNIX) | docker.sock, ssh-agent, systemd socket activation | net_connect takes "host:port" string, no AF_UNIX path | TRUE |
17+
| 3 | **O_EXCL not checked in fdOpen** | Atomic lock file creation (SQLite, Make, pkg managers) | kernel.ts fdOpen only handles O_CREAT; O_EXCL stored but never checked | TRUE |
18+
| 4 | **O_TRUNC not applied in kernel fdOpen** | Shell `>` redirect, log rotation, any "w" mode open | kernel.ts fdOpen never truncates; Node bridge does but WasmVM kernel doesn't | TRUE |
19+
| 5 | **readdir missing "." and ".."** | tar, find, shell globbing, POSIX compliance | in-memory-fs.ts listDirEntries never synthesizes "." or ".." | TRUE |
20+
| 6 | **Blocking flock() returns EAGAIN immediately** | File-based locks (databases, build tools) | file-lock.ts:61 comment: "Blocking not implemented — treat as EAGAIN" | TRUE |
21+
| 7 | **Pipe write EAGAIN without O_NONBLOCK** | Large pipelines (`tar \| gzip \| aws s3 cp`) | pipe-manager.ts:107 throws EAGAIN on full buffer; no retry in fd_write handler | TRUE |
22+
| 8 | **Unlink open file doesn't defer delete** | Temp file pattern (create, unlink, keep writing) | in-memory-fs.ts removeFile unconditionally deletes; no refcount check | TRUE |
23+
| 9 | **No signal handlers** (sigaction/signal) | Servers, databases, graceful shutdown | No sigaction syscall; signals delivered by default actions only | TRUE |
24+
| 10 | **WASM can't be interrupted mid-compute** | Ctrl+C during tight loops hangs | Tight loops bypass Atomics.wait; worker.terminate() is hard kill | TRUE |
25+
| 11 | **All inodes are 0** | Hard link detection, backup tools, `find -inum` | in-memory-fs.ts lines 156,172,332 hardcode ino:0; no inode allocator | TRUE |
26+
27+
## TOO_THIN — Works for simple cases, breaks complex ones
28+
29+
Fact-check found 4 claims were FALSE or EXAGGERATED.
30+
31+
| # | Issue | What breaks | Verified | Notes |
32+
|---|-------|-------------|----------|-------|
33+
| 12 | **O_NONBLOCK not settable via fcntl F_SETFL** | Async I/O, event loops | ~~TRUE~~ **FALSE** | WasmVM fcntl.c DOES implement F_SETFL via __wasi_fd_fdstat_set_flags; only Node kernel lacks it |
34+
| 13 | **PIPE_BUF atomicity not guaranteed** | Concurrent pipe writers | ~~TRUE~~ **FALSE** | JS is single-threaded — all pipe writes are atomic by definition; no interleaving possible |
35+
| 14 | **pread/pwrite load entire file into memory** | Large file random access | ~~TRUE~~ **FALSE** | It's an InMemoryFS — files are already in memory; pread just slices a view. This is by design, not a bug |
36+
| 15 | **/dev/ptmx stub** (doesn't allocate real PTY) | `script`, PTY-allocation | ~~TRUE~~ **FALSE** | Real PtyManager exists with full master/slave pairs, line discipline, and signal delivery. /dev/ptmx device-layer entry is VFS stub but PTY allocation happens via kernel ioctl |
37+
| 16 | **poll() timeout -1 capped to 30s** | Event loops expecting indefinite blocking | **TRUE** | driver.ts:1098 hardcodes `timeout < 0 ? 30000 : timeout` |
38+
| 17 | **No UDP sockets** | ping, DNS raw queries, DHCP | **TRUE** | SOCK_DGRAM accepted but silently creates TCP socket stub |
39+
| 18 | **Socket send/recv ignore flags** (MSG_PEEK, MSG_DONTWAIT) | Protocol libraries | **TRUE** | Flags passed via RPC but never used in driver handlers |
40+
| 19 | **setsockopt not implemented** (SO_REUSEADDR, TCP_NODELAY) | Port reuse, latency tuning | **TRUE** | kernel-worker.ts returns ENOSYS unconditionally |
41+
| 20 | **Hard links don't increment nlink** | `ls -l`, backup dedup tools | **TRUE** | hardLinks Map tracked but stat always returns nlink:1 |
42+
| 21 | **pthread_cond/barrier/rwlock/once not patched** | Python GIL, any C++ std::thread code | **TRUE** | Only mutex/key/attr patched; cond/barrier/rwlock/once use unpatched musl stubs that assume futex |
43+
| 22 | **No iconv()** | Character set conversion | ~~TRUE~~ **FALSE** | musl's iconv IS compiled into wasi-libc; iconv.h available; charset support limited but present |
44+
| 23 | **Timezone limited to UTC** | Locale-aware time formatting | **TRUE** | musl timezone code ifdef'd out for WASI; no tzdata in VFS; localtime returns UTC |
45+
| 24 | **fcntl cloexec tracking limited to 256 FDs** | Programs with many open files | **TRUE** | fcntl.c:32 MAX_FDS=256; FDs >= 256 get EBADF |
46+
47+
## MISSING — Not implemented at all
48+
49+
Fact-check found several claims EXAGGERATED — the C interfaces exist but fail at WASI syscall layer (ENOSYS). The distinction matters: programs that check for availability at link time will succeed; programs that check at runtime will get clean errors.
50+
51+
| # | Issue | Verified | Notes |
52+
|---|-------|----------|-------|
53+
| 25 | No fork() | **EXAGGERATED** | fork() callable via musl but returns ENOSYS from WASI layer |
54+
| 26 | No epoll | **EXAGGERATED** | epoll stubs exist in musl; fail with ENOSYS at SYS_epoll_create1 |
55+
| 27 | No named pipes (mkfifo) | **EXAGGERATED** | mkfifo/mknod exist; fail with ENOSYS at SYS_mknodat |
56+
| 28 | No shared memory | **EXAGGERATED** | shm_open/shmget exist; fail with ENOSYS at syscall layer |
57+
| 29 | No /proc population | **TRUE** | /proc directory created but empty; no self/exe/cpuinfo |
58+
| 30 | No mmap | **EXAGGERATED** | mmap available via `-lwasi-emulated-mman` emulation layer |
59+
60+
## What works well
61+
62+
- Text processing pipelines (grep, sed, awk, jq, sort)
63+
- Shell scripting (bash 5.x compatible via brush-shell)
64+
- Build systems (make with subcommand spawning)
65+
- HTTP/HTTPS clients (curl, wget, git clone, npm install)
66+
- Interactive terminals (real PTY with line discipline, Ctrl+C for children)
67+
- File I/O basics (read, write, create, delete, rename)
68+
- Cross-runtime process spawning (WasmVM <-> Node <-> Python)
69+
- SQLite (single-threaded, file-based)
70+
- iconv (character set conversion via musl)
71+
- Full PTY allocation with master/slave pairs and signal delivery
72+
- O_NONBLOCK settable via fcntl F_SETFL in WasmVM
73+
74+
## Corrected honest claim
75+
76+
> "POSIX shell scripts, text processing tools, and HTTP clients run unmodified. Server sockets (bind/listen/accept), custom signal handlers, and true multi-threading are not supported. Most POSIX C interfaces exist and link correctly but several kernel-level operations (O_EXCL, O_TRUNC, readdir ./.. , blocking flock, deferred unlink) are missing and will break programs that depend on them."
77+
78+
## Severity summary
79+
80+
- **11 confirmed WILL_BREAK** issues (all verified true)
81+
- **9 confirmed TOO_THIN** issues (4 original claims debunked as false)
82+
- **1 confirmed MISSING** issue (5 original claims were exaggerated — interfaces exist, ENOSYS at runtime)
83+
- **4 claims were FALSE** (O_NONBLOCK works, PIPE_BUF atomic in JS, pread fine for InMemoryFS, real PTY exists)
84+
- **1 claim was FALSE** (iconv exists in musl)
85+
- **5 claims were EXAGGERATED** (fork/epoll/mkfifo/shm/mmap exist as C stubs returning ENOSYS)

docs/docs.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,8 @@
125125
"group": "Reference",
126126
"pages": [
127127
"posix-compatibility",
128-
"posix-conformance-report",
128+
"os-test-conformance-report",
129+
"libc-test-conformance-report",
129130
"nodejs-conformance-report",
130131
"python-compatibility"
131132
]
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
title: libc-test Conformance Report
3+
description: musl libc-test kernel behavior conformance results for WasmVM.
4+
icon: "chart-bar"
5+
---
6+
7+
{/* AUTO-GENERATED — do not edit. Run scripts/generate-libc-test-report.ts */}
8+
9+
## Summary
10+
11+
musl libc-test tests actual kernel behavior — file locking, socket operations, stat edge cases,
12+
and process management. Unlike os-test (which tests libc function correctness), these tests
13+
exercise the runtime and kernel layer.
14+
15+
| Metric | Value |
16+
| --- | --- |
17+
| libc-test version | master |
18+
| Total tests | 75 |
19+
| Passing | 69 (92.0%) |
20+
| Expected fail | 6 |
21+
| Skip | 0 |
22+
| Native verified | 68 of 69 passing tests verified against native output (98.6%) |
23+
| Last updated | 2026-03-26 |
24+
25+
## Per-Suite Results
26+
27+
| Suite | Total | Pass | Fail | Skip | Pass Rate |
28+
| --- | --- | --- | --- | --- | --- |
29+
| functional | 41 | 37 | 4 | 0 | 90.2% |
30+
| regression | 34 | 32 | 2 | 0 | 94.1% |
31+
| **Total** | **75** | **69** | **6** | **0** | **100.0%** |
32+
33+
## Exclusions by Category
34+
35+
### WASM Limitations (4 entries)
36+
37+
Features impossible in wasm32-wasip1.
38+
39+
| Test | Reason | Issue |
40+
| --- | --- | --- |
41+
| `functional/dlopen_dso` | dlopen/dlsym not available in wasm32-wasip1 — no dynamic linking support | [#48](https://github.com/rivet-dev/secure-exec/issues/48) |
42+
| `functional/tls_align_dso` | Thread-local storage with dynamic shared objects requires dlopen — not available in WASM | [#48](https://github.com/rivet-dev/secure-exec/issues/48) |
43+
| `functional/tls_init_dso` | Thread-local storage initialization with DSOs requires dlopen — not available in WASM | [#48](https://github.com/rivet-dev/secure-exec/issues/48) |
44+
| `regression/tls_get_new-dtv_dso` | TLS dynamic thread vector with DSOs requires dlopen — not available in WASM | [#48](https://github.com/rivet-dev/secure-exec/issues/48) |
45+
46+
### WASI Gaps (1 entry)
47+
48+
WASI Preview 1 lacks the required syscall.
49+
50+
| Test | Reason | Issue |
51+
| --- | --- | --- |
52+
| `regression/statvfs` | statvfs/fstatvfs not part of WASI — no filesystem statistics interface | [#48](https://github.com/rivet-dev/secure-exec/issues/48) |
53+
54+
### Implementation Gaps (1 entry)
55+
56+
Features we should support but don't yet. Each has a tracking issue.
57+
58+
| Test | Reason | Issue |
59+
| --- | --- | --- |
60+
| `functional/strptime` | strptime fails on timezone-related format specifiers (%Z, %z) — musl timezone code is ifdef'd out for WASI | [#48](https://github.com/rivet-dev/secure-exec/issues/48) |

0 commit comments

Comments
 (0)