Standardize run tests by vidyalakshmir · Pull Request #280 · Lind-Project/lind-wasm-apps

vidyalakshmir · 2026-08-17T17:19:13Z

Adds TESTING.md which specifies the app test requirements
Adds scripts/test_lib.sh which is a common script to skip tests
Adds standardize app test scripts for all apps (except cpython and tinycc).

Uk-jake · 2026-08-20T18:30:09Z

Thanks for the work. I’ll start on the apps pipeline using this.

I just confirmed my understanding. Are 11 apps (awk, bash, coreutils, curl, git, grep, lmbench, make, nginx, perl, sed) standardized, while cpython and tinycc are excluded?

Additionally, to run make test for all 11 apps, I think we should also update TESTABLE_APPS in the Makefile

vidyalakshmir · 2026-08-20T19:17:42Z

Yes, need to update TESTABLE_APPS in the Makefile

There are two categories of test suites for apps :

Default test suite shipped from the app developers
Test suite we developed for the apps.

Adapting test suite of apps to run with lind-wasm requires some porting effort. Hence, we have
Cpython - Category 1
Coreutils - Category 1 and 2
All other apps - Category 2

We have Cpython and coreutils (from category 1) and tinycc pending.

Regarding postgres, I am not completely sure about which category its tests are. @rishabhBudhouliya Do you have more information about postgres test suites.

Uk-jake · 2026-08-21T14:59:29Z

Default test suite shipped from the app developers

Test suite we developed for the apps.

Just curious, what is the reason for dividing the tests into Category 1 and 2?

vidyalakshmir · 2026-08-21T15:08:14Z

Default test suite shipped from the app developers

Test suite we developed for the apps.

Just curious, what is the reason for dividing the tests into Category 1 and 2?

Adapting test suite of apps to run with lind-wasm requires some porting effort. Depends on if the app comes with test suite and the porting effort.

Uk-jake · 2026-08-21T15:18:38Z

Default test suite shipped from the app developers

Test suite we developed for the apps.

Just curious, what is the reason for dividing the tests into Category 1 and 2?

Adapting test suite of apps to run with lind-wasm requires some porting effort. Depends on if the app comes with test suite and the porting effort.

In that case, is the ideal approach to test both Category 1 and 2? It looks like Coreutils is listed under both categories. Or is it fine to just run either Category 1 or Category 2?

qianxichen233 · 2026-08-22T01:04:53Z

Yes, need to update TESTABLE_APPS in the Makefile

There are two categories of test suites for apps :

Default test suite shipped from the app developers

Test suite we developed for the apps.

Adapting test suite of apps to run with lind-wasm requires some porting effort. Hence, we have Cpython - Category 1 Coreutils - Category 1 and 2 All other apps - Category 2

We have Cpython and coreutils (from category 1) and tinycc pending.

Regarding postgres, I am not completely sure about which category its tests are. @rishabhBudhouliya Do you have more information about postgres test suites.

postgres is runnings its own regression testsuite

rishabhBudhouliya · 2026-08-23T16:53:48Z

@vidyalakshmir As Qianxi mentioned, Postgres has its own test suite called pg_regress. The usual way to run it is to spawn a postgres server and then run the pg_regress harness which itself spawns a psql client to talk to the main server and then run a bunch of tests.
It also has a benchmarking test suite called pg_bench but I think that might not be related to the standardization effort.

rishabhBudhouliya

Thanks Vidya for leading the standardization effort. I mostly have app specific behavior related comments. Please let me know how I can help.

Other app related behavior I saw:
Difference in how PASS/FAIL is represented in different app tests results:
[PASS]/[FAIL] is PASS:/FAIL: in curl, git, nginx, lmbench, sed

rishabhBudhouliya · 2026-08-23T17:04:11Z

I suspect nearly every test name contains spaces. Should we use a newline/ pipe delimeter?

Yeah makes sense

rishabhBudhouliya · 2026-08-23T17:05:42Z

It seems like some timeout configs are inconsistent:

awk/make/perl/bash/coreutils hardcode TIMEOUT_SECS=N;

sed/grep/curl/git use ${TIMEOUT_SECS:-N}.

rishabhBudhouliya · 2026-08-23T18:21:16Z

 echo "================================"
+
+if [ "$FAIL" -gt 0 ]; then
+    exit 1


Since this exits 1 on failure, with Makefile .SHELLFLAGS := -eu -o pipefail -c + .ONESHELL, a coreutils failure aborts the entire make test run. Is that the intended behavior?

No, even if one test fails, need to continue to the next test.

Uk-jake · 2026-08-27T19:52:00Z

@rishabhBudhouliya, @qianxichen233
I'd appreciate it if you could confirm whether the following result is expected when running 'make test'

I was running 'make test' on thestandardize-run-tests branch.

First, about running the tests at all. The current test target loops over TESTABLE_APPS, but it stops as soon as one suite exits 1. For a CI/CD pipeline, we need a trigger that runs every app's tests even when some of them fail. If we want to use make test as that trigger, the Makefile needs a change along these lines.

test:
	@rc=0; \
	for app in $(APP); do \
	    case " $(TESTABLE_APPS) " in \
	      *" $$app "*) ;; \
	      *) echo "ERROR: unsupported test app '$$app'"; exit 1 ;; \
	    esac; \
	    if [[ -x '$(APPS_ROOT)/'"$$app"'/run_tests.sh' ]]; then \
	        if ! '$(APPS_ROOT)/'"$$app"'/run_tests.sh' "$$app"; then \
	            rc=1; \
	            echo "[FAIL] $$app"; \
	        fi; \
	    else \
	        echo "[SKIP] $$app: missing run_tests.sh"; \
	    fi; \
	done; \
	exit $$rc

Wrapping the call in a conditional keeps set -e from aborting the loop, and make test still exits non-zero if anything failed.

This a result of 'make test' with the change.

app	result	passed/total
awk	PASS	45/45
sed	PASS	7/7
curl	FAIL	6/25 (5 skipped)
grep	FAIL	1/5
git	FAIL	4/36
make	FAIL	1/22
coreutils	FAIL	1/27
bash	FAIL	3/162
perl	FAIL	1/107
lmbench	FAIL	did not run
nginx	FAIL	failed to start

make-test-patched.log

That's more failures than I expected, so I'd appreciate it if you could point out anything I might have missed.
Here's what I ran.

Image: securesystemslab/lind-wasm-dev:latest

make preflight
make merge-sysroot
make <app> and make install-<app> for: awk bash coreutils curl git grep lmbench make nginx perl sed
make test

vidyalakshmir · 2026-08-27T19:59:19Z

@rishabhBudhouliya, @qianxichen233 I'd appreciate it if you could confirm whether the following result is expected when running 'make test'

I was running 'make test' on thestandardize-run-tests branch.

First, about running the tests at all. The current test target loops over TESTABLE_APPS, but it stops as soon as one suite exits 1. For a CI/CD pipeline, we need a trigger that runs every app's tests even when some of them fail. If we want to use make test as that trigger, the Makefile needs a change along these lines.
test:
	@rc=0; \
	for app in $(APP); do \
	    case " $(TESTABLE_APPS) " in \
	      *" $$app "*) ;; \
	      *) echo "ERROR: unsupported test app '$$app'"; exit 1 ;; \
	    esac; \
	    if [[ -x '$(APPS_ROOT)/'"$$app"'/run_tests.sh' ]]; then \
	        if ! '$(APPS_ROOT)/'"$$app"'/run_tests.sh' "$$app"; then \
	            rc=1; \
	            echo "[FAIL] $$app"; \
	        fi; \
	    else \
	        echo "[SKIP] $$app: missing run_tests.sh"; \
	    fi; \
	done; \
	exit $$rc
Wrapping the call in a conditional keeps set -e from aborting the loop, and make test still exits non-zero if anything failed.

This a result of 'make test' with the change.

app result passed/total
awk PASS 45/45
sed PASS 7/7
curl FAIL 6/25 (5 skipped)
grep FAIL 1/5
git FAIL 4/36
make FAIL 1/22
coreutils FAIL 1/27
bash FAIL 3/162
perl FAIL 1/107
lmbench FAIL did not run
nginx FAIL failed to start
make-test-patched.raw.log make-test-patched.log

That's more failures than I expected, so I'd appreciate it if you could point out anything I might have missed. Here's what I ran.

Image: securesystemslab/lind-wasm-dev:latest

make preflight

make merge-sysroot

make <app> and make install-<app> for: awk bash coreutils curl git grep lmbench make nginx perl sed

make test

Thanks @Uk-jake You could make necessary changes to Makefile to adapt it to the CI/CD pipeline.

Also, thanks for running the test suites.

It would also help if you could attach or post the errors you face for the failed apps.

vidyalakshmir · 2026-08-27T20:15:26Z

So I tried testing some of the apps (curl) which had failures. And seems like the issue is mostly with env::__lind_debug_import has not been defined. I think this is because of debug build being made default for lind-boot. How could we fix this for the apps? @qianxichen233 @Yaxuan-w

lind_run --preload env=lib/libz.so --preload env=lib/libcrypto.so usr/local/bin/curl --version
failed to run main module

Caused by:
    0: failed to invoke command default
    1: unknown import: `env::__lind_debug_import` has not been defined

rishabhBudhouliya · 2026-08-30T22:39:47Z

@vidyalakshmir @qianxichen233 have created a postgres PR for app test standardization: #282

vidyalakshmir · 2026-08-31T16:55:27Z

@vidyalakshmir @qianxichen233 have created a postgres PR for app test standardization: #282

Thanks @rishabhBudhouliya

vidyalakshmir · 2026-08-31T16:56:55Z

@Uk-jake After running the tests, the two cases worth noting are :

The lind-wasm should NOT have being built enabling debug mode to run with apps.
For apps that require fpcast, lind-wasm should be compiled with fpcast.

Uk-jake · 2026-08-31T18:43:54Z

@vidyalakshmir, @rishabhBudhouliya
Could you let me know which apps require FPCast?

rishabhBudhouliya · 2026-08-31T18:47:19Z

@Uk-jake
Apps that require fpcast
Bash, coreutils, perl, postgres, nginx, grep, awk, make lmbench

I am not 100% confident but these apps don't require fpcast
binutils, curl, diffutils, ed25519, gcc, git, gnulib, libtirpc, llvm-project, openssl, sed, tinycc, zlib.

vidyalakshmir requested review from qianxichen233 and rishabhBudhouliya August 20, 2026 19:17

rishabhBudhouliya reviewed Aug 23, 2026

View reviewed changes

rishabhBudhouliya mentioned this pull request Aug 30, 2026

Add postgres test suite with smoke tests #282

Open

vidyalakshmir added 3 commits August 30, 2026 18:38

Add testing requirements

e0b88da

Common script to skip tests

18e8cb0

Standardize app test scripts

8b16e83

rishabhBudhouliya force-pushed the standardize-run-tests branch from f7339f7 to 8b16e83 Compare August 30, 2026 22:38

Uh oh!

Conversation

vidyalakshmir commented Aug 17, 2026

Uh oh!

Uk-jake commented Aug 20, 2026

Uh oh!

vidyalakshmir commented Aug 20, 2026

Uh oh!

Uk-jake commented Aug 21, 2026

Uh oh!

vidyalakshmir commented Aug 21, 2026

Uh oh!

Uk-jake commented Aug 21, 2026

Uh oh!

qianxichen233 commented Aug 22, 2026

Uh oh!

rishabhBudhouliya commented Aug 23, 2026

Uh oh!

rishabhBudhouliya left a comment

Choose a reason for hiding this comment

Uh oh!

rishabhBudhouliya Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

vidyalakshmir Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

rishabhBudhouliya Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

rishabhBudhouliya Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

vidyalakshmir Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uk-jake commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vidyalakshmir commented Aug 27, 2026

Uh oh!

vidyalakshmir commented Aug 27, 2026

Uh oh!

rishabhBudhouliya commented Aug 30, 2026

Uh oh!

vidyalakshmir commented Aug 31, 2026

Uh oh!

vidyalakshmir commented Aug 31, 2026

Uh oh!

Uk-jake commented Aug 31, 2026

Uh oh!

rishabhBudhouliya commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uk-jake commented Aug 27, 2026 •

edited

Loading