Browser-driven smoke tests for the webapp (extend the render-harness conventions to real user flows)

[thedavidmeister]

Ask

Browser-driven smoke tests for the webapp: drive a real browser through a few critical user flows and assert on what actually happens, in CI.

Why this is still open after #476

#476 asked for this with tauri-driver + selenium. The desktop app it targeted was deleted (#2417), so that framing is dead — but the underlying gap is real and survived the rewrite. Today:

• Component/unit testing is well covered — 122 frontend test files (24 in packages/webapp, 98 in packages/ui-components) on vitest + jsdom + @testing-library/svelte, gated by test-webapp.yaml and test-ui-components.yaml, both green on main.
• Nothing drives a real browser. A tree-wide search for playwright|cypress|webdriver|selenium|puppeteer returns one unrelated Rust file (crates/common/src/raindex_client/take_orders/e2e_tests.rs).

So every assertion about the running app is either jsdom-level or a human looking at a picture.

What already exists to build on

This should extend our existing render conventions, not introduce a parallel stack beside them.

The component render harness — packages/ui-components/harness/ (PR #2816, merged 2026-07-25):

• Entry: packages/ui-components/harness/screenshot.sh <scene> <out.png> [width] [height]
• screenshot.mjs spawns a standalone vite dev server (harness/vite.config.ts, plain Svelte plugin, not sveltekit()), then runs Chromium once: --headless=new --disable-gpu --no-sandbox --hide-scrollbars --force-device-scale-factor=2 --virtual-time-budget=6000 --screenshot=<out>
• Alias stubs make it hermetic: $app/stores, $app/navigation, $app/environment, and @rainlanguage/raindex (a recursive Proxy so no wasm ever initialises), with $lib pointing at the real component source
• Scenes are fixture components in scenes/ keyed by a registry, selected via ?scene=<name>
• Deliberately outside src/ and excluded from lint, svelte-check, the package build and CI — it is an evidence tool, not a test
• Its README already anticipates this work: it states the same directory pattern is meant to be copied to packages/webapp/harness, pointing $lib at webapp's src/lib and stubbing webapp's own stores. That copy does not exist yet.

The dashboard render app — rain-org-health's nix run .#screenshot, a writeShellApplication with chromium pinned through rainix.pkgs.${system}, a local http server with a real readiness poll (not a fixed sleep) plus a cleanup trap, an explicit fontconfig, and --force-color-profile=srgb so two runs are byte-comparable. That determinism is what lets a PR make falsifiable claims: #129 proved "every sibling sparkline is untouched" with a thresholded ImageMagick diff reduced to bbox=4x4+1016+389; #121 proved "this refactor moved no pixels" with identical sha256 sums.

Delivery — PNGs go to the pr-screenshots orphan branch under a directory named for the PR number, embedded in a PR comment via raw.githubusercontent.com.

Written rules these harnesses serve:

• rainlanguage/.github/pull_request_template.md L24 — "included screenshots (if this involves a front-end change)"
• raindex/.github/copilot-instructions.md L44-45 — "If you modify frontend code or functionality affecting the frontend, you MUST provide a screenshot of the built webapp reflecting your change", and L53 — "If you are unable to build the webapp, you MUST provide the concrete reasons and errors. Workarounds are not acceptable."
• raindex/AGENTS.md L70-71 — PRs "include screenshots/GIFs for UI changes"

What is actually missing (the honest gap)

Both harnesses are one-shot renderers. Neither opens a DevTools-protocol channel — grepping screenshot.mjs for CDP/WebSocket/Page./Runtime. finds nothing. They render one frame and exit. There is no way to click, type, wait on a selector, or read state back, and no exit code meaning "assertion failed" (the only non-zero path is "server never came up"). Settling is a fixed --virtual-time-budget, not a wait-for-condition.

The second wall is the stubs. @rainlanguage/raindex is replaced by an inert Proxy precisely so the harness stays fast and hermetic — which is right for rendering one component, and wrong for exercising a flow that depends on real client behaviour.

Suggested shape

1. Reuse the scaffolding, not the capture step. The pinned chromium, fontconfig, readiness poll, cleanup trap and srgb determinism are already written and pinned in both harnesses — an e2e runner needs exactly those four things.
2. Add a driver that can interact and assert — a CDP session or Playwright — instead of --screenshot fire-and-forget. This is the genuinely new capability.
3. Decide the stub question deliberately. Smoke tests want the real client (or high-fidelity fakes), not the inert Proxy. This is the main design call and worth settling before writing flows.
4. Follow the anticipated layout — packages/webapp/harness, as the ui-components README already describes, so the two stay siblings rather than rivals.
5. Run it in CI, unlike the evidence harnesses which are deliberately excluded — a smoke test that nothing runs is not a smoke test. Keep the pr-screenshots convention for failure artifacts so a red run leaves a picture.
6. Start with a handful of flows, not a suite: load the app, navigate to orders and vaults, open a deploy page, and assert the page renders with real data rather than an error boundary.

Context

Split out of #476, which is closed — its literal ask (tauri-driver/selenium) targets a deleted app, and its title's ask (frontend testing harness + basic tests) is satisfied by the vitest suites. This issue carries forward the one part that was never delivered.

[linear]

RAI-1541