test(sp3): visual-regression infrastructure (Playwright + SSIM) + golden refresh - #10
Conversation
Wires shader:check to a real Playwright + SSIM suite and lands the shader:update-goldens maintenance script. Goldens are not committed in this PR; the suite skip-warns when one is missing so the default test run stays green until they're captured. The user runs shader:update-goldens --confirm once to populate tests/golden/, then commits the PNGs in a follow-up. The capture pipeline drives the dev server through Playwright headless, hashes the simulation config into the URL (mass, spin, zoom, diskTemp, etc., the same keys useUrlState already reads), waits for the canvas to mount and complete enough requestAnimationFrame ticks for TAA to converge, then screenshots at the manifest viewport. Deterministic seeding is not yet wired; the stabilization-frames count is the lever that controls run-to-run variance for now. The diff stage uses pixelmatch for the per-pixel count plus ssim.js for the structural similarity score and a manual max-pixel-delta sweep. Per-frame thresholds live in tests/golden/manifest.json so tightening or relaxing one frame doesn't ripple through the others; the threshold ladder (0.998 / 0.997 / 0.996) reflects increasing visual variance with spin. The runner is its own vitest config (vitest.shader.config.ts) so the expensive boot path stays opt-in; the default vitest run excludes tests/visual-regression entirely. SHADER_CHECK_BASE_URL + SHADER_CHECK_SKIP_DEV environment knobs let an already-running dev server serve the suite without a second boot. shader:update-goldens captures every frame in the manifest, stamps captured_at + captured_commit on each entry, and refuses to run without --confirm. The user diffs the PNG output before committing; a wrong golden silently shifts the regression baseline.
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (11)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 32 minutes and 30 seconds.Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e72731990a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| devProc = spawn("bun", ["run", "dev"], { | ||
| stdio: "ignore", | ||
| shell: true, | ||
| detached: false, | ||
| }); |
There was a problem hiding this comment.
When SHADER_CHECK_SKIP_DEV is not set, this always starts bun run dev and then only probes BASE_URL for readiness. If port 3000 is already occupied (for example, a stale dev server from another branch), the new process can fail to bind while waitForServerReady still succeeds against the old server, so the suite captures/diffs the wrong build and can report misleading pass/fail results.
Useful? React with 👍 / 👎.
1 participant
Closes the last two SP-3 groups in one PR: 3B wires the visual-regression
suite to a real Playwright + SSIM pipeline, 3D lands the maintenance
script that re-captures goldens behind a --confirm flag. The shader:check
alias used to match zero tests; it now invokes a real suite that the
user activates by running shader:update-goldens --confirm once.
Goldens themselves are NOT committed in this PR. The runner skip-warns
when a golden is missing so the suite stays green until they're captured;
that capture step needs the dev server up and a one-time visual review of
each PNG, which is the right shape for an operator workflow rather than
an autonomous PR step.
Group 3B - infrastructure:
Kerr a=0.5 inclined, near-extremal a=0.99 edge-on) with per-frame
ssim_threshold and max_pixel_delta. Threshold ladder (0.998 / 0.997 /
0.996) reflects increasing visual variance with spin.
dev server, hashes the simulation config into the URL (mass, spin,
zoom, diskTemp, etc., the same keys useUrlState reads), waits for
canvas mount + a configurable number of requestAnimationFrame ticks
for TAA to converge, screenshots at the manifest viewport.
ssim.js for the structural similarity score, manual sweep for the
max-pixel-delta.
dev server, walks every manifest entry, and asserts SSIM + pixel
delta against the committed PNG. Skip-warns if golden missing.
out of the default suite (the boot path is expensive). The default
vitest run now excludes tests/visual-regression entirely.
Group 3D - golden refresh:
stamps captured_at + captured_commit on each entry, refuses to run
without --confirm. Reuses an already-running dev server if one is up
at SHADER_CHECK_BASE_URL.
Test plan
Follow-on PRs
The goldens themselves want their own PR so the visual review and the
infrastructure changes don't hide behind each other in the diff.