perf(test): clone a migrated template DB per TestD1Database instead of replaying 178 migrations#8533
Merged
Merged
Conversation
…f replaying 178 migrations Executing the full migrations/*.sql chain into :memory: on every construction (~640ms x ~1500 sites) was ~950s of pure schema replay per full run and put ordinary tests in the 3s+ range. Build one content-hash-keyed template file per migration-chain content (atomic rename, shared across workers, page_size=1024 + VACUUM), then copyFileSync a clone per instance: ~1.5ms per construction with a verified write. Full suite: 510s -> 171s wall, 4288s -> 870s aggregate, same 21472-test pass set. Clones stay on disk until a single exit sweep -- SQLite fails writes with SQLITE_READONLY_DBMOVED if the main file is unlinked while open -- and all memoization lives on globalThis so vitest's per-file module isolation cannot silently rebuild the template per test file. serialize()/deserialize() remain unusable on the pinned Node 22 (the prior attempt crashed CI).
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
❌ 3 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every
TestD1Database()construction executed the fullmigrations/*.sqlchain (178 files, growing with every merge) into a fresh:memory:database — measured at ~640ms per construction, across ~1,500 construction sites: ~950s of pure schema replay per full run, and the reason ordinary tests sat in the 3s+ range (observed live on PR #8524's validate-tests).This replaces the per-construction replay with a content-hash-keyed, fully-migrated template database file built once per migration-chain content and shared across all vitest workers; each
TestD1Databasegets its owncopyFileSyncclone (~1.5ms including a verified write — ~440× faster per construction).Measured on the full unsharded suite (same machine, same tree):
Design constraints the shape answers (each verified the hard way, documented in the helper):
serialize()/deserialize()don't exist on pinned Node 22 (the prior attempt crashed CI) — file copy is Node-22-safe.SQLITE_READONLY_DBMOVED. Clones stay on disk until a single per-process exit sweep.globalThis— module-scope memos would rebuild the template once per test file and silently give back most of the win..tmpsibling then atomicallyrenameSyncd — concurrent workers double-build harmlessly; no reader ever copies a half-written template; a schema change gets a fresh hash key.page_size=1024+VACUUMshrink the empty-schema template 1.4MB → ~410KB, bounding worst-case tmpdir usage for a run's clones to a few hundred MB, swept at exit.Validation
npm run test:coverage(full, unsharded): 21,472 passed / 0 failed — identical pass set to the pre-change baseline run, at 171s vs 510s.test/helpers/d1.tschanges (Codecov-ignored path); no runtime code touched.