Skip to content

feat(miner): add schema-version migration runner across local stores#5364

Merged
JSONbored merged 1 commit into
JSONbored:mainfrom
real-venus:feat/gittensory-miner-schema-versioning-v2
Jul 12, 2026
Merged

feat(miner): add schema-version migration runner across local stores#5364
JSONbored merged 1 commit into
JSONbored:mainfrom
real-venus:feat/gittensory-miner-schema-versioning-v2

Conversation

@real-venus

Copy link
Copy Markdown
Contributor

Summary

The miner's local SQLite stores only ever ran CREATE TABLE IF NOT EXISTS, with no user_version/migration mechanism at all — so an older on-disk database file was silently reused with a stale shape, unlike the main product's migrations/ discipline. This adds a lightweight, shared schema-versioning convention across every store.

New: packages/gittensory-miner/lib/schema-version.js (+ .d.ts) exporting applySchemaMigrations(db, migrations):

  • Each store's bootstrap schema is treated as version 1 (BASELINE_SCHEMA_VERSION); a store's migrations array holds only post-baseline changesmigrations[i] upgrades the schema from version i+1 to i+2.
  • The runner reads PRAGMA user_version, runs exactly the pending migrations in order, then stamps the new version — so opening an older file runs its outstanding migrations instead of continuing on an incompatible shape. Re-opening an up-to-date file runs none.
  • A pre-versioning file (user_version 0) already carries the baseline tables (the idempotent CREATE TABLE IF NOT EXISTS ran), so it advances from the baseline.
  • Upgrade-only: a file written by newer code with more migrations is never stamped back down — it keeps its higher version.
  • Atomic + resumable: each migration and its version stamp run in one transaction (PRAGMA user_version is transactional in SQLite), so a failure part-way through the sequence leaves the file at the last fully-applied version — the next open resumes at the failed migration instead of re-running the ones that already succeeded (which, for a non-idempotent ALTER, would be a duplicate-column error).
  • Pure control flow over an injected DatabaseSync handle — deterministic, no IO of its own beyond the PRAGMA read/write.

Wired into all seven stores: claim-ledger, event-ledger, governor-ledger, plan-store, run-state, laptop-init, portfolio-queue.

Portfolio-queue's existing ad-hoc leased_at ALTER is expressed as its first real migration (1 → 2) — kept defensive (checks table_info) so a version-0 file that already ran the pre-convention ad-hoc ALTER is not re-altered into a duplicate-column error. This both removes the ad-hoc branch and demonstrates the convention on a real migration.

Kept lightweight per the issue's boundary — no heavyweight migration framework.

Scope

  • Narrow, one coherent change — one shared runner + uniform per-store wiring + tests
  • In scope (packages/), no blockedPaths, no secrets/private terms
  • Hand-authored .d.ts companion added, matching the package's existing convention

Validation

  • npm run typecheck
  • npm run test:coverage (full unsharded suite)
  • New unit test test/unit/miner-schema-version.test.ts (in-memory DatabaseSync): baseline stamp with no migrations, full run in order on a pre-versioning file, idempotent re-apply, partial catch-up from an intermediate version, upgrade-only (no downgrade of a newer file), mid-sequence failure leaves the file at the last applied version (no re-run of succeeded migrations), atomic rollback of a failed migration's partial changes, and coercion of an absent/non-integer/negative user_version
  • Every existing store test still passes (verified against a clean-main baseline: identical results — the Windows-only chmod/path test failures are pre-existing and unrelated); portfolio-queue's leased_at/markFailed tests confirm the migrated behavior is preserved

Safety

  • Behaviour-preserving: stores gain a user_version stamp and a migration hook; the only converted logic (portfolio-queue leased_at) keeps its exact defensive semantics
  • No secrets, tokens, wallets, trust scores, or reward values in code, tests, or this description

Closes #4832

The miner's local SQLite stores only ever ran CREATE TABLE IF NOT EXISTS,
with no user_version/migration mechanism — so an older on-disk file was
silently reused with a stale shape. Add a lightweight shared runner,
applySchemaMigrations(db, migrations), and wire it into every store
(claim-ledger, event-ledger, governor-ledger, plan-store, run-state,
laptop-init, portfolio-queue).

Each store's bootstrap schema is treated as version 1; a store's
migrations array holds only post-baseline changes (migrations[i] upgrades
version i+1 to i+2). The runner reads PRAGMA user_version, runs exactly the
pending migrations in order, and stamps the new version, so opening an
older file runs its outstanding migrations instead of continuing on an
incompatible shape. Pre-versioning files (user_version 0) already carry the
baseline tables, so they advance from the baseline. Kept lightweight per
the issue — no heavyweight migration framework.

Portfolio-queue's existing ad-hoc leased_at ALTER is expressed as its first
real migration (1 to 2), staying defensive so a file that already ran the
pre-convention ALTER is not re-altered into a duplicate-column error.

Closes JSONbored#4832
@real-venus
real-venus requested a review from JSONbored as a code owner July 12, 2026 15:03
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 12, 2026
@loopover-orb

loopover-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-12 15:09:51 UTC

10 files · 2 AI reviewers · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This adds a shared `applySchemaMigrations(db, migrations)` runner (PRAGMA user_version-based, atomic per-migration transactions, upgrade-only, resumable-on-failure) and wires it into all seven local SQLite stores, converting portfolio-queue's ad-hoc `leased_at` ALTER into migration[0]. The logic is sound: baseline stamping, pending-only execution, non-downgrade, and transactional atomicity are all correctly implemented and covered by a thorough unit test suite that exercises fresh files, partial-history files, mid-sequence failures, and newer-than-target files. The one substantive risk is that `PRAGMA user_version = ${value}` interpolates a computed integer directly into `exec()` — safe here since both interpolated values (`BASELINE_SCHEMA_VERSION` and `version + 1`) are internally computed integers never derived from external input, so this isn't an injection vector despite the string interpolation.

Nits — 5 non-blocking
  • schema-version.js:34/47/58 use `db.exec(`PRAGMA user_version = ${value}`)` string interpolation instead of a parameterized statement; PRAGMA doesn't support `?` binding so this is unavoidable, but a comment noting the value is always an internally-computed integer (already present) is the right mitigation — worth double-checking no future caller ever threads a non-integer into `migrations.length`.
  • The six stores with `applySchemaMigrations(db, [])` (claim-ledger.js, event-ledger.js, governor-ledger.js, laptop-init.js, plan-store.js, run-state.js) add a PRAGMA write on every single open even though nothing changes after the first stamp — negligible cost, but worth confirming this doesn't measurably slow high-frequency store opens.
  • README.md's 'Local storage' table (referenced in context) isn't updated in this diff to mention the new schema-version convention, though that's a doc-completeness nit rather than a code defect.
  • Consider a fast-path early-return in `applySchemaMigrations` when `current >= target` to skip the `db.exec` PRAGMA write entirely on already-versioned opens, avoiding a no-op disk write on every store initialization.
  • Since this is described as closing Add schema versioning to the local SQLite stores #4832, confirm the issue is fully satisfied — the PR description says 'partial' issue coverage was found; verify no additional store or migration was expected.
Signal Result Evidence
Code review ✅ No blockers 2 reviewers, synthesized
Linked issue ✅ Linked #4832
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 98 registered-repo PR(s), 54 merged, 8 issue(s).
Contributor context ✅ Confirmed Gittensor contributor real-venus; Gittensor profile; 98 PR(s), 8 issue(s).
Gate result ✅ Passing No configured blocker found.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence. LLM value judgment: moderate — This is a well-targeted infrastructure improvement that turns previously implicit local-store schema drift into an explicit, tested upgrade path while preserving the existing store boundaries.
Linked issue satisfaction

Addressed
The PR introduces a shared applySchemaMigrations/readSchemaVersion convention using PRAGMA user_version and wires it into all seven local stores, so opening an older-schema file runs pending migrations (demonstrated for portfolio-queue's leased_at column) instead of silently continuing on a stale shape; tests cover pending/partial/failed/downgrade scenarios.

Review context
  • Author: real-venus
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 98 PR(s), 8 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 12, 2026
@JSONbored
JSONbored merged commit e8e068d into JSONbored:main Jul 12, 2026
14 checks passed
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.07%. Comparing base (fd6427d) to head (9db5291).
⚠️ Report is 3 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (fd6427d) and HEAD (9db5291). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (fd6427d) HEAD (9db5291)
shard-6 1 0
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5364      +/-   ##
==========================================
- Coverage   94.41%   85.07%   -9.35%     
==========================================
  Files         550      551       +1     
  Lines       44143    44168      +25     
  Branches    14632    14632              
==========================================
- Hits        41677    37575    -4102     
- Misses       1791     5219    +3428     
- Partials      675     1374     +699     
Flag Coverage Δ
shard-1 43.76% <85.71%> (-0.23%) ⬇️
shard-2 34.04% <82.14%> (-0.59%) ⬇️
shard-3 32.24% <85.71%> (+0.64%) ⬆️
shard-4 31.07% <89.28%> (-0.18%) ⬇️
shard-5 32.86% <89.28%> (-0.43%) ⬇️
shard-6 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/gittensory-miner/lib/claim-ledger.js 94.87% <100.00%> (+0.06%) ⬆️
packages/gittensory-miner/lib/event-ledger.js 87.32% <100.00%> (+0.18%) ⬆️
packages/gittensory-miner/lib/governor-ledger.js 98.00% <100.00%> (+0.04%) ⬆️
packages/gittensory-miner/lib/laptop-init.js 97.10% <100.00%> (+0.04%) ⬆️
packages/gittensory-miner/lib/plan-store.js 92.45% <100.00%> (+0.07%) ⬆️
packages/gittensory-miner/lib/portfolio-queue.js 95.60% <100.00%> (+0.04%) ⬆️
packages/gittensory-miner/lib/run-state.js 80.00% <100.00%> (-20.00%) ⬇️
packages/gittensory-miner/lib/schema-version.js 100.00% <100.00%> (ø)

... and 113 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add schema versioning to the local SQLite stores

2 participants