fix(selfhost): normalize SQLite text for Postgres migration

[JSONbored]

Summary

• Normalize embedded NUL bytes in SQLite text values before the self-host Postgres migrator writes them to Postgres.
• Apply the same normalization to migration validation comparisons so dry-runs and non-empty target checks use the exact values Postgres can store.
• Add focused regression coverage for NUL normalization and non-string passthrough.

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue, or this is small enough that the summary explains why an issue is not needed.

No issue because this is a narrow operational follow-up to the Postgres cutover tooling added in #1986. A migration dry-run found cached repo chunk text containing embedded NUL bytes, which SQLite allows but Postgres rejects for text/json inputs.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥97% coverage of the lines AND branches you changed (aim for 98%+ on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

Additional focused checks:

• npx vitest run test/unit/selfhost-migrate.test.ts test/unit/selfhost-grafana-reporting.test.ts test/unit/selfhost-pg-dialect.test.ts
• Temporary Postgres smoke: migrated a SQLite repo_chunks row containing an embedded NUL byte with --execute, then verified Postgres stored the replacement-character-normalized text.

If any required check was skipped, explain why:

• None.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed.
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

Not applicable; this is a self-host migration script/test change only.

Notes

• scripts/** is outside Codecov's measured src/** patch surface, but the behavior is covered by a focused unit regression and an actual SQLite-to-Postgres smoke.
• The migrator replaces only the unrepresentable NUL byte so cached text rows remain migratable without dropping the row.

[gittensory-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-01 06:36:30 UTC}

2 files · 1 AI reviewer · no blockers · readiness 93/100 · CI green · clean

⏸️ Suggested Action - Manual Review

• Touches a guarded path — held for manual review

Review summary
The change centralizes SQLite-to-Postgres value normalization and applies it consistently to copy parameters and validation comparisons, so embedded NUL bytes are transformed before they hit Postgres while equality checks compare against the stored representation. The import-safe main guard is necessary for the new unit-level helper import and is implemented with a direct file URL comparison. The visible diff is coherent and does not introduce a reachable correctness break.

Nits — 5 non-blocking

• nit: test/unit/selfhost-migrate.test.ts:109 covers the helper directly but not the actual migrator binding path that feeds normalized values into copy/validation SQL parameters, so a small integration-style assertion would better protect the production path.
• nit: scripts/migrate-selfhost-sqlite-to-postgres.ts:248 hard-codes the replacement policy without documenting why U+FFFD is preferred over removal or escaped substitution, which makes future migration audits harder.
• Add a focused migrator test around a temporary SQLite row containing `\0` that exercises `copyTable`/validation behavior through the script path rather than only `normalizePostgresValue`.
• Add one short comment or constant name note in `scripts/migrate-selfhost-sqlite-to-postgres.ts` explaining that the migration intentionally stores U+FFFD as a lossy Postgres-safe replacement.
• Touches a guarded path — held for manual review — A maintainer must review and merge this change.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ No-issue rationale	PR body explains why no issue is linked.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (size label size:S; no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 68 registered-repo PR(s), 58 merged, 209 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 68 PR(s), 209 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 68 PR(s), 209 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Triage stale or unlinked PRs.
• No action.
• Link the issue being solved, or explicitly explain why this is a no-issue PR.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.62%. Comparing base (6f042a2) to head (29b8233).
⚠️ Report is 7 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1995   +/-   ##
=======================================
  Coverage   95.62%   95.62%           
=======================================
  Files         218      218           
  Lines       24317    24317           
  Branches     8819     8819           
=======================================
  Hits        23253    23253           
  Misses        436      436           
  Partials      628      628           

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.