perf(KnowledgeHarvester): single-pass backlink index instead of per-note O(n^2) rg scan#1574
Open
asdf8675309 wants to merge 1 commit into
Open
Conversation
…(n^2) rg scan regenerateMOC and expireStaleSeedlings counted each note's inbound [[wikilinks]] by shelling out `rg -c '[[<slug>'` across the entire KNOWLEDGE archive once per note. On a 6k-note corpus that is ~6,400 full-archive scans (one subprocess spawn each) every time a MOC is regenerated or a harvest expires seedlings — O(n^2), heavy I/O. computeBacklinks() now reads every note once and tallies each [[target]] into a slug->count Map, computed once per command run and passed into regenerateMOC/expireStaleSeedlings for O(1) lookups. Measured on a live 6,419-note archive: 619 s -> 0.13 s (about 2,000-4,900x). The single pass also tightens the count to true note-to-note references: exact target match (the old `[[slug` prefix let [[foo]] count toward [[foo-bar]]), distinct source notes rather than raw line hits, and it skips generated _*.md MOC files and self-links -- all of which the per-note rg over-counted. "Most Referenced" in a MOC is a cosmetic ranking and seedling expiry only branches on >0 references, so the tighter count is a safe, in-the-right-direction change. Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
regenerateMOCandexpireStaleSeedlingseach counted a note's inbound[[wikilinks]]by shelling outrg -c '[[<slug>'across the entireKNOWLEDGE archive once per note. Regenerating the MOCs (
index,promote)or expiring seedlings (
harvest) therefore runs ~N full-archive scans for Nnotes — O(n^2), with one subprocess spawn each. On a real 6,419-note archive
that is 6,419
rginvocations over a 42 MB corpus, measured at ~10 minutes fora single
index.Change
One new helper,
computeBacklinks(), reads every note once and tallies each[[target]]into aslug -> countMap. It's computed a single time per commandrun and passed into
regenerateMOC(domain, backlinks)andexpireStaleSeedlings(backlinks), which now do O(1) Map lookups instead ofspawning
rg. Both formerrg -csites are removed. One file, no API change, nonew dependency, and it removes a
child_processdependency from the hot path.Why it's safe
The single pass is more accurate than the per-note
rg, not merely equal —and the difference is in the safe direction (it removes over-counting):
rg -c '[[<slug>'was a prefix match, so[[foo]]wrongly counted toward[[foo-bar]]. The new index matches the exacttarget.
rg -c); anote that linked a target on 3 lines counted as 3. The new index counts
distinct source notes.
_*.mdfiles and self-links. The old scan ran over thewhole
KNOWLEDGE_DIR, so each domain's auto-generated_index.md(which listsnearly every note as a
[[link]]) inflated real note-to-note counts, and a noteciting its own slug counted itself. The new index skips
_*.mdand self-links.Both consumers tolerate the tighter count safely: "Most Referenced" in a MOC is a
cosmetic ranking, and seedling expiry only branches on
> 0references. Thepractical effect is that expiry now keys off real inbound note-links rather
than MOC self-listing, which is the intended behavior. Empirically the new count
is always
<=the old (see table): across the full 6,419-note corpus, 6,257differed and every difference was the old count being higher (NEW is never
higher).
Verification
Benchmarked against the real 6,419-note archive (READ-ONLY), old approach
reproduced faithfully from the pre-change source, new approach run verbatim.
rg -c, full corpus)computeBacklinks, single pass)cloudflare689 -> 5)Tested on macOS 25.5, Bun 1.3.14, ripgrep 15.2.0. File transpiles clean
(
bun build --no-bundle --target=bun, exit 0).Scope
One file (
KnowledgeHarvester.ts). No change to the CLI surface, note schema,staging/promote flow, or any other function.
computeBacklinksis a pure,self-contained function (no I/O beyond the reads the code already did). No new
dependencies; a
child_process/rgdependency is removed from the MOC path.