Skip to content

perf(KnowledgeHarvester): single-pass backlink index instead of per-note O(n^2) rg scan#1574

Open
asdf8675309 wants to merge 1 commit into
danielmiessler:mainfrom
asdf8675309:perf/knowledgeharvester-backlink-index
Open

perf(KnowledgeHarvester): single-pass backlink index instead of per-note O(n^2) rg scan#1574
asdf8675309 wants to merge 1 commit into
danielmiessler:mainfrom
asdf8675309:perf/knowledgeharvester-backlink-index

Conversation

@asdf8675309

Copy link
Copy Markdown
Contributor

What

regenerateMOC and expireStaleSeedlings each counted a note's inbound
[[wikilinks]] by shelling out rg -c '[[<slug>' across the entire
KNOWLEDGE archive once per note. Regenerating the MOCs (index, promote)
or expiring seedlings (harvest) therefore runs ~N full-archive scans for N
notes — O(n^2), with one subprocess spawn each. On a real 6,419-note archive
that is 6,419 rg invocations over a 42 MB corpus, measured at ~10 minutes for
a single index.

Change

One new helper, computeBacklinks(), reads every note once and tallies each
[[target]] into a slug -> count Map. It's computed a single time per command
run and passed into regenerateMOC(domain, backlinks) and
expireStaleSeedlings(backlinks), which now do O(1) Map lookups instead of
spawning rg. Both former rg -c sites are removed. One file, no API change, no
new dependency, and it removes a child_process dependency from the hot path.

Why it's safe

The single pass is more accurate than the per-note rg, not merely equal —
and the difference is in the safe direction (it removes over-counting):

  • Exact target match. The old rg -c '[[<slug>' was a prefix match, so
    [[foo]] wrongly counted toward [[foo-bar]]. The new index matches the exact
    target.
  • Distinct source notes. The old code summed matching lines (rg -c); a
    note that linked a target on 3 lines counted as 3. The new index counts
    distinct source notes.
  • Skips generated MOC/_*.md files and self-links. The old scan ran over the
    whole KNOWLEDGE_DIR, so each domain's auto-generated _index.md (which lists
    nearly every note as a [[link]]) inflated real note-to-note counts, and a note
    citing its own slug counted itself. The new index skips _*.md and self-links.

Both consumers tolerate the tighter count safely: "Most Referenced" in a MOC is a
cosmetic ranking, and seedling expiry only branches on > 0 references. The
practical effect is that expiry now keys off real inbound note-links rather
than MOC self-listing, which is the intended behavior. Empirically the new count
is always <= the old (see table): across the full 6,419-note corpus, 6,257
differed and every difference was the old count being higher (NEW is never
higher).

Verification

Benchmarked against the real 6,419-note archive (READ-ONLY), old approach
reproduced faithfully from the pre-change source, new approach run verbatim.

Metric Value
Corpus (4 domains) 6,419 notes / 42 MB
OLD (per-note rg -c, full corpus) 619 s (~10 min), 96.5 ms/slug
NEW (computeBacklinks, single pass) 0.13 s warm (0.30 s cold)
Speedup ~2,000–4,900x
Count parity (OLD vs NEW), all 6,419 162 identical, 6,257 differ, all OLD-higher (NEW never higher) = the over-count corrections above (e.g. cloudflare 689 -> 5)

Tested on macOS 25.5, Bun 1.3.14, ripgrep 15.2.0. File transpiles clean
(bun build --no-bundle --target=bun, exit 0).

Scope

One file (KnowledgeHarvester.ts). No change to the CLI surface, note schema,
staging/promote flow, or any other function. computeBacklinks is a pure,
self-contained function (no I/O beyond the reads the code already did). No new
dependencies; a child_process/rg dependency is removed from the MOC path.

…(n^2) rg scan

regenerateMOC and expireStaleSeedlings counted each note's inbound
[[wikilinks]] by shelling out `rg -c '[[<slug>'` across the entire
KNOWLEDGE archive once per note. On a 6k-note corpus that is ~6,400
full-archive scans (one subprocess spawn each) every time a MOC is
regenerated or a harvest expires seedlings — O(n^2), heavy I/O.

computeBacklinks() now reads every note once and tallies each [[target]]
into a slug->count Map, computed once per command run and passed into
regenerateMOC/expireStaleSeedlings for O(1) lookups. Measured on a live
6,419-note archive: 619 s -> 0.13 s (about 2,000-4,900x).

The single pass also tightens the count to true note-to-note references:
exact target match (the old `[[slug` prefix let [[foo]] count toward
[[foo-bar]]), distinct source notes rather than raw line hits, and it
skips generated _*.md MOC files and self-links -- all of which the
per-note rg over-counted. "Most Referenced" in a MOC is a cosmetic
ranking and seedling expiry only branches on >0 references, so the
tighter count is a safe, in-the-right-direction change.

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant