feat(search): index markdown section bodies for BM25 full-text search (#518)

[ShauryaaSharma]

Slice 1 of the #617 split, per the review guidance there: the FTS body infrastructure plus Markdown Section indexing. #617 stays open as the ledger; the #519 YAML/JSON Module description promotion follows as a second PR stacked on this one.

Rebuilt against current main rather than rebased — the original branch was ~1100 commits behind and pipeline.c, mcp.c and store.c had all been reworked around the touched regions, so the old diff served as the design spec.

What & why

search_graph BM25 only matched node names and headings, so it was blind to markdown prose. Section nodes exposed only their heading text and were excluded from BM25 results entirely. This indexes the section body so content is searchable.

Closes #518.

Changes

store — adds a body column to the nodes_fts FTS5 table. New cbm_store_fts_rebuild() drops and recreates the table (which is what upgrades legacy 4-column databases) and backfills body from each node's docstring, guarded by json_valid() so malformed-JSON rows degrade to name-only indexing instead of aborting the whole INSERT...SELECT.

CBM_SQL_FTS_BODY_EXPR is exported from store.h so every nodes_fts write site feeds body through one expression.

pipeline — the wholesale backfill delegates to cbm_store_fts_rebuild() (net −17 lines). The row-level delta-merge insert in pipeline_delta.c writes body through the same shared expression.

That third write site is the one worth reviewing closely. nodes_fts now has five columns, and an INSERT naming only the original four is still perfectly valid SQL that silently leaves body NULL — prose arriving through delta merge would have been unsearchable while a full reindex looked correct. It has its own test rather than relying on the full-index path.

mcp — stops excluding Section from BM25 results in both the search and count queries. Section falls in the unboosted ELSE 0.0 bucket of the label-boost CASE, so code symbols keep their ranking advantage by construction rather than by exclusion. Module stays excluded here and is handled in the #519 slice.

internal/cbm — captures the markdown section body beneath each heading, stopping at the first nested subsection (each subsection gets its own Section node and its own body). Capped at the existing MAX_COMMENT_LEN (500) with a UTF-8-safe backoff, reusing the existing docstring property.

Testing

3 extraction cases and 4 store FTS cases:

• markdown_section_body_captured — body captured, sibling section excluded
• markdown_section_no_body — empty heading yields no docstring
• markdown_section_body_capped — body ≤ 500 bytes
• fts_rebuild_indexes_body_content — body tokens searchable, names still searchable
• fts_rebuild_upgrades_legacy_schema — legacy 4-column table upgraded in place
• fts_rebuild_tolerates_malformed_properties — json_valid() guard holds
• fts_delta_insert_populates_body — the delta-path guard described above

The extractor's grammar assumption was verified directly against tree_sitter_markdown rather than assumed: for nested headings the parser produces section [0-81] containing the heading, a paragraph, and a nested section [48-81], and the helper correctly stops at byte 48.

Notes

Backward compatible — additive column, legacy databases keep working read-only and are upgraded by the DROP+recreate at the next index run, which was already a full FTS rebuild.

One operational caveat worth stating rather than leaving to be discovered: the DROP+recreate opens a brief window during the schema upgrade where a concurrent bm25_search finds no table and degrades to the regex path. It is one-time per database and degrades gracefully, but with daemon-era concurrent sessions it is observable.

No MCP tool changes, no new dependencies, no new system()/popen()/network calls.

Performance evidence — still outstanding

I owe measured numbers for this slice and do not have them yet. scripts/benchmark-index.sh needs a built product binary and shells out to python3; my local toolchain is 32-bit MinGW.org GCC 6.3.0 with no make, and a 32-bit address space will not hold anything near the reference corpus, so any figure I produced here would be the wrong measurement at the wrong scale.

What I can measure is the narrowed question: since the full-index path already did delete-all plus a full re-INSERT, this change makes an existing rebuild index more text rather than adding a pass, so the real delta is per-row body tokenisation and the storage the column adds. I can build a standalone harness against the vendored SQLite with FTS5 measuring index size and tokenisation wall-clock with and without the body column, and the same pair with a WHERE label IN (...) lever applied, and commit it so it is reproducible.

Happy to build that if it is the evidence you want, or to defer to a run of benchmark-index.sh on the reference corpus or in CI. Flagging it up front rather than having it discovered in review.

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[DeusData]

Read in full. The structure is right and slice 1 is complete with no slice-2 leakage. But the delta test has exactly the hole it was written to close, and there is a second behavioural change the description does not mention. Both are fixable; neither is a rethink.

The delta test does not guard the delta call site

This is the important one, and I checked it rather than reading the name and moving on.

fts_delta_insert_populates_body never calls cbm_delta_patch() or any production entry point. It hand-prepares its own INSERT INTO nodes_fts (...) inside the test body — and the copy is not even identical to production, since it uses plain name where pipeline_delta.c uses cbm_camel_split(name).

So: revert pipeline_delta.c's INSERT to the original four columns and this test still passes. It guards CBM_SQL_FTS_BODY_EXPR; it does not guard the site that consumes it. That is precisely the failure mode you described better than I did — no compile error, nothing red, body silently NULL on the path users hit daily — and the test written to catch it cannot.

The fix is to drive the production delta path: build a base index, add nodes, run the real delta merge, then assert fts_match_count finds body text from the newly merged rows. If that is awkward to reach from test_store_search.c, it belongs wherever cbm_delta_patch is already exercised. A test that reimplements the thing it is testing proves the reimplementation.

A fourth write site exists

tests/test_mcp.c:2489 does a delete-all plus a four-column INSERT INTO nodes_fts(rowid, name, qualified_name, label, file_path). It stays valid SQL against a five-column table, so nothing goes red — it just leaves body NULL and stops mirroring production shape.

Not a correctness bug today, but the store.h comment claims CBM_SQL_FTS_BODY_EXPR covers every nodes_fts write site, and that is now inaccurate. Either route that site through the macro too or narrow the claim, because a comment that overstates its own coverage is how the next person gets caught.

BM25 ranking changes for every query, including nodes with no body

bm25(nodes_fts) is called at mcp.c:3025 and :3062 with no column weights, so every column carries the default 1.0. Adding a fifth column changes the score for every row in the table — including rows whose body is empty. Existing search_graph result ordering shifts for queries that have nothing to do with prose.

The PR body says "backward compatible — additive column". That is true of the schema and not of the ranking. This is worth an explicit decision rather than an accident: either accept the shift and say so, or pass explicit column weights to bm25() so the four original columns keep their current relative influence and body is added deliberately. I lean towards explicit weights — it makes the ranking a stated design rather than a side effect of column count — but I would rather hear your reasoning first, since you have been closer to the FTS behaviour than I have.

Alongside that: Section is now returned by BM25 and counted in the total, which changes both the result set and the count for existing callers. That is the intended effect of the direction call I gave you, so it is correct — it just belongs in the description as a behavioural note.

Two test weaknesses

• markdown_section_no_body is vacuous. On main every Section docstring is already NULL, so ASSERT(body == NULL || body[0] == '\0') passes unchanged with or without your production code. It needs to assert something only true after the change — for instance that a sibling section in the same file does have a body while this one does not.
• markdown_section_body_capped hardcodes 500 rather than MAX_COMMENT_LEN, which reintroduces exactly the parallel-constant coupling you removed from production. And its corpus is "alpha " repeated — pure ASCII — so the UTF-8 backoff never executes. A multi-byte character straddling the cap is the case that backoff exists for.

Mechanical

lint fails on src/store/store.h:361: CBM_SQL_FTS_BODY_EXPR is one 273-byte physical line. Valid C via adjacent string concatenation, but past the column limit — split it across continuations and it clears.

Worth knowing that this is currently blocking everything else: the test job is skipped the moment lint fails, so none of your seven tests have executed in CI even once. Fix the line and you get a real signal.

What is right, and it is most of it

CBM_SQL_FTS_BODY_EXPR as a single definition consumed by both production sites is the correct structural answer — it converts "remember to update both" into "there is only one", which is the durable fix for the class of bug we found. The DDL is likewise single-source. fts_rebuild_tolerates_malformed_properties genuinely guards the json_valid() wrapper, and fts_rebuild_upgrades_legacy_schema proving a four-column database upgrades in place is the test most people would have skipped. Documenting the DROP+CREATE window in the body, the commit message and at the function is three places more than it needed to be.

Scope is clean: Module stays excluded in both queries, no YAML/JSON promotion, nothing from slice 2 leaked in.

On the benchmark harness — you committed it about twenty minutes after asking whether it was wanted. I have since said yes on #617, so keep it; I mention it only because "asked, then shipped before the answer" is a habit worth not forming. It is 477 of the 859 lines here, which is also worth a sentence in the description so a reviewer knows 55% of the diff is a measurement tool rather than the feature.

Fix the delta test, decide the BM25 weights question, tighten the two tests and split that header line, and I will merge this. Then I will run the real-corpus benchmark on my side as promised.