Skip to content

docs: add holdout numbers and CMM zero-retrieval detail to v52 tables #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
suhaanthayyil wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: add holdout numbers and CMM zero-retrieval detail to v52 tables #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,15 +156,24 @@ competitor.
\* CMM v0.9.0 indexed the Markdown conversations as `Section` nodes, but its
public natural-language `search_graph(query=...)` BM25 path excludes that node
type. Its separate `search_code` verb can find literal source phrases, but the
unchanged benchmark question is not a literal source pattern. The CMM column
therefore reports that public route on an off-domain corpus and is not a claim
about CMM's code-search quality.
unchanged benchmark question is not a literal source pattern. That route
retrieved nothing at all here: all 1,050 CMM cells returned a byte-identical
empty context (24 bytes, zero hits), so the CMM QA percentages are the shared
reader answering from the question alone. The CMM column therefore reports that
public route on an off-domain corpus and is not a claim about CMM's code-search
quality.

**Holdout (tune-disjoint).** The 300 + 50 full set is not tune-disjoint: 35 of
the 350 cases overlap the tune phase. The clean generalization estimate is the
sealed holdout, where Entire Graph scored 75.6% LOCOMO QA accuracy and 0.900
recall@10 (n=30; the LongMemEval-S holdout is n=5).

All 3,150 raw cells, 3,150 blinded primary grades, and 630 fixed audit cells
passed the sealed integrity gates. The Opus audit agreed with the primary judge
on 98.41% of cells (Cohen's kappa 0.9682), and there were zero invalid attempts.
Across all 350 paired memory cases, Entire minus Graphify semantic accuracy was
+0.1648 (cluster 95% CI +0.1034 to +0.2146; McNemar p=1.70e-11).
+0.1648 (95% CI +0.1034 to +0.2146, a bootstrap clustered by conversation over
10 effective LOCOMO clusters; McNemar p=1.70e-11).

This is a **public-protocol reimplementation**, not a reproduction of
Graphify's historical README run: Graphify's advertised memory harness and
Expand Down
10 changes: 9 additions & 1 deletion docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,15 @@ LOCOMO QA accuracy, and 76.0% vs. 68.0% on LongMemEval-S QA accuracy. All
systems used zero build-time LLM credits. Codebase Memory MCP is present in the
release as an off-domain native diagnostic: v0.9.0 indexes the Markdown content
as `Section` nodes but excludes that node type from its public natural-language
BM25 route, so its column is not a third apples-to-apples memory comparison.
BM25 route, so its column is not a third apples-to-apples memory comparison. On
that route it retrieved nothing at all — all 1,050 CMM cells returned a
byte-identical empty context (24 bytes, zero hits), so its QA number is the
shared reader answering from the question alone.

Those 300 + 50 cases are not tune-disjoint: 35 of the 350 overlap the tune
phase. The tune-disjoint holdout is the clean generalization estimate — Entire
Graph scored 75.6% LOCOMO QA accuracy and 0.900 recall@10 on it (n=30; the
LongMemEval-S holdout is n=5).

The product under test is commit
`c9641bf1caaf41d64ce8a4a421f041939feecca3`. Its native JSON result contains a
Expand Down