docs: add holdout numbers and CMM zero-retrieval detail to v52 tables - #86
Open
suhaanthayyil wants to merge 1 commit into
Open
docs: add holdout numbers and CMM zero-retrieval detail to v52 tables#86suhaanthayyil wants to merge 1 commit into
suhaanthayyil wants to merge 1 commit into
Conversation
suhaanthayyil
force-pushed
the
docs/v52-quoting-rules
branch
from
August 7, 2026 20:37
1779623 to
8de1498
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://entire.io/gh/entireio/entire-graph/trails/39
Precision follow-up to the v52 documentation in #84. Two facts established by offline
recomputation of the
memory-native-v52raw cells are not currently disclosed next to thetables, and both change how the numbers should be quoted.
CMM's column is literally zero retrieval. The existing footnote already explains that
CMM v0.9.0 indexes the conversations as
Sectionnodes and that its public natural-languageBM25 route excludes that node type. What it does not say is the consequence: on that route all
1,050 CMM cells returned a byte-identical empty context (24 bytes, zero hits). The CMM QA
percentages are therefore the shared reader answering from the question alone, not a retrieval
score. One sentence added to the footnote in
README.mdand to the equivalent prose indocs/benchmarks.md.The 300 + 50 full set is not tune-disjoint. 35 of the 350 cases overlap the tune phase, so
the headline figures are not a clean generalization estimate. The sealed holdout is: 75.6%
LOCOMO QA accuracy and 0.900 recall@10 at n=30 (the LongMemEval-S holdout is n=5). Added beside
the full numbers in both files with the n visible, so a reader quoting the table can see which
number answers which question.
No numbers are changed or removed; this only adds the qualifiers that make the existing ones
safe to quote.