Skip to content

docs: publish verified Graphify native-memory comparison - #83

Closed
suhaanthayyil wants to merge 2 commits into
codex/eg-graphify-advertised-benchfrom
codex/eg-graphify-native-memory-v45
Closed

docs: publish verified Graphify native-memory comparison#83
suhaanthayyil wants to merge 2 commits into
codex/eg-graphify-advertised-benchfrom
codex/eg-graphify-native-memory-v45

Conversation

@suhaanthayyil

@suhaanthayyil suhaanthayyil commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • publish the verified full Graphify/Entire Graph/CMM native-memory comparison in the README
  • add the frozen revisions, fairness boundary, integrity record, and external evidence link to benchmark docs
  • state the split result honestly: Entire Graph wins LOCOMO retrieval recall; Graphify wins final-answer accuracy
  • skip the max-RSS guard assertion on Windows, where the benchmark explicitly reports RSS as unavailable; production code and the benchmarked binary are unchanged

Result

  • LOCOMO recall@10: Entire Graph 0.918, Graphify 0.844
  • LOCOMO QA: Entire Graph 51.0%, Graphify 67.4%
  • LongMemEval-S QA: Entire Graph 44.7%, Graphify 62.7%
  • graph-build LLM credits: 0 for Entire Graph, Graphify, and CMM

Boundary

This PR is stacked on the exact benchmarked candidate commit dea450b. Its only non-documentation change is a test-only Windows skip for a metric that rss_windows.go deliberately cannot measure. It does not change the measured binary. The comparison is explicitly labeled a public-protocol reimplementation because Graphify does not publish its advertised memory harness or selectors.

The protocol, 1%/10%/100% sequence, fairness audit, and artifact hashes are under review in entirehq/graphmark#63.

Validation

  • go vet ./...: passed (upstream tree-sitter Lua compiler warning only)
  • go test ./...: passed locally
  • git diff --check: passed
  • prior Windows CI failure was isolated to the unsupported RSS assertion; follow-up CI is the cross-platform confirmation

Stacked on #77. Review-only: do not merge until both benchmark PRs are approved.

@suhaanthayyil

Copy link
Copy Markdown
Collaborator Author

Superseded by #84, the single Entire Graph release PR now targeting main. The older PR is preserved for history; its branch is not deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant