Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,12 @@ The most robust result is retrieval, which is measured without an agent and so i
file the gold patch edits reaches the agent in **96.3%** of sessions versus **81.5%** for the
closest comparable tool, at equal payload bytes and with fewer search calls.

In a separate full conversational-memory comparison, Entire Graph leads Graphify on LOCOMO
recall@10 (**0.918 vs 0.844**) but trails on final-answer accuracy: LOCOMO **51.0% vs 67.4%** and
LongMemEval-S **44.7% vs 62.7%**. That split matters: better evidence-session retrieval did not
translate into better answers under the shared reader. See [Numbers, honestly](#numbers-honestly)
for the exact public-protocol boundary.

Read [Numbers, honestly](#numbers-honestly) before quoting any of that. In short: the figure is
Haiku-specific and we **cannot** measure it on stronger models at these sample sizes; a **≥35%**
saving is *refuted*, not merely unproven; two earlier claims (55%, then 31.6%) were withdrawn after
Expand Down Expand Up @@ -144,6 +150,42 @@ You (a human) will mostly experience it *through* those surfaces. Your agents ca

## Numbers, honestly

### Native conversational-memory retrieval

The frozen candidate at `dea450b` was evaluated against Graphify v8/v0.9.34 at `07b9143` and
Codebase Memory MCP v0.9.0 on the same official 300 LOCOMO and 50 cleaned LongMemEval-S cases,
three repetitions per case. Kimi K3 was the shared reader and blinded primary grader; Opus 5
audited a deterministic 20%. Every arm had top 10 and a 128,000-byte ceiling, separate caches,
identical questions, and no shared source reread.

| Full native-memory run | Entire Graph | Graphify | cmm |
|---|---:|---:|---:|
| LOCOMO recall@10 (n=300) | **0.918** | 0.844 | 0.000 |
| LOCOMO QA accuracy (n=300) | 51.0% | **67.4%** | 22.3% |
| LongMemEval-S QA accuracy (n=50) | 44.7% | **62.7%** | 6.0% |
| graph-build LLM credits | 0 | 0 | 0 |

There is no blanket winner. Entire Graph retrieves the evidence session more often; Graphify
answers more questions correctly. All 3,150 raw cells, 3,150 Kimi grades, and 630 Opus audits
passed the sealed gates with zero invalid attempts; audit agreement was 97.94% (kappa 0.9583).
The list-price equivalent for reader, grading, and audit was $50.82. Provider-billed spend was not
available for every Fireworks cell, so the smaller known-actual subtotal is not reported as total
spend.

This is a **public-protocol reimplementation**, not a reproduction of Graphify's historical
claims. Graphify's advertised memory harness and selectors are not public. The Graphify reader
received its native BFS-rendered graph text; a separate projection of the same traversal onto
session files supplied recall@10. Entire Graph supplied only native search snippets and locator
headers. cmm's shipped Markdown search excludes `Section` nodes from BM25, so its result describes
that public Markdown path, not its quality on programming-language graphs.

The full protocol, fairness explanation, exact artifact hashes, 1%/10%/100% sequence, and results
are in [GraphMark PR #63](https://github.com/entirehq/graphmark/pull/63). The benchmarked binary is
from `dea450b`; these documentation changes do not change it.

See [docs/benchmarks.md](docs/benchmarks.md#external-native-memory-comparison) for the frozen
identities and integrity record.

All savings are measured against the **baseline**: the same agent, same model, same task, with
no code tool at all. For scale, the same suites also ran
[codebase-memory-mcp](https://github.com/DeusData/codebase-memory-mcp) ("cmm"), an open-source
Expand Down
4 changes: 4 additions & 0 deletions cmd/graph-bench/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"encoding/json"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
Expand Down Expand Up @@ -55,6 +56,9 @@ func TestValidateExecutionModeRejectsParentOnlyCPUProfile(t *testing.T) {
}

func TestMaxRSSGuardFailsRunEvenWhenViolatingRowIsExcludedFromAggregates(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("process RSS is not available on this platform")
}
dir := t.TempDir()
manifestPath := filepath.Join(dir, "manifest.json")
if err := os.WriteFile(manifestPath, []byte(`{"languages":{"Go":["owner/repo"]}}`), 0o644); err != nil {
Expand Down
50 changes: 50 additions & 0 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,56 @@ Treat the numbers as historical; re-run with the current streaming benchmark
tiny constant per relation rather than the full payload, which is what kept
the in-memory path from finishing on the largest repos.

## External native-memory comparison

GraphMark's `memory-native-v2` suite evaluated this repository's candidate
commit `dea450b924e0ba4e9b5a7dc3ae5db72cf3aa857a` alongside Graphify and
Codebase Memory MCP. This is an external public-protocol reimplementation of
Graphify's advertised memory surfaces, not a `cmd/graph-bench` run and not a
reproduction of Graphify's unpublished harness.

Frozen product identities:

- Entire Graph candidate: `dea450b924e0ba4e9b5a7dc3ae5db72cf3aa857a`
- Entire Graph stable reference: `90a3346a624d76f7fee21bd894721e5438dd9ac2`
- Graphify v8/v0.9.34: `07b9143d4b90b1e1cb88dc71423f742a501efd29`
- Codebase Memory MCP v0.9.0: `b637e3330c96cfe452da623db068c241aaa3ec01`
- GraphMark harness: `09d14f2f149e8459d98b30ecf6bf1a31c757f04d`
- protocol SHA-256: `524e64ce2b0e8c888169d68d990466bed71336e2d7cd30b2723a78ba679fdb1b`
- full completion SHA-256: `30c8c19c24d7077abc28b30bb5853391b3f86700801f5e9a371acec0c1938a76`

The full run used 300 LOCOMO and 50 LongMemEval-S cases, three repetitions,
Kimi K3 reader/grader, a deterministic 20% Opus 5 audit, top 10, and a shared
128,000-byte context ceiling. It followed a sealed 1% validation and untouched
10% validity holdout; the full run was outcome-independent. Each product has
exactly 1,050 raw cells with identical case/question/repetition topology.

Entire Graph receives no benchmark-specific query rewrite. For each original
question the adapter executes `entire graph search --format json --top-k 10`
and gives the shared reader only the returned native snippets and neutral
locator headers. It does not open surrounding files after search. The build and
query cache is isolated from every other arm.

The verified full result is:

| Metric | Entire Graph | Graphify | cmm |
|---|---:|---:|---:|
| LOCOMO recall@10 | **0.918** | 0.844 | 0.000 |
| LOCOMO QA accuracy | 51.0% | **67.4%** | 22.3% |
| LongMemEval-S QA accuracy | 44.7% | **62.7%** | 6.0% |
| graph-build LLM credits | 0 | 0 | 0 |

Integrity record: 3,150/3,150/630 raw/grade/audit cells, zero invalid
attempts, promotion passed with zero findings, final artifact verification
passed, and Opus agreement was 97.94% (kappa 0.9583). Five transient raw-reader
timeouts and two transient grader timeouts were recorded and recovered within
the sealed retry policy; no audit retry occurred.

The publication bundle and review are in
[entirehq/graphmark#63](https://github.com/entirehq/graphmark/pull/63). Raw
artifacts remain outside Git because they occupy about 420 MB; the bundle
publishes their SHA-256 values and the exact reproduction contract.

## Notes

- A full-tier run is heavy (the manifest includes linux, tensorflow, vscode,
Expand Down