Skip to content

feat: prepare Entire Graph for Graphify advertised benchmark parity - #77

Closed
suhaanthayyil wants to merge 18 commits into
codex/eg-graphify-stable-basefrom
codex/eg-graphify-advertised-bench
Closed

feat: prepare Entire Graph for Graphify advertised benchmark parity#77
suhaanthayyil wants to merge 18 commits into
codex/eg-graphify-stable-basefrom
codex/eg-graphify-advertised-bench

Conversation

@suhaanthayyil

@suhaanthayyil suhaanthayyil commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • adds deterministic entity/phrase-aware search and progressive preselection for the Graphify advertised benchmark comparison
  • prefilters constraint-agreement scoring without changing bonuses, signals, or ordered retrieval results
  • parses snapshot files with a bounded deterministic worker pipeline while preserving original-order graph emission
  • adds lossless deterministic compact snapshot/query support with explicit cold-build phase and RSS metrics
  • preserves legacy search limits/coverage semantics and isolates failed worker metrics

Fairness scope

This branch is the candidate arm. The frozen stable arm remains commit 90a3346. The benchmark harness uses generic tools, identical prompts and limits, blinded grading, and disjoint tune/holdout selectors. No benchmark-specific query strings, prompts, candidate caps, or approximate retrieval paths are used. Results will be added only after the sealed tune and disjoint holdout complete.

Verification

  • combined semantic and CLI suites pass after integration; vet and semantic race suite pass on the provider branch
  • provider output is byte-identical to base on pinned ERPNext: 239,930 NDJSON lines, 121,318,933 bytes, SHA-256 9e963febbce6cf10c9fb97ead494db43c2678e740049b35a7d078489c1301c4c
  • controlled 600-file provider benchmark: 308.6 ms at one worker and 121.7 ms at eight workers (2.54x); no full-ERPNext speedup claim because the sequential cold pair was contaminated by one-time warmup
  • four old-vs-new full-profile CLI comparisons produced identical ordered result JSON for entity, quoted phrase, Unicode/compound, and unrelated queries
  • synthetic 60,000-candidate stress mix with 5 percent plausible matches: constraint agreement about 79 percent faster and about 85 percent fewer temporary bytes than the pre-change implementation
  • all-match synthetic worst case remains faster; the sealed ERPNext diagnostic rerun is still required before any production-performance claim
  • gofmt and diff checks clean
  • native, compact, and query smoke tests pass
  • compact snapshot is lossless/queryable and 76.7 percent smaller on the verified fixture

Current benchmark status

The 10 percent tune raw run completed 840 memory cells, 24 code-QA cells, and 72 temporal pairs with zero failed runs. Blinded scoring is still running. This draft does not claim that Entire Graph wins; promotion requires exact-equivalence diagnostics and a disjoint sealed holdout.

Known limitation

The self-repository snapshot was degraded (332/342 files parsed); the benchmark report will retain this rather than hiding it.

@suhaanthayyil
suhaanthayyil changed the base branch from main to codex/eg-graphify-stable-base August 1, 2026 21:28
@suhaanthayyil

Copy link
Copy Markdown
Collaborator Author

Superseded by #84, the single Entire Graph release PR now targeting main. The older PR is preserved for history; its branch is not deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant