Skip to content

DETAILS.md

Latest commit

 

History

History
355 lines (251 loc) · 21.4 KB

DETAILS.md

File metadata and controls

355 lines (251 loc) · 21.4 KB
Raw

🧠 entire-graph

License Languages Graph Local Deterministic Accuracy Plugin

A deterministic, local-first code graph for coding agents, and the semantic layer behind Entire checkpoints. entire-graph parses your repository with tree-sitter and answers structural questions — where is this defined, who calls it, what changed at the entity level, what will this break — straight from a persistent knowledge graph of functions, classes, call chains, routes, and cross-service links. No model calls, no embeddings, no network. The same commit always produces the same graph.

It ships as an Entire CLI plugin, invoked as entire graph ..., and doubles as a local-only semantic provider that streams a machine-readable graph of symbols and relations for downstream tools such as Entire Brain.

entire graph search   --repo . --query "where is webhook retry handled?"   # 🔍 ranked code for a task
entire graph snapshot --repo . --format ndjson                             # 🕸️  full symbol + relation graph
entire graph snapshot --repo . --format compact-ndjson > graph.compact.ndjson # compact full graph artifact
entire graph snapshot-query --input graph.compact.ndjson --symbol Cache.Refresh --format ndjson
entire graph diff     --base main --head HEAD                               # 🧬 what changed, at the entity level
entire graph capabilities --json                                           # 🧭 languages + relation types

🔬 Accuracy first. On a frozen 21-repo multi-language board of fixed-answer semantic tasks (definition lookup, call graph, imports, change-impact) scored on real checkouts, entire-graph scored 265/283 (94%) versus 191/283 (68%) for the leading tree-sitter code-memory tool. A wrong edge is worse than a missing one, so the graph is built to be right about symbols and impact, not just fast at emitting them.

🔒 Security and trust. entire-graph reads your codebase and writes only to Entire's managed plugin directory. All processing happens 100% locally: no network, no telemetry, no API keys, no grammar downloads at runtime. Your code never leaves your machine. entire graph doctor --json reports no_egress=true.

Are you a coding agent (or configuring one)? The operating instructions — the parts of the graph, the exact commands, and the query-before-grep doctrine — live in AGENTS.md (mirrored in CLAUDE.md). This README is for humans installing and running the plugin.

📑 Contents

🎯 What it is

entire-graph builds a code graph — nodes for files, packages, functions, classes, methods, types, routes, and resources; 30 relation types for calls, construction, inheritance, field access, service boundaries, and config/infra dependencies — and answers structural questions from it:

  • Where is this defined? → symbols
  • Who calls this / what does it call? → neighbors, edges
  • Find the code for this task → search (hybrid, ranked)
  • What changed, and what does it put at risk? → diff / commit / checkpoint
  • Give me the whole graph → snapshot

Everything is deterministic (pure tree-sitter static analysis — no LLM, no vectors, no similarity thresholds), git-native (every result keyed by (repo, commit, tree), so it is content-addressed and cacheable per commit), and 100% local (no keys, no network, no telemetry). It scales to millions of relations on one machine and releases memory after each build.

🚀 Install

Prerequisites: the Entire CLI · Git · a Go toolchain with CGO enabled (tree-sitter uses native parser bindings).

# 1. Install the plugin binary and register it with Entire's managed plugin dir
go install github.com/entireio/entire-graph/cmd/entire-graph@main
entire plugin install "$(go env GOPATH)/bin/entire-graph" --force

# 2. Verify the install and the local-only environment
entire graph version
entire graph doctor --json        # includes "no_egress": true

# 3. Run your first semantic diff, from any git repository
entire graph commit HEAD

Entire plugins are local executables, not a hosted marketplace: entire graph works because Entire finds an entire-graph binary in its managed plugin directory or on $PATH. If $(go env GOPATH)/bin is already on your PATH, Entire can discover the binary directly after go install.

🛠️ Install from source

git clone https://github.com/entireio/entire-graph.git
cd entire-graph
mise run build                        # go build -o entire-graph ./cmd/entire-graph
entire plugin install ./entire-graph --force

scripts/install-local.sh does this in one command (builds, installs, prints entire graph version, and fails early if the parent entire CLI is not on PATH). For release archives with SHA256SUMS, run scripts/release.sh. See docs/operations.md for target and cgo details.

🔄 Updating (and migrating from entire-sem)

This plugin was renamed from entire-sem to entire-graph after v0.1.0 — both the binary and the cmd/ path changed. To update, or to switch over from the old entire-sem:

go install github.com/entireio/entire-graph/cmd/entire-graph@latest
entire plugin install "$(go env GOPATH)/bin/entire-graph" --force
rm -f "$(go env GOPATH)/bin/entire-sem"   # remove the old binary if you had it

…/cmd/entire-graph@latest needs v0.2.0 or newer — v0.1.0 predates the rename and only ships cmd/entire-sem, so @latest against it fails with "module found (v0.1.0), but does not contain package …/cmd/entire-graph". If a fresh tag hasn't propagated to the Go module proxy yet, pin it (@v0.2.0) or use @main. Graph and query behavior is unchanged — only the name.

🔌 MCP & Entire Brain integration

entire-graph is not itself an MCP server, and it exposes no MCP tools. It is a local CLI plugin and a machine-readable semantic provider. There is nothing to configure as an MCP endpoint here, and no daemon to run.

MCP access to code intelligence comes from Entire Brain (entire-brain), a separate downstream product. Entire Brain ingests entire-graph's NDJSON output (snapshot / symbols / edges) into its own persistent store and is what exposes MCP tools to agents (brain_search, brain_code, brain_impact, brain_context, and so on). So the split is:

Layer Role How you use it
entire-graph (this repo) Deterministic graph engine + NDJSON provider entire graph <command> on the CLI (see AGENTS.md)
Entire Brain (separate) Persistence, indexing, query, MCP server MCP tools in your agent, backed by the graph above

If you want MCP-style querying inside an agent, install and configure Entire Brain (its own docs) — entire-graph is the graph it builds on. If you just want direct, no-egress graph queries from an agent or the terminal, call entire graph ... directly; no MCP layer is required.

♻️ Refresh & keeping the graph current

entire-graph has no watch mode, no daemon, and no file-watcher, and it needs none. The graph is deterministic and content-addressed by (repo, commit, tree), so "refreshing" is just re-running a query — the result reflects whatever state you point it at.

There are two modes, and neither can go stale on you:

  • Working tree (default). Every search, neighbors, and snapshot re-reads your live files, so results always include your uncommitted edits. Nothing to refresh — just run the command again.
  • Committed tree (--head). Results are cached by the git tree hash. The cache directory defaults to the platform's per-user cache directory (~/Library/Caches/entire-graph on macOS, $XDG_CACHE_HOME/entire-graph or ~/.cache/entire-graph elsewhere); --cache-dir and ENTIRE_PLUGIN_DATA_DIR override it, in that order. Make a new commit and the tree hash changes, so the cache automatically misses and rebuilds — you can never read a committed-tree answer that is stale for that commit. --no-cache disables the cache entirely. Caching only ever changes latency, never results: measured on this repository, a cold --head --profile full search is 9.2s wall and the same query warm is 1.0s, with output byte-identical to --no-cache.

To warm the cache for a large repo before a latency-sensitive session, prebuild it once:

entire graph index --repo . --head --profile full --cache-dir /path/to/cache

Re-running index is the refresh: same tree → instant cache hit; changed tree → a fresh build. After that, search and neighbors on the same committed tree report the cache hit directly. This is the whole "keep it current" story — deterministic re-index, no background process.

⚡ Quick Start

# What can this graph do here?
entire graph capabilities --json          # languages (semantic vs inventory-only) + relation types
entire graph doctor --json                # environment, repo resolution, no_egress=true

# Find the code for a task (ranked, with source and file:line)
entire graph search --repo . --query "retry logic for webhook delivery" --format text --top-k 8

# See what changed in the last commit, at the entity level
entire graph commit HEAD

# Stream the whole graph for another tool to ingest
entire graph snapshot --repo . --format ndjson

Running outside an Entire session? Point the plugin at a repo with --repo . (or ENTIRE_REPO_ROOT=/path/to/repo).

🧩 Commands

Command What it does
entire graph search --query "..." 🔍 Ranked source regions for a natural-language task
entire graph neighbors --symbol NAME_OR_ID Callers / callees / relations for one symbol (impact)
entire graph symbols --format ndjson Full stream of symbol definitions
entire graph edges --format ndjson Full stream of relations (all 30 types)
entire graph snapshot --format ndjson 🕸️ Full graph: header + files + symbols + relations
entire graph commit <rev> Entity-level semantic diff of a commit vs its first parent
entire graph diff --base <a> --head <b> Semantic diff between two refs (analyze is an alias)
entire graph checkpoint <id> Semantic diff for the commit behind an Entire checkpoint trailer
entire graph index --head Prebuild / warm the durable committed-tree cache
entire graph capabilities --json Languages, relation types, features, network requirements
entire graph doctor --json Environment, repo resolution, plugin data dir, no_egress=true
entire graph version [--json] Provider name and plugin version

🔍 How search ranks (file-class prior + duplicate collapsing)

search answers a code question: the caller's next action is an edit to a source file. Two classes of result are systematically over-scored by a body-text match and almost never the edit site, so both are corrected explicitly:

Class Examples Prior
source src/**, headers, any parsed language file 1.0
documentation .md .mdx .rst .adoc .txt, docs/, website/, versioned_docs/, README, CHANGELOG, man pages 0.5
vendored vendor/, node_modules/, third_party/, site-packages/ 0.5
generated dist/, generated/, single_include/, *-lock.json, Cargo.lock 0.5
example examples/, samples/, demo/ 0.75

The prior is a multiplier on the positive part of a hit's score, never a filter. 0.5 means "a documentation hit has to be twice as relevant as the best source hit to outrank it" — a doc still ranks first when it is genuinely the only match, and the prior is switched off completely when the query itself asks for that class (documentation, readme, changelog, example, dist, vendored, …). Demoted hits are labelled doc-prior / vendored-prior / generated-prior / example-prior in signals.

Near-duplicate collapsing. Repositories carry deliberate copies of the same content — versioned documentation trees (version-2.x/x beside version-3.0.1/x), vendored snapshots, generated mirrors — and every copy scores identically, so one document can consume the whole result budget. Copies are merged into the best-ranked one, which reports a +N similar signal; the freed slots go to genuinely different code. Collapsing requires distinct files with the same basename, outside different monorepo units, and either identical normalized region text or paths equal modulo version-like segments with ≥ 90% token overlap.

Full flags and the agent-facing operating guide are in AGENTS.md. Diff commands print human-readable text by default and structured output with --json:

When a neighbor lookup by name is ambiguous, each definition is listed with the narrowest selector that picks it out — --symbol NAME --file <path> --line <n>, plus --kind <kind> when two records share a name and a line. Copy that selector verbatim.

--symbol also accepts a definition's stable compound-v1 ID (repoKey:language:path:kind:qualifiedName, with a #sig: suffix for overloads), which symbols --format ndjson emits. An exact ID match wins over every other filter — it already encodes the file and kind, so a stale --file cannot veto it — and it is the one selector that survives edits shifting line numbers. The ambiguity listing prints selectors rather than IDs because an ID repeats the path and name the same line already shows.

Semantic changes HEAD~1..HEAD

auth.py
  ~ function validate_token signature changed (14 dependents)
  + class TokenClaims added
  - function parse_token removed (0 dependents)

🗣️ Language support

capabilities --json exposes two honest tiers: semantic_languages (parser-backed call/type/data-flow analysis) and inventory_only_languages (stable file/document records, no semantic claims). Current counts: 185 recognized names, 36 semantic, 149 inventory-only.

36 semantic languages: Bash, C, C#, C++, CUE, Clojure, ClojureScript, Dart, Elixir, Erlang, F#, Go, Groovy, HCL/Terraform, Haskell, Java, JavaScript, Julia, Kotlin, Lua, OCaml, Objective-C, PHP, Perl, Protocol Buffers, Python, R, Ruby, Rust, SQL, Scala, Swift, TypeScript, YAML (including GitHub Actions workflow sections and jobs), Zig, Zsh.

Everything else is reported as an honest partial failure rather than dropped silently. See docs/language-support.md for the full two-tier matrix.

📊 Benchmarks

All figures below are measured on real, public repositories, not estimated. Reproduce them with cmd/graph-bench (see docs/benchmarks.md).

🎯 Accuracy vs codebase-memory-mcp

Fixed-answer semantic tasks (definition lookup, call graph, imports, change-impact) on real checkouts, scored as correctness counts.

System Score Corpus
entire-graph 265 / 283 (94%) 21 repos, multi-language
codebase-memory-mcp 191 / 283 (68%) same board

⚡ Throughput (full profile)

Repo Language Files Symbols Relations Build time
caddyserver/caddy Go 214 1,857 23,681 1.5s
redis/redis C 633 10,312 358,082 5.5s
laravel/framework PHP 2,019 23,182 1,872,087 17.0s
micropython/micropython C 3,447 18,788 2,268,173 25.6s

🪶 Peak memory (vs codebase-memory-mcp)

Repo entire-graph codebase-memory-mcp
nlohmann/json 80 MB 599 MB
koin 64 MB 285 MB
Dapper 43 MB 225 MB

Memory is released back to the OS after the build completes.

💸 Token efficiency

Answering the five structural questions for a core symbol, graph query vs reading every referencing file (tiktoken cl100k):

Task entire-graph file-by-file Reduction
redisCommand (redis), 23 referencing files ~2,300 tokens ~654,000 tokens 283x

Savings scale with symbol connectivity: single-digit for narrow symbols, 280x+ for core symbols.

⚡ Performance

  • Semantic diff: about 0.1s for a typical HEAD~1..HEAD on redis.
  • Full-graph build: linear in repository size; 23K relations in 1.5s up to 2.27M relations in 25.6s across the repos above.
  • Streaming output: snapshot emits records as it parses, so memory stays bounded on very large repositories.
  • Cached committed-tree search: reuses a tree-keyed compressed index across invocations, in the platform's per-user cache directory unless --cache-dir/ENTIRE_PLUGIN_DATA_DIR redirect it, so repeated queries on an unchanged tree skip re-parsing. The working tree is never cached. A complete prepared index derives the exact query-selected view, so relation expansion cannot escape that file set.
  • Explicit preindex: index --head builds and verifies that query-independent artifact before latency-sensitive work; cached search and neighbors calls then report the hit directly.

Compact snapshot NDJSON v1

Normal snapshot --format ndjson remains the interoperable default and retains its object-per-line schema. snapshot --format compact-ndjson is a separate, complete-snapshot-only native artifact: positional rows tagged f, x, s, and r reference deterministic first-seen dictionaries emitted as d rows; h is the required first header row and carries the sole v1 version marker; m is the required final summary. A decoder rejects unknown versions, malformed arity, duplicate headers, and missing summaries.

Every header, dictionary, data, and summary line counts toward raw compact bytes—size claims never subtract dictionary overhead. Compact output uses a separate cache namespace. Consumers load it through snapshot-query, which returns deterministic native NDJSON symbol/relation records:

entire graph snapshot --repo . --format compact-ndjson > graph.compact.ndjson
entire graph snapshot-query --input graph.compact.ndjson --symbol Cache.Refresh --format ndjson
entire graph snapshot-query --input graph.compact.ndjson --from '<stable-id>' --relation CALLS --format ndjson

The canonical SHA-256 is calculated from normalized native records in record order, not compact bytes. A valid compact artifact must match native NDJSON in both that hash and decoded public projection; a matching hash alone is not a proof of losslessness.

Absolute numbers are environment-sensitive (measured on Apple Silicon). Read them as relative signals and reproduce locally with the harness.

🔒 Security & local-first

  • Zero egress. No API keys, no network calls, no telemetry, no grammar downloads at runtime. All 36 grammars are vendored and compiled in.
  • Verifiable. entire graph doctor --json reports no_egress=true. Pass --no-network to make the no-egress contract explicit to callers.
  • Writes only to Entire's managed plugin directory. Your code is read locally and never leaves the machine — safe to point at private repositories.
  • Reads committed HEAD by default for provider/graph streams; pass --worktree to include live edits.

🏗️ Architecture

cmd/
  entire-graph/     Plugin entry point (semantic diff, search, provider commands)
  graph-bench/      Reproducible benchmark driver
internal/
  cli/              Hand-rolled command dispatch and flag parsing
  sem/              Tree-sitter parsing, symbol + relation extraction, rename reconciliation, search
  gitutil/          Git subprocess wrappers (NUL-delimited output parsing)
  bench/            Measurement core (throughput, memory, coverage)
docs/               Language support matrix, benchmarks, operations, provider requirements, ADRs
scripts/            Local install and checksum-backed release archives

The parser is isolated behind internal/sem, so the command surface stays stable while the semantic model gets richer. The wire schema is a frozen 1.x GA contract (additive-only minors; see docs/adr/0001-ga-schema-contract.md).

⚠️ Current limits

  • Dependent counts are heuristic, not compiler / type-checker accurate.
  • CALLS, HANDLES_ROUTE, and HANDLES_TOOL relations are heuristic.
  • Rename detection is heuristic.
  • Unsupported languages are reported as partial failures, not parsed semantically.
  • The plugin is invoked as entire graph ...; it requires no changes to the main Entire CLI.

📄 License

MIT. The runtime parser dependency is github.com/smacker/go-tree-sitter (MIT). Three tree-sitter grammars are vendored as source under their own upstream MIT licenses: Dart (internal/sem/grammars/dart/), PostgreSQL (internal/sem/pgsql/), and Zsh (internal/sem/zsh/). This implementation does not copy or vendor Ataraxy Labs code.