code-intelligence-bench

An independent, apples-to-apples benchmark for seven code-intelligence tools built to cut an AI agent's code-exploration token cost:

Tool	Lang	What it is	Search modality
semble	Python	Code search library (chunks)	semantic + lexical (static embeddings + BM25 + RRF)
code-review-graph (crg)	Python	Code graph for review / blast-radius	lexical (FTS5 + keyword) by default
codegraph	TypeScript	Code knowledge graph for agents	lexical (FTS5)
soop (RPG)	TypeScript	Repository Planning Graph (search + dep graph + generation)	RPG features (LLM or heuristic) + optional vector
code-search (csp)	Rust	Code search CLI (chunks) — Rust port of semble's algorithm	semantic + lexical (static embeddings + BM25 + RRF)
lsp	—	Language servers (pyright / tsserver / gopls) via code lsp-server	structural (workspace/symbol + callHierarchy)
@ttsc/graph	TypeScript	Compiler-exact code graph (tsc Program → MCP)	compiler graph (lexical lookup + typed trace)

soop is itself a graph-for-agents tool (it also does semantic search, a dependency graph, and — uniquely — repo generation), so it participates as both a retriever (Arm A) and a graph (Arm B). It is benchmarked in its free/deterministic no-LLM heuristic mode here; its LLM-feature + vector mode is a paid/non-deterministic extension (roadmap).

Each project benchmarks itself, its own way, on its own data — so the headline numbers (semble NDCG@10 0.854 · crg 38–528× token reduction & 100% impact recall · codegraph 57% fewer tokens) cannot be put in one table. This repo measures all three on the same corpus, same gold set, same harness.

What it answers

The three tools overlap only partially. semble is a pure semantic retriever; the two graph tools add structural queries (callers/callees/impact/trace) that semble has no notion of. So we measure along separate, honest axes — and one of them tests whether semble + a graph tool beats either alone:

Arm	Tools	Question	Cost
A. Search	semble, crg, codegraph, soop, csp	NL query → relevant code?	free / deterministic
B. Graph	crg, codegraph, soop, lsp, ttsc	blast-radius & multi-hop accuracy?	free / deterministic
C. Combined	semble→crg, semble→codegraph, semble→soop	does semble as anchor-finder help the graph tools?	free / deterministic
H. Hybrid	csp→lsp, csp→soop, csp→lsp+soop	do three complementary tools compose end-to-end?	free / deterministic
D. Agent E2E (opt-in)	all + no-tool	real agent tokens/cost/time/tool-calls?	paid (Claude API)
Perf (cross-cutting)	all	index time, query latency, footprint	free

Headline findings (this corpus)

• Search (Arm A). The two semantic retrievers dominate NL queries: semble NDCG@10 0.72–0.96, and csp — the Rust port of the same algorithm — matches it within ±0.03 everywhere (0.69–0.97), winning on 2 of 7 repos, with ~45% fewer returned tokens, the fastest index build (0.6–3.7 s), and no Docker. codegraph's FTS5 follows (0.40–0.72); soop's no-LLM heuristic is next (0.29–0.57), beating crg's lexical default everywhere (crg 0.10–0.30) at the lowest token cost. The graph tools return far fewer tokens per query (60–740 vs semble's ~2,500–3,500) because they return symbol references, not code bodies.
• Graph (Arm B). crg reproduces its ~100% impact recall claim (blast-radius). But all three graph tools' built-in search fails to locate the right anchor from a verbose NL query (crg's FTS-AND → 0 hits; codegraph latches onto common tokens like "Flask"; soop's no-LLM features likewise miss).
• Combined (Arm C) — the point of this benchmark. Using semble to localize the query, then letting the graph tool resolve the symbol there and traverse, fixes the anchor problem where it was broken (flask & express: anchor-found 0.00 → 1.00, neighbor-recall up to 0.00 → 1.00) — but can regress where the graph tool's own search already nailed it (gin/httpx codegraph). Net: semble is a strong complement on hard NL cases, not a universal replacement.
• Hybrid (Arm H). The strongest composed pipeline measured: csp localizes, LSP traverses — anchor-found 0.00 → 1.00 and perfect neighbor recall on flask & gin, all native/free/deterministic. soop confirms anchors but its no-LLM graph has no function-level edges, so it adds no neighbors; the remaining misses (httpx/fastapi) are chunk-granularity anchor ambiguity (csp localizes to a plausible twin symbol), not traversal failures.

Full tables (all arms, with per-repo numbers and interpretation) are in RESULTS.md. They come from the committed reference run in results/; regenerate with cibench report. Multi-hop tasks are few per repo (1–3), so Arm B/C are directional, not precise.

Corpus & gold set

5 dual-labeled core repos — flask, fastapi, httpx, express, gin — have both semble retrieval labels and crg graph/impact labels, plus tokio, alamofire (Rust/Swift) for retrieval-only language breadth. See bench/corpus.json.

Gold labels are reused from upstream (both MIT, attributed in bench/goldset/SOURCES.md):

• retrieval (bench/goldset/retrieval/*.json) — semble's {query, relevant, secondary, category} annotations (categories: semantic / architecture / symbol).
• graph (bench/goldset/graph/*.yaml) — crg's test_commits (git-derived blast-radius), multi_hop_tasks, search_queries, entry_points.

semble and crg pinned each repo at different commits, so each repo is cloned once per SHA under checkouts/<repo>@<sha>/; each gold set is only evaluated on its own snapshot.

How tools are run (and why it's fair)

All three index the whole repo root (same input), so hit paths are repo-root-relative and directly comparable. Tools run in their shipped default mode, not tuned. Because two tools have x86_64-macOS wheel gaps, runtimes differ:

• semble → Docker (linux/amd64). semble needs tree-sitter-language-pack <1.8.0, which has no x86_64-macOS wheel or sdist, so it can't run natively on Intel Macs. The image (docker/semble.Dockerfile) pins the correct wheel and bakes the embedding model in (offline). Its index time includes container startup.
• crg → native isolated venv (.venv-crg), lexical mode. This is faithful to crg's own eval (build → post-process → hybrid_search, no embeddings), which is also offline and avoids the torch x86_64-mac gap.
• codegraph → native CLI (bundled Node). FTS5-only by design. Its query latency is measured as CLI wall time, so it includes Node startup (~240 ms) — unlike semble/crg's in-process latency. Flagged in Perf; quality metrics unaffected.
• csp → native single Rust binary (csp). Indexes with csp index -o into the bench scratch dir; queried via csp search --index. Query latency is CLI wall time (includes process startup, like codegraph). Caveat: csp downloads its embedding model (potion-code-16M, Hugging Face) and tree-sitter grammars on first use and silently falls back to a stub embedder offline — warm the cache with one online search before benchmarking, or the semantic scores are garbage. Parity check: its flask NDCG@10 matches semble's (same algorithm), confirming the real model ran.
• soop → Docker (linux/amd64), under Node. soop's native tree-sitter backend has no x86_64-macOS build, and its semantic cache needs better-sqlite3 (unsupported by Bun) — so the worker runs under Node in a container. Benchmarked as the published @pleaseai/soop in no-LLM heuristic mode (its no-LLM floor; LLM-feature + vector mode is the paid/non-deterministic roadmap variant).
• lsp → native language servers through pleaseai/code-intelligence's code lsp-server <id> --project=<repo> (a transparent stdio pipe), driven by a minimal Python LSP client. Servers: pyright (python), typescript-language-server (javascript; classic TS 5 kept in .toolchain/ since TS 7 has no tsserver.js), gopls (go). LSP has no NL retrieval — its Arm B "own search" is workspace/symbol over the NL query, expected to fail like crg's FTS-AND; its value is compiler-precise callHierarchy traversal (Arm H). Go snapshots get go mod tidy first (untidy go.mod silently breaks gopls package loading).
• ttsc → native Go binary + MCP stdio server, driven with its real typed requests (lookup, trace). TypeScript-only by design — this corpus has no TS repo, so it runs only on express (plain CommonJS) via an injected allowJs tsconfig, an upstream-unsupported path: its NL lookup still finds anchors, but the compiler graph over CommonJS has almost no call edges, so traversal recall is a compatibility floor, not representative. A TS corpus repo with gold labels is the fair-arena roadmap item.

Every tool is reached through one uniform adapter interface (bench/adapters/base.py) that returns normalized Hits, so the metric/matching code is identical across tools.

Metrics

Ported verbatim from the upstream tools so numbers stay comparable to their self-reports: NDCG@k / Recall@k / MRR from semble (bench/metrics.py), precision/recall/F1 from crg's scorer, multi-hop score = anchor_found × neighbor_recall from crg. Returned tokens = payload chars ÷ 4 (semble: code body; graph tools: symbol reference). Hit↔target matching (bench/matching.py) is path + line-span overlap + symbol name.

Setup

# harness (orchestrator only)
uv venv && uv pip install -e ".[dev]"

# isolated tool runtimes
uv venv .venv-crg     --python 3.13 && uv pip install --python .venv-crg "code-review-graph"
npm i -g @colbymchenry/codegraph
brew install pleaseai/tap/csp        # or: npm i -g @pleaseai/csp
docker build --platform linux/amd64 -f docker/semble.Dockerfile -t cibench-semble:latest .
docker build --platform linux/amd64 -f docker/soop.Dockerfile   -t cibench-soop:latest .

# lsp arm: language servers + the code-intelligence pipe
npm i -g @pleaseai/code pyright typescript-language-server
npm install --prefix .toolchain typescript@5   # classic tsserver for tsls
brew install gopls go                          # gopls needs a modern go on PATH

# ttsc arm (TypeScript-only; see caveats)
npm i -g ttsc @ttsc/graph
chmod +x "$(npm root -g)"/ttsc/node_modules/@ttsc/*/bin/ttscgraph

cibench goldset          # validate the gold set
cibench fetch            # clone pinned snapshots into checkouts/

Run

cibench run a            # Arm A — search quality (all repos / tools)
cibench run b            # Arm B — graph: multi-hop + impact accuracy
cibench run c            # Arm C — combined: semble anchor → graph traverse
cibench run h            # Arm H — hybrid: csp anchor → lsp/soop traverse
cibench run perf         # Perf — index time / latency / footprint
cibench run a --repo flask --tool semble   # filter repo / tool

cibench report           # render latest results/*.json as markdown tables

# Arm D is opt-in and spends Claude API credits:
cibench run d --confirm --budget 2 --model sonnet

Reproducibility

Pinned SHAs + CPU-only embeddings + cold-index-once + median-of-N query latency make Arm A/B/C deterministic (re-running yields identical quality numbers). Perf varies run to run; record your machine (CPU/RAM/OS) alongside results. The reference run in results/ was produced on x86_64 macOS 15.

Limitations & roadmap

• crg semantic mode. crg can use embeddings ([embeddings], local all-MiniLM-L6-v2) but that needs torch, which has no x86_64-mac/py3.13 wheel. A cibench-crg Docker image (linux/amd64, embeddings) would let crg's NL retrieval compete fairly — currently crg is reported in its lexical default.
• soop LLM-feature mode. soop is run in no-LLM heuristic mode (free, deterministic). Its real strength — the LLM-generated feature/intent layer plus vector search — is a paid, non-deterministic mode not exercised here; expect soop's Arm A numbers to climb meaningfully with it enabled. soop is now also an Arm C/H traversal target via its combined worker op.
• Multi-hop sample size is small (1–3 tasks/repo); treat Arm B/C/H as directional.
• ttsc needs a TypeScript corpus repo. @ttsc/graph is compiler-exact and TS-only; on this corpus it can only run against express-as-CommonJS (near-empty edge set). Adding a TS repo with retrieval+graph gold labels would give it a fair arena — its own benchmark (8 TS repos, agent token A/B, common vs dedicated prompt lanes, no-source-read trace gate) is the model to port for Arm D there.
• soop's no-LLM graph has no function-level dependency edges (file-level only), so its Arm B/H traversal cannot recover symbol-granular neighbors in free mode — its LLM mode presumably adds them (untested, paid).
• LSP timings aren't index times. Language servers analyze lazily in the background; the harness waits for progress-quiet before traversing, and LSP participates only in quality arms (B/H), not Perf.
• Perf does not yet capture peak RSS or incremental-update time.
• codegraph latency includes Node process startup (CLI), not in-process.
• Arm D (live agent A/B) is implemented but unrun here (costs money); semble's MCP-in-Docker entrypoint is a TODO.

Attribution

Gold-set labels © their projects (MinishLab/semble, tirth8205/code-review-graph), both MIT, redistributed with attribution. Harness code MIT.