Arborist MCP

Arborist MCP is a mixed Rust + Python workspace for semantic code analysis, patch validation, persisted symbol indexing, and a lightweight stdio MCP gateway.

Current layers:

• crates/arborist-core: Rust parsing core with Tree-sitter based semantic extraction, symbol graph indexing, patch validation, VFS state, and SQLite persistence.
• crates/arborist-py: PyO3 bridge that exposes the Rust core to Python as _arborist_core.
• python/arborist_mcp: MCP-compatible JSON-RPC gateway over stdio.

Documentation

• Development guide: setup, checks, CI profiles, test suites, build artifacts, and common failures.
• Protocol guide: MCP usage, legacy JSON-RPC compatibility, tool catalog generation, and protocol validation.
• Tool guide: supported tool families, source overlays, patch preview, symbol indexes, trace/context workflows, and C/C++ status.
• Generated tool catalog: exact tools/list snapshot.

Language Support

Arborist currently supports Python, C, and C++ source files. Language routing is extension-based:

• Python: .py, .pyi
• C grammar: .c, .h
• C++ grammar: .cc, .cpp, .cxx, .c++, .tpp, .tcc, .ipp, .inl, .hpp, .hh, .hxx, .h++

Python overload groups retain one compatibility semantic_path while exposing unique IDs for each declaration and implementation, such as /repo/store.py::Store.get#overload[1] and /repo/store.py::Store.get#implementation. Arborist recognizes standard typing and typing_extensions overload decorators, including directly imported aliases and module aliases declared before the decorated definition. Those aliases are tracked in source order through top-level control-flow bodies and the enclosing class/function scopes, including typing and typing_extensions wildcard imports, imports, loop targets, assignments, deletes, match captures, parameters, and other rebinding events. Nested functions inherit overload aliases from enclosing function scopes, while global and nonlocal declarations keep rebinding behavior aligned with Python name resolution; wildcard imports from unknown modules conservatively invalidate the bare overload name. Rebinding the bare overload name also invalidates later @overload decorators. Arbitrary decorators such as custom.overload are not treated as standard overloads. Non-unique Python semantic-path selectors are rejected with candidate IDs rather than silently selecting the first overload. Rebuild indexes created by older Arborist builds to materialize these identities. Incremental refreshes rewrite every affected file when a cross-file collision changes its ID.

C++ files use the dedicated Tree-sitter C++ grammar. C-family indexing, tracing, query ownership, and patch targets support free functions in named namespaces plus named methods declared or defined in class bodies, with qualified semantic paths such as outer::Class::method. Class definitions are also indexed with their namespace and enclosing-class scope. Named class methods defined outside the class are also matched to their declarations. Explicit constructors and destructors are supported as Class::Class and Class::~Class; defaulted/deleted methods are indexed with their full declaration signatures. Named function and class-method templates are indexed and traced with their template declaration text. Explicit function template specializations have distinct paths such as increment<int> and Box<int>::value. Non-type template parameters are treated as local bindings during patch validation and reference tracing. C++ callable semantic_path values remain overload-set paths, while exact symbol_id values include normalized parameter types and member qualifiers, such as api::convert(int), api::convert(double), and api::Counter::value() const. Basic operator methods use paths such as Class::operator+ and Class::operator bool with the same exact-ID convention. C++ graph resolution filters direct function calls by argument count before choosing an overload; defaulted and variadic parameters are considered when matching candidates. Namespace-qualified calls such as api::convert(value) are resolved relative to enclosing namespaces before overload filtering. Explicit template calls such as convert<int>(value) prefer an indexed exact specialization and otherwise fall back to the primary template through the same direct-call graph path. Calls through this->method(value), (*this).method(value), and dependent member-template syntax such as this->template method<T>(value) resolve against the enclosing class's method overloads by argument count; const member callers prefer matching const overloads, including declarations whose top-level cv qualifiers are written as either const volatile or volatile const. Because this receivers are lvalues, matching & and const & member overloads are preferred over && overloads. Explicit rvalue self calls through std::move(*this).method(value) or static_cast<T&&>(*this).method(value) prefer matching && member overloads; const-qualified casts select matching const & or const && overloads. std::as_const(*this).method(value) selects a matching const & member overload. std::forward<T>(*this).method(value) follows the explicit template argument's value category and top-level const qualification. Direct C++ type constructions such as Counter(value), Counter{value}, and new api::Counter and new api::Counter(value) resolve to the matching constructor overload by argument count. Template constructions such as api::Box<int>{value} fall back to the primary class template when an explicit specialization is not indexed; this applies to new api::Box<int>(value) as well. Member calls on direct temporary constructions, such as api::Counter{}.adjust(value), resolve against the constructed type's member overloads and prefer matching && qualifiers; the same applies when the temporary is wrapped in std::move or an explicit static_cast<T&&>. A static_cast<const T&> or static_cast<const T&&> temporary selects matching const-qualified member overloads; std::forward<T> follows its template argument's value category and const qualification. Type aliases are expanded for direct temporary member calls, so using Alias = api::Counter; Alias{}.adjust(value) resolves against api::Counter overloads. Member calls on explicitly typed local C++ objects and function parameters are resolved too: after Alias current{}; or Alias& current, current.adjust(value) follows the & overload, while const Alias current{} or const Alias& current follows const & and std::move(current).adjust(value) follows &&. Local bindings are selected lexically, so an inner declaration with the same name shadows an outer object for graph tracing; range-for bindings follow the same rules. Directly typed raw pointers are also resolved through ->, so Alias* current; current->adjust(value) follows the pointee's & overload and const Alias* current follows const &; the equivalent (*current).adjust(value) form is resolved as well. auto bindings from std::addressof(value) or &value retain the same pointee receiver behavior. auto&, const auto&, auto const&, and named auto&& bindings retain the referenced object's lvalue and const receiver behavior, including bindings initialized with std::move(value), std::as_const(value), std::forward<T>(value), or static_cast<T&>(value). Bindings from *pointer retain the raw pointee's lvalue and const receiver behavior. decltype(auto) bindings preserve the same local receiver behavior for parenthesized lvalues, xvalues, pointer and optional dereferences, and reference-wrapper .get() calls; a bare identifier follows its declared decltype type, including top-level const. Equivalent address-expression aliases such as *std::addressof(value), *std::addressof(std::as_const(value)), and *&value retain the addressed object's lvalue and const receiver behavior. Direct -> calls through those same address expressions are resolved as well. An explicit static_cast<T&>(value) inside the address expression preserves T as the member lookup type, including when combined with std::as_const. For std::forward<T>(value), the explicit T determines the alias's static member lookup type and const receiver behavior. Bindings from std::reference_wrapper<T>::get(), std::ref(value).get(), and std::cref(value).get() retain the wrapped object's receiver behavior. Bindings from std::optional<T>::value() or *optional retain the selected value's lvalue and const receiver behavior, including std::move, std::as_const, and std::forward<T> wrappers around the selected value. std::expected<T, E> follows the same selected-value receiver behavior through ->, .value(), and dereference, including const and rvalue wrappers and direct auto construction. Its .error() accessor resolves against E with the error object's own const and value category; references bound from it retain the same behavior. std::expected<T, std::unique_ptr<U>> and std::expected<T, std::shared_ptr<U>> also resolve .error()->member() against U. Bindings from *std::unique_ptr<T> or *std::shared_ptr<T> retain the pointee's lvalue and const receiver behavior. std::weak_ptr<T>::lock() resolves through the returned shared pointer, both for direct lock()->member() calls and auto bindings; const on the weak pointer wrapper does not make T const. Direct std::get<N>(tuple_like) calls resolve member calls on supported std::tuple, std::pair, and std::variant elements. The analyzer preserves the container expression's const and value category through std::move, std::as_const, and std::forward, including .value() / .error() on selected std::optional and std::expected elements; operator-> continues to model the pointed-to object as an lvalue. Type-based std::get<T> follows the same rules only when T identifies exactly one top-level element, avoiding false edges for invalid or ambiguous tuple-like calls. Braced local initializers such as api::Counter counter{value} and api::Box<int> box{value} also resolve to constructor overloads by argument count. Indexed using and typedef aliases declared earlier in the same source file or in a local header included before the caller, such as using Alias = api::Counter; Alias counter{value}; or typedef api::Counter CounterAlias;, resolve to the aliased constructor; alias chains are expanded transitively. Template aliases such as template <typename T> using BoxAlias = api::Box<T>; resolve to the primary template constructor. Top-level const and volatile qualifiers are ignored for construction lookup; pointer and reference aliases do not create constructor dependencies. For conditional local includes, static analysis follows only branches with literal #if 0 or #if 1 conditions and leaves macro-dependent branches unresolved. Namespace aliases are expanded for direct qualified calls, so an alias such as namespace vendor = detail; resolves vendor::convert(value) to detail; alias chains are expanded transitively. Qualified namespace aliases and using declarations must be declared before the caller in the same source file or in a local header included before it. Qualified calls through using api::function; declarations resolve to the imported callables rather than the declaration symbols themselves; local and imported overloads remain part of the same argument-count-filtered set. Unqualified direct calls also resolve through scoped using api::function; declarations before global fallback candidates are considered, including declarations from local headers included before the caller. Direct unqualified C++ calls also honor using namespace vendor; imports from the enclosing namespace scopes before falling back to global candidates, including namespace-alias targets such as using namespace alias; when the alias is declared earlier in the same source file. C++ using aliases and declarations are indexed with namespace and class scope, for example api::Size, api::Config::Count, and api::convert. Namespace aliases are indexed at their definition scope, for example api::vendor. See the tool guide for the current scope. C++20 concept definitions, named enum definitions and members, and named struct/union definitions are also indexed by qualified name, such as api::Incrementable, api::Status, api::Status::ready, api::Counter, and api::Counter::Storage. C definitions such as struct Packet { ... };, union Payload { ... };, and named enum members are indexed without a typedef alias. C++ anonymous-namespace members use file-anchored identities so symbols with the same name in separate translation units remain isolated. extern "C" function declarations and definitions are indexed through their linkage wrapper. Declarations in #if/#else branches are also indexed without evaluating preprocessor conditions. Inline friend functions, including function templates, are indexed in their enclosing namespace rather than as class methods. Explicit class and function template instantiations are indexed with their specialized paths, such as api::Vector<int> and api::increment<int>.

Implemented Tool Families

The MCP catalog currently returns 58 tools:

• Read tools: 29, including batch reads, semantic skeletons, patch previews, bounded raw Tree-sitter queries with cooperative timeout budgets, symbol reads (including bounded neighborhood reads), symbol list/search, and graph-backed read bundles.
• Write tools: 2, patch_ast_node and patch_ast_node_at_position.
• VFS tools: 10, including open/change/close, virtual patching, byte edits, commit/discard, and virtual reads.
• Index tools: 9, covering register, unregister, list, inspect, migrate, rebuild, workspace refresh, and file refresh for persisted symbol indexes.
• Trace tools: 8, covering graph/neighborhood traces plus trace-backed replay and validation.

batch runs up to 32 read-only Arborist calls in order and accepts an optional shared timeout_ms budget capped at 300000 milliseconds. Before execution, the gateway validates every inner call's structure and any explicit inner timeout. Every batch-eligible tool accepts a cooperative timeout, so the gateway forwards the smaller of the caller's explicit inner timeout and the batch's remaining budget, or injects the remaining budget when none was supplied. A single blocking step inside an inner tool remains non-preemptible. Expiration fails the whole batch without returning partial results, and input argument objects are not modified.

The two write patch tools and the two VFS-only patch tools accept an optional cooperative timeout_ms budget capped at 300000 milliseconds. The budget covers target resolution and patch validation, and VFS-backed writes restore the prior buffer when it expires before persistence. Once an atomic disk write starts, the operation reports the write/index-sync outcome rather than a timeout after the source may already have changed.

did_open, did_change, read_virtual_file, list_virtual_files, apply_buffer_edit, commit_virtual_file, discard_virtual_file, and did_close accept the same optional timeout cap. Open and read budgets cover loading, parsing, clean-buffer refresh, and result validation; a timeout restores the exact prior entry or removes one loaded only for the failed request. Listing refreshes loaded entries in deterministic order and rolls all refreshes back if its budget expires. Byte and position edits share one request budget across loading, range and position validation, source splicing, incremental parsing, syntax collection, and result validation. They stage each edit before a final mutation gate, and a failed batch restores the exact prior buffer. Commit and discard retain their final pre-mutation gates, while did_close follows the commit path when persist=true and the discard path otherwise. After persistence or buffer replacement starts, these operations report their final outcome instead of a late timeout.

list_symbol_indexes and unregister_symbol_index accept the same timeout cap. Listing checks it while collecting and validating registrations and around deterministic sorting. Unregister retains a final gate after path normalization and immediately before mutation; a timeout through that gate preserves the registration, while a started removal returns its actual outcome.

migrate_symbol_index also accepts the timeout_ms cap. Its cooperative budget covers path and database setup, schema and workspace metadata checks, legacy row loading, persisted-path validation, and a final gate before the schema migration transaction. A timeout before that gate leaves the database unchanged. Once the transaction begins, Arborist completes the required source rebuild and final health inspection and returns their actual outcome rather than a late timeout. Individual SQLite queries, source reads, the schema transaction, and rebuild persistence remain non-preemptible.

The offline replay_patch_evidence_against_trace, validate_patch_commit_with_trace, and export_patch_diagnostics_sarif tools accept the same timeout cap. Their cooperative native budgets cover validated patch/trace traversal and result construction after strict JSON decoding; individual decodes, source parses, model-validation calls, and serialization steps remain non-preemptible.

Use python -m arborist_mcp.gateway --dump-tool-catalog or read docs/tool-catalog.json for exact names, input schemas, output schemas, defaults, and categories.

Quick Start

On Windows:

python -m venv .venv
. .\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install "maturin>=1.7,<2.0"
maturin develop --locked
.\scripts\sync-extension.ps1 -SkipBuild
python scripts\gateway_smoke.py --require-core
python -m arborist_mcp.gateway --help

Or run:

.\scripts\bootstrap.ps1

Index Watch

Use the polling watch command to keep one persisted index synchronized without rewriting it while it is healthy:

arborist-index-watch --version
arborist-index-watch --workspace-root . --db-path .\symbols.db
arborist-index-watch --workspace-root . --db-path .\symbols.db --once
arborist-index-watch --workspace-root . --db-path .\symbols.db --once --timeout-ms 5000
arborist-index-watch --workspace-root . --db-path .\symbols.db --once --dry-run
arborist-index-watch --workspace-root . --db-path .\symbols.db --check

--version reports the installed Arborist package version without requiring a watch target. The watcher refreshes missing indexes and current-schema freshness issues through the incremental workspace refresh path. It exits without writing when inspection requires manual intervention, such as an unsupported or foreign SQLite schema. --timeout-ms bounds health freshness reads and workspace reconciliation scans as well as refresh indexing work. --dry-run reports would_refresh or would_migrate without changing an index. --check performs that no-write inspection once and returns a nonzero exit status unless every target is healthy, which is useful for CI and deployment checks. --check is mutually exclusive with --once and cannot be combined with --dry-run or a non-default --interval-seconds. Emitted health summaries include issue, stale, missing, unreadable, and unindexed file counts.

To watch several registered workspace/index pairs, provide a JSON manifest:

{
  "indexes": [
    {"workspace_root": "./workspace-a", "db_path": "./indexes/a.db"},
    {"workspace_root": "./workspace-b", "db_path": "./indexes/b.db"}
  ]
}

Run it with arborist-index-watch --config .\watch.json. Relative paths in the manifest are resolved from the manifest directory. Each target is inspected and reconciled in deterministic workspace order; an unsupported or foreign index stops the command without rewriting it. Duplicate workspace or database paths are rejected before the first refresh.

On Linux or macOS:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install "maturin>=1.7,<2.0"
maturin develop --locked
python -m pip install .
python scripts/gateway_smoke.py --require-core
python -m arborist_mcp.gateway --help

Quick Validation

For the normal local loop:

.\scripts\test.ps1 -Suite inner-loop

Useful suite variants:

.\scripts\test.ps1 -Suite python-fast
.\scripts\test.ps1 -Suite python-native
.\scripts\test.ps1 -Suite python
.\scripts\test.ps1 -Suite rust,inner-loop -ShowPlan
python scripts/python_suite_manifest.py

For the full gate:

.\scripts\check.ps1

Useful profile variants:

.\scripts\check.ps1 -Profile python-fast
.\scripts\check.ps1 -Profile gateway-fast
.\scripts\check.ps1 -Profile gateway-native
.\scripts\check.ps1 -Profile python-discovery
.\scripts\check.ps1 -Profile gateway-smoke
.\scripts\check.ps1 -Profile python-native
.\scripts\check.ps1 -Profile full,python-native -ShowPlan

Useful direct commands:

cargo fmt --check
cargo test --locked
cargo clippy --locked --all-targets -- -D warnings
python scripts\tool_catalog.py --check
python scripts\gateway_smoke.py --require-core
python scripts\gateway_smoke.py --launcher console --require-core
python -m unittest tests.gateway_protocol.request_validation
python -m arborist_mcp.gateway --help

See the development guide for profiles, suite names, native-extension sync behavior, CI coverage, and release wheel builds.

MCP Usage

Minimal MCP server configuration:

{
  "mcpServers": {
    "arborist": {
      "command": "python",
      "args": ["-m", "arborist_mcp.gateway"],
      "cwd": "E:/workspace/arborist-mcp"
    }
  }
}

If Arborist is installed as a package, arborist-mcp is equivalent:

{
  "mcpServers": {
    "arborist": {
      "command": "arborist-mcp",
      "args": []
    }
  }
}

Minimal MCP messages:

{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"example-client","version":"0.1.0"}}}
{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"arborist/get_semantic_skeleton","arguments":{"file_path":"tests/fixtures/sample.py","depth_limit":2}}}

Legacy direct arborist/* JSON-RPC calls remain supported over the same newline-delimited stdio transport. See the protocol guide for response shapes, error behavior, and examples.

Core Capabilities

• Semantic skeletons with stable selectors, symbol IDs, signatures, byte ranges, parameters, return types, and docstrings when available.
• One-shot source overlays for unsaved-file analysis, including persisted-index read/trace/list/search overlays when index_db_path is supplied.
• Patch preview tools that return validation plus unified diff without writing to disk, with optional cooperative budgets across target resolution, validation, and diff generation.
• Semantic skeleton extraction with bounded depth and expansion selectors plus cooperative budgets for file reads and post-parse symbol traversal.
• MCP resources expose the generated tool catalog snapshot for clients that prefer resource reads over tools/list.
• Semantic patching with structured binding decisions, commit gates, bypass auditing, trace-backed replay validation, and cooperative budgets across context validation and multi-file edit previews.
• Session-scoped VFS with open/change/close, virtual patching, commit/discard, and incremental Tree-sitter edits.
• Python/C workspace symbol graph indexing, listing, searching, reading, tracing, bounded neighborhood context, and optional cooperative budgets for direct read and trace queries.
• SQLite-backed persisted symbol indexes with transactional v1-v3-to-v4 schema migration plus source reindexing, health inspection, response schema versioning, stale/missing/unreadable/unindexed file diagnostics, bounded workspace scans, optional per-file byte limits and cooperative time budgets, partial refresh, and fail-closed handling for damaged or unrelated databases.
• C include-family tracing and patch disambiguation for header/source projects, including duplicate globals and file-local static symbols.

Current Status

Phase 1 is complete for the Python/C read path. The current Phase 2 foundation includes patch validation, trace-backed validation, VFS-backed editor flows, persisted indexes, source overlays, index health inspection, and generated MCP tool schemas.

Remaining larger work includes:

• Splitting large Rust modules such as lib.rs, symbols.rs, and model.rs.
• Reducing PyO3 wrapper repetition with parameter/context objects.
• Extending C++ semantic support beyond overload-aware callable identities to fuller language-aware overload resolution and remaining grammar coverage.
• Adding broader fuzz/property coverage and cancellation for remaining native symbol-resolution operations.