The extraction pipeline turns a filesystem snapshot into a coherent graph artifact set. This page follows the cold path, incremental path, and failure boundaries through the owning crates.
Who this page is for: contributors changing discovery, languages, resolution, graph construction, or output publication.
You will learn: pipeline stages, core types, incremental invalidation, concurrency, semantic merge, and the tests required for safe changes.
Prerequisites: Workspace tour.
Reading time: 15–20 minutes.
At the CLI:
compass update
compass extract
compass watch
compass-cli parses arguments into compass-core::BuildOptions and selects a
BuildPurpose. The core exposes:
build_local_graph
build_graph_with_semantic
build_graph_with_layers
build_graph_with_layers_and_tiebreaker
The implementation keeps CLI display policy outside the core build transaction.
Options cover:
- root and output directory;
- clustering and visualization;
- ignore and explicit exclusions;
- resolution and hub thresholds;
- force/cold versus incremental work;
- Cargo/PostgreSQL/Google Workspace inputs;
- semantic provider, model, mode, budget, concurrency, timeout;
- partial and deduplication policy;
- timing.
Not every option affects every stage. Meaning-affecting options must be represented in fingerprints or manifest compatibility where reuse could otherwise produce a wrong graph.
compass-files::detect walks the requested root under DetectOptions.
It returns classified files and incremental information:
source code
project/config manifests
semantic documents/media
integration shortcuts
ignored or unsupported paths
Important invariants:
- paths stay within the scan root;
- file kinds are deterministic;
- ignore policy is explicit;
- watch filters use compatible classification;
- very large or invalid inputs fail at the owning boundary.
The current manifest and stat/content fingerprints divide files into:
unchanged
changed
added
renamed
deleted
An unchanged file can reuse cached extraction when:
- content/fingerprint matches;
- cache format matches;
- parser/extractor configuration matches;
- relevant project-wide inputs have not invalidated it.
A file-local cache is only one input. Cross-file resolution and graph analysis still operate on a complete logical set.
compass-languages::Registry maps a file to a LanguageSpec and extractor.
The engine produces an Extraction:
nodes
edges
hyperedges
raw calls / language facts
universal evidence batches
File work can run in parallel. To keep results deterministic:
- collect facts in stable source order where ordering is contractual;
- give nodes stable IDs;
- avoid global mutable parser state;
- keep errors associated with source paths;
- separate parsing from project-wide resolution.
An extractor should emit evidence available from that file and its explicit local/project metadata:
- declarations and containment;
- imports/re-exports;
- calls and references with unresolved facts as needed;
- source locations;
- relation context and provenance;
- language-specific identifiers.
It should not pick one arbitrary project-wide target when evidence is ambiguous.
Established direct adapters publish their current graph records and unresolved facts through this boundary. Independently qualified universal adapters emit the versioned evidence contract and use shared projection instead. See Universal evidence implementation architecture for the current and planned boundaries.
Configured sources can add extractions:
- Cargo dependency graph;
- PostgreSQL schema;
- Google Workspace shortcuts;
- SCIP data;
- documents/media after semantic extraction.
Every integration returns graph-compatible records and remains bounded at its I/O boundary.
compass-resolve merges per-file facts and resolves:
- module/import targets;
- JavaScript re-exports;
- cross-file calls;
- members and receiver types;
- language-specific call facts;
- colliding IDs;
- header/implementation pairs;
- unique placeholders.
The resolver owns project-wide evidence. It marks derived relations appropriately and preserves ambiguity when a unique target cannot be justified.
Peak-memory paths use ownership transfer (resolve_owned_with_root) so large
per-file buffers are appended/dropped rather than cloned.
compass-graph combines resolved extractions into a GraphDocument.
It:
- maps facts to node-link records;
- enforces direction;
- preserves parallel edges where required;
- removes only duplicates allowed by the compatibility contract;
- applies entity deduplication;
- optionally uses a semantic tiebreaker where explicitly configured;
- records graph-level metadata.
Deduplication must not merge two same-labeled but distinct entities merely for a smaller graph.
The core can receive a SemanticLayer. The semantic pipeline:
- reads bounded units;
- packs chunks;
- invokes the selected backend;
- parses and validates untrusted JSON fragments;
- binds evidence and normalizes IDs;
- records partial/completeness status;
- merges results with structural facts.
Provider failure or invalid fragments must not publish a complete semantic
graph. --allow-partial is an explicit policy change, not silent recovery.
compass-graph:
- clusters nodes into communities;
- computes community cohesion/signatures;
- remaps communities stably for incremental output where possible;
- identifies god nodes and surprising connections;
- suggests questions;
- detects import cycles;
- generates data for reports.
--no-cluster can isolate deterministic AST behavior in qualification, but the
normal user artifact includes analysis when enabled.
compass-output converts the graph and analysis to:
- graph JSON;
- Markdown report;
- optional HTML;
- requested export sidecars.
Derived output should not change graph identity. A renderer change can still affect historical artifact registries and must be versioned when reproducible reconstruction depends on it.
compass-files atomic helpers and build guards publish the new artifact set.
Required behavior:
- no successful marker for incomplete work;
- no partial JSON replacing a valid graph;
- diagnostics identify the source/stage;
- manifest advances only with the graph it describes;
- disposable cache failure does not redefine graph truth.
The fast path is correct only if it produces a graph equivalent to a clean rebuild for the same inputs.
Cases to test:
| Change | Expected invalidation |
|---|---|
| edit function body | changed file extraction plus affected resolution/analysis |
| add file | new extraction and project merge |
| delete file | remove its nodes/edges and re-resolve dependents |
| rename file | path/ID/import effects handled |
| change manifest | project metadata and dependent resolution invalidated |
| change ignore rule | corpus membership recalculated |
| parser/extractor version | incompatible caches invalidated |
| semantic profile change | semantic cache/profile invalidated |
Performance qualification compares cold, warm unchanged, single-file change, rename, and delete cases against Compass-owned baselines.
Extractor semantics use the Compass-owned AST compatibility namespace v1.
This cache version is independent of the Compass package release and advances
only when changed extractor behavior makes existing AST entries unsafe to
reuse. The first update after such a version change refreshes AST facts; later
unchanged updates return to the normal warm path. The nested e6 namespace is
the binary cache encoding version, not part of the AST compatibility version.
compass-core::watch_local_graph combines compatible watch filtering,
debouncing, and the update pipeline.
Watch correctness requires:
- no missed relevant path;
- no infinite loop on generated
compass-out/; - coalesced bursts;
- observable failure;
- clean shutdown;
- optional polling fallback.
The one-shot update remains the recovery oracle.
Keep stage context:
FileError discovery/read/write/cache
language error parse/extract source
CoreError orchestration/completeness
DedupError graph identity conflict
semantic error provider/parse/validation
output error rendering/publication
The CLI maps these to useful stderr and exits. Avoid stringifying too early.
At minimum:
- smallest unit fixture for the new fact or invalidation;
- cold build artifact check;
- unchanged warm update;
- changed/renamed/deleted cases if relevant;
- graph equivalence against a clean rebuild;
- direction, relation, provenance, attributes, and multiplicity assertions;
- malformed/oversized/interrupted input failure;
- CLI stdout/stderr/exit/output-tree test;
- parity fixture if the behavior is in the compatibility ledger;
- performance qualification if the hot/cold pipeline changes.
- How Compass works
- Language architecture
- Workspace tour
- Universal evidence implementation
- Semantic pipeline
- Performance qualification
Next step: trace one fixture from detection through its Extraction, then
inspect the resolved edge and final graph.json record.