feat: Hierarchical Knowledge Aggregation — layered DAG build, multi-strategy summarization, and intelligent query routing

[ljinshuan]

Background

Graphify currently produces a single flat knowledge graph — all source files are extracted, merged into one graph.json, and served via MCP as a flat graph. This works well for small-to-medium projects, but breaks down at scale:

1. Information Overload: A 500+ file project produces thousands of nodes. LLM query token budgets are wasted on irrelevant details, and key architectural information is buried in noise.

2. No Abstraction Levels: Real-world architectures are naturally hierarchical (functions → modules → services → system). A flat graph cannot distinguish "how does auth work?" (implementation detail) from "what is the system architecture?" (high-level overview). Both questions hit the same undifferentiated graph.

3. Full-Graph Search Overhead: Every query traverses the entire graph. There's no way to narrow scope based on the question's abstraction level.

Proposed Solution

Introduce Hierarchical Knowledge Aggregation — knowledge should be layered like geographic maps (street-level → city-level → country-level), where each upper layer contains its own content plus a summarized view of the layer below.

Core Concepts

• Layered DAG: A layers.yaml config defines a strict parent-child hierarchy. Parents are built before children (topological order). No cycles, no cross-layer dependencies.
• Bottom-up Aggregation: Each layer builds its own graph from its sources, then merges a summarized sub-graph from its parent. Summary nodes are prefixed with summary:<parent_id>: for collision avoidance and provenance tracking.
• Intelligent Query Routing: Questions are automatically routed to the most appropriate layer based on keyword matching and abstraction-level heuristics. Abstract questions ("system architecture") go to upper layers; concrete questions ("function implementation") go to lower layers. Auto-zoom drills down when results are sparse.
• 5 Aggregation Strategies: none (no-op), topk_filter (degree-based selection), community_collapse (community detection → abstract nodes), llm_summary (LLM semantic summarization), composite (CC → LLM pipeline).

Value Proposition

Before	After
Single flat graph, all detail levels mixed	Layered graphs with clear abstraction boundaries
Every query searches the full graph	Queries routed to the right abstraction level
No way to compress/reduce graph size	5.3x compression from code layer to overview layer (validated on real data)
No provenance for how summaries were derived	Full provenance: aggregation/from_<parent>.json per layer
No structural comparison between views	layer-diff command for cross-layer comparison
Sequential build only	Parallel same-depth builds with graceful fallback

Validation

Built and tested on 3 real corpora (example, httpx, mixed-corpus) with a 3-layer architecture:

L0 (Code): 266 nodes, 357 edges — raw code extraction
L1 (Docs): 146 nodes, 154 edges — docs + topk_filter summary from L0
L2 (Overview): 50 nodes, 64 edges — code + community_collapse summary from L1

5.3x node compression from L0 → L2, with query routing correctly distinguishing abstract vs. concrete questions.

103 unit tests + real-data integration tests, all passing.

Implementation (4 Phases)

Phase	Scope	Tasks
1. Layer Config Foundation	layers.yaml parsing, DAG validation, topological sort, LayerRegistry	36
2. Aggregation Engine	5 strategies (none, topk, collapse, llm, composite) with LLM fallback	32
3. Query Routing	QueryRouter, keyword + level-weighted scoring, auto-zoom, MCP integration	27
4. CLI Polish	layer-info/layer-tree/layer-diff commands, provenance, parallel build, auto-detection	22

Reference: Full design doc and test records in README_TEAM.md and README_TEAM.zh-CN.md.

New CLI Commands

graphify build --layers layers.yaml              # Build all layers
graphify build --layers layers.yaml --layer L2   # Incremental rebuild
graphify layer-info --layers layers.yaml         # Table of layer stats
graphify layer-tree --layers layers.yaml         # ASCII tree
graphify layer-diff L0 L1 --layers layers.yaml   # Cross-layer comparison
graphify query "architecture?" --layers layers.yaml  # Auto-routed query
graphify query "auth function" --layer L0        # Direct layer query

New MCP Tools

• layer_info — List all layers with stats
• drill_down — Query a specific layer by ID
• query_graph — Auto-routed query (multi-layer mode, auto-detected)

Backward Compatibility

100% backward compatible. Without --layers, Graphify behaves exactly as before — single flat graph, no changes to existing workflows. Multi-layer mode is opt-in via layers.yaml.

[3esmit]

Looking forward this implemented.

[ljinshuan]

Looking forward this implemented.

#264

This has been implemented in this PR. Looking forward to its merge.

[3esmit]

There is something I can help with?

[ljinshuan]

There is something I can help with?

Not for now. I'll resolve the conflicts tomorrow and then ask the owner to review the code.

[3esmit]

I've abandoned this skill due the lack of this feature.