feat: jam intel — codebase intelligence

[sunilp]

Summary

• jam intel — a new codebase intelligence feature that scans repositories, builds a semantic knowledge graph, generates architecture diagrams, and supports natural language queries
• 6 language analyzers (TS/JS, Python, COBOL, SQL, Docker, OpenAPI) — all regex-based, zero new dependencies
• Framework detection for Express, React, dbt, Django, Flask, Airflow, Spark, Prisma, Kafka, SQLAlchemy
• Progressive LLM enrichment with priority queue and budget controls
• Mermaid diagram export for architecture, deps, flow, impact, and framework views
• 6 CLI subcommands: scan, status, query, impact, diagram, explore

Commands

jam intel scan                    # Scan repo, generate architecture diagram
jam intel scan --no-enrich        # Structural scan only, no LLM
jam intel scan --dry-run          # Show token estimate
jam intel status                  # Graph stats + enrichment progress
jam intel query "auth" --no-ai    # Keyword search (offline)
jam intel query "what handles auth?"  # NL query via LLM tool-use
jam intel impact src/models/user.ts   # What breaks if I change this?
jam intel diagram --type architecture # Mermaid diagram to stdout
jam intel diagram --type deps -o deps.mmd  # Export to file
jam intel explore                 # Open interactive viewer in browser

Architecture

Source Files → Pluggable Analyzers → Knowledge Graph → Consumers (CLI/Mermaid/Browser)
                                   ↑
                            LLM Enrichment (progressive, background)

• Hybrid storage: JSON on disk + in-memory graph at runtime
• Incremental scans: mtime-based, only re-analyzes changed files
• Zero new dependencies: all analysis is regex-based, graph is hand-rolled
• Lazy loading: intel code only loads when jam intel * is invoked — no startup impact

Numbers

Metric	Value
New source code	3,215 lines (19 files)
New tests	3,198 lines (15 files)
Total tests	647 (was 388)
New dependencies	0
Scan time (this repo)	0.1s → 588 nodes, 997 edges
Dist size (intel)	360KB of 2.8MB total

Test plan

• All 647 tests pass (npm test)
• Clean TypeScript build (npm run build)
• jam intel scan --no-enrich scans this repo successfully
• jam intel status shows correct stats
• jam intel query "provider" --no-ai returns matching nodes
• jam intel impact src/providers/base.ts traces 65 dependent files
• jam intel diagram outputs valid Mermaid
• jam intel explore opens browser viewer
• No startup time regression for non-intel commands

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added Intel command suite for codebase intelligence and analysis
  • Support for analyzing TypeScript/JavaScript, Python, COBOL, SQL, Docker, and OpenAPI files
  • Generate architecture and dependency diagrams as Mermaid visualizations
  • Query codebase knowledge graph with keyword search or natural language
  • Analyze code impact and dependencies across files
  • LLM-based enrichment to add semantic insights and patterns to your codebase

• Configuration

  • New .jam/intel storage directory for graph and enrichment data
  • Configurable enrichment depth, token budgets, and diagram output formats

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Introduces a comprehensive Intel subsystem for codebase analysis, including static file analyzers for six languages (TypeScript, Python, COBOL, SQL, Docker, OpenAPI), a graph-based knowledge representation, LLM-powered semantic enrichment, natural language and keyword-based querying, and Mermaid diagram generation for visualization and browser-based exploration.

Changes

Cohort / File(s)	Summary
Configuration & Initialization .gitignore, src/config/defaults.ts, src/config/schema.ts	Added .jam/ to gitignore and introduced Intel config block with enrichment depth, token budget, storage directory, auto-scan, exclusion patterns, diagram format, and browser-open flags.
CLI Command Layer src/index.ts, src/commands/intel.ts	Added top-level intel command group with subcommands: scan, status, query, impact, diagram, and explore. Mapped CLI options to function handlers with dynamic imports and global option integration.
Core Data Model & Graph src/intel/types.ts, src/intel/graph.ts, src/intel/graph.test.ts	Defined node/edge/enrichment types (NodeType, EdgeType, EnrichDepth, IntelNode, IntelEdge, SemanticMetadata) and implemented IntelGraph with node/edge management, BFS pathfinding, impact subgraph computation, serialization, and statistics.
File Analysis & Scanning src/intel/analyzers/base.ts, src/intel/analyzers/registry.ts, src/intel/scanner.ts, src/intel/scanner.test.ts	Established AnalyzerPlugin contract, created registry for plugin lookup, and implemented Scanner for incremental workspace analysis with framework detection, file filtering, and per-analyzer project-level enrichment hooks.
Language-Specific Analyzers src/intel/analyzers/typescript.ts, src/intel/analyzers/python.ts, src/intel/analyzers/cobol.ts, src/intel/analyzers/sql.ts, src/intel/analyzers/docker.ts, src/intel/analyzers/openapi.ts + corresponding .test.ts files	Implemented six analyzers using regex-based pattern matching to extract nodes (files, classes, functions, endpoints, tables, configs) and edges (imports, calls, reads, writes, contains, depends-on) specific to each language/format.
Framework Detection src/intel/frameworks/profiles.ts, src/intel/frameworks/detector.ts, src/intel/frameworks/*.test.ts	Defined FrameworkMarker and FrameworkProfile types, created a framework database (Express, React, Django, dbt, etc.), and implemented async framework detection via package.json, requirements.txt, and file/directory markers.
LLM Enrichment Engine src/intel/enrichment.ts, src/intel/enrichment.test.ts	Built EnrichmentEngine for node prioritization (in/out degree and churn-based scoring), dynamic prompt generation (shallow vs. deep depth), streamed LLM response parsing with robust JSON handling, and batch enrichment with token-budget enforcement and progress callbacks.
Querying & Visualization src/intel/query.ts, src/intel/query.test.ts, src/intel/mermaid.ts, src/intel/mermaid.test.ts, src/intel/viewer.ts	Implemented keyword-based offline and NL-based tool-calling query modes against the graph, Mermaid diagram generators (architecture, deps, flow, impact, framework-specific), and a browser-based viewer with auto-polling for diagram updates.
Persistence & Public API src/intel/storage.ts, src/intel/storage.test.ts, src/intel/index.ts	Added graph/enrichment serialization with file-based locking, Mermaid file saving, .gitignore verification, and a barrel export consolidating the entire Intel module surface.
End-to-End Integration Test src/intel/intel.integration.test.ts	Validated full scan → query → diagram pipeline using a real Express fixture with TypeScript and SQL files, verifying framework detection, incremental scanning, and query/diagram functionality.

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI as CLI (src/index.ts)
    participant Scanner
    participant Analyzers
    participant Graph
    participant Enrichment as Enrichment Engine
    participant LLM as LLM Provider
    participant Query
    participant Mermaid as Mermaid/Viewer
    
    User->>CLI: jam intel scan
    CLI->>Scanner: scan(rootDir, options)
    Scanner->>Scanner: collectFiles(excludePatterns)
    Scanner->>Analyzers: analyzeFile(content, path)
    Analyzers->>Graph: addNode/addEdge
    Analyzers->>Graph: analyzeProject (if provided)
    Scanner->>Graph: setFrameworks, setLanguages
    CLI->>Graph: serialize → storage
    
    User->>CLI: jam intel query "text"
    alt AI Enabled
        CLI->>Query: query(text, graph, provider)
        Query->>LLM: chatWithTools(tools, prompt)
        LLM->>Query: tool calls (findNode, getNeighbors, etc.)
        Query->>Graph: executeToolCall
        Graph->>Query: nodes, edges
        Query->>Mermaid: formatQueryResultAsMermaid (if flag)
    else Offline Mode
        Query->>Graph: keyword search
        Graph->>Query: matched nodes, edges
    end
    Query->>User: QueryResult
    
    User->>CLI: jam intel scan --enrich deep
    CLI->>Enrichment: enrichAll(graph, options)
    Enrichment->>Enrichment: prioritize(graph)
    Enrichment->>LLM: stream enrichment for node
    LLM->>Enrichment: purpose, pattern, risk, semanticEdges
    Enrichment->>Graph: attach SemanticMetadata
    CLI->>Graph: serialize enrichment → storage
    
    User->>CLI: jam intel diagram --type architecture
    CLI->>Mermaid: generateArchitectureDiagram(graph)
    Mermaid->>Graph: allNodes, allEdges
    Mermaid->>User: Mermaid string
    
    User->>CLI: jam intel explore
    CLI->>Mermaid: generateArchitectureDiagram(graph)
    CLI->>Viewer: generateViewerHtml(mermaid, path)
    Viewer->>Viewer: auto-poll .mmd file
    Viewer->>User: open in browser

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Poem

🐰 Whiskers twitch with delight!
A whole new world of code takes flight,
With graphs and nodes, and enrichment deep,
The codebase secrets we now shall keep,
Analyzers hopping through every file,
Making sense of it all with style! 🌟

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 67.31% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main feature: introducing a new 'jam intel' module for codebase intelligence with graph building, diagram generation, and querying capabilities.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/jam-intel

📝 Coding Plan

• Generate coding plan for human review comments

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}