🔍 Repo Explainer Agent

Point it at any codebase — local or GitHub URL — and ask it anything. It maps the architecture, reads the code, and gives you grounded answers.

---

Overview

Repo Explainer is an agentic AI system built on the Claude Agent SDK. It uses multi-step reasoning and file-system tools to deeply understand any codebase and answer natural language questions about it.

It is built around three pillars:

Pillar	What it does
🤖 Agent	Multi-step reasoning agent — reads files, searches code, maps dependencies, synthesizes answers
📊 Eval Harness	Systematic test suite with 8 test cases and 4 weighted metrics to measure answer quality
⚡ Optimizer	Automated prompt tuning loop that uses eval scores to iteratively improve the agent

---

Architecture

┌─────────────────────────────────────────────────────┐
│                    CLI Entry Point                   │
│              scripts/run_agent.py                    │
└─────────────────────┬───────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────┐
│                  Agent Core                          │
│            agent/repo_explainer.py                   │
│                                                      │
│  ┌─────────────┐     ┌──────────────────────────┐   │
│  │System Prompt│     │      Agentic Loop         │   │
│  │agent/       │────▶│  send → tool_use →        │   │
│  │prompts.py   │     │  execute → send result →  │   │
│  └─────────────┘     │  repeat → final answer    │   │
│                      └──────────────────────────-┘   │
└──────────────────────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────┐
│                    Tools Layer                       │
│                  agent/tools.py                      │
│                                                      │
│  read_file │ list_directory │ search_code │          │
│  get_file_tree │ detect_language_and_framework       │
└─────────────────────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────┐
│                  Claude API                          │
│     (Anthropic direct or any compatible gateway)     │
└─────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────┐
│                  Eval + Optimizer                    │
│                                                      │
│  evals/harness.py ──▶ evals/metrics.py              │
│         │                                            │
│         ▼                                            │
│  optimizer/optimizer.py                              │
│  (prompt tuning loop — uses eval scores to improve) │
└─────────────────────────────────────────────────────┘

---

Quick Start

Prerequisites

• Python 3.10+
• An API key for Claude (see setup options below)

Installation

# Clone the repo
git clone https://github.com/AxaySharma/repo-explainer.git
cd repo-explainer

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Set up environment
cp .env.example .env
# Edit .env and fill in your credentials (see options below)

---

API Setup Options

The agent uses the Anthropic Python SDK. You can point it at any compatible endpoint.

Option 1 — Anthropic API (Direct)

Get a key from console.anthropic.com.

# .env
ANTHROPIC_API_KEY=sk-ant-your-key-here

Option 2 — OpenRouter (pay-per-use, many Claude models available)

Get a key from openrouter.ai. Costs fractions of a cent per run.

# .env
ANTHROPIC_API_KEY=sk-or-your-key-here
ANTHROPIC_BASE_URL=https://openrouter.ai/api

Then in agent/repo_explainer.py set:

model = "anthropic/claude-3.5-haiku"

Option 3 — Claude Code (Pro/Max subscribers)

If you have a Claude Pro or Max subscription, you can run scripts as bash tasks from within an active Claude Code session. Claude Code injects its auth into subprocesses it spawns directly:

# Start Claude Code in your project directory
claude

# Then ask Claude Code to run it as a task:
# "Please run: python scripts/run_agent.py --repo evals/fixtures/sample_project --question 'What is the architecture?'"

Note: This requires running the command from within the Claude Code session itself, not from a separate terminal window.

---

Usage

Ask about a local repo

python scripts/run_agent.py --repo /path/to/any/repo --question "How does authentication work?"

Ask about a GitHub repo (no cloning needed)

python scripts/run_agent.py --repo https://github.com/any/public-repo --question "What is the overall architecture?"

Example output

🔍 Repo Explainer Agent
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📁 Repo: /path/to/repo
❓ Question: How does authentication work?

🤖 Thinking...

✅ Answer:
The authentication system uses JWT tokens issued at /auth/login.
Tokens are validated in auth.py via the decode_token() function,
which is called by the require_auth decorator applied to protected routes.

📊 Stats: 4 iterations | Tools used: get_file_tree, read_file, search_code

Run the Eval Suite

python scripts/run_evals.py

Run the Optimizer

python scripts/run_optimizer.py --iterations 3

---

The Agent

The agent runs a proper agentic loop — not a single-shot API call:

1. Receives repo path or GitHub URL + question
2. Always starts with get_file_tree + detect_language_and_framework
3. Reads relevant files, searches for patterns
4. Synthesizes a grounded answer from actual code it has read
5. Stops when it has sufficient context (max 10 iterations)

Tools available:

Tool	Purpose
read_file	Read any file with line numbers
list_directory	Browse directory contents
search_code	Grep patterns across the codebase
get_file_tree	Full ASCII repo structure
detect_language_and_framework	Identify stack from config files

---

Eval Harness

8 test cases run against a sample FastAPI project in evals/fixtures/sample_project/.

Metrics (weighted):

Metric	Weight	What it measures
Topic Coverage	40%	Are expected concepts present in the answer?
Hallucination Penalty	30%	Does the answer contain fabricated information?
Answer Length	15%	Is the answer appropriately detailed?
Groundedness	15%	Did the agent actually read the code to answer?

A test case passes if overall score ≥ 0.6.

---

Optimizer

The optimizer runs an automated prompt tuning loop:

baseline eval → identify failed cases → ask Claude to improve system prompt
→ re-run evals → keep if better → revert if worse → repeat N times

Cost-efficient by design — uses the fastest available Claude model throughout.

---

Results

Run	Pass Rate	Avg Score	Topic Coverage	Groundedness
Baseline	87.5%	0.853	0.834	0.963
After Optimization (iter 1)	100%	0.952	0.984	1.000
Δ Improvement	+12.5%	+0.099	+0.150	+0.037

The optimizer identified a failure in framework_detection, improved the system prompt in iteration 1, and recovered to 8/8 passing with a +0.099 score improvement. Further variance across runs is due to LLM non-determinism — a known characteristic of probabilistic models that averaging across multiple runs mitigates. See optimizer/results/ for full JSON reports.

---

Design Decisions

• Claude Agent SDK over raw API — proper agentic loop with tool use, not a single-shot prompt
• Compatible with any Claude-compatible endpoint — Anthropic direct, OpenRouter, or Claude Code
• Grounded answers only — agent is instructed never to claim something it has not read in actual code
• GitHub URL support — clones to a temp directory transparently so remote repos work out of the box
• Weighted metrics — hallucination penalty weighted heavily (30%) because a wrong answer is worse than an incomplete one
• Optimizer reverts on regression — keeps the best prompt found, never degrades below baseline

---

Project Structure

repo-explainer/
├── agent/
│   ├── repo_explainer.py      # Core agent + agentic loop
│   ├── tools.py               # File system tools
│   ├── prompts.py             # System prompt + templates
│   └── prompts_optimized.py   # Best prompt found by optimizer
├── evals/
│   ├── harness.py             # Eval runner
│   ├── metrics.py             # Scoring functions
│   ├── test_cases.py          # 8 test cases
│   └── fixtures/
│       └── sample_project/    # Sample FastAPI project for testing
├── optimizer/
│   ├── optimizer.py           # Prompt tuning loop
│   └── results/               # Before/after JSON reports
├── scripts/
│   ├── run_agent.py           # CLI: run the agent
│   ├── run_evals.py           # CLI: run eval suite
│   └── run_optimizer.py       # CLI: run optimizer
├── tests/
│   └── test_metrics.py        # 25 unit tests (all passing)
├── .env.example
├── requirements.txt
├── CONTRIBUTING.md
├── CHANGELOG.md
└── README.md

---

Built with the Claude Agent SDK · MIT License