A Retrieval-Augmented Generation (RAG) pipeline for technical documentation. Point it at any docs website — it scrapes the pages, chunks and embeds them, stores them in Chroma, and then answers your questions using the site as context.
Input: URL of a documentation site. Output: an LLM that answers questions grounded in that site's content.
The repo ships with two sites already scraped as demos — Hyperliquid docs and HuggingFace Transformers docs — but the pipeline is generic and works on any docs site you feed into Scrapper.ipynb.
Scrapper → Chunker → ChromaExporter → Querier → TestGenerator → Evaluator
Each stage is a Jupyter notebook. Run them in order the first time; after that any stage can be re-run in isolation.
| Notebook | Purpose |
|---|---|
Scrapper.ipynb |
BFS-crawl a docs site, save raw text per page under scraped_texts/<host>/raw/ |
Chunker.ipynb |
Split raw text into overlapping chunks (multiple size/overlap presets) |
ChromaExporter.ipynb |
Embed chunks with a HuggingFace model, push into a running Chroma collection |
Querier.ipynb |
Ask a question → retrieve top-k chunks → generate an answer via OpenAI |
TestGenerator.ipynb |
Auto-generate Q&A test cases from the corpus using OpenAI |
Evaluator.ipynb |
Sweep retrieval k, score answers, plot results |
Requirements: Python 3.12, a running Chroma server, an OpenAI API key.
python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m ipykernel install --user --name=rag-scrapper --display-name="Python 3.12 (rag-scrapper)"Copy .env.example → .env and fill in your OpenAI key:
cp .env.example .env
# edit .env, set OPENAI_API_KEY=sk-...Start a Chroma server (default http://localhost:8000):
chroma run --path ./chroma-datajupyter labIn each notebook: Kernel → Change Kernel → Python 3.12 (rag-scrapper).
chromadb==0.4.24 pulls in older API surfaces; on Python 3.12 it requires numpy<2 and scipy<1.14. If you rebuild the venv from requirements.txt, add those constraints or upgrade chromadb.