feat(retrieval): DB-native RRF via Supabase RPC, drop Python-side fusion

.gitignore

@@ -27,3 +27,9 @@ rag-agent/models/
 
 # Temporary generated files
 /pr_comments.json
+
+# Claude Code local session state (user-specific)
+.claude/
+
+# Root-level personal utility scripts (not part of RAG)
+/tools/

CLAUDE.md

@@ -0,0 +1,102 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+DOC-KB-RAG is a Retrieval-Augmented Generation system. It ingests Markdown/JSON documentation into a Supabase pgvector store and answers questions about it using Google Gemini embeddings and LLM. The `docs/` corpus is fully interchangeable — the RAG engine makes no assumptions about the content.
+
+## Working Directory
+
+All commands run from `rag-agent/`. That directory owns the `.env`, `venv/`, and Supabase config. Never run `ingest.py` or `query.py` from the repo root.
+
+## Commands
+
+### Environment Setup
+```bash
+# Windows Git Bash (the active shell on this machine)
+source venv/Scripts/activate
+
+# Start local Supabase (requires Docker Desktop running)
+npx supabase start
+
+# Stop when done
+npx supabase stop
+```
+
+### Running the Pipeline
+```bash
+# Ingest docs into vector store (re-run when corpus changes; SHA-256 dedup skips unchanged files)
+python ingest.py
+
+# Query the indexed docs
+python query.py "Your question here"
+```
+
+### Verifying Changes
+Before marking any feature complete, run a live query to confirm the pipeline is unbroken:
+```bash
+python query.py "What is X?"
+```
+
+### CI
+The CI pipeline (`.github/workflows/ci.yml`) currently only runs a syntax check (`py_compile`) on `ingest.py` and `query.py` against Python 3.10 and 3.11. There are no automated tests yet.
+
+### Helper Tools
+```bash
+python tools/list_models.py       # List available embedding models
+python tools/list_llm_models.py   # List available LLM models
+python tools/check_dim.py         # Check embedding dimensions
+```
+
+## Architecture
+
+### Data Flow
+
+**Ingest:**
+`Docs (.md/.mdx/.json)` → `SimpleDirectoryReader` → `MarkdownNodeParser` (header-aware splits) → `SentenceSplitter` (512 tokens, 64 overlap) → `GoogleGenAIEmbedding` (3072 dims) → `SupabaseVectorStore` + `docstore.json` (SHA-256 dedup cache)
+
+**Query:**
+`Question` → `GoogleGenAIEmbedding` → `asyncpg` RPC `hybrid_search_rrf()` (dense cosine + BM25 full-text fused via RRF, top_k=5) → `RAG_PROMPT_TEMPLATE` → `GoogleGenAI LLM` → Answer + source attribution with RRF scores
+
+### Key Files
+- `rag-agent/ingest.py` — Full ingestion pipeline with path validation and dedup
+- `rag-agent/query.py` — Query engine with prompt injection protection and source attribution
+- `rag-agent/.env` — Secrets/config (gitignored; copy from `.env.example`)
+- `rag-agent/migrations/001_hnsw_index.sql` — HNSW cosine index (m=16, ef_construction=64)
+- `rag-agent/migrations/003_hybrid_search_rrf.sql` — DB-native hybrid search RPC (RRF fusion)
+- `rag-agent/supabase/config.toml` — Local Supabase ports (DB: 54322, API: 54321, Studio: 54323)
+
+### Configuration (`rag-agent/.env`)
+| Variable | Default | Purpose |
+|---|---|---|
+| `GOOGLE_API_KEY` | — | Gemini API key (required) |
+| `DB_CONNECTION_STRING` | `postgresql://postgres:postgres@127.0.0.1:54322/postgres` | Supabase local DB |
+| `DOCS_PATH` | — | Directory to ingest (validated against path-traversal list) |
+| `EMBED_MODEL` | `models/gemini-embedding-2-preview` | Embedding model |
+| `LLM_MODEL` | `models/gemini-3.1-flash-lite-preview` | Generation model |
+| `EMBED_DIMENSIONS` | `3072` | Must match the embedding model output |
+| `COLLECTION_NAME` | `openclaw_docs` | pgvector collection (`vecs.<name>`) |
+| `CHUNK_SIZE` / `CHUNK_OVERLAP` | `512` / `64` | SentenceSplitter params |
+| `SIMILARITY_TOP_K` | `5` | Number of results from hybrid search RPC |
+
+## Development Rules
+
+- **Scope:** Work is confined to `ingest.py`, `query.py`, the Supabase integration, and Gemini configuration. Do not build tangential functionality outside RAG scope unless explicitly asked.
+- **Modularity:** Keep the vector DB layer and LLM layer loosely coupled and pluggable even as you add features.
+- **Virtual environment:** All Python execution must use `rag-agent/venv`. Never install global pip packages.
+- **Docker required:** Supabase (`npx supabase ...`) requires Docker Desktop to be running.
+- **No destructive migrations:** Never drop or permanently alter the vector DB schema without explicit user approval.
+- **Secrets:** Never log, print, or commit `GOOGLE_API_KEY` or `DB_CONNECTION_STRING`.
+- **README as source of truth:** If a script argument changes in a breaking way, update `README.md` immediately.
+
+## Roadmap Context
+
+- **Phase 1** (done): Foundation — MarkdownNodeParser chunking, top-k tuning, source attribution, config centralization
+- **Phase 2** (done): Incremental ingestion — `IngestionPipeline` + SHA-256 dedup via `docstore.json`
+- **Phase 3** (done): Hybrid search — BM25 + dense vector RRF via Postgres RPC (`hybrid_search_rrf`)
+- **Phase 4** (planned): MCP server + FlashRank reranking + mimalloc allocator (WSL2 deployment)
+
+## Agent Framework
+
+`.agent/` contains 8 specialist agent personas, 11 skills, and 14 workflow slash commands for coordinating multi-agent development. See `.agent/ARCHITECTURE.md` for the coordination model. Workflows include `/orchestrate`, `/plan`, `/debug`, `/test`, `/refactor`, `/create-pr`, and others.

README.md

@@ -10,16 +10,14 @@ documentation corpus in one place.
 - `docs/` is the documentation corpus currently being indexed.
 - `.agent/` provides agent personas, skills, and workflows for coordinating
   multi-agent development work. See `.agent/ARCHITECTURE.md`.
-- Root-level files such as `AGENTS.md`, `GEMINI.md`, and `CHANGELOG.md` are
-  project guidance and imported reference material.
+- Root-level files such as `AGENTS.md` and `GEMINI.md` are project guidance.
 
 ## Structure Map
 
 ```text
 DOC-KB-RAG/
 |-- .agent/                  # Agent coordination framework (see ARCHITECTURE.md)
 |-- AGENTS.md                # Local agent instructions for this repo
-|-- CHANGELOG.md             # Imported/source project changelog
 |-- GEMINI.md                # Gemini-specific project context
 |-- LICENSE
 |-- README.md
@@ -58,7 +56,15 @@ application root and contains the local `.env`, `venv`, and Supabase config.
 - Docker Desktop running
 - Node.js
 
-### 2. Activate the virtual environment
+### 2. Install Python dependencies
+
+From `rag-agent/`:
+
+```bash
+pip install -r requirements.txt
+```
+
+### 3. Activate the virtual environment
 
 From `rag-agent/`:
 
@@ -80,12 +86,25 @@ source venv/Scripts/activate
 .\venv\Scripts\Activate.ps1
 ```
 
-### 3. Start local Supabase
+### 4. Start local Supabase
 
 ```bash
 npx supabase start
 ```
 
+### 5. Run database migrations
+
+After the first ingestion, run the SQL migrations once:
+
+```bash
+psql postgresql://postgres:postgres@127.0.0.1:54322/postgres \
+  -f migrations/001_hnsw_index.sql \
+  -f migrations/002_pg_trgm.sql \
+  -f migrations/003_hybrid_search_rrf.sql
+```
+
+See `migrations/README.md` for details.
+
 ## Configuration
 
 Set these values in `rag-agent/.env`:
@@ -141,3 +160,12 @@ python ingest.py
   usage depend on those entrypoints.
 - Avoid moving `rag-agent/venv/` or `rag-agent/supabase/` unless you are also
   rebuilding the environment and command assumptions around them.
+
+## Known Limitations (v0.1.0-beta)
+
+- CI runs syntax checks only (`py_compile`) — no automated test suite yet.
+- The collection name `openclaw_docs` is hardcoded in the SQL migrations; update
+  manually if you change `COLLECTION_NAME`.
+- `hybrid_search_rrf()` computes `tsvector` on-the-fly; no stored/indexed
+  tsvector column (planned optimisation for Phase 4).
+- No reranking stage — planned for Phase 4 (FlashRank cross-encoder).

rag-agent/.env.example

@@ -16,9 +16,8 @@ EMBED_BATCH_SIZE=100
 CHUNK_SIZE=512
 CHUNK_OVERLAP=64
 
-# Query
+# Query (SIMILARITY_TOP_K maps to hybrid_search_rrf match_count)
 SIMILARITY_TOP_K=5
-SIMILARITY_CUTOFF=0.65
 
 # Logging (DEBUG, INFO, WARNING, ERROR)
 LOG_LEVEL=INFO

rag-agent/migrations/002_pg_trgm.sql

@@ -0,0 +1,9 @@
+-- Migration 002: Enable pg_trgm extension for trigram-based text search support.
+--
+-- Run this ONCE in the Supabase SQL editor (or via psql) BEFORE migration 003.
+-- pg_trgm provides trigram similarity functions and operators that complement
+-- tsvector full-text search for fuzzy matching scenarios.
+--
+-- This is safe to run multiple times (IF NOT EXISTS).
+
+CREATE EXTENSION IF NOT EXISTS pg_trgm;

rag-agent/migrations/003_hybrid_search_rrf.sql

@@ -0,0 +1,92 @@
+-- Migration 003: Create hybrid_search_rrf() RPC function.
+--
+-- Run this ONCE in the Supabase SQL editor (or via psql) AFTER migrations 001 and 002.
+-- This function fuses dense vector search (cosine) with full-text BM25-style search
+-- using Reciprocal Rank Fusion (RRF).  Both retrieval methods run inside the DB;
+-- only the final top-K rows cross the network.
+--
+-- NOTE: If you changed COLLECTION_NAME in .env from the default "openclaw_docs",
+-- replace every occurrence of "openclaw_docs" in this file with your collection name.
+--
+-- Schema note: The vecs library stores node text inside metadata->'_node_content'
+-- (a serialized JSON string).  The text is extracted via:
+--   (metadata->>'_node_content')::jsonb->>'text'
+--
+-- Parameters:
+--   query_text       — raw user question (for full-text search via websearch_to_tsquery)
+--   query_embedding  — dense embedding vector from Gemini (3072 dims)
+--   match_count      — number of fused results to return (default 5)
+--   rrf_k            — RRF smoothing constant (default 60, standard value)
+--   semantic_weight  — weight for semantic (vector) results in fusion (default 0.5)
+--   fulltext_weight  — weight for full-text (BM25) results in fusion (default 0.5)
+
+CREATE OR REPLACE FUNCTION hybrid_search_rrf(
+    query_text       text,
+    query_embedding  vector(3072),
+    match_count      int     DEFAULT 5,
+    rrf_k            int     DEFAULT 60,
+    semantic_weight  float   DEFAULT 0.5,
+    fulltext_weight  float   DEFAULT 0.5
+)
+RETURNS TABLE (
+    id       text,
+    content  text,
+    metadata jsonb,
+    score    float
+)
+LANGUAGE sql
+STABLE
+AS $$
+    -- 1. Dense vector search: cosine distance, uses HNSW index from migration 001
+    WITH semantic AS (
+        SELECT
+            s.id,
+            ROW_NUMBER() OVER (ORDER BY s.vec <=> query_embedding) AS rank_ix
+        FROM vecs.openclaw_docs s
+        ORDER BY s.vec <=> query_embedding
+        LIMIT (match_count * 2)
+    ),
+
+    -- 2. Full-text search: BM25-style ranking via tsvector/tsquery
+    --    Text is extracted from the serialized _node_content JSON inside metadata.
+    --    Computed on-the-fly (no stored tsvector column — we do not own the vecs schema).
+    fulltext AS (
+        SELECT
+            f.id,
+            ROW_NUMBER() OVER (
+                ORDER BY ts_rank_cd(
+                    to_tsvector('english', (f.metadata ->> '_node_content')::jsonb ->> 'text'),
+                    websearch_to_tsquery('english', query_text)
+                ) DESC
+            ) AS rank_ix
+        FROM vecs.openclaw_docs f
+        WHERE
+            f.metadata ->> '_node_content' IS NOT NULL
+            AND to_tsvector('english', (f.metadata ->> '_node_content')::jsonb ->> 'text')
+                @@ websearch_to_tsquery('english', query_text)
+        LIMIT (match_count * 2)
+    ),
+
+    -- 3. Reciprocal Rank Fusion: score = sum of weight/(k + rank) per method.
+    --    Documents found by only one method get 0 for the missing term.
+    fused AS (
+        SELECT
+            COALESCE(sem.id, ft.id) AS id,
+            (COALESCE(semantic_weight / (rrf_k + sem.rank_ix), 0.0)
+           + COALESCE(fulltext_weight / (rrf_k + ft.rank_ix), 0.0)) AS score
+        FROM semantic sem
+        FULL OUTER JOIN fulltext ft ON sem.id = ft.id
+        ORDER BY score DESC
+        LIMIT match_count
+    )
+
+    -- 4. Re-join to fetch content and lightweight metadata (strip _node_content).
+    SELECT
+        fused.id,
+        (doc.metadata ->> '_node_content')::jsonb ->> 'text' AS content,
+        doc.metadata - '_node_content'                        AS metadata,
+        fused.score
+    FROM fused
+    JOIN vecs.openclaw_docs doc ON doc.id = fused.id
+    ORDER BY fused.score DESC;
+$$;

rag-agent/migrations/README.md

@@ -0,0 +1,27 @@
+# Database Migrations
+
+Run these SQL files **in order** against your Supabase database
+(SQL Editor or `psql`). Each file is idempotent (safe to re-run).
+
+| File | Purpose |
+|---|---|
+| `001_hnsw_index.sql` | HNSW cosine distance index on `vecs.openclaw_docs.vec` |
+| `002_pg_trgm.sql` | Enable `pg_trgm` extension for trigram text search |
+| `003_hybrid_search_rrf.sql` | `hybrid_search_rrf()` RPC — fuses vector + full-text via RRF |
+
+## Running via psql
+
+```bash
+# From rag-agent/ with local Supabase running:
+psql postgresql://postgres:postgres@127.0.0.1:54322/postgres \
+  -f migrations/001_hnsw_index.sql \
+  -f migrations/002_pg_trgm.sql \
+  -f migrations/003_hybrid_search_rrf.sql
+```
+
+## Notes
+
+- Migrations target the default collection `openclaw_docs`. If you use a
+  different `COLLECTION_NAME`, update the table references in each file.
+- None of these migrations are destructive — they only add indexes,
+  extensions, and functions.

rag-agent/package.json

@@ -1,6 +1,6 @@
 {
   "name": "rag-agent",
-  "version": "1.0.0",
+  "version": "0.1.0-beta",
   "description": "",
   "main": "index.js",
   "scripts": {