An LLM-first, guardrailed question-answering assistant over your own tabular data (CSV → DuckDB) and documents (Markdown/text → hybrid retrieval). Ask a question in plain English; the assistant decides whether to run a safe SQL query, search the documents, or both — then answers with its sources.
It is LLM-first: every answer comes from a live model (a cloud API or a local CLI) running a native tool-calling loop — there is no deterministic "fake answer" mode. Drop new files into the data folder and it re-embeds and re-indexes them automatically.
This is a clean, general-purpose reference implementation. It ships with synthetic demo data only.
A normal chatbot hard-codes intents and bolts an LLM on as an afterthought. Here the LLM is the brain:
it plans, calls tools (run_sql, search_docs), reads the results, and synthesizes an answer — while a
deterministic safety layer wraps every call so the model can never run unsafe SQL, leak secrets, or be
hijacked by injected instructions.
- Agentic core — native tool-calling loop (
run_sql+search_docs), multi-turn conversation memory. - Safe SQL — single read-only
SELECTonly; DDL/DML, file-readers, internal tables, and stacked statements are rejected; row limits are clamped. - Pluggable LLM providers — OpenAI (or any OpenAI-compatible endpoint), AWS Bedrock, or any local CLI (e.g. the ChatGPT/Codex CLI on an existing subscription). One interface, swap freely.
- Hybrid retrieval — BM25 (lexical) + dense vectors fused with reciprocal-rank fusion.
- Guardrails in & out — input firewall (injection / exfiltration / code-request / unsafe-SQL / format-hijack) and an output scrubber that redacts leaked secrets and a leaked system prompt.
- LLM-first (required) — every analytical answer is produced by a live model; with no provider
configured the app refuses to start rather than guessing (override with
REQUIRE_LLM=false). - Auto-ingest — drop a CSV / Markdown / text file into the data dir and a background watcher embeds and indexes it on the fly; no restart, no manual step. Local hashing embeddings mean vectorization needs no model download or API key.
- Atomic ingestion — table reloads build a temp table and swap, so a failed load never destroys data.
- Hardened HTTP — body-size limit, per-IP rate limiting, optional API key, security headers.
- Tested, linted, and CI-wired.
flowchart LR
UI[Chat UI] --> API[FastAPI]
API --> MW[Security middleware]
API --> Guard[Input firewall]
Guard --> Engine[Agent engine]
Engine -->|tool-calling| LLM[LLM provider]
Engine -->|offline| Fallback[Deterministic fallback]
Engine --> Tools
Tools --> SQL[(DuckDB · safe SELECT)]
Tools --> RAG[BM25 + vectors]
Engine --> Scrub[Output scrubber] --> UI
make setup # create venv + install deps
make test # run the test suite
make run # serve at http://127.0.0.1:8000The repo ships with synthetic demo data already generated in data/sample/. Regenerate it any time
with make sample-data.
This assistant is LLM-first, so before make run copy env.example to .env and configure one
provider (below) — without a model the server refuses to start. Then open http://127.0.0.1:8000 and
ask: "What was total revenue by region?" or "What does baseline mean?"
To add your own data, just drop .csv (tables) or .md / .txt (documents) into the data dir — the
running app picks them up, embeds, and indexes them automatically within seconds.
Set LLM_PROVIDER (or leave it auto) and configure one backend:
| Provider | LLM_PROVIDER |
Configure | Notes |
|---|---|---|---|
| OpenAI / compatible | openai |
OPENAI_API_KEY, OPENAI_MODEL, OPENAI_BASE_URL |
Native tool-calling. Works with any OpenAI-compatible endpoint. |
| AWS Bedrock | bedrock |
pip install boto3, BEDROCK_MODEL_ID, AWS_REGION + AWS creds |
Native tool use via the Bedrock converse API. |
| Local CLI | cli |
LLM_CLI_COMMAND |
Any CLI that reads the prompt on stdin and writes the answer on stdout — e.g. the ChatGPT/Codex CLI on an existing subscription (no API key). Uses a text-based tool protocol. |
| None | none |
— | Disables the model. The app refuses to start unless REQUIRE_LLM=false, in which case it answers every question with a "configure a provider" message. |
auto picks the first that's configured: OpenAI → Bedrock → CLI. If none is configured the app
refuses to start (set REQUIRE_LLM=false to boot without a model).
All settings come from environment variables (or .env). See env.example for the
full list — API key, rate limits, LLM providers, embedding provider, and ingestion limits.
- Guard inspects the question (length, injection/exfiltration/scope) before anything else.
- Engine builds a system prompt (armor + live schema + document summary) and runs the LLM tool-calling loop; tools execute against DuckDB and the retriever.
- Scrubber redacts any leaked secret/prompt/code from the final text.
- Auto-ingest watches the data dir: add or change a CSV / Markdown / text file and it is
re-embedded and re-indexed on the fly (the
/refreshendpoint runs the same reindex manually).
app/
main.py FastAPI app + routes
middleware.py body limit, rate limit, API key, security headers
config.py env-driven settings
security/ input firewall, output scrubber, system-prompt armor
data/ DuckDB store (safe SELECT, atomic load), ingestion, auto-reindex watcher, BM25
rag/ embeddings, in-memory vector index, hybrid retriever
agent/ tools, LLM providers, memory, engine
ui/chat.html single-file chat UI
scripts/ synthetic data generator
tests/ pytest suite
The deterministic safety layer holds even if the model misbehaves. SQL safety is enforced in three
layers rather than a (bypassable) denylist: the DuckDB connection has external file access
disabled, every query is parsed and checked against a table allowlist, and results are
hard-capped by an outer LIMIT. Untrusted tool/document content is never treated as instructions,
and outputs are scrubbed for leaked secrets/prompt. See app/security/ and app/data/store.py.
MIT — see LICENSE.