Spec: RAG architecture deepening

[elkaix]

Design spec for the RAG architecture deepening effort. Synthesized from the 2026-07-12 architecture review (three parallel explorations: core RAG modules, API layer, eval harness) and the wayfinder map's charting decisions. Resolves all eight map tickets with their recommended answers.

Problem Statement

The developer maintains a public portfolio RAG system whose architecture has drifted into three kinds of friction:

1. Proven improvements can't ship. The eval harness demonstrated gains from hybrid BM25 retrieval, cross-encoder reranking, query rewriting, and refusal handling — but production has no retrieval seam, so none of these levers can run for real users. Worse, the eval pipeline re-implements retrieve→generate with hand-copied prompts that have already drifted from production, so eval results measure a pipeline that isn't the one being served (including different chunking defaults).
2. The same question gets different treatment depending on the path. The answer system prompt exists in three copies that have diverged — the streaming chat path instructs the model differently than the synchronous query path — and reported token costs are computed against a reconstructed copy of the prompt rather than the one actually sent.
3. The codebase resists testing and navigation. One 1,491-line facade mixes query orchestration, conversation CRUD, and evaluation; an 812-line LLM module repeats the same provider dispatch four times with no injection point (two of three real providers have zero test coverage); routes hand-copy fields across the API seam (a key-drift bug already happened once); and dead modules sit around failing the deletion test.

Solution

Deepen the system's modules so each has a small interface hiding real behaviour, placed at a clean seam:

• One Retriever seam through which dense, hybrid, reranked, and multi-query retrieval are interchangeable adapters — making the eval-proven levers shippable by configuration.
• One QueryEngine module owning retrieve→generate — one prompt, one context format, one telemetry assembly — that both production and the eval harness call, so eval provably measures the shipped pipeline.
• Provider adapters inside LLMHandler with injected clients, collapsing four dispatch tables and sixteen provider methods into one adapter per provider, each testable with a fake client.
• A split of the backend facade into ConversationStore, Evaluator, and query orchestration, with the facade kept thin for API compatibility.
• Usage-returning generation so cost telemetry reports the tokens actually billed, with token/pricing utilities owned by the core (not the eval package).
• A WebSocket streaming adapter so the streaming protocol is testable without a live socket.
• Typed objects across the backend→API seam and a single dependency-injection seam for routes.
• Deletion of the modules that fail the deletion test, and closure of the one seam breach into the vector store's internals.

User Stories

1. As an end user, I want the streaming chat answer and the synchronous query answer to be generated with identical instructions, so that answer quality doesn't depend on which path I used.
2. As an end user, I want retrieval quality improvements (hybrid search, reranking) that were validated offline to actually reach the product, so that my questions get better sources.
3. As an end user, I want unanswerable questions handled by the validated refusal behaviour, so that the system says "I don't know" instead of hallucinating.
4. As an end user, I want the cost/token footer under each answer to reflect what was actually sent to the model, so that the numbers I see are trustworthy.
5. As the developer, I want one Retriever interface with dense/hybrid/reranked/multi-query adapters, so that I can switch retrieval strategies by configuration instead of by rewrite.
6. As the developer, I want the eval harness to call the same QueryEngine as production, so that a measured improvement is an improvement in the shipped system.
7. As the developer, I want eval configuration defaults derived from production configuration, so that "baseline" eval runs benchmark the pipeline users actually get.
8. As the developer, I want the answer prompt defined in exactly one module, so that prompt changes can't silently drift between paths.
9. As the developer, I want each LLM provider behind its own adapter with an injected client, so that I can unit-test every provider path with a fake instead of monkeypatching module globals.
10. As the developer, I want generation calls to return usage alongside text, so that no caller ever rebuilds a prompt just to count tokens.
11. As the developer, I want token counting and pricing owned by the core, so that production no longer imports internals from the eval package.
12. As the developer, I want conversation CRUD in its own store module that depends only on the database session, so that conversation bugs concentrate in one place with focused tests.
13. As the developer, I want the realtime evaluation logic in its own module with a direct test, so that its skip/dedup branches are verified instead of mocked away.
14. As the developer, I want the WebSocket event framing behind an adapter, so that the streaming protocol has unit tests that don't open sockets.
15. As the developer, I want the streaming event vocabulary defined once, so that the backend and the WebSocket handler can't disagree about event names.
16. As the developer, I want typed objects crossing the backend→API seam, so that a renamed field is a type error, not a silent production bug.
17. As the developer, I want every route to obtain the backend through one dependency-injection seam, so that tests can override it uniformly.
18. As the developer, I want routes to stop reimplementing existence checks and run-id derivation, so that domain logic lives behind the domain interface.
19. As the developer, I want dead modules deleted, so that readers and agent sessions stop navigating code with zero callers.
20. As the developer, I want the vector store to expose document-chunk lookup on its interface, so that no caller reaches into its private collection and parses raw result shapes.
21. As an eval operator, I want to enable retrieval levers per run config against the shared engine, so that A/B comparisons isolate the lever rather than incidental pipeline differences.
22. As a portfolio reviewer, I want module boundaries that each teach one concept with visible design rationale (ADRs, context glossary), so that the repo demonstrates architectural judgment.
23. As a future agent session, I want each locked decision recorded as an ADR and each new module named in the domain glossary, so that settled decisions aren't re-litigated.

Implementation Decisions

Vocabulary note: decisions below use the codebase-design glossary — module, interface, depth, seam, adapter, leverage, locality.

1. Retriever seam (resolves "Decide the Retriever seam")

• A runtime-checkable Retriever Protocol: retrieve(query, top_k) -> list[SearchResult]. SearchResult remains the shared result type.
• Adapters: a dense adapter wrapping the existing vector store; the existing BM25-hybrid retriever conforms directly.
• The cross-encoder reranker composes rather than conforms: a reranking adapter wraps an inner Retriever, over-fetches candidates, and re-scores — presenting the same Retriever interface outward.
• Multi-query rewriting also composes: a multi-query adapter wraps an inner Retriever, fans out rewritten queries, and dedups — again presenting the same interface.
• The seam is constructor injection on the QueryEngine, defaulting to the dense adapter. Production lever activation is configuration-driven (central config), so a validated eval chain is promoted by config, not by code changes.

2. QueryEngine (resolves "Decide the shared QueryEngine module and prompt ownership")

• A new deep module, QueryEngine, owns retrieve→generate end to end: retrieval via the injected Retriever, context assembly, prompt construction, generation, and telemetry assembly. Interface: a synchronous ask returning (results, answer, telemetry) and a streaming ask yielding typed events — mirroring the two existing query paths.
• Prompt ownership: the markdown-formatting system prompt (currently only on the streaming path) becomes the single answer prompt for all paths — it is what the product's renderer expects, and eval must measure the shipped prompt. One prompt-builder inside the engine produces the (system, user) pair; no other module assembles answer prompts.
• Context format: filename-prefixed chunks win (citations depend on them); the eval pipeline's bare join is retired.
• EvalPipeline becomes an adapter over QueryEngine: it configures retriever chains and judges, but delegates retrieve→generate to the engine. Its divergent copies of prompt/context/top-k logic are deleted.
• EvalConfig defaults derive from production config (single source of truth): baseline eval runs use production's chunking and top-k unless a lever explicitly overrides them.
• Telemetry assembly (token counts, cost, stage latencies) happens once, inside the engine — collapsing the four duplicated assembly sites.

3. Provider adapters (resolves "Decide the LLM provider adapter design")

• A ProviderAdapter Protocol (Protocol over ABC, per project standard) with two methods: generate over an OpenAI-style messages list returning a generation result, and stream yielding tokens with terminal usage.
• Adapter inventory: one OpenAI-compatible adapter serving both OpenAI and GLM (base-url swap and the token-parameter/temperature quirks live inside it), an Anthropic adapter (owning the system-message split), an Ollama adapter (owning its payload shape), and the dummy fallback adapter.
• Client construction is injected (a client factory per adapter), so tests supply fakes per provider instead of monkeypatching module globals. LLMHandler selects one adapter at construction; the four duplicated dispatch tables collapse to that single selection.
• LLMHandler's public interface (generate / stream, prompt and messages variants) is preserved for callers; the prompt variants become thin translations to the messages form.

4. Backend split (resolves "Decide the RAGBackend split")

• Extract ConversationStore — all conversation CRUD, search, export, and sharing — depending only on the database session factory.
• Extract Evaluator — realtime faithfulness scoring and message evaluation — depending on the judge LLM handler and the database.
• RAGBackend remains as a thin facade delegating to QueryEngine, ConversationStore, Evaluator, and ingestion — preserving the API layer's contract while each extracted module gets its own focused interface and tests. Routes keep addressing the facade; nothing user-facing changes.
• Ingestion (load → chunk → upsert → metadata, with the ChromaDB-before-SQLite write order) stays with the facade for now; it is already a coherent responsibility.

5. Telemetry ownership (resolves "Decide telemetry ownership and the usage-returning LLM interface")

• Token counting and pricing move to a core telemetry module; the eval package imports from core, never the reverse.
• Generation returns usage: provider-reported usage where the provider supplies it, adapter-internal counting as fallback — callers never count tokens against reconstructed prompts. Streaming reports usage in its terminal event.
• The engine's telemetry assembly consumes these reported numbers directly.

6. WebSocket streaming adapter (resolves "Decide the WebSocket streaming adapter interface")

• An async streaming adapter owns the sync-generator-to-event-loop bridge (executor sentinel-pull) and the event-tuple→JSON envelope framing. The route handler keeps request parsing, validation, persistence wiring, and disconnect handling.
• The event vocabulary becomes a shared constants/enum definition consumed by both the engine's streaming events and the adapter's framing — deleting the duplicated event-name knowledge.
• The post-stream faithfulness evaluation stays in the handler (it is persistence orchestration, not protocol).
• Error layering: the adapter owns stream-error framing and generator closure; the handler owns request validation and socket lifecycle.

7. Typed API seam (resolves "Decide typed returns and the single DI seam at the API")

• The backend facade returns shared typed objects (Pydantic models owned alongside the domain modules); routes validate/return them directly instead of hand-copying fields. The twice-enumerated source-info shape and double excerpt truncation collapse to one definition with one truncation rule.
• One dependency-injection seam: the annotated backend dependency becomes the only way routes obtain the backend; the direct app-state reads are deleted; the eval run registry gets an equivalent dependency.
• The facade gains the narrow methods routes currently fake: document existence/lookup, display-shaped chunk retrieval, and an eval-facade for run-id derivation and headline-metric selection — so routes stop shelling out to git or mining aggregate dicts.

8. Deletions and the seam breach (resolves "Decide traced_stage adopt-vs-delete; author the dead-code deletion ticket")

• Delete traced_stage. The orchestrator's inline spans with dynamically-set attributes are the pattern actually in use; a decorator can't set per-call attributes cleanly, which is why it was declined. Keeping both is the worst of both. (Simplicity first; the span sites consolidate inside QueryEngine anyway.)
• Delete the dead response-generator module and the unused utilities module (zero callers; the deletion test passes trivially).
• Delete the three reshape-only judge passthrough wrappers in the eval metrics; keep the substantive metrics (answer correctness, context recall, factual match). Hoist the duplicated fence-stripping JSON parser into one helper owned by the canonical judge module.
• Add a document-chunk lookup method to the vector store interface and delete the one call site that reaches into its private collection and parses raw batch dicts.

Sequencing

Leaf-first, behavior-preserving steps, tests green after each:

1. Deletions + vector-store lookup method (no design dependencies).
2. Provider adapters with injected clients + usage-returning generation.
3. Telemetry utilities move to core; backend consumes reported usage (drops the reconstructed prompt).
4. Retriever Protocol + adapters; QueryEngine extracted; both production paths and EvalPipeline converge on it; eval config defaults derive from production config.
5. Backend split (ConversationStore, Evaluator) behind the thin facade.
6. API layer: typed returns, single DI seam, narrow facade methods, WebSocket streaming adapter, shared event vocabulary.

Each step lands as its own commit series with an ADR for the decision it implements; new module names enter the domain glossary as they appear.

Testing Decisions

• Good tests exercise external behaviour through a module's interface — the interface is the test surface. No test reaches past an interface into implementation details; if a test wants to, the module is the wrong shape.
• Primary seam: the backend facade's public interface — the existing integration tests (ephemeral vector store + in-memory database) continue to pass unchanged through every step; they are the behavior-preservation proof.
• New seams get interface-level tests:
  • Retriever adapters: contract tests run against every adapter (dense, hybrid, reranking-wrapped, multi-query-wrapped) asserting the shared Protocol semantics; prior art is the existing vector-store tests with explicit embeddings.
  • QueryEngine: tested with a fake Retriever and fake LLM handler — asserting prompt assembly, context format, telemetry assembly, and that sync and streaming paths produce identical instructions. Prior art: the judge-module tests that inject a mock LLM.
  • Provider adapters: each tested with a fake client factory (replacing the module-global monkeypatch); Anthropic and Ollama paths gain their first real coverage.
  • ConversationStore and Evaluator: direct tests against in-memory database sessions; the evaluation skip/dedup logic gets its first direct test (today it is only ever mocked).
  • WebSocket streaming adapter: driven with a scripted fake generator and captured envelopes — no socket. Prior art: the existing WS telemetry test's framing assertions.
  • API routes: with the single DI seam, route tests override one dependency; the conversation/document/upload routes gain their first HTTP-level tests where the hand-assembly bugs historically lived.
• Eval-production parity test: one test asserts the eval pipeline and production path produce the same prompt and context for the same inputs — the regression guard for the drift this spec eliminates.

Out of Scope

• New retrieval or generation features beyond promoting the four existing eval levers behind the new seam.
• Frontend changes beyond consuming unchanged API contracts (the API surface is preserved; only its internals deepen).
• Horizontal scaling, external vector databases, or replacing ChromaDB/SQLite.
• Authentication, authorization, or multi-tenancy.
• Re-running or extending the eval harness's datasets/metrics (only its pipeline plumbing converges on the shared engine).
• Ingestion-path redesign (document loading and chunking strategies stay as they are, minus config-default unification).

Further Notes

• This spec is the destination of the wayfinder map "Wayfinder map: RAG architecture deepening spec" (Wayfinder map: RAG architecture deepening spec #7); its eight grilling tickets are resolved by the sections above (each ticket's closing comment points at its section).
• Each sequencing step authors its ADR in the repo's ADR directory as it lands, and adds new module names (QueryEngine, Retriever, ConversationStore, Evaluator, ProviderAdapter) to the domain glossary — per the map's recording convention.
• Standing constraints from the repo's operating manual apply throughout: public portfolio repo (educational code style, no secrets, no AI co-author trailers), behavior-preserving refactors with the full test suite green after each step, ≤250 lines per module, typed boundaries, Google-style docstrings.
• Evidence for every claim in the Problem Statement (file/line references, duplication counts, drift examples) lives in the map's ticket bodies; it is deliberately excluded here so the spec doesn't rot as line numbers move.

[elkaix]

Sequencing step 1 merged: #17. Deletes generator.py, utils.py, and observability.traced_stage (verified zero-caller); rewires EvalRunner to call src.evaluation's judges directly, removing the three reshape-only wrapper functions in eval/metrics/generation.py; adds ChromaVectorStore.get_by_doc_id and closes the backend->_collection leak. 279/279 tests pass.

Merged past a required CI check via admin override — that failure is pre-existing and unrelated (squad_v2 dataset loading broken by an upstream datasets/huggingface_hub incompatibility, not by this PR's changes). Filed separately as #18.

Steps 2-6 (Retriever seam, QueryEngine, provider adapters, backend split, telemetry ownership, WebSocket adapter, typed API seam) remain — issue stays open.