feat(rag): make chunk_size tunable in tests and tooling (closes #932)

[arcgod-design]

What

Makes the RAG text splitter's chunk_size parameter fully tunable — closing issue #932's "Add chunking tuning to tests and tooling".

Previously chunk_size was hardcoded to 600 in backend/services/rag_service.py while only rag_chunk_overlap was exposed via the settings panel. This PR exposes both, validates both, and surfaces both in the UI + a new offline CLI tuner.

Why

The local sentence-transformers embedder (all-MiniLM-L6-v2, 384 dims) and the SQLite-ChromaDB retrieval pipeline prefer different chunk sizes depending on document type:

• Short Q&A / dense technical docs → ~200 chars (fine-grained retrieval, more chunks, more accurate pinpoint citations)
• Long-form prose / chapter-sized content → ~2000 chars (more context per chunk, fewer chunks, faster top-k retrieval at slightly lower pinpoint precision)

Without a tunable chunk_size users were stuck with the 600 default. The pre-existing rag_chunk_overlap slider was there but sized for one chunk size — without a paired chunk size control the overlap's relative percentage drifted wildly. This PR binds the two together with a cross-validation check.

What changed

Backend

• backend/models/schemas.py — new rag_chunk_size: int = 600 field on AppSettings. Default matches the previous hardcoded behaviour. Docstring documents bounds + best-effort splitter semantics.

• backend/routes/settings.py — two new validation steps in the PUT /api/settings/ body validator:

  1. rag_chunk_size must lie in [200, 2000] inclusive (422 outside)
  2. rag_chunk_overlap must be strictly less than rag_chunk_size (422 if equal or greater — a zero-progress splitter window is unsplittable). Cross-checked after the standalone 0..200 bounds.

• backend/services/rag_service.py — index_document() now reads rag_chunk_size + rag_chunk_overlap from db_service.get_settings() at call-time. Both are defensively clamped:

  • chunk_size out of [200, 2000] → reset to default 600.
  • overlap out of [0, 200] → reset to default 50.
  • overlap >= chunk_size (impossible via the API since Add chunking tuning to tests and tooling #932-merge, but theoretically present in a stale DB row) → clamp to chunk_size // 10.

  Logger line updated to print both values.

Frontend

• frontend/src/components/SettingsPanel.jsx — a new rag_chunk_size field in the form initial state (?? 600) and a new slider (200..2000, step 100) above the existing chunk-overlap slider so users see a coherent pair.

Tooling

• scripts/chunk_tuner.py — 130-line zero-dependency CLI that empirically tests the splitter at any (chunk_size, chunk_overlap) combination against a markdown / text file (stdin or --). Used for offline experimentation without spinning up FastAPI / Ollama / ChromaDB.
  • Modes: --json (default; structured report), --plain (human-readable)
  • Emits chunk_count, first_chunk_preview, last_chunk_preview, min_chunk_chars, max_chunk_chars, mean_chunk_chars
  • Same 200..2000/0..200/overlap < chunk_size bounds as the API endpoint
  • Exit 1 on 0-chunk output (input too short / all whitespace)

Tests

• backend/tests/test_chunking_tuning.py — 23 pytest cases (all pass):
  • TestSchema (3): rag_chunk_size defaults to 600 through model_dump(); defaults match legacy chunk_overlap=50.
  • TestSettingsValidation (10): chunk_size 199/2001 → 422 with correct loc + msg; boundaries 200/2000 → 200; overlap == chunk_size → 422 with "strictly less than chunk size" message; overlap < 0 rejected by the standalone bounds; default payload → 200.
  • TestServiceUsesChunkSize (4): rag_service reads chunk_size from get_settings(); empty settings dict → falls back to 600/50; cross-validation clamps badly-stored values; out-of-range chunk_size resets to default.
  • TestSplitterBehaviour (6): smaller chunk_size → more chunks; zero overlap → no shared tail prefix; non-zero overlap documents that separators may absorb; empty content → 0 chunks; single-short-paragraph → 1 chunk; \n\n-separated paragraphs align with RecursiveCharacterTextSplitter's first-tier separator.
  • TestChunkTunerTool (2): the script runs as both an importable module (module.tune(content, chunk_size, chunk_overlap)) and a subprocess CLI. Tests skip if scripts/chunk_tuner.py is not present (e.g. a shallow clone).

Docs

• docs/chunking-tuning.md — operator-facing reference: the settings table with bounds, the API request shape, the 422-body contract, the CLI tooling examples (with --plain for human inspection and a real-world diff workflow against two chunk_size settings), and a test command summary.

Verification

$ cd backend && pytest tests/test_chunking_tuning.py -v
======================= 23 passed in 27.71s ========================

$ cd backend && pytest tests/test_settings_timeout.py -v
============================== 3 passed in 2.11s ==============  (existing suite still green)

$ ruff check backend/{routes/settings,models/schemas,services/rag_service}.py \
           backend/tests/test_chunking_tuning.py scripts/chunk_tuner.py
All checks passed!

$ ruff format --check <same files>
5 files already formatted

$ python scripts/chunk_tuner.py --chunk-size 300 --chunk-overlap 30 --plain docs/csrf-protection.md
chunk_count:            23
min_chunk_chars:       13
max_chunk_chars:       298
mean_chunk_chars:      191.9
(first_chunk_preview + last_chunk_preview omitted for brevity)

The existing test_settings_timeout.py (3 cases) continues to pass unmodified — the new validation steps are additive and don't conflict with the existing rag_top_k / rag_chunk_overlap / temperature bounds.

Acceptance criteria

Issue #932 acceptance criteria:

• Implement the change in the tests and tooling.
• Add or update tests, docs, or validation for the touched path.

Closes #932.

[vercel]

@arcgod-design is attempting to deploy a commit to the Darshan's projects Team on Vercel.

A member of the Team first needs to authorize it.