feat: answer feedback capture and a quality-improvement loop (#43)

.env.example

@@ -33,3 +33,9 @@ GEMINI_API_KEY=your_api_key_here
 # ZAKAT_NISAB_USD=6000               # fallback when no gold price is reachable
 # NISAB_CACHE_TTL_SECONDS=21600      # how long a fetched gold price is reused
 # GOLD_PRICE_TIMEOUT=8
+
+# Answer feedback loop (optional)
+# ADMIN_TOKEN=generate_a_long_random_value   # enables /feedback/stats and /feedback/records
+# FEEDBACK_DB_PATH=feedback.db               # SQLite path when REDIS_URL is unset
+# FEEDBACK_RATE_LIMIT_MAX=20
+# FEEDBACK_RATE_LIMIT_WINDOW=60

.github/workflows/ci.yml

@@ -28,10 +28,10 @@ jobs:
           pip install pytest flake8 pytest-asyncio
 
       - name: Run linting
-        run: flake8 main.py memory stellar.py nisab.py safety telemetry.py tests/redteam study.py fiqh.py hadith.py confidence.py review.py review_store.py tafsir.py semantic_cache.py tests/test_confidence.py tests/test_review_queue.py tests/test_tafsir.py tests/test_zakat.py tests/test_telemetry.py tests/test_multilingual.py tests/test_memory_profile.py tests/test_memory_extraction.py tests/test_memory_integration.py scripts/build_hadith_data.py scripts/build_surah_index.py --max-line-length=120 --ignore=E501,W503
+        run: flake8 main.py memory stellar.py nisab.py safety telemetry.py tests/redteam study.py fiqh.py hadith.py confidence.py review.py review_store.py tafsir.py semantic_cache.py feedback.py scripts/export_eval_candidates.py tests/test_feedback.py tests/test_confidence.py tests/test_review_queue.py tests/test_tafsir.py tests/test_zakat.py tests/test_telemetry.py tests/test_multilingual.py tests/test_memory_profile.py tests/test_memory_extraction.py tests/test_memory_integration.py scripts/build_hadith_data.py scripts/build_surah_index.py --max-line-length=120 --ignore=E501,W503
 
       - name: Check syntax
-        run: python -m compileall -q main.py memory stellar.py nisab.py safety telemetry.py tests/redteam study.py fiqh.py hadith.py confidence.py review.py review_store.py tafsir.py semantic_cache.py scripts/build_hadith_data.py scripts/build_surah_index.py
+        run: python -m compileall -q main.py memory stellar.py nisab.py safety telemetry.py tests/redteam study.py fiqh.py hadith.py confidence.py review.py review_store.py tafsir.py semantic_cache.py feedback.py scripts/export_eval_candidates.py scripts/build_hadith_data.py scripts/build_surah_index.py
 
       - name: Run offline safety and red-team tests
         run: pytest -q tests/redteam
@@ -60,6 +60,9 @@ jobs:
       - name: Run zakat and nisab tests
         run: pytest -q tests/test_zakat.py
 
+      - name: Run feedback loop tests
+        run: pytest -q tests/test_feedback.py
+
       - name: Run LLM telemetry tests
         run: pytest -q tests/test_telemetry.py
 

.gitignore

@@ -38,3 +38,10 @@ ENV/
 *.log 
 # Scholar-review export (contains user questions)
 data/review/
+
+# Local feedback database (SQLite fallback store)
+feedback.db
+feedback.db-*
+
+# Committed virtual environments are machine-specific — never commit them
+venv_linux/

README.md

@@ -58,6 +58,9 @@ The platform is composed of three services:
 | `GET` | `/review/reviewed` | Answers that already carry a verdict (reviewer token) |
 | `GET` | `/review/{id}` | A single review item (reviewer token) |
 | `POST` | `/review/{id}/verdict` | Record approve / correct / reject (reviewer token) |
+| `POST` | `/feedback` | Rate a specific answer and flag failure categories |
+| `GET` | `/feedback/stats` | Aggregate answer-quality metrics (admin token) |
+| `GET` | `/feedback/records` | Browse flagged records, filterable (admin token) |
 
 ## 🚀 Getting Started
 
@@ -146,6 +149,10 @@ services:
 | `ZAKAT_NISAB_USD` | Fallback nisab when no gold price can be fetched | `6000` |
 | `NISAB_CACHE_TTL_SECONDS` | How long a fetched gold price is reused | `21600` (6h) |
 | `GOLD_PRICE_TIMEOUT` | Gold price request timeout in seconds | `8` |
+| `ADMIN_TOKEN` | Enables the feedback admin endpoints; required as `X-Admin-Token` | — (endpoints disabled) |
+| `FEEDBACK_DB_PATH` | SQLite path for the feedback store (when Redis is not used) | `feedback.db` |
+| `FEEDBACK_RATE_LIMIT_MAX` | Max feedback submissions per IP per window | `20` |
+| `FEEDBACK_RATE_LIMIT_WINDOW` | Feedback rate-limit window in seconds | `60` |
 | `QURAN_API_BASE` | Base URL for tafsir/ayah retrieval | `https://api.quran.com/api/v4` |
 | `QURAN_API_TIMEOUT` | Tafsir request timeout in seconds | `15` |
 | `TAFSIR_MAX_AYAT` | Maximum ayat per `/tafsir` request | `10` |
@@ -379,6 +386,60 @@ validation like any other malformed input, so one can never reach Horizon. If a
 message looks like it contains a secret key, the assistant refuses to use it and
 warns the user to treat it as compromised — without repeating it back.
 
+### Answer feedback & the quality loop
+
+Every chat answer carries a stable `message_id`, so the frontend can rate a
+specific turn. `POST /feedback` attaches an up/down rating and optional failure
+categories to that answer:
+
+```bash
+curl -sX POST http://localhost:8000/feedback \
+  -H 'Content-Type: application/json' \
+  -d '{"chat_id": "…", "message_id": "…", "rating": "down",
+       "categories": ["wrong_or_missing_citation"], "comment": "Ayah number is off"}'
+```
+
+- **Snapshot resolution.** The prompt and the *displayed* answer (after any
+  safety, hadith, or confidence shaping) are resolved server-side from what the
+  user actually saw — the client never has to be trusted for them. On a
+  free-tier restart the snapshot is gone; the client then supplies `prompt` and
+  `answer`, and a request missing both is a `422`.
+- **Validation.** `rating` must be `up`/`down`, categories are checked against a
+  fixed taxonomy, and `comment` is length-capped — bad input is a `422`.
+- **Idempotent.** Resubmitting for the same `(chat_id, message_id)` overwrites,
+  so a user changing their mind never double-counts.
+- **Rate-limited** per IP (in-process sliding window), and durably stored in
+  SQLite locally or Redis when `REDIS_URL` is set — the same store direction the
+  session and scholar-review work use, not a parallel one.
+
+Maintainers read the aggregate signal through two **admin** endpoints, gated on
+`X-Admin-Token` and disabled entirely until `ADMIN_TOKEN` is set:
+
+```bash
+curl -s http://localhost:8000/feedback/stats   -H "X-Admin-Token: $ADMIN_TOKEN"
+curl -s "http://localhost:8000/feedback/records?rating=down&category=too_vague" \
+     -H "X-Admin-Token: $ADMIN_TOKEN"
+```
+
+#### Eval-candidate export
+
+Down-rated answers become candidates for the evaluation dataset (issue #16
+format). Each carries `needs_review: true` and an `answer_draft` for the
+reviewer to judge — the script **never** fabricates an expected answer for
+religious content:
+
+```bash
+# Reads whichever store the service is configured to use (Redis or SQLite):
+python scripts/export_eval_candidates.py --output candidates.jsonl
+REDIS_URL=redis://localhost:6379 python scripts/export_eval_candidates.py --output candidates.jsonl
+
+# …or force a specific SQLite file:
+python scripts/export_eval_candidates.py --db feedback.db --output candidates.jsonl
+```
+
+Near-duplicate prompts are deduplicated; approved candidates feed the harness
+and, via #56, the semantic cache.
+
 ### Content-safety testing
 
 The versioned policy lives in [`safety/policy.yaml`](safety/policy.yaml), with