[Enhancement] Hallucination detection via self-consistency sampling and claim-agreement checks

[zeemscript]

Summary

The assistant answers every question with the same confident tone whether it is repeating Surah al-Fatiha or inventing a hadith number on the spot. There is no mechanism to notice when the model is guessing. This issue adds inference-time hallucination detection via self-consistency: for high-stakes questions, sample the model more than once (and/or re-ask it to verify its own factual claims), measure agreement across samples, and treat disagreement as a hallucination signal — so a shaky answer can be softened, flagged for verification, or abstained on instead of served as fact. This is a general fabrication guard that complements the citation-vs-corpus check (#40) and the offline eval harness (#16): #40 catches wrong Quran text, #16 measures quality in the lab; this catches the model making things up at request time, before the user sees it.

Current state

• /chat calls chat.send_message exactly once (main.py, lines 143–151) at temperature: 0.7 and returns response.text verbatim (lines 176–179). A single stochastic sample is trusted completely — there is no second look, no agreement check, nothing between the model's first guess and the user.
• The only guardrails are Gemini's built-in safety settings (get_safety_settings, lines 91–109), which police harm categories, not factual accuracy.
• ISLAMIC_CONTEXT (lines 57–71) says "Acknowledge when a question is beyond your scope," but nothing measures whether the model actually knows the answer — so it rarely does.
• No structured claim extraction exists; the response is an opaque blob of text.

What to build

1. A self-consistency sampler — for questions routed as high-stakes (factual/ruling/citation-bearing; reuse intent signals from [Enhancement] Question-understanding pipeline: intent classification, clarifying questions, answer-length calibration, and suggested follow-ups #42 or a cheap heuristic, and skip sampling for greetings/small-talk to protect latency and cost), draw N samples (or one answer plus a verification pass) and compare their key factual claims — citations, numbers, names, rulings. Make N and the trigger policy configurable, and justify the latency/cost tradeoff in the PR.
2. A claim-agreement scorer — extract the load-bearing claims from each sample (a structured-output Gemini call is acceptable) and compute an agreement score: do the samples cite the same verses, the same collections, the same conclusion? Divergence on citations/numbers is a strong hallucination signal.
3. A response policy — below an agreement threshold, either abstain ("I'm not certain enough to answer this reliably; please consult a scholar / a cited source"), hedge with an explicit uncertainty note, or strip the unsupported claim. Never silently serve a low-agreement answer as confident fact. Emit a machine-readable signal (see #ai-19's confidence work — coordinate so they share one score, not two competing ones).
4. Latency control — sampling is expensive; run samples concurrently (note that [Bug] /chat blocks the event loop and turns every Gemini failure into a leaky 500 #7 already flags that the current send_message blocks the event loop — build on the async fix, don't fight it), cache verification results with [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27's cache, and keep single-sample fast paths for low-stakes chat.
5. Metadata — an optional consistency: {samples, agreement_score, action_taken} block on ChatResponse, additive and optional.
6. Tests — offline tests with a mocked client that returns deliberately divergent samples, asserting the scorer flags them and the policy abstains/hedges; and convergent samples pass through unchanged. No live API calls in CI.

Acceptance criteria

• High-stakes questions trigger multi-sample (or answer-plus-verification) generation; low-stakes chat keeps a single fast path — trigger policy is configurable and documented.
• Load-bearing claims (citations, numbers, names, conclusions) are extracted and compared across samples to produce an agreement score.
• Below the agreement threshold the service abstains, hedges, or strips the unsupported claim — never serves it as confident fact.
• Sampling runs concurrently and reuses caching so the added latency is bounded and measured (report the numbers in the PR).
• The confidence/agreement signal is shared with the confidence-scoring work (#ai-19), not duplicated.
• ChatResponse carries the optional consistency block; existing clients are unaffected.
• Tests with mocked divergent and convergent samples pass offline; CI stays green.

Pointers

• main.py — the single send_message call and generation config (lines 143–151), response.text return (lines 153–179), get_safety_settings (lines 91–109), ISLAMIC_CONTEXT (lines 57–71).
• Adjacent issues to compose with, not duplicate: [Enhancement] Citation verification: check quoted Quran text against a bundled corpus and flag unverifiable references #40 (citation-vs-corpus verification — a complementary deterministic check; where a corpus exists, prefer it over self-consistency), [Enhancement] Evaluation harness for Islamic answer quality #16 (offline eval — this is the online counterpart), [Bug] /chat blocks the event loop and turns every Gemini failure into a leaky 500 #7 (async fix — build on it), [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27 (cache), [Enhancement] Question-understanding pipeline: intent classification, clarifying questions, answer-length calibration, and suggested follow-ups #42 (intent signals for the trigger), and #ai-19 (confidence scoring — share one score).
• CI (.github/workflows/ci.yml) has no pytest step — add one and extend flake8 to new modules.
• PRs target the dev branch (see CONTRIBUTING.md).

Difficulty

High — self-consistency and claim-level agreement scoring are subtle to get right (paraphrase-tolerant claim matching, threshold tuning, avoiding both over- and under-abstention), and doing it without wrecking latency requires careful concurrency and caching on top of the async fix. This is real ML-systems work, not a wrapper.

---

🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.

💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0