[Enhancement] Prompt template registry with versioning and A/B experimentation

[zeemscript]

Summary

The single most important lever on answer quality — the system prompt — is a hardcoded string literal in the middle of main.py, with no version, no history, and no way to change it without a code deploy or to test whether a change actually helped. Improving the assistant means iterating on this prompt constantly, and right now every iteration is a blind, unmeasured edit. This issue turns the prompt into managed infrastructure: a versioned template registry with named, parameterized templates, and an A/B experimentation harness that can run two prompt versions against real traffic and measure which produces better answers. This is the substrate that makes all the other answer-quality work measurable.

Current state

• The prompt is ISLAMIC_CONTEXT, a triple-quoted constant at main.py lines 57–71. There is exactly one, it has no version tag, and it is even applied inconsistently — the /chat handler only injects it when a context is supplied, and does so via a buggy f-string that interpolates a tuple (f"...{ISLAMIC_CONTEXT, request.prompt}", line 139). ([Bug] Islamic system prompt is skipped for plain requests and easily overridden by prompt injection #5 is fixing the injection/skip bug by moving it to system_instruction; this issue builds the management layer around whatever prompt [Bug] Islamic system prompt is skipped for plain requests and easily overridden by prompt injection #5 lands.)
• There is no template engine, no registry, no variables — a prompt that needs the user's madhhab ([Enhancement] Madhhab-aware fiqh answers: present the positions of the major schools instead of one flattened ruling #39), language ([Enhancement] First-class Arabic and multilingual support #14), or knowledge level ([Enhancement] Per-user long-term memory and conversation summarization for personalized answers #41) has nowhere to compose those in.
• There is no experimentation mechanism: no way to route a fraction of requests to a variant, no way to record which variant produced which answer, no metric to compare them.

What to build

1. A prompt registry — extract prompts out of main.py into a managed store (versioned files or a small module, e.g. prompts/ with a loader) where each template has a stable name, a semantic version, a changelog entry, and typed variables (madhhab, language, knowledge_level, intent, retrieved_context). Compose with [Bug] Islamic system prompt is skipped for plain requests and easily overridden by prompt injection #5's system_instruction change rather than reverting it. Rendering is deterministic and unit-tested.
2. Template parameterization — a small, safe rendering layer (no arbitrary code execution, injection-safe) so other issues fill variables instead of string-concatenating into the handler. Publish the variable contract so [Enhancement] First-class Arabic and multilingual support #14/[Enhancement] Madhhab-aware fiqh answers: present the positions of the major schools instead of one flattened ruling #39/[Enhancement] Per-user long-term memory and conversation summarization for personalized answers #41 target it.
3. An A/B experiment harness — assign each request (by chat_id/user_id hash, so a session is sticky) to a prompt variant per an experiment config, record the assignment with the response, and expose the variant in optional response metadata for offline analysis. Support a control + one-or-more variants and a kill switch.
4. Measurement hook — join experiment assignments to a quality signal (feedback ratings from [Enhancement] Answer feedback capture and a quality-improvement loop feeding the evaluation dataset #43, eval-harness scores from [Enhancement] Evaluation harness for Islamic answer quality #16, or confidence from #ai-19) so a variant's win/loss is computable. Do not build a new metrics store — emit to the ones those issues define.
5. Config-driven — experiment definitions and the active prompt version live in configuration (coordinate with the pydantic-settings work [Enhancement] Centralize configuration with pydantic-settings (model, generation params, CORS origins) #10) so changing the live prompt or starting an experiment needs no code change.
6. Tests — offline tests for deterministic rendering, variable substitution, sticky variant assignment (same session → same variant), the kill switch, and that metadata records the variant. No live API calls in CI.

Acceptance criteria

• Prompts live in a versioned registry outside the handler, each with a name, version, changelog, and typed variables; rendering is deterministic and unit-tested.
• The rendering layer is injection-safe and publishes a variable contract that [Enhancement] First-class Arabic and multilingual support #14/[Enhancement] Madhhab-aware fiqh answers: present the positions of the major schools instead of one flattened ruling #39/[Enhancement] Per-user long-term memory and conversation summarization for personalized answers #41 can target instead of concatenating strings.
• The A/B harness assigns requests to variants stickily by session, records the assignment with the response, and has a kill switch.
• Variant performance is computable by joining assignments to an existing quality signal ([Enhancement] Answer feedback capture and a quality-improvement loop feeding the evaluation dataset #43/[Enhancement] Evaluation harness for Islamic answer quality #16/#ai-19) — no new metrics store.
• The active prompt version and experiment config are configuration, not code (coordinated with [Enhancement] Centralize configuration with pydantic-settings (model, generation params, CORS origins) #10).
• The layer composes with [Bug] Islamic system prompt is skipped for plain requests and easily overridden by prompt injection #5's system_instruction change (does not revert the skip/injection fix).
• Tests cover rendering, substitution, sticky assignment, kill switch, and metadata offline; CI stays green.

Pointers

• main.py — ISLAMIC_CONTEXT (lines 57–71), the buggy tuple interpolation and conditional injection (lines 136–139), the /chat handler (lines 118–184).
• Adjacent issues to compose with, not fight: [Bug] Islamic system prompt is skipped for plain requests and easily overridden by prompt injection #5 (moves prompt to system_instruction + injection hardening — build on it), [Enhancement] Centralize configuration with pydantic-settings (model, generation params, CORS origins) #10 (pydantic-settings config — put experiment/version config there), [Enhancement] Evaluation harness for Islamic answer quality #16 (eval scores), [Enhancement] Answer feedback capture and a quality-improvement loop feeding the evaluation dataset #43 (feedback ratings), #ai-19 (confidence) for the measurement join, and [Enhancement] First-class Arabic and multilingual support #14/[Enhancement] Madhhab-aware fiqh answers: present the positions of the major schools instead of one flattened ruling #39/[Enhancement] Per-user long-term memory and conversation summarization for personalized answers #41 as consumers of the variable contract.
• CI (.github/workflows/ci.yml) lints/compiles only main.py and has no pytest step — add one and extend flake8 to the new prompts/ module.
• PRs target the dev branch (see CONTRIBUTING.md).

Difficulty

Medium — the template registry and sticky variant assignment are standard, well-scoped engineering, but doing it right means designing a clean variable contract that a half-dozen other issues will build on, composing carefully with the in-flight system_instruction change, and wiring measurement to existing signals rather than sprawling a new metrics system.

---

🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.

💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0