[Enhancement] Arabic morphology toolkit: diacritization, root lookup, and word-by-word Quranic analysis

[zeemscript]

Summary

A serious Islamic learning platform has to handle Arabic as a language, not just as UTF-8 bytes it passes through to the model. A learner studying an ayah wants to know the root of a word, its morphology (verb form, tense, person), and how to read it correctly with diacritics (tashkeel). The model guesses at all of this today and is frequently wrong about roots and grammar. This issue adds an Arabic morphology toolkit: reliable diacritization, root-word extraction, morphological analysis, and word-by-word breakdown of Quranic text — deterministic linguistic tooling the assistant can lean on instead of hallucinating grammar. This is explicitly the analysis layer that complements the general Arabic/multilingual I-O support in #14 (which covers detecting Arabic, replying in Arabic, and RTL handling) — different problem, no overlap.

Current state

• There is no Arabic processing anywhere. requirements.txt lists fastapi, uvicorn, python-dotenv, google-generativeai, pydantic, stellar-sdk — no NLP, no Arabic library, no Quranic corpus.
• Any Arabic question (roots, i'rab/grammar, tashkeel) is answered by chat.send_message (main.py, lines 143–151) from the model's memory, which is unreliable for morphology and root derivation.
• ISLAMIC_CONTEXT (lines 57–71) says nothing about Arabic linguistics, and the service has no verse text to analyze against.
• [Enhancement] First-class Arabic and multilingual support #14 ("First-class Arabic and multilingual support") owns language detection and Arabic-language responses; it does not provide a morphological analyzer, root lookup, or word-by-word Quran breakdown — that gap is this issue.

What to build

1. A morphology module (e.g. arabic.py) exposing: diacritization (add tashkeel to unvowelled text), root extraction (return the triliteral/quadriliteral root), and morphological features (part of speech, verb form/pattern, tense, person, number, gender) — built on an established resource (e.g. a Quranic Arabic morphology corpus, CAMeL Tools, Farasa, or a comparable library/dataset), not on the LLM. Document the source and its accuracy characteristics.
2. A word-by-word Quran analysis endpoint (e.g. POST /arabic/analyze or GET /quran/{surah}/{ayah}/words) returning, per word: the surface form, root, lemma, morphological gloss, and translation — the backbone of a "study this ayah" feature. Reuse the bundled Quran corpus from [Enhancement] Citation verification: check quoted Quran text against a bundled corpus and flag unverifiable references #40 for the source text rather than shipping a second copy.
3. A root-lookup capability — given a root, list its Quranic occurrences and derived forms, so a learner can explore a root across the Quran.
4. Chat integration — when a user asks "what's the root of…" or "parse this word," call the deterministic analyzer and have the model explain the verified analysis, rather than inventing morphology. Reuse [Enhancement] Question-understanding pipeline: intent classification, clarifying questions, answer-length calibration, and suggested follow-ups #42 intent detection; do not duplicate language detection already owned by [Enhancement] First-class Arabic and multilingual support #14.
5. Caching — analysis of a fixed corpus word is immutable; cache it (coordinate with [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27).
6. Tests — offline unit tests against known-answer cases (a set of words with textbook roots and morphology, a few fully-parsed ayahs), asserting the analyzer matches. No live API calls in CI.

Acceptance criteria

• The morphology module returns diacritization, root, lemma, and morphological features for Arabic input, backed by a documented linguistic resource (not the LLM), with known-answer tests passing.
• A word-by-word analysis endpoint breaks down an ayah into per-word root/lemma/morphology/translation, sourcing verse text from the [Enhancement] Citation verification: check quoted Quran text against a bundled corpus and flag unverifiable references #40 corpus (no duplicate Quran copy).
• Root lookup returns a root's Quranic occurrences and derived forms.
• Chat questions about roots/grammar/tashkeel are answered from the deterministic analyzer, with the model explaining (not inventing) the analysis.
• Analysis of immutable corpus words is cached (coordinated with [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27).
• The work does not re-implement language detection or Arabic-response handling owned by [Enhancement] First-class Arabic and multilingual support #14; it composes with it.
• Tests run offline; CI stays green.

Pointers

• main.py — router wiring (line 24), /chat handler (lines 118–184), ISLAMIC_CONTEXT (lines 57–71).
• stellar.py — typed APIRouter pattern (lines 25, 65–81, 106–118) as the endpoint template.
• requirements.txt — currently no NLP deps; justify any Arabic library added (size, license, install footprint on Render — see render.yaml).
• Adjacent issues to compose with, not overlap: [Enhancement] First-class Arabic and multilingual support #14 (Arabic/multilingual I-O — this is the analysis layer, cite the boundary explicitly), [Enhancement] Citation verification: check quoted Quran text against a bundled corpus and flag unverifiable references #40 (Quran corpus — reuse for source text), [Enhancement] Migrate off the deprecated google-generativeai SDK and the retired preview model #17 (tafsir — word analysis pairs naturally with tafsir display), [Enhancement] Semantic response cache with embedding-similarity matching for repeated questions #27 (cache), [Enhancement] Question-understanding pipeline: intent classification, clarifying questions, answer-length calibration, and suggested follow-ups #42 (intent).
• CI (.github/workflows/ci.yml) lints/compiles only main.py and has no pytest step — extend flake8 to new modules and add a pytest step.
• PRs target the dev branch (see CONTRIBUTING.md).

Difficulty

High — Arabic morphology is a hard NLP domain (roots, patterns, clitics, diacritization ambiguity), integrating a real linguistic resource has license/size/deployment implications on a small Render service, and the word-by-word feature must line up exactly with the Quran corpus. Done well it is a standout learning capability; done from the LLM it would just move the hallucination around.

---

🏆 GrantFox OSS — Official Campaign | FWC26. Apply for this issue through the GrantFox campaign page. The maintainer assigns one contributor before work starts; unassigned PRs may not be reviewed. PRs target the dev branch. Quality bar: CI must stay green.

💬 Questions or need help? Reach the maintainers and other contributors on the DeenBridge Telegram: https://t.me/+nst9lXNj1wc4ZDE0