A compact, recruiter‑friendly codebase for evaluating LLMs on specialized Talmudic multiple‑choice questions.
This repository contains the code and methodology for building and evaluating a Talmudic QA benchmark. All data assets are intentionally excluded.
Large language models excel on general benchmarks but struggle in specialized religious and cultural domains. This project evaluates LLMs on authentic Talmudic questions and highlights a large accuracy‑confidence gap in Hebrew/Aramaic legal reasoning tasks.
The Talmud’s language, interpretive style, and historical depth present unique computational challenges. This work contributes a clean pipeline for dataset generation, difficulty scoring, and evaluation, designed to scale across tractates.
- Built an end‑to‑end pipeline for generating, normalizing, and evaluating Talmudic multiple‑choice questions.
- Designed a difficulty heuristic combining question length, option length, lexical rarity, option similarity, and negation signals.
- Implemented a validation workflow to reduce label noise and prevent train/eval leakage.
- Evaluated models using SimpleQA‑style metrics to quantify overall correctness and confidence calibration.
check.py
evaluate.py
split.py
stats.py
README.md
- Data files are intentionally excluded to avoid distributing restricted or licensed content.
- This repo is meant to showcase methodology and code quality rather than data assets.