Skip to content

Commit 373ae41

Browse files
committed
Port the ReWOO loaders instead of writing three from scratch
StrategyQA, SportsUnderstanding and PhysicsQuestions are adapted from billxbf/ReWOO at 9cd02830, from utils/DataLoader.py and utils/Evaluator.py. That repository is where the datasets come from rather than BigBench itself, for two reasons. We are comparing against its numbers, and going back to the origin would introduce a version difference that is invisible in the result while giving the comparison an unexplainable cause. And strategy_qa is no longer in BigBench main at all, so its vendored CSV is the copy that is actually obtainable. The sample is reproducible. ReWOO draws with pandas sample(random_state=seed) and run_eval.py defaults seed to 2024, which overrides the loader's own 2023. RandomState(2024).permutation(n)[:k] matches pandas bit for bit, verified, so seed 2024 is ReWOO's exact set of questions without depending on pandas. The first ten indices are pinned in a test. Three divergences are recorded in the module because any of them would otherwise turn into a comparison that looks valid and is not. ReWOO gives all five tools including Google, Wikipedia and WolframAlpha, three of which need paid services we cannot reproduce, so these loaders give none by default and say so in code. ReWOO also reports an LLM-judge accuracy whose judge is unpinned -- langchain's default, which is now a retired model -- so we report em and f1 and leave that column out. And its Evaluator inserts an all-NaN row on exception, which nanmean then skips, so crashed tasks are dropped rather than counted wrong; we count them. Porting turned up two defects in my own work. normalize_answer has a required order, lower then strip articles, and I had it backwards, so a capitalised "The" survived and the same trajectories would have scored differently while both versions looked like they were computing em. The parity test copies ReWOO's two scoring functions verbatim and compares, which is what caught it; "I copied it" is not itself a guarantee. And the physics tolerance has to derive from the gold set: the gold's own variants ['7187 m', '7 km'] differ by 2.67 percent, so a fixed one percent tolerance rejects an answer that the gold itself lists as correct. Also implements the http download kind, which fetch() declared in its own table but never handled -- so a loader could declare a download nobody could run. The shared tarball is cached by URL rather than per loader, since three loaders share one source and a fetch that takes three times as long is a fetch people skip. Python tests: 199 -> 215.
1 parent 04b88d1 commit 373ae41

6 files changed

Lines changed: 1027 additions & 0 deletions

File tree

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
"""三个 BigBench 任务(StrategyQA / SportsUnderstanding / PhysicsQuestions)。
2+
3+
**改编自 ReWOO 的 `utils/DataLoader.py` 与 `utils/Evaluator.py`** ——
4+
出处、抽样方式、以及三处必须记住的偏离,全在 `bigbench.py` 的头部。
5+
"""
6+
7+
from experiments.benchmark.bigbench.bigbench import * # noqa: F401,F403

0 commit comments

Comments
 (0)