Repository navigation
Commit 373ae41
committed
Port the ReWOO loaders instead of writing three from scratch
StrategyQA, SportsUnderstanding and PhysicsQuestions are adapted from
billxbf/ReWOO at 9cd02830, from utils/DataLoader.py and utils/Evaluator.py.
That repository is where the datasets come from rather than BigBench itself,
for two reasons. We are comparing against its numbers, and going back to the
origin would introduce a version difference that is invisible in the result
while giving the comparison an unexplainable cause. And strategy_qa is no
longer in BigBench main at all, so its vendored CSV is the copy that is
actually obtainable.
The sample is reproducible. ReWOO draws with pandas sample(random_state=seed)
and run_eval.py defaults seed to 2024, which overrides the loader's own 2023.
RandomState(2024).permutation(n)[:k] matches pandas bit for bit, verified,
so seed 2024 is ReWOO's exact set of questions without depending on pandas.
The first ten indices are pinned in a test.
Three divergences are recorded in the module because any of them would
otherwise turn into a comparison that looks valid and is not. ReWOO gives all
five tools including Google, Wikipedia and WolframAlpha, three of which need
paid services we cannot reproduce, so these loaders give none by default and
say so in code. ReWOO also reports an LLM-judge accuracy whose judge is
unpinned -- langchain's default, which is now a retired model -- so we report
em and f1 and leave that column out. And its Evaluator inserts an all-NaN
row on exception, which nanmean then skips, so crashed tasks are dropped
rather than counted wrong; we count them.
Porting turned up two defects in my own work. normalize_answer has a required
order, lower then strip articles, and I had it backwards, so a capitalised
"The" survived and the same trajectories would have scored differently while
both versions looked like they were computing em. The parity test copies
ReWOO's two scoring functions verbatim and compares, which is what caught it;
"I copied it" is not itself a guarantee. And the physics tolerance has to
derive from the gold set: the gold's own variants ['7187 m', '7 km'] differ by
2.67 percent, so a fixed one percent tolerance rejects an answer that the
gold itself lists as correct.
Also implements the http download kind, which fetch() declared in its own
table but never handled -- so a loader could declare a download nobody could
run. The shared tarball is cached by URL rather than per loader, since three
loaders share one source and a fetch that takes three times as long is a
fetch people skip.
Python tests: 199 -> 215.1 parent 04b88d1 commit 373ae41
6 files changed
Lines changed: 1027 additions & 0 deletions
File tree
- experiments
- benchmark/bigbench
- core
- dataset
- scripts
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
0 commit comments