Repository navigation
Commit 11b70c1
committed
fix(evaluation): do not score LCB samples 0.0 when the judge cannot run
evaluate_livecodebench_worker swallows every exception and returns
False, so import/setup failures get scored as wrong answers with subset
status "ok". That currently always happens in this env:
lcb_runner.prompts imports anthropic (not installed) and datasets>=3
cannot load the script-based code_generation_lite dataset. Observed:
335 samples "graded" 0/335 in 83s; the same outputs score 297/349
through lcb-service. A fake zero on 349/4388 samples is enough to drop
a passing DeepSeek-R1 run below the MLPerf accuracy threshold.
- re-raise ImportError/FileNotFoundError/RuntimeError from the worker
and the gather loop; the runner already marks the subset failed and
sets complete=false
- call load_lcb_benchmark() once before spawning workers so setup
errors fail fast with the real traceback
Sample-level grading failures (bad generated code, timeouts) still
score as incorrect. This does not fix the local path itself (deps are
still missing); it just stops it from reporting a wrong score.1 parent 111cf76 commit 11b70c1
1 file changed
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
567 | 567 | | |
568 | 568 | | |
569 | 569 | | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
570 | 574 | | |
571 | 575 | | |
572 | 576 | | |
573 | 577 | | |
574 | 578 | | |
575 | 579 | | |
| 580 | + | |
| 581 | + | |
576 | 582 | | |
577 | 583 | | |
578 | 584 | | |
| |||
655 | 661 | | |
656 | 662 | | |
657 | 663 | | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
658 | 668 | | |
659 | 669 | | |
660 | 670 | | |
| |||
679 | 689 | | |
680 | 690 | | |
681 | 691 | | |
| 692 | + | |
| 693 | + | |
682 | 694 | | |
683 | 695 | | |
684 | 696 | | |
| |||
0 commit comments