Skip to content

Commit 11b70c1

Browse files
committed
fix(evaluation): do not score LCB samples 0.0 when the judge cannot run
evaluate_livecodebench_worker swallows every exception and returns False, so import/setup failures get scored as wrong answers with subset status "ok". That currently always happens in this env: lcb_runner.prompts imports anthropic (not installed) and datasets>=3 cannot load the script-based code_generation_lite dataset. Observed: 335 samples "graded" 0/335 in 83s; the same outputs score 297/349 through lcb-service. A fake zero on 349/4388 samples is enough to drop a passing DeepSeek-R1 run below the MLPerf accuracy threshold. - re-raise ImportError/FileNotFoundError/RuntimeError from the worker and the gather loop; the runner already marks the subset failed and sets complete=false - call load_lcb_benchmark() once before spawning workers so setup errors fail fast with the real traceback Sample-level grading failures (bad generated code, timeouts) still score as incorrect. This does not fix the local path itself (deps are still missing); it just stops it from reporting a wrong score.
1 parent 111cf76 commit 11b70c1

1 file changed

Lines changed: 12 additions & 0 deletions

File tree

  • src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/mlperf_eval

‎src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/mlperf_eval/eval_accuracy.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -567,12 +567,18 @@ def evaluate_livecodebench(code: Optional[str], question_id: str) -> bool:
567567
os.environ.pop('TQDM_DISABLE', None)
568568

569569

570+
# Infrastructure failures (broken judge) must not be scored as wrong answers.
571+
_LCB_INFRA_ERRORS = (ImportError, FileNotFoundError, RuntimeError)
572+
573+
570574
def evaluate_livecodebench_worker(args: Tuple[str, str]) -> Tuple[str, bool]:
571575
"""Worker function for parallel LiveCodeBench evaluation."""
572576
code, question_id = args
573577

574578
try:
575579
return question_id, evaluate_livecodebench(code, question_id)
580+
except _LCB_INFRA_ERRORS:
581+
raise
576582
except Exception:
577583
return question_id, False
578584

@@ -655,6 +661,10 @@ def process_livecodebench_parallel(
655661
if not work_items:
656662
return 0, 0
657663

664+
# Pre-flight: a broken judge (missing dependency/dataset) must raise here
665+
# with a real traceback, not be scored as 0.0 by every worker.
666+
load_lcb_benchmark()
667+
658668
# Process in parallel
659669
max_workers = min(multiprocessing.cpu_count(), len(work_items))
660670
logger.info(
@@ -679,6 +689,8 @@ def process_livecodebench_parallel(
679689
total_evaluated += 1
680690
if is_correct:
681691
correct_count += 1
692+
except _LCB_INFRA_ERRORS:
693+
raise
682694
except Exception as e:
683695
logger.error(f"Error evaluating row {idx}: {e}")
684696
df.at[idx, 'prompt_accuracy'] = 0.0

0 commit comments

Comments
 (0)