Skip to content

Handle empty EVMbench summaries - #161

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-evmbench-empty-summary
Open

Handle empty EVMbench summaries#161
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-evmbench-empty-summary

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • make EVMbench.get_full_summary() handle an empty result set
  • use the eval's own run_group_id rather than indexing the first result task
  • preserve existing score, health, and per-audit aggregation behavior

Nanoeval's base Eval.self_test() explicitly calls get_full_summary([]). EVMbench currently builds a task list and then reads tasks[0] solely to populate the run-group ID, which raises IndexError when no results are available.

The eval already owns the canonical run_group_id, so the summary can use that value directly without depending on a completed task.

Regression coverage verifies an empty summary reports zero samples and scores while retaining the eval run-group ID.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant