Skip to content

Group binary agent outcomes for standard metrics - #162

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-agent-binary-answer-groups
Open

Group binary agent outcomes for standard metrics#162
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-agent-binary-answer-groups

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • group agent-summary answers by their binary correctness outcome instead of assigning every rollout a unique answer group
  • deduplicate the per-instance correctness mapping for the two binary groups
  • preserve existing outcome counts and system-error handling

get_summary_error_aware() only has a binary result for each rollout: correct or incorrect. _compute_metrics_plus_outcome_aggregations() currently assigns answer_group_id with groupby(...).cumcount(), so every attempt is treated as a different answer even when multiple attempts have the same outcome.

That does not affect the repository's minimal accuracy-only backend, but it breaks the answer-group contract for richer standard metric backends that use group identity for consensus-style metrics. Two correct attempts should belong to the same answer group, as should two incorrect attempts.

The fix uses 1 for correct and 0 for incorrect, and keeps one correctness row per (instance, answer_group_id) so repeated outcomes do not duplicate the correctness mapping.

Regression coverage verifies that repeated correct attempts share one group while incorrect attempts use the other group.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant