Summary
handle_system_errors_and_compute_metrics() documents top-level metrics as invalid when excluded system errors leave too few rollouts for an instance, but its is_valid check compares only the set of question_id values before and after excluding errors.
If an instance has multiple attempts and only one attempt fails, the question ID still exists in both sets and is_valid remains True even though a rollout was removed.
Reproducer
For one question with attempts 0 and 1:
- attempt 0 returns a normal result;
- attempt 1 returns
RolloutSystemError.
The full result set has two rollouts for the instance, while the top-level metrics see only one. The current check still reports is_valid=True because both sets contain the same single question_id.
Impact
Consumers can treat metrics computed from an incomplete rollout set as valid, including consensus/pass@k settings where the missing attempt is precisely the condition the function's validity contract is intended to surface.
Proposed resolution
Compare per-instance rollout counts before and after excluding system errors, rather than only distinct question IDs. is_valid should be true only when every instance retains the same number of rollouts.
Add a regression covering a partial per-instance system error and a fully successful multi-attempt instance.
Summary
handle_system_errors_and_compute_metrics()documents top-level metrics as invalid when excluded system errors leave too few rollouts for an instance, but itsis_validcheck compares only the set ofquestion_idvalues before and after excluding errors.If an instance has multiple attempts and only one attempt fails, the question ID still exists in both sets and
is_validremainsTrueeven though a rollout was removed.Reproducer
For one question with attempts 0 and 1:
RolloutSystemError.The full result set has two rollouts for the instance, while the top-level metrics see only one. The current check still reports
is_valid=Truebecause both sets contain the same singlequestion_id.Impact
Consumers can treat metrics computed from an incomplete rollout set as valid, including consensus/pass@k settings where the missing attempt is precisely the condition the function's validity contract is intended to surface.
Proposed resolution
Compare per-instance rollout counts before and after excluding system errors, rather than only distinct question IDs.
is_validshould be true only when every instance retains the same number of rollouts.Add a regression covering a partial per-instance system error and a fully successful multi-attempt instance.