Summary
SWELancerTask._grade_swe_ic() records -1.0 when it cannot parse /app/tests/logs/$ISSUE_ID/pytest_exit_code. The retry loop continues, but if every test attempt is missing that exit code, test_results is still non-empty and the current if not test_results guard does not fire.
The task is then returned with score=0.0 and the note Tests ran smoothly, even though no test run produced a valid pytest status.
Impact
A grading/infrastructure failure can be counted as a model failure, lowering SWE-Lancer scores and contaminating evaluation results instead of surfacing as RolloutSystemError.
Proposed resolution
Treat a run set as valid only if at least one attempt produced a real pytest result (0.0 failure or 1.0 success). If all attempts are the -1.0 missing-status sentinel, raise RolloutSystemError after retries are exhausted.
Add focused coverage distinguishing all-invalid attempts from a valid failing test run.
Summary
SWELancerTask._grade_swe_ic()records-1.0when it cannot parse/app/tests/logs/$ISSUE_ID/pytest_exit_code. The retry loop continues, but if every test attempt is missing that exit code,test_resultsis still non-empty and the currentif not test_resultsguard does not fire.The task is then returned with
score=0.0and the noteTests ran smoothly, even though no test run produced a valid pytest status.Impact
A grading/infrastructure failure can be counted as a model failure, lowering SWE-Lancer scores and contaminating evaluation results instead of surfacing as
RolloutSystemError.Proposed resolution
Treat a run set as valid only if at least one attempt produced a real pytest result (
0.0failure or1.0success). If all attempts are the-1.0missing-status sentinel, raiseRolloutSystemErrorafter retries are exhausted.Add focused coverage distinguishing all-invalid attempts from a valid failing test run.