Skip to content

Treat invalid SWE-Lancer pytest runs as rollout errors - #153

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-swelancer-invalid-pytest-results
Open

Treat invalid SWE-Lancer pytest runs as rollout errors#153
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-swelancer-invalid-pytest-results

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • require at least one real pytest pass/fail result before scoring an IC SWE-Lancer task
  • treat a retry set containing only the -1.0 missing-status sentinel as a grading/infrastructure failure
  • preserve ordinary failing test runs (0.0) as valid model failures

Previously, repeated missing pytest_exit_code values left test_results non-empty, so the evaluator returned score=0.0 and reported that tests ran smoothly. The validity check now accepts only run sets containing at least one real pass/fail status.

The code change is intentionally limited to the existing post-retry guard.

Fixes #151.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SWE-Lancer grades all-missing pytest exit codes as a model failure instead of rollout error

1 participant