Skip to content

Propagate external cancellation from EVMbench agent runs - #167

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-evmbench-cancellation-propagation
Open

Propagate external cancellation from EVMbench agent runs#167
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-evmbench-cancellation-propagation

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • preserve local agent-timeout handling
  • allow external asyncio.CancelledError to propagate through EVMbench agent execution
  • avoid suppressing cancellation with a return from finally

execute_agent_in_computer() currently catches asyncio.CancelledError, even though asyncio.timeout() also uses cancellation internally. More importantly, run_agent_in_computer() returns AgentOutput from its finally block. A return in finally suppresses an in-flight external cancellation, so shutting down or cancelling the evaluation can be converted into an apparently normal agent run and grading can continue.

The fix handles the timeout as asyncio.TimeoutError outside the timeout context, leaves external cancellation uncaught, and returns AgentOutput only after normal completion of the wrapper.

Regression coverage verifies that an ordinary agent timeout remains locally handled while externally cancelling the running task raises CancelledError to the caller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant