Skip to content

Preserve nanoeval max concurrency when resuming run sets - #163

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-nanoeval-preserve-max-concurrency
Open

Preserve nanoeval max concurrency when resuming run sets#163
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix-nanoeval-preserve-max-concurrency

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • preserve an existing run set's stored max_concurrency when the database is reopened
  • initialize the default limit only when the metadata key does not yet exist
  • keep the existing run_set_id metadata behavior unchanged

open_run_set_db() currently upserts max_concurrency with ON CONFLICT ... DO UPDATE, resetting it to 1_000_000 every time an existing run-set database is opened.

This conflicts directly with nanoeval.bin.concurrency, which stores a user-selected limit in the same metadata key. Reopening or resuming the run set therefore silently discards that value before workers read it, and even the concurrency command itself resets the previous value as part of opening the database.

The fix changes initialization to ON CONFLICT DO NOTHING, so new databases still receive the default while existing run sets retain their configured limit.

Regression coverage sets a custom limit, closes the run set, reopens the same run-set ID, and verifies the value is preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant