Skip to content

Validate LLM judge concurrency - #2995

Open
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix/llm-judge-concurrency
Open

Validate LLM judge concurrency#2995
sylvesterkaczmarek wants to merge 1 commit into
openai:mainfrom
sylvesterkaczmarek:fix/llm-judge-concurrency

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Summary

  • Reject non-positive concurrency values at the judge_folder() boundary.
  • Keep all valid worker counts and judging behavior unchanged.

Motivation

judge_folder() accepts concurrency directly and later passes it to ThreadPoolExecutor(max_workers=concurrency). Values such as 0 or -1 are therefore accepted by the function and CLI, perform some setup work, and then fail with an executor-internal ValueError.

This parameter is part of the evaluator's public helper/CLI surface, so invalid worker counts should fail immediately with a clear, local validation error before creating output directories, loading prompts, or constructing the client.

Validation

The new guard enforces the executor's existing requirement explicitly:

  • concurrency < 1 -> ValueError("concurrency must be at least 1")
  • concurrency >= 1 -> existing behavior unchanged

No API requests, judge prompts, retry behavior, or result formats are changed.

Self-review

  • Three added lines in one file.
  • Valid concurrency values preserve existing behavior.
  • No dependencies, notebooks, registry entries, or documentation changed.
  • Searched open PRs for an existing concurrency-validation fix and found none.

Maintainers may modify the branch if needed.

@sylvesterkaczmarek
sylvesterkaczmarek requested a review from a team as a code owner August 21, 2026 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant