Add people_search benchmark (scorer-based, provider-agnostic) - #12
Open
diya-kejriwal wants to merge 8 commits into
Open
Add people_search benchmark (scorer-based, provider-agnostic)#12diya-kejriwal wants to merge 8 commits into
diya-kejriwal wants to merge 8 commits into
Conversation
Ship a CSV dataset and Nyne/PDL/Exa people samplers that emit structured people[] output, graded by field-fill scorers instead of gold answers. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace Nyne/PDL/Exa-specific clients with http_people_search so any endpoint returning people[] JSON can run the shared dataset and scorers. Co-authored-by: Cursor <cursoragent@cursor.com>
Port the Braintrust judge rubrics into local prompts scored via the shared grader model; disable with PEOPLE_SEARCH_LLM_JUDGES=0. Co-authored-by: Cursor <cursoragent@cursor.com>
Expand the README so outsiders can run the benchmark end-to-end from the shared eval runner against any people[] endpoint. Co-authored-by: Cursor <cursoragent@cursor.com>
Leave accuracy_score blank in analyzed results; emphasize mean_field_fill and mean_judge_*; use has_people/no_people as the row evaluation_result. Co-authored-by: Cursor <cursoragent@cursor.com>
Fix judge defaults, --clean True, Gemini env var, HTTP contract details, metrics wording, and clarify that no provider clients ship in-repo. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
diya-kejriwal
had a problem deploying
to
Run Tests
July 24, 2026 18:16 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Assemble scoring metadata from dedicated columns at dataset load time so the runner and graders keep working without shipping gold strings. Co-authored-by: Cursor <cursoragent@cursor.com>
diya-kejriwal
had a problem deploying
to
Run Tests
July 24, 2026 18:35 — with
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
people_searchdataset (enrichment + open search across 6 buyer personas) with no gold answers.people[]via deterministic field-fill / persona-field-fill scorers and optional LLM judges (overall + persona), using the sharedeval_runner.py.http_people_searchsampler: pointPEOPLE_SEARCH_API_URLat any people-search HTTP endpoint that speaks the documented request/response contract.accuracy_scoreis blank for this dataset; primary signals aremean_field_fill/mean_judge_*andhas_people_rate.Test plan
pytest tests/test_people_search.pyPEOPLE_SEARCH_API_URLto an endpoint returning{people, person_count}and run:python src/evals/eval_runner.py --samplers http_people_search --datasets people_search --limit 5field_fill/has_people(and judge columns whenPEOPLE_SEARCH_LLM_JUDGES=1)analyzed_results.csvhas blankaccuracy_scoreand populatedmean_field_fill/has_people_ratePEOPLE_SEARCH_LLM_JUDGES=0run skips LLM judge columnsMade with Cursor