Skip to content

feat: Phase W W4 benchmark challenge registry schema (BCR-) - #1006

Merged
cschanhniem merged 1 commit into
mainfrom
feat/w4-benchmark-challenge-registry
Jul 10, 2026
Merged

cschanhniem merged 1 commit into
mainfrom
feat/w4-benchmark-challenge-registry

Conversation

@cschanhniem

Copy link
Copy Markdown
Collaborator

Summary

  • Adds BCR- (benchmark challenge registry) schema: machine-readable registry of NCH/CMC/SCH challenge results for a batch+pipeline version
  • Maps raw challenge verdicts to pass/marginal/fail/not_run (e.g. novel_batch→pass, mixed_novelty→marginal, near_neighbor_dominated→fail)
  • Hardness grade: A (all pass), B (all pass+marginal), C (some pass/marginal), D (all fail/not_run)
  • ChallengeEntry helper stores challenge_type, artifact_id, raw_verdict, challenge_verdict
  • 64 tests covering constants, build happy paths, verdict mapping, grade computation, validate rejection, format

Test plan

  • python3 -m pytest tests/evidence/test_benchmark_challenge_registry.py -- 64 passed

…ates NCH+CMC+SCH verdicts into hardness grade A-D; REQUIRED_CHALLENGE_TYPES (3), VALID_CHALLENGE_VERDICTS (4), ChallengeEntry helper; verdict mapping from raw challenge outputs; grade A=all pass; dry_lab_only=True; 64 tests
@cschanhniem
cschanhniem merged commit 278f805 into main Jul 10, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant