Fix the Ouroboros CUDA benchmark protocol and budget accounting - #5
Draft
ouroboros-agent wants to merge 10 commits into
Draft
Fix the Ouroboros CUDA benchmark protocol and budget accounting#5ouroboros-agent wants to merge 10 commits into
ouroboros-agent wants to merge 10 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes the Ouroboros arm of the CUDA benchmark and defines a separately labeled H200 cohort. Every acting and review slot is pinned to the arm model, task acceptance remains automatic with advisory enforcement, and the benchmark runs as a strict single root with schedule_subagent and plan_task disabled. Each run records the exact Ouroboros source, image, applied settings, provider routing, raw usage ledger, and shared-GPU evidence.
The evaluator runs each submitted candidate twice from a read-only mount and binds both CUDA receipts to the candidate and evaluator process. Non-final provider usage keeps the independent capability score while charging the full $50 cell cap. The report shows reported cost, cost finality, and charged budget exposure separately, and fails closed when exposure authority is missing or malformed.
A local corrected H200 campaign completed all 20 preregistered cells. GPT and Kimi each passed 9/10, for 18/20 overall, with $72.696679 of charged exposure. These are diagnostic results and are kept separate from the published RTX A6000 table because the hardware, prompts, evaluator, agent configuration, budget, runtime version, and replicate count differ.
Validation on this branch: 95 harness tests pass. Ruff F checks for harnessblog and its tests, Python compileall, and git diff --check also pass.