Fix the Ouroboros CUDA benchmark protocol and budget accounting

harnessblog/README.md

@@ -1,51 +1,139 @@
 # harnessblog
 
-Reproducible CUDA coding-agent benchmark for Pi, Cline, Hermes Agent, Claude Code, and Codex, with GPT-5.6 Sol and Kimi K3 through OpenRouter. Events, scores, images, and videos are spooled locally and uploaded to Tracehouse.
+Experimental CUDA coding-agent harness from OpenRsi. The corrected Ouroboros
+cohort in this tree is a new H200 diagnostic cohort; it must not be pooled with
+the published RTX A6000 results.
 
-## Safety first
+## Corrected Ouroboros cohort
 
-Never reuse credentials pasted into chat or committed to a repository. Revoke them, create a dedicated OpenRouter inference key with a lifetime `$100` limit, and create a new Tracehouse key. Export them only on the runner host:
+- Two tasks: `three-body` and `heat-2d`.
+- Two OpenRouter models: `openai/gpt-5.6-sol` and `moonshotai/kimi-k3`.
+- Five preregistered replicates per task/model: 20 scored cells.
+- USD 50 per cell, USD 1,000 nominal campaign ceiling.
+- The declared H200 cohort may opt into a selected shared GPU; foreign-process
+  evidence is retained and timing remains diagnostic.
+- One cell is dispatched per invocation so its raw evidence can be inspected
+  before another cell starts.
+- Ouroboros is built from the exact branch snapshot recorded in the Dockerfile
+  and manifest. The image attests its commit, tree, describe string, and version.
+
+Each cell pins main, heavy, light, vision, consciousness, deep-review,
+web-search, task-acceptance, triad, scope-review, and native child routes to the
+arm model. OpenRouter routing is additionally restricted to the model owner's
+first-party provider (`openai` or `moonshotai`), so endpoint retries cannot
+silently move the cell onto a third-party implementation. Cross-model fallback,
+hosted delegate routes, Claude/Cursor/Claudexor credentials, and model switching
+are absent. The task
+review mode is `auto`, enforcement is `advisory`, the reviewer panel has one
+same-arm `api_chat` slot, context is `max`, runtime is `advanced`, and safety
+LLM calls are off. Tools with per-call model overrides (`web_search`,
+`vlm_query`, and `analyze_screenshot`) are disabled for these offline tasks;
+the inert web-search backend is additionally pinned to pure-retrieval DDGS.
+The benchmark is strict single-root: subagent depth is zero,
+and both `schedule_subagent` and the optional pre-implementation `plan_task`
+tool are disabled. The cross-task worker pool remains 10 and per-model call
+concurrency remains 3. Post-delivery task acceptance remains enabled.
+
+The direct OpenRouter credential lives only in the owner-only per-run settings
+file while Ouroboros runs. It is scrubbed afterward and exact-key persistence is
+scanned fail-closed. The candidate workspace and publication bundle never
+receive the credential. Native usage-ledger checks prove Ouroboros' requested
+routes; raw-network shell calls remain a disclosed limitation of this
+experimental benchmark.
+
+## Evaluation
+
+After the agent exits, the evaluator runs `run.sh` twice with:
+
+- the submitted project mounted read-only;
+- two distinct host-owned output directories;
+- the authoritative config mounted read-only;
+- no network or model credentials;
+- CUDA evidence bound to the exact evaluator container and host GPU PID.
+
+The scorer validates fresh regular artifacts, stored FP64 arrays, fixed task
+grids, numerical thresholds, two-run determinism, media signatures, and both
+CUDA receipts. Agent-reported parity, determinism, ensemble size, or device
+names never own a hard gate. Agent exit status and Ouroboros task/review outcome
+are retained separately from the independent score.
+
+An OpenRouter request whose usage cannot be authoritatively settled remains
+marked non-final. It does not suppress independent evaluation: the campaign
+charges the full USD 50 cell cap as conservative exposure and reports exact
+cost as unknown. A task-level provider failure is still classified as
+infrastructure failure and stops the next dispatch.
+
+## Rootless setup and free canaries
+
+Use only the owner's rootless Docker daemon. Never run these commands against a
+shared system daemon.
 
 ```bash
-# Run these inside an interactive shell on eva02, never in shell history.
-read -rsp 'OpenRouter key: ' OPENROUTER_API_KEY; export OPENROUTER_API_KEY; echo
-read -rsp 'Tracehouse key: ' TRACEHOUSE_API_KEY; export TRACEHOUSE_API_KEY; echo
-export TRACEHOUSE_API_BASE='https://tracehouse.ai'
-```
+export DOCKER_HOST=unix:///run/user/1006/docker.sock
+export TMPDIR=/mnt/data/a.razzhigaev/openrsi-tmp
+mkdir -p "$TMPDIR"
 
-The runner starts a private host-side gateway automatically, generates a random run token, and keeps the real OpenRouter key out of agent containers. `OPENROUTER_PROXY_URL` and `RUN_PROXY_TOKEN` may instead select an externally managed gateway.
+bash scripts/build-images.sh ouroboros
+bash scripts/smoke-images.sh ouroboros
+bash scripts/cuda-smoke.sh "$GPU_UUID"
+```
 
-## Setup and run
+Run a dry manifest under a throwaway campaign id. A dry campaign must not be
+resumed as a paid campaign because paid manifests bind immutable image digests.
 
 ```bash
-python3 -m venv .venv
-. .venv/bin/activate
-pip install -e '.[dev]'
-harnessblog preflight --host eva02
-harnessblog build-images --host eva02
-ssh -t eva02
-# The sync directory suffix is the UID of the machine where harnessblog was invoked.
-cd /tmp/harnessblog-501
-python3 -m harnessblog.cli run --budget-usd 100
+python3 -m harnessblog.cli run \
+  --campaign-id corrected-dry \
+  --harness ouroboros \
+  --replicates 5 \
+  --gpu "$GPU_UUID" \
+  --allow-shared-gpu \
+  --budget-usd 1000 \
+  --reserve-usd 0 \
+  --per-run-budget-usd 50 \
+  --timeout 3900 \
+  --max-new-runs 1 \
+  --dry-run
 ```
 
-Replace `501` with the local UID printed by `id -u`. Keeping the paid command in
-that interactive remote shell ensures neither API key is forwarded in an SSH
-argument or stored by this project.
+## Paid execution
+
+Load the OpenRouter credential without placing it in argv, shell history,
+settings committed to git, or a publication artifact. Immediately before the
+campaign, verify both key and account remaining credit and run real burst probes
+against both exact models.
 
-Dry-run the complete 20-run manifest without model calls:
+The internal Ouroboros solve deadline remains 3450 seconds. The 3900-second
+outer timeout covers the documented acceptance/finalization and cost-finality
+tail; it does not add LLM solve time. Each independent evaluator execution has
+its own 900-second bound.
 
 ```bash
-harnessblog run --campaign-id smoke --dry-run
-harnessblog report smoke
+python3 -m harnessblog.cli run \
+  --campaign-id "$CAMPAIGN_ID" \
+  --harness ouroboros \
+  --replicates 5 \
+  --gpu "$GPU_UUID" \
+  --allow-shared-gpu \
+  --budget-usd 1000 \
+  --reserve-usd 0 \
+  --per-run-budget-usd 50 \
+  --timeout 3900 \
+  --max-new-runs 1
+
+# After manually inspecting the previous cell's raw evidence:
+python3 -m harnessblog.cli resume "$CAMPAIGN_ID" --max-new-runs 1
 ```
 
-Campaign state and the idempotent trace spool live under `.harnessblog/campaigns/<id>/`. Re-run with `harnessblog resume <id>` after a failure.
+Campaign state, manifests, reservations, workspaces, task results, runtime
+logs, physical usage ledgers, evaluator receipts, and local trace spools live
+under `.harnessblog/campaigns/<id>/`. Resume is append-only: an existing run id
+is never overwritten or selectively retried.
 
-## Benchmark policy
+## Reporting limitations
 
-- One measured run for each task × harness × model combination.
-- Native harness defaults; identical English task prompts and resource limits.
-- Sequential runs on GPU 0, 16 CPUs, 64 GB RAM, one hour, and a `$4.50` soft run allocation.
-- Correctness is scored; timing and GPU utilization are diagnostic because `eva02` is shared.
-- A score requires independent evaluator evidence. Self-reported model text is never a pass condition.
+The tasks and prior results were public before this rerun and network access is
+retained, so every root tool trajectory must be audited for benchmark or result
+lookup. Timing is diagnostic on the shared H200 host. This is an
+experimental personal harness rather than an official leaderboard benchmark;
+scores need full config, hardware, evaluator, trace, and denominator disclosure.

harnessblog/docker/ouroboros.Dockerfile

@@ -1,10 +1,24 @@
 ARG BASE_IMAGE=harnessblog/base:cuda12.8
 FROM ${BASE_IMAGE}
-ARG VERSION=6.100.0
+ARG VERSION=6.101.1
+ARG OUROBOROS_SHA=b5e216975e8c5653f4b24886e7dcc878399b8124
+ARG OUROBOROS_TREE=0aad326e2b6f1595687b9d13296058acd2aab3a7
+ARG OUROBOROS_DESCRIBE=v6.100.0-35-gb5e21697
 USER root
-RUN git clone --depth 1 --branch "v${VERSION}" https://github.com/razzant/ouroboros.git /opt/ouroboros \
-    && sed -i '/^pip==/d' /opt/ouroboros/requirements-runtime.lock \
-    && python3 -m pip install --no-cache-dir -r /opt/ouroboros/requirements.txt \
+RUN git clone https://github.com/razzant/ouroboros.git /opt/ouroboros \
+    && git -C /opt/ouroboros checkout -q --detach "${OUROBOROS_SHA}" \
+    && test "$(git -C /opt/ouroboros rev-parse HEAD)" = "${OUROBOROS_SHA}" \
+    && test "$(git -C /opt/ouroboros rev-parse 'HEAD^{tree}')" = "${OUROBOROS_TREE}" \
+    && test "$(git -C /opt/ouroboros describe --tags --always)" = "${OUROBOROS_DESCRIBE}" \
+    && test "$(cat /opt/ouroboros/VERSION)" = "${VERSION}" \
+    && test -z "$(git -C /opt/ouroboros status --porcelain)" \
+    && test -n "$(git -C /opt/ouroboros describe --tags --always)" \
+    && python3 -m pip install --no-cache-dir --ignore-installed "$(grep '^pip==' /opt/ouroboros/requirements-runtime.lock)" \
+    && grep -v '^pip==' /opt/ouroboros/requirements-runtime.lock > /tmp/ouroboros-requirements.lock \
+    && python3 -m pip install --no-cache-dir -r /tmp/ouroboros-requirements.lock \
     && python3 -m pip install --no-cache-dir --editable /opt/ouroboros --no-deps \
+    && rm -f /tmp/ouroboros-requirements.lock \
+    && test -z "$(git -C /opt/ouroboros status --porcelain)" \
+    && case "$(git -C /opt/ouroboros describe --tags --always --dirty)" in *-dirty) exit 1;; *) true;; esac \
     && install -d -o agent -g agent /home/agent/Ouroboros/data
 USER agent