test(multiharness): prove the agy fence with a negative control and pin the LFB score - #1045
Draft
ai-agent-on-behalf-of-john-hughes[bot] wants to merge 5 commits into
Conversation
The agy image's fence note said the 2026-08-31 runs could not distinguish
"the fence stopped a search that would have worked" from "the search would
have failed anyway", because the PreToolUse hook blocks search_web before any
request leaves the machine. Two runs of the committed image settle it.
The control needs no image change: entrypoint.sh writes a hook file only when
it is ABSENT, so pre-seeding both customization roots with {} runs the same
image with no fence loaded. Asked for a stock index close on a named past date
-- a number in no file in the container and in no model's weights -- plus a
token planted in a workspace file, the unfenced arm answered with the correct
figure and named the news domain it read; the fenced arm answered WEB-BLOCKED
and journalled search_web. Both echoed the planted token, so local tools were
live in both. The cited domain is not on the allowlist and appears in neither
arm's refusal list, so the search ran provider-side inside the model's own TLS
session: this hook is the only thing between an agy run and a live lookup of
the outcome it is scored on.
The Dockerfile note records that. The three test_agy_entrypoint_* cases pin
the seed-only-if-absent rule the control rests on, by running the image's real
entrypoint against a stub CLI; they live beside the egress tests because the
hook is this package's answer to the one path the allowlist cannot reach.
…unscoreable input The release-backed chain already ran end to end, but the test stopped at "a summary exists for this model_id". It now pins the number the chain actually produces: a 0.25 forecast against an outcome of 0 is a micro and macro Brier of 0.0625, and the 0.5 base rate over the label file makes that a skill score of 0.75. Both are exact in binary floating point, so a silent change in how a lane row reaches lfb_brier moves the value and fails here rather than passing on a row that merely exists. The other half is the operator's. Picking --packets instead of --forecast-release produces an honest plumbing run and no lfb/runs.jsonl, and until now the only place that was said was --help. A run resolved on the packet source now says it on stderr before the run starts: packets carry no release identity or per-unit packet/prompt commitment, so release_harness projects no score row, and the invocation that would score is named. It warns rather than refuses, because a packet-backed plumbing run is a legitimate thing to want.
Both shapes came out of a real run and neither had a remedy a contributor could reach, so a legitimate submission was simply unpackageable. A plain-text file the harness wrote -- codex-cli writes codex-last-message.txt, a row that failed under the continue policy writes private-logs/error.txt -- refused the package on its suffix. Rather than carve an exception into the rule that stops case text being republished, the packager stops producing the suffix: answer.txt is carried as answer.txt.json, a record holding the original path and the text. The bytes are kept and still scanned, since .json is a text suffix to the guardrails too, and a .txt that is not valid UTF-8 is refused by name rather than mangled. A transcript that echoed a credential refused on the scrubber's own output: the value became sk-ant-[redacted] and the scan then found the credential name beside it. An assignment whose value is already the blessed placeholder is now elided before the credential patterns run. Only the matched span goes, so an unscrubbed secret sharing that line still refuses -- the existing backstop test now asserts exactly that -- and a credential name with no redacted value attached still fires. Also stages the first harness-lane submission package, built from a live containerized Claude Code row over a synthetic release under subscription auth (tools_observed ["Bash"], zero metered spend). Not an official result: the record carries official=false and the not-official attestation.
…scan The community intake's whole claim is that a contributor's full results are checked before any byte is offered for upload, and the secret scan was the fail-closed backstop behind a deliberately light redaction. It was not reading the one file a leaked credential would actually land in: a harness CLI that echoes its own login echoes it to a stream, those streams are captured as .stdout and .stderr, and neither suffix was in TEXT_SUFFIXES -- an AWS key sitting plainly in a live transcript sailed straight through. .transcript joins them, because a contributor's run directory can hold one too. The second half of the same hole was silent: the scan skips a text file larger than max_text_bytes and says nothing, and its 2 MB default sits below the 8 MiB per-artifact cap this intake admits, so a long agentic transcript passed the caps and skipped the scan. The ceiling is now pinned to the cap, in community_upload beside the caps it belongs with. The existing residual-secret test moves its planted AWS key from leak.log to leak.stdout: before this it did not raise at all, which is the proof the suffix mattered. Verified against the staged live submission: the real 40 KB Claude Code transcript now reaches the scan and returns zero findings, where before it was never read.
…y for the harness lane The layer-5 lane grew tests/test_multiharness_harness_lane_run.py by nine lines (970 -> 979) and tests/test_container_harness_egress.py was never recorded in the CLI-corpus path identity. Both diffs are attributable to files this workflow added or extended; no other entry moved.
ai-agent-on-behalf-of-john-hughes
Bot
changed the title
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 5 of stack #1042. Review #1040, #1041, #1043, #1044 first.
Closes the three things the earlier layers left honestly open.
The agy fence is load-bearing — proven by negative control
Same committed image (
sha256:bf878580…), same prompt, same five-host allowlist, same model. The only variable is whether the entrypoint's hook is seeded.The prompt asked for the S&P 500 close on 2026-08-31 — unknowable from the model's weights or from anything in the container, and the workspace file states explicitly that it holds no market data. Unfenced, agy retrieved 7,686.14, independently confirmed as the real closing level. Fenced, it could not.
egress_allowed_hostsandegress_refusedare byte-identical between the two arms, so the proxy is not the variable — the hook is.This answers the question the previous layer could not: agy genuinely does retrieve when unfenced, so the fence is the control standing between a run and a live search, not decoration. For a benchmark of real federal cases whose outcomes are one search away, that distinction is the whole integrity argument.
A mutation check backs the pinned rule: flipping the entrypoint's write-only-if-absent guard makes
test_agy_entrypoint_leaves_a_hook_file_that_is_already_there_alonefail.Standing limit, unchanged: the control shows agy retrieves on this version, at this login tier, with this allowlist. It does not establish that the hook matcher covers every tool name a future agy may add.
LFB scoring is pinned to a number
A correction to earlier framing in this stack: the release-backed LFB path was never broken.
task_sources.pysupports two inputs — an authenticatedforecast-release.v1that scores, and a legacy model-packet JSONL that cannot, because it carries no release id, no release digest, and no per-unit packet/prompt commitment. Only the packet path was refused, deliberately.The full chain now runs to an actual
lfb_brierresult:Fixture release with an injected runner, so this proves the wiring and the arithmetic, not any model's skill. There is no real forecast release in this repository — only
docs/schemas/forecast-release-v1.md— which is stated rather than fabricated around. The CLI now warns at the point of use when the unscoreable input is chosen, instead of leaving an operator to discover an absentruns.jsonllater.Both contributor papercuts fixed
The
.txt/.textrefusal outsidecontainer-workspace/and the scrubbed-key transcript that still tripped the guardrail both had no CLI remedy. Fixed without weakening the scan — tests prove the legitimate case now packages and that a genuinely unscrubbed secret is still caught. A related commit closes a gap found along the way: the intake was promising a transcript scan it did not perform.Reported, not fixed
row-results.jsonlrecords"container_execution": {"mode": "plan_only", "status": "not_run"}on rows where a container demonstrably did run (real image digest, 27.73s, live transcript). The field is wrong in published output.runner.pyis under the hard no-growth line ratchet, so this is reported rather than worked around, and needs its own change.Still the owner's call
No workflow dispatch, no Hugging Face upload, no variable read or write, no AWS. Publishing to a hosted dataset is outward-facing, so a validated real submission is staged and the exact dispatch command is recorded for the owner to run.
Gates
Full suite 10878 passed / 22 skipped / 0 failed, verified independently of the implementing agents.
ruff format --check,ruff check,contracts.ratchet,config.fence,pyright(0/0/0), andactionlinton all changed workflows pass. Across the whole stackcli.pyis net −41 lines;claude_code.pyandcodex_cli.pyare untouched.Refs legalforecastbench-wm0b