feat(multiharness): wire the community harness-lane submission flow end to end - #1044
Draft
ai-agent-on-behalf-of-john-hughes[bot] wants to merge 2 commits into
Conversation
…nd to end A contributor could package a run and could validate a package, but the submit path was not a flow and its destination did not exist. Three things closed that. One command. `multiharness harness submit` packages a finished run with the light-touch redaction, writes the upload plan, and then runs the same checks the publishing job runs -- caps, declared digests, undeclared bytes, symlinks, the publication secret scan, and the destination -- before a pull request is opened, then prints the exact files to commit and the exact dispatch inputs. The local check does not replace the server-side one: it is the same code called earlier, and `community-harness-intake.yaml` still re-validates from scratch at the dispatched commit, because bytes that travelled through a pull request are not the bytes the contributor checked. It registers through `add_harness_parser` and is reached by entry point rather than import, so the lane does not drag the publication guardrails into the CLI's import cycle. The destination. The owner's dataset repository is johnhughes3/legal-quants-community-submissions, and the intake's mirror constant named a different repository -- with the publish step proving the plan's mirror against the variable, every real publish would have refused. It is now bound in code, and `resolve_community_dataset_repo` fails closed on an unset variable, on a variable naming any other repository, and on a variable pointed at the official dataset, so a community submission cannot land where official results live. LFB_HF_COMMUNITY_DATASET_REPO is not set yet; the job stops until it is. The `.txt` refusal. A real run could not be packaged at all: every row stages a tool-use sentinel token into `container-workspace/`, and the publication guardrails refuse a `.txt` in a public artifact. That rule is right -- a Harvey LAB row stages the projected corpus documents into the same directory -- so the answer is not an exception but a boundary: `container-workspace` is what the lane staged *into* the container, not what the harness produced, and it never travels. The count left behind is recorded rather than dropped silently. Nothing is lost on the LFB path, where the answer and transcript live in `private-logs/` and `container-logs/`; on the LAB path the written deliverable is lost, and the docs say so plainly rather than leaving it to be discovered.
ai-agent-on-behalf-of-john-hughes
Bot
changed the title
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| #: in code so the error a contributor or a workflow sees can name the exact | ||
| #: value the variable should hold rather than only the variable. | ||
| COMMUNITY_HARNESS_DATASET_REPO: Final = "johnhughes3/legal-quants-community-submissions" | ||
| COMMUNITY_HARNESS_ARTIFACT_MIRROR: Final = ( |
|
|
||
|
|
||
| class _ParserRegistration(Protocol): | ||
| def __call__(self, harness_commands: Any) -> None: ... |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Layer 4 of stack #1042. Review #1040, #1041, #1043 first.
Gives a contributor a one-step path from a finished harness-lane run to a submission a maintainer can publish, and binds the destination to a dataset repository separate from the official one.
Contributor flow
submitpackages the run with light-touch redaction, writeshf-upload-plan.json(per-artifact sha256 + size_bytes), runs every server-side check locally, and prints the exact files to commit plus the dispatch inputs. The local check is a convenience — the server-side check remains authoritative and is not skipped because the local one ran.Registered through
harness_lane/cli_parser.py::add_harness_parser, socli.pyis untouched.Adversarial results
Each tampered package was made self-consistent first — digests and sizes recomputed into the plan, the way a real submitter's would be — then attacked:
/etc/passwd/etc/shadowDestination binding fails closed on an unset variable, on the wrong repository, and when the community and official variables name the same repository — the last is what stops a community submission landing where official results live.
Packaging was proven against a real run directory produced by the test runner (22 artifacts, 35,680 bytes, 1 file redacted), not a hand-authored fixture.
Redaction stays light
A containerized run has a clean HOME, no host bind-mounts beyond the task workspace and a throwaway credential copy, and no egress but allowlisted provider hosts — so there is structurally nothing of the operator's environment in the transcript. Redaction scrubs credentials and the container's own throwaway paths and keeps the substance: prompts, reasoning, tool calls, tool outputs, answers.
Two papercuts, reproduced and deliberately not papered over
.txt/.textfile anywhere outsidecontainer-workspace/refuses packaging (hit on a real run atcontainer-logs/answer.txt). The guardrail exists to stop raw case text being republished, so no blanket exception was carved.sk-ant-[redacted], but the guardrail still refuses on the pattern.Neither blocks the path; both are real contributor friction.
The live end is unverified
No workflow dispatch, no Hugging Face upload, no variable set, no container run, no provider call. The OIDC claim (GitHub OIDC exchanged for a repository-scoped token of at most 60 minutes, no durable
HF_TOKEN) is verified only as structural consistency with the official lane, not exercised.One-time owner setup
johnhughes3/legal-quants-community-submissions.legalforecastbench-community-artifacts, restricted tomain— the intake job declares it and will not run without it.LFB_HF_COMMUNITY_DATASET_REPO=johnhughes3/legal-quants-community-submissions(plainnamespace/repository). A repo-scoped write was refused terminally by secure-gate (Purpose is not allowed for GitHub token elevation: repository-variable);LFB_HF_OFFICIAL_DATASET_REPOis environment-scoped too.main, workflow filenamecommunity-harness-intake.yaml. Entries are per filename — thefan-in-publish-legacy.yamlentry will not authorize this workflow.Gates
Full suite 10873 passed / 22 skipped / 0 failed, verified independently.
ruff format --check,ruff check,contracts.ratchet,config.fence,pyright(0/0/0) all pass.claude_code.pyandcodex_cli.pyuntouched.Refs legalforecastbench-wm0b