Consolidated into #5386: exact proof workspaces and verification - #5387
Closed
williamjblair wants to merge 168 commits into
Closed
williamjblair wants to merge 168 commits into
williamjblair wants to merge 168 commits into
Conversation
A second-pass semantic review for Formal Conjectures pull requests. It answers one question, whether the Lean statement says what its cited source says, and leaves style, naming and imports to CI and AGENTS.md. The default path is one pass: bind the scope, run the focused build, read the cited source, compare meanings, report or stop. Escalation has named triggers, so the expensive work (whole-document reads, cross-references, Lean witnesses, history) runs when something calls for it rather than on every file. Findings carry a witness, what that witness shows and does not show, and the smallest proposed change. The output is publishable to GitHub as-is, and the review is advisory: it does not approve, merge, label, or touch a contributor branch. evals/evals.json records the measurement history, including a three-arm run against no skill and against an earlier version of this skill, over eight cases built by reverting merged fixes onto main. This skill matches the unaided reviewer's cost and wall clock, catches a case the unaided reviewer misses, and files a quarter as many items. The false-positive control returns CLEAN with no findings, where both other arms file nits. That run also records the measure the earlier iterations lacked. A suite where every case has a planted defect rewards recall and charges nothing for a finding that should not have been filed, so the file now reports findings filed alongside pass rate.
The monolithic defect-classes reference becomes four rubric files - _common, source-fidelity, statement-soundness, metadata-hygiene - the shape TauCetiProject/TauCetiReview uses, so a finding names the kind of defect it is, coverage is checkable per angle, and the rubrics could be judged independently by a runner without rewriting them. SKILL.md's compare step routes through the angles; the fast path is otherwise unchanged. The restructure initially regressed exactly the verification behaviours the assertion hardening of iterations 9-11 exists to detect, and the eval loop caught and repaired it: witnesses are checked rather than argued, escalation is mandatory for any verdict-setting finding, a source construction runs as a positive control when faithfulness rests on it, and a formal_proof root link stays a finding even after private resolution. Post-fix runs restore the ceiling on both active cases, 9/9 on 940 and 8/8 on ClaudesCycles, with each rule attributable to the single assertion it flipped. iteration_16 records the loop.
# Conflicts: # toolkit/conjectures/cli.py
Collaborator
Author
|
Consolidated into #5386. On 16 September the toolkit was rebuilt on current |
williamjblair
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generate a workspace for an exact FC declaration, import only permitted submission files, and retain a typed verification result. Candidate code cannot choose trusted files, tool pins or verification policy.
Active draft. Depends on #5386, #5337 and leanprover/comparator#87. The upstream diff still includes prerequisites. Replay only this delivery's changes after they merge, then request review.
init TARGET --out DIRUnfinished proofs and imported assumptions remain distinct from verifier crashes and missing results. Definition holes can require semantic assessment even after kernel success. A missing workspace now returns
directory_missingbefore target lookup.Qualified on the fork: 94 focused tests pass. RC3 qualification records fresh macOS initialization, explicit public workspace commit/push, hosted proof success/rejection/error, and real Linux trust controls. Missing or malformed results retain unevaluated policy. No outcome is inferred from log text.
Remaining: adopt accepted dependency revisions, rerun affected qualification, and replay this delivery after prerequisites merge. The Harbor adapter remains experimental on
codex/fc-toolkit-evals; it does not block the core CLI.Roadmap: #4394. Acceptance: #5376. No upstream executor activation is authorized by this draft.