diff --git a/docs/overhaul-completion-plan.md b/docs/overhaul-completion-plan.md index 53b1120f..72fdbb17 100644 --- a/docs/overhaul-completion-plan.md +++ b/docs/overhaul-completion-plan.md @@ -65,6 +65,14 @@ The whole overhaul is complete only after production launch **and**: This distinction allows the useful new system to launch without pretending that the historical migration is finished. +Production launch is the immediate operating milestone. Once the launch gates +in section 7 are satisfied, proceed directly through the ordered production +steps in section 8. Do not delay launch for a fresh exact-candidate live matrix +of optional lifecycle routes, historical completion, the issue-intake overlap, +or the production canary's later release date. Full overhaul completion remains +calendar-bound by the overlap and notice periods and by verification of the +production canary's first automatic release at its two-calendar-month due time. + ## 3. Retained product scope The following remain part of the overhaul. @@ -200,27 +208,33 @@ ownership is not a launch gate. Launch with most of the implemented lifecycle surface available. -| Function | Launch disposition | Minimum staging acceptance | +| Function | Launch disposition | Minimum launch evidence | | --- | --- | --- | -| New browser submission | Enable | One successful private synthetic submission and one invalid/unauthorized request | -| Headless-agent submission | Enable | One source-bound successful submission and one challenge/source mismatch denial | -| Metadata backfill | Enable | One owner success and one non-owner denial | -| Repair and retraction requests | Enable | One valid owner request and one invalid or non-owner denial | -| Maintainer decisions | Enable | One configured-maintainer success and one non-maintainer denial | -| Model alias and rename | Enable | One owner success and one collision or non-owner denial | -| Publication opt-in | Enable | One post-result private-to-scheduled transition with its atomic release schedule checked | +| New browser submission | Enable | Exact launch-candidate private submission through archive, evaluation, immutable Result, and append-only State | +| Headless-agent submission | Enable | Exact launch-candidate source-bound submission through archive, evaluation, immutable Result, and append-only State | +| Metadata backfill | Enable | Repository tests and the retained prior bounded staging owner-success/non-owner-denial evidence | +| Repair and retraction requests | Enable | Repository tests and the retained prior bounded staging valid/invalid or non-owner evidence | +| Maintainer decisions | Enable | Repository tests and the retained prior bounded staging maintainer/non-maintainer evidence | +| Model alias and rename | Enable | Repository tests and the retained prior bounded staging success/collision or non-owner evidence | +| Publication opt-in | Enable | Packet-bound production canary, initially private, then irreversibly opted in with its atomic release schedule and presentation checked | | Model consolidation | Keep disabled or remove | Not a launch test | Publication opt-out is not part of the launch surface. Keep its implementation gate disabled and omit it from user-facing forms and documentation. -This is intentionally a bounded smoke, not a qualification campaign. Existing -repository tests must stay green, but launch does not require a persistent -harness, a full combinatorial route matrix, repeated live contention, or -failure injection at every State write boundary. +The exact-candidate live prelaunch smoke is deliberately limited to the browser +and headless archive-evaluate-Result paths. Existing repository tests and prior +bounded staging route evidence are sufficient for the optional lifecycle route +families; do not rerun them as an exact-candidate live matrix. Publication +opt-in is proved once on the production canary, not by another staging +submission. This remains a bounded acceptance exercise, not a qualification +campaign: launch does not require a persistent harness, repeated live +contention, or failure injection at every State write boundary. -For each enabled route family, also prove that its tracked feature flag can -disable it and that public health reports the expected effective state. +Keep the optional lifecycle APIs launch-enabled. For each enabled route family, +retain an independent tracked feature flag, prove the flag can disable it, and +require public health and the emergency rollback path to report the expected +effective state. ## 7. Production-launch gates @@ -265,17 +279,27 @@ Historical legacy-archive migration is not a launch gate for new submissions. ### 7.4 Entry page and bounded lifecycle smoke - Publish and verify the no-DNS entry page. -- Complete the bounded route-family staging cases in section 6 against the - exact proposed launch commit. -- Complete one exact-version staging lifecycle from archive through accepted - result, State, scheduled release, staging reconstruction, and rollback. +- At exact deployed submissions commit + `f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`, complete one browser and one + source-bound headless submission through schema-version-3 archive before + evaluation, terminal evaluation, immutable Result, and append-only State. +- Accept the green repository tests and retained prior bounded staging route + evidence for metadata backfill, repair/retraction, maintainer decisions, and + model alias/rename. Do not require a new exact-f03 live route-family matrix. +- Use the credentialed, publication-disabled staging reconstruction already + required by section 7.3; do not repeat it solely to bind it to f03. +- Restore intake and every public lifecycle gate to disabled, verify effective + health and State, and clean up the temporary staging proof, tag, and branch. +- Prove the live private-to-scheduled publication opt-in and its atomic release + schedule on the packet-bound production canary in section 8, before public + announcement. For this gate, the exact launch candidate is the submissions commit deployed for the lifecycle-API launch step. The later single-purpose intake-enable commit is verified separately by the protected staging promotion canary, finite-lease transition, and health and State readbacks in section 8; it does -not require repeating the bounded route-family smoke. Historical image and -profile commits are independent of this gate. +not require repeating the browser/headless smoke or any optional route-family +case. Historical image and profile commits are independent of this gate. ### 7.5 Production launch readiness packet @@ -304,8 +328,13 @@ After the launch packet is complete, make capability changes separately: 1. enable the automatic release controller initially when no release is due; 2. enable the approved lifecycle route families; 3. enable production intake through the finite-lease controller; and -4. verify the public entry path, effective health, State consistency, release - scheduling, and leaderboard presentation read-only. +4. submit the packet-bound production canary as private, verify archive, + evaluation, Result, State, and the private leaderboard presentation, then + exercise the visible irreversible publication opt-in and verify its atomic + release schedule and scheduled presentation; +5. prove the production emergency pause and ordered restore; and +6. verify the public entry path, effective health, State consistency, release + scheduling, and leaderboard presentation before announcing server intake. Do not mix refactoring, replay expansion, documentation cleanup, or unrelated features into an enablement change. @@ -469,6 +498,8 @@ The lifecycle overhaul is finished when all of the following are true: - accepted and rejected lifecycle transitions are coherent and recoverable; - automatic two-calendar-month releases are operating with initial scheduled/private choice and one-way private-to-scheduled opt-in; +- the production canary's first automatic release has published successfully + at or after its exact two-calendar-month due timestamp; - the launch-approved backfill, repair/retraction, maintainer, alias, and rename functions are available; - the leaderboard correctly exposes lifecycle, statements, standings, diff --git a/docs/overhaul-execution-runbook.md b/docs/overhaul-execution-runbook.md index 87f90111..65a63d01 100644 --- a/docs/overhaul-execution-runbook.md +++ b/docs/overhaul-execution-runbook.md @@ -145,48 +145,57 @@ effective flags; Section 5.2 and the current operational ledgers record those. | Repository | Commit | Protection state | | --- | --- | --- | -| `lean-eval` | `90ab83b75eabb5604c611f0ba9bb6a4d66b02311` | Required `verify` | +| `lean-eval` | `0c566d62bd247879885dfb70061e085ff03daba6` | Required `verify` | | `lean-eval-submissions` | `f03f5cde4f1ac83b13ce78f294fc2273980dbf0a` | Required `verify` | -| `lean-eval-leaderboard` | `b6df2533e2a6ceea8a6ed6eff5527cc3aef3e7c2` | Required `build` | +| `lean-eval-leaderboard` | `d7f0de9d9b5abbb62a4080df31002825a1afa814` | Required `build` | | `lean-eval-state` | `7ffb7ffb78d79847137785c65df25770f41b62ef` | Required `validate`; append-only | -| `lean-eval-state-staging` | `2398558b8de9e98fc330153862b5e6e0a809b577` | Required `validate`; append-only | +| `lean-eval-state-staging` | `c604bb446a51fc833c96887053ec64672c912d8c` | Required `validate`; append-only | | `lean-eval-releases` | `c0bcb97d87eeb17c0a2f1ef7e8bfc76502deb798` | Required `validate` | | `lean-eval-generator` | `010b01634cccda2db538cf9b09e6f26ddc453743` | Required `check` | -| `lean-eval-audit` | `2681d179d515b6843ee4a4f862d76983f09ea2e9` | Reviewed changes; non-rewritable linear history | +| `lean-eval-audit` | `666950ce7702d1d2a1392b12f9104781ac9446e3` | Reviewed changes; non-rewritable linear history | ### 5.2 Deployed services The current protected submissions and deployed staging commit is `f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`. Its protected CI, immutable dispatch tag, staging deployment, promotion canary, and corrected bounded-smoke -workflow pass. Staging intake and the launch lifecycle gates are enabled only -for the bounded final smoke; model consolidation and publication opt-out remain -disabled, and the promotion canary remains enabled. The exact-commit watchdog -is active, automatic all-false restoration is armed, and the browser receipt is -pending. The deployed commit binds production State contract +workflow pass. Staging intake and every public lifecycle gate are disabled; +model consolidation and publication opt-out remain disabled, and only the +staging promotion canary remains enabled. The bounded watchdog is cancelled. +The deployed commit binds production State contract `235a96c96462438c7680e6fb90fa0e6044ec1774` and staging State contract `6105a6255ec40409bcce66c6cf6b6764e0e93ed4`. Current protected staging State -`2398558b8de9e98fc330153862b5e6e0a809b577` is a validated append-only -descendant of that contract. Current protected production State +`c604bb446a51fc833c96887053ec64672c912d8c` is a validated append-only +descendant of that contract. Protected staging Results is +`1deb87414faf64edfa31639a8430fcf98fb2ccb5`, and the current audit head is +`666950ce7702d1d2a1392b12f9104781ac9446e3`. Current protected production State `7ffb7ffb78d79847137785c65df25770f41b62ef` is a validated append-only descendant of its deployed minimum contract. +The exact-f03 browser and source-bound headless submissions both completed +schema-version-3 archive-before-evaluation, terminal evaluation, immutable +Result, and append-only State. The scheduled headless path has a materialized +release at `2026-11-02T00:57:18.002Z`. The temporary staging branch and tag are +absent, the exact identity proof is restored, and the separately bound +production-canary branch remains present. + Protected releases commit `c0bcb97d87eeb17c0a2f1ef7e8bfc76502deb798` predates the current production State projection script. Publication remains disabled; reviewed release PR #36 -repins that exact script and State head and is green but held until the bounded -staging smoke finishes against the current release fixture. +repins that exact script and State head and is green. Final staging acceptance +is complete, so the ordered release-controller merge and no-op qualification +are the next launch action. -The compact launch packet, exact release fixture, and publication-opt-in -all-false watchdog are part of the protected final candidate. Only the bounded -final staging smoke and cleanup remain before the packet can be finalized. +The compact launch packet and exact release fixture are part of the protected +final candidate. The packet is being refreshed with the terminal staging +bindings before its launch changes merge. Production remains deployed from exact all-false baseline `451856ebdd4ca4d875e43be7cd113678dea9e1b7`. Intake, replay, lifecycle APIs, model consolidation, the promotion canary, publication opt-in, publication opt-out, and publication are disabled. That dark runtime predates the selected -production State contract; production deployment remains gated on terminal -staging acceptance and a refreshed exact-head launch packet. +production State contract; production deployment remains gated on a refreshed +exact-head launch packet. - [x] Read staging and production intake health. - [x] Read staging and production broker/replay health and current versions. @@ -254,22 +263,28 @@ These lanes can proceed in parallel after Phase 1. - [x] maintainer decision; - [x] model alias/rename; and - [x] publication opt-in success, including post-result atomic scheduling. +- [x] Accept the green repository tests and retained prior bounded staging + route evidence as the prelaunch evidence for backfill, + repair/retraction, maintainer decisions, and model alias/rename. Do not + run a fresh exact-f03 live matrix for those optional routes. +- [x] Move the required live publication-opt-in proof to the packet-bound + production canary; keep its focused repository coverage green before + launch. - [x] Verify every launch gate can be returned to disabled and health reports the effective state. - [x] Do not build a persistent staging harness. -### 6.4 Exact-version lifecycle rehearsal +### 6.4 Exact-version core lifecycle rehearsal The operational-baseline table in section 5.1 records the current repository family. Protected submissions `main` and the exact deployed staging basis are `f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`, with immutable tag `lean-eval-dispatch/f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`. Protected CI, staging deployment, and promotion canary pass. Protected releases -commit `c0bcb97d87eeb17c0a2f1ef7e8bfc76502deb798` is the exact staging -reconstruction fixture. -The final candidate contains the exact release fixture repin and corrected -publication-opt-in all-false watchdog. The bounded final smoke is active, with -the browser receipt pending and automatic all-false restoration armed. +commit `c0bcb97d87eeb17c0a2f1ef7e8bfc76502deb798` remains the credentialed, +publication-disabled staging reconstruction fixture qualified by section 7. +The bounded final core smoke, all-false restoration, validation, and fixture +cleanup are complete. Production remains on exact all-false baseline `451856ebdd4ca4d875e43be7cd113678dea9e1b7`, and production deployment is withheld. @@ -286,7 +301,7 @@ launch-candidate prerequisite. - [x] Use a synthetic private source repository owned for staging. - [x] Move the final source fixture to a temporary, non-default fixture branch in private allowlisted `lean-eval-state-staging`. -- [ ] Select the private staging fixture repository in both contents-read org +- [x] Select the private staging fixture repository in both contents-read org App installations, preflight both Apps against that branch, use a runtime-unique tag, and remove the staging branch and tag after the terminal run. The separately tracked production canary uses its own @@ -296,14 +311,16 @@ launch-candidate prerequisite. to the individual GitHub login. Apply only the exact runtime-generated Gist file CAS write/restore under standing authorization. - [x] Prepare one browser and one source-bound headless submission. -- [x] Include one deliberate invalid or unauthorized case. -- [ ] Confirm archive-before-evaluation and schema-version-3 binding against - the exact final candidate. -- [ ] Confirm the exact-candidate accepted path produces an immutable Result, - append-only State, release scheduling, and a redacted leaderboard - projection. -- [ ] Confirm the bounded rejection and authorization-denial cases against the - exact final candidate. +- [x] Retain repository coverage and prior bounded staging evidence for invalid + and unauthorized optional-route cases. +- [x] Confirm archive-before-evaluation and schema-version-3 binding against + the exact final candidate for both browser and headless submissions. +- [x] Confirm both exact-candidate core paths produce immutable Results and + append-only State, including the initially scheduled headless release + readiness and redacted projection. +- [x] Do not require an exact-f03 live matrix for metadata backfill, + repair/retraction, maintainer decisions, model alias/rename, or + publication opt-in. The production canary owns the live opt-in proof. - [x] Prepare the rollback/disable steps for the same exact version. Exit condition: repository changes and staging fixtures are ready; bounded @@ -360,31 +377,24 @@ release path has passed a credentialed staging boundary. ## 8. Phase 3 — final staging acceptance -Temporary staging feature flags and their lifecycle/intake all-false recovery -are autonomous. -Before the bounded run, tell the maintainer that the browser and headless -canaries permanently add synthetic staging archives, Results, and append-only -State events. The maintainer deliberately performs the browser submission as -an operator handoff; the exact unavoidable secret-Gist CAS mutation for the -headless identity proof is covered by standing authorization. - -The deployed protected staging candidate is -`f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`. Its protected CI, immutable tag, -staging deployment, and promotion canary pass. Staging intake and the launch -lifecycle gates are enabled for the bounded final smoke; model consolidation -and publication opt-out remain false, while the promotion canary is -intentionally true. Its exact-commit watchdog is active and automatic all-false -restoration is armed. Protected releases commit -`c0bcb97d87eeb17c0a2f1ef7e8bfc76502deb798` remains the exact staging -reconstruction fixture; publication remains disabled. +The exact-f03 final acceptance is complete. It was limited to the browser and +source-bound headless core paths; optional lifecycle route families require no +new live matrix. Protected submissions commit +`f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`, its immutable tag, staging +deployment, promotion canary, and both core paths pass. Staging intake and all +public lifecycle gates are false, model consolidation and publication opt-out +remain false, and the promotion canary is intentionally true. The watchdog is +cancelled, the temporary staging branch and tag are absent, and the exact +identity proof is restored. Protected releases commit +`c0bcb97d87eeb17c0a2f1ef7e8bfc76502deb798` remains the already qualified +publication-disabled staging reconstruction fixture. - [x] Merge and deploy the exact final candidate through the normal protected path and qualify the corrected bounded-smoke workflow. -- [ ] Receive the browser submission, then complete the bounded final smoke: - the source-bound headless submission, launch lifecycle route families, - one authorization or validation denial, and publication-disabled archive - reconstruction. -- [ ] Run the all-false recovery and fixture cleanup; verify public-health +- [x] Reconcile the exact-f03 browser and source-bound headless submissions + through archive-before-evaluation, terminal evaluation, immutable Result, + append-only State, and release readiness where selected. +- [x] Run the all-false recovery and fixture cleanup; verify public-health readback, staging State validation, and absence of exposed source or credential material. @@ -414,8 +424,15 @@ green staging run to substitute for launch readiness. The compact launch packet is part of protected final candidate `f03f5cde4f1ac83b13ce78f294fc2273980dbf0a`. Finalize it only after the bounded -staging smoke and all-false cleanup complete. Fill only its post-action fields -after the corresponding Phase 4 actions. +browser/headless core acceptance and all-false cleanup complete. Repository +tests and retained prior staging evidence satisfy the optional route-family +prelaunch fields; the packet binds publication opt-in to the production canary. +Fill only its post-action fields after the corresponding Phase 4 actions. + +Production launch is the immediate milestone after packet `GO`. Do not hold it +for another optional-route staging matrix, historical completion, overlap +elapsed time, or the canary's later release date. Those calendar-bound checks +remain required for full overhaul completion. A production-intake candidate has been rebased locally onto the protected final candidate, but draft PR #1603 still points to its stale predecessor and must be @@ -443,6 +460,9 @@ After the launch packet is complete: user-facing forms and documentation. - [ ] Keep model consolidation disabled. - [ ] Verify effective public health and one non-mutating authorization denial. +- [ ] Retain a separately reversible feature flag for every enabled family and + verify the all-false rollback; do not substitute another staging matrix + for this production readback. ### 10.3 Intake @@ -456,9 +476,11 @@ After the launch packet is complete: - [ ] Use the packet-bound visible archived problem and exact previously accepted Kim-owned source with a distinct canary model identity. - [ ] Submit the canary as private, verify that choice on the live problem page, - exercise the visible irreversible publication opt-in, force an immediate - leaderboard build, and verify the scheduled choice, release schedule, - and exact source-State commit on the live page. + then perform the required live proof of the visible irreversible + publication opt-in. Verify the same atomic State append creates its + release schedule, force an immediate leaderboard build, and verify the + scheduled choice, release schedule, and exact source-State commit on the + live page. - [x] Keep the periodic read-only leaderboard State-drift deployment path active so later State-only lifecycle events cannot leave the public site stale indefinitely. @@ -486,6 +508,11 @@ the system can be paused through the documented path. ## 11. Phase 5 — four-week overlap +Production launch starts this phase immediately after the Phase 4 checks and +announcement. Full overhaul completion remains calendar-bound: it cannot occur +before both the overlap/notice gates and the canary's exact two-calendar-month +automatic-release checkpoint pass. + - [ ] Monitor severity-high incidents and readiness failures. - [ ] Monitor State validation, archive completion, evaluation dispatch, release scheduling, and automatic releases. @@ -726,9 +753,10 @@ disabled, and no migration or replay run or temporary executor remains. - [ ] Mark this runbook complete and summarize current operation and ordinary maintenance ownership. -The overhaul is not complete merely because a token budget, agent session, or -calendar period ends. It is complete only when the completion-plan criteria are -actually satisfied. +The overhaul is not complete merely because a token budget or agent session +ends. Production launch is the immediate milestone, but full completion is +deliberately calendar-bound and occurs only when every completion-plan +criterion is actually satisfied. ## 15. Compact status table @@ -740,10 +768,10 @@ Update this table in place; do not append a history beneath it. | 1. Disabled baseline | Complete | — | | 2. Repository launch preparation | Complete | — | | Credential boundary | Complete | — | -| 3. Final staging acceptance | In progress | Receive the browser submission, finish the bounded final smoke, then clean up and verify automatic all-false recovery | -| Production launch readiness | In progress | Finalize the protected candidate's compact packet from the terminal staging result and refresh stale intake PR #1603 | -| 4. Launch | Not started | Production remains on exact all-false baseline `451856ebdd4ca4d875e43be7cd113678dea9e1b7` until packet `GO` | -| 5. Four-week overlap | Not started | Production launch and overlap announcement | +| 3. Final staging acceptance | Complete | — | +| Production launch readiness | In progress | Merge the refreshed compact packet and ordered release-controller/submissions launch changes; publication opt-in is bound to the production canary | +| 4. Launch | Immediate next milestone | Production remains on exact all-false baseline `451856ebdd4ca4d875e43be7cd113678dea9e1b7` until packet `GO` | +| 5. Four-week overlap | Calendar-bound after launch | Production launch and overlap announcement | | 6. Historical completion | In progress; not an initial-launch gate | All 63 private profiles and the final plan are canonical and the temporary qualifier is retired; complete the packet-bound rewrap/replay, with the final delta after cutoff | | 7. Remaining product completion | In progress | Open problems and editorial work are complete; final leaderboard readback waits for live release and replay data, and issue closure retains its overlap, notice, stability, adoption, final-delta, and readiness gates | | Final audit | Preparatory cleanup complete; final audit pending | Repeat the audit after all phases and confirm only explained launch and retirement work remains |