diff --git a/GOLD-STANDARD.html b/GOLD-STANDARD.html index 25dd7de..ef426c9 100644 --- a/GOLD-STANDARD.html +++ b/GOLD-STANDARD.html @@ -1234,7 +1234,8 @@

6. The harness standard

│ ├── correctness.yml # fresh agent (≠ grader, ≠ author) hunts changed lines for logic │ │ # defects; blocks on a high-confidence defect, named override on record │ ├── security.yml # security-reviewer on gated paths or risk:high label; blocks on HIGH -│ ├── deploy-dev.yml # merge to main -> client dev environment +│ ├── deploy-dev.yml # merge to main -> client dev environment (automatic, unattended) +│ ├── deploy-promote.yml # dev -> test -> prod; manual only, named approver every time │ └── eval-*.yml # eval-regression + eval-suite (§11 agentic work) ├── scripts/rails/ # diff-anchors.sh, apply-branch-protection.sh ├── eval-datasets/ + prompts/ # golden-set template + versioned judge prompts (§11 work only) @@ -1509,6 +1510,9 @@

10. The kit (what's in this repo)

│ ├── CLAUDE.md.template │ ├── spec-template.md │ ├── spike-template.md # the written finding a spike leaves behind (§5.3a) +│ ├── rollback-template.md # Phase 8: the written "roll back if X" + rehearsal record +│ ├── alert-definitions-template.md # Phase 9: baselines, thresholds, who is woken +│ ├── incident-playbook-template.md # Phase 9: detect/diagnose/escalate/communicate │ ├── settings.json │ ├── mcp.json # team MCP server set; packs merge additions │ ├── HARNESS.md # developer-facing tour; installs to docs/harness.md @@ -1518,6 +1522,8 @@

10. The kit (what's in this repo)

│ ├── hooks/ # stop-gate, review-gate, save-review-receipt (.ps1 + .sh each) │ │ # + sensitive-edit-nudge (advisory example, unregistered) │ ├── workflows/ # ci.yml, grader.yml, correctness.yml, security.yml, deploy-dev.yml +│ │ # + deploy-promote.yml (the deploy rail's second half: dev→test→prod, +│ │ # manual only, human go/no-go every time — §7) │ │ # (+ eval-regression.yml, eval-suite.yml for agentic specs — §11) │ ├── packs/ # composable additions: stacks/dotnet, cicd/github, cicd/azure-devops, │ │ # frontend/generic, frontend/react, tools/gitnexus diff --git a/GOLD-STANDARD.md b/GOLD-STANDARD.md index dd80c43..5c50294 100644 --- a/GOLD-STANDARD.md +++ b/GOLD-STANDARD.md @@ -443,6 +443,18 @@ change. never in specs. The Anthropic API key is client-procured (section 8). - **Environments:** merge -> dev (automatic), dev -> test (on demand, smoke-tested), test -> prod (Phase 8 ceremony and thereafter on the client's release cadence, human go/no-go every time). + Two workflows, deliberately separate: `deploy-dev` is automatic and unattended; `deploy-promote` + is manual-trigger only and cannot run without a named approver, because promotion beyond dev is + the standard's most protected stop. The go/no-go is the target environment's own approval + mechanism (GitHub required reviewers / Azure DevOps environment checks) rather than anything + hand-rolled — the client's security team can already audit it, and `deploy-promote` refuses to + run against an environment that has no approver configured. Neither workflow rebuilds: both ship + the exact artifact a named CI run produced, and a promotion is rejected unless the source + environment has already run those same bytes. +- **Rollback:** every deploy captures the last known-good version and restores it on failure. The + human path — the deploy that succeeded and went wrong an hour later — is written down in advance + in `ROLLBACK.md` (`kit/rollback-template.md`), including what a rollback does **not** undo, and + is proven by the client's own operators rehearsing deploy -> roll back -> redeploy in test. --- @@ -541,6 +553,9 @@ intent-driven-development/ # cloned locally as delivery-standard/ on some ma │ ├── CLAUDE.md.template │ ├── spec-template.md │ ├── spike-template.md # the written finding a spike leaves behind (§5.3a) +│ ├── rollback-template.md # Phase 8: the written "roll back if X" + rehearsal record +│ ├── alert-definitions-template.md # Phase 9: baselines, thresholds, who is woken +│ ├── incident-playbook-template.md # Phase 9: detect/diagnose/escalate/communicate │ ├── settings.json │ ├── mcp.json # team MCP server set; packs merge additions │ ├── HARNESS.md # developer-facing tour; installs to docs/harness.md @@ -550,6 +565,8 @@ intent-driven-development/ # cloned locally as delivery-standard/ on some ma │ ├── hooks/ # stop-gate, review-gate, save-review-receipt (.ps1 + .sh each) │ │ # + sensitive-edit-nudge (advisory example, unregistered) │ ├── workflows/ # ci.yml, grader.yml, correctness.yml, security.yml, deploy-dev.yml +│ │ # + deploy-promote.yml (the deploy rail's second half: dev→test→prod, +│ │ # manual only, human go/no-go every time — §7) │ │ # (+ eval-regression.yml, eval-suite.yml for agentic specs — §11) │ ├── packs/ # composable additions: stacks/dotnet, cicd/github, cicd/azure-devops, │ │ # frontend/generic, frontend/react, tools/gitnexus diff --git a/docs/companion/the-rails.html b/docs/companion/the-rails.html index 3e8b931..4f6411c 100644 --- a/docs/companion/the-rails.html +++ b/docs/companion/the-rails.html @@ -367,6 +367,7 @@

Five checks — and knowing which ones can say "no"

correctnessevery PR that changes sourceBlocks on a high-confidence defectA fresh AI agent — separate from the grader — hunts the changed lines for plain logic defects: the bug class ci can't see (it compiles, the tests pass) and security doesn't look for (it's not exploitable, just wrong). Passes trivially when no source changed; a named human can override on the record. securitythe risk:high label or any PR touching a registered gated pathBlocks on HIGH; advises otherwiseRuns the security-reviewer agent. Path-triggered: fires on any PR touching a guarded path (auth, migrations, the pipeline, infra), independent of the spec's tier. deploy-devmerge to mainn/a (it ships)Deploys the merged artifact to the client's dev environment, and restores the last good version when a deploy fails. + deploy-promotemanual only — never a triggern/a (it ships)Promotes that same artifact onward to test and production. Holds until a named approver signs, and refuses a target environment that has no approver configured. Restores the last good version when a promotion fails.

The grader advises; it never blocks. It is tempting to let a confident AI @@ -1690,6 +1691,7 @@

The five workflows

correctnessevery PR that changes sourceBlocks on a high-confidence defectA fresh AI agent, separate from the grader, hunts the changed lines for plain logic defects — the bug class ci can't see (it compiles, the tests pass) and security doesn't look for (just wrong, not exploitable). Trivial pass when no source changed; named-human override on the record. securitythe risk:high label or any PR touching a registered gated pathBlocks on HIGH; advises otherwiseRuns the security-reviewer agent. Path-triggered: fires on any PR touching a guarded path (auth, migrations, the pipeline, infra), independent of the spec's tier. deploy-devmerge to mainn/a (it ships)Deploys the merged artifact to the client's dev environment, and restores the last good version when a deploy fails. + deploy-promotemanual only — never a triggern/a (it ships)Promotes that same artifact onward to test and production. Holds until a named approver signs, and refuses a target environment that has no approver configured. Restores the last good version when a promotion fails.

The grader advises; it never blocks — a polished, plausible @@ -1740,8 +1742,8 @@

Deploy and promotion

StepHowGate Merge → devThe deploy-dev workflow ships every merged change to devAutomatic — the merge already cleared the bar - Dev → testPromoted deliberately, smoke-tested on arrivalOn demand; the test environment is added at the first hardening pass - Test → prodThe first promotion is the Phase 8 go/no-go; every promotion after rides the client's release cadenceA human go/no-go every time — the single most protected stop in the standard + Dev → testThe deploy-promote workflow, run by hand, smoke-tested on arrivalOn demand — and it will not start until a named approver signs + Test → prodThe same deploy-promote workflow. The first promotion is the Phase 8 go/no-go; every one after rides the client's release cadenceA human go/no-go every time — the single most protected stop in the standard, and enforced by the environment's own approval rule rather than by convention

Two rules govern every promotion:

diff --git a/docs/phase-8-deployment.md b/docs/phase-8-deployment.md index c4001bf..f054679 100644 --- a/docs/phase-8-deployment.md +++ b/docs/phase-8-deployment.md @@ -218,7 +218,7 @@ change-control board adds latency. | ------------------------------ | -------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------ | | Release notes | Claude (from the merged specs) | Pod Lead | Readable by people who never saw a spec: features, fixes, limitations, what changes for users | | Deployment checklist | Claude (from the proven RUNBOOK) | Setup Owner | Every step ordered, observable, and verified in rehearsal; sign-off lines for the ceremony roles | -| Rollback procedure + evidence | Claude (drafts), client ops (executes) | Setup Owner | Executed in test by the client's own operators — deploy, roll back, redeploy — with the trigger condition written down | +| Rollback procedure + evidence | Claude (drafts from `kit/rollback-template.md`), client ops (executes) | Setup Owner | Executed in test by the client's own operators — deploy, roll back, redeploy — with the trigger condition written down, and what a rollback does **not** undo answered for this release | | Rollout-shape decision | Product owner + operations | Product Owner | Cutover/pilot/parallel chosen; in-flight work answered; fallback and trigger conditions recorded | | Smoke results (test + prod) | Claude (runs), Quality Engineer (owns) | Quality Engineer | One non-destructive journey per top-priority story, green in both environments, results recorded per test | | The go/no-go record | Pod Lead | Pod Lead | Every named role asked and answered, decision and rationale recorded — the durable proof a human said go | diff --git a/docs/phase-9-monitoring.md b/docs/phase-9-monitoring.md index f0264fe..8b32752 100644 --- a/docs/phase-9-monitoring.md +++ b/docs/phase-9-monitoring.md @@ -242,8 +242,8 @@ gate falls at hypercare's end, closing both together. | ------------------------- | -------------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------- | | Monitoring configuration | Claude (drafts), Setup Owner (wires) | Setup Owner → client | Dashboards live in the client's stack, each with a named owner; every top-priority feature observable | | Production baseline | Claude (measures) | Quality Engineer | Normal recorded per key metric with its measurement period; modeled values flagged with revisit dates | -| Alert definitions | Claude (drafts), ops (confirm) | Client operations | Every critical failure mode covered; every threshold derived from baseline and confirmed by the people being paged | -| Incident playbook | Claude (drafts), ops (correct) | Client operations | Detect-diagnose-escalate-communicate per alert; templates included; cross-referenced to the RUNBOOK | +| Alert definitions | Claude (drafts from `kit/alert-definitions-template.md`), ops (confirm) | Client operations | Every critical failure mode covered; every threshold derived from baseline and confirmed by the people being paged | +| Incident playbook | Claude (drafts from `kit/incident-playbook-template.md`), ops (correct) | Client operations | Detect-diagnose-escalate-communicate per alert; templates included; cross-referenced to the RUNBOOK | | The drill record | Quality Engineer | Quality Engineer | Every critical alert fired and answered by the client's on-call from the playbook; failures fixed and re-drilled | | Engagement retrospective | Claude (evidence base), humans (the candor) | Pod Lead | Product and process findings with receipts; debt log; harvest list — concrete items, not platitudes | | Close & Transfer handoff | Claude (drafts) | Pod Lead | Monitoring inventory, drill record, debt log, open items with owners | diff --git a/docs/the-rails.md b/docs/the-rails.md index 04f192b..98f3434 100644 --- a/docs/the-rails.md +++ b/docs/the-rails.md @@ -314,6 +314,36 @@ if X") written down in advance, not invented mid-incident. The rails are not pro being present; they are proven by a deploy failing and the rollback catching it. A rail that has never failed safely has not been proven (section 9). +The trigger belongs in `ROLLBACK.md` (from `kit/rollback-template.md`), written while nobody is +under pressure, because the person deciding at 2 a.m. should be *executing* a decision rather than +making one. That file also forces the question teams skip: **what a rollback does not undo.** Code +reverts cleanly; state does not. A release carrying a destructive schema migration, a one-way data +transform, or a published message cannot simply be reversed, and the restored version may not +understand the data it now finds. If that question has no answer at the Phase 8 go/no-go, the +release is not ready to promote — that is a finding, not a footnote. + +### The two halves of the deploy rail + +Deploy is **two** workflows, and the split is the point: + +- `deploy-dev` is automatic and unattended. It fires on a successful CI run on the protected + branch and needs no human, because the merge bar has already been cleared. +- `deploy-promote` is manual-trigger only. It cannot fire on its own, and it holds until a named + person approves. + +The go/no-go is the target environment's **own** approval mechanism — GitHub Environment required +reviewers, Azure DevOps environment checks — not logic invented in a workflow file. Two reasons: +the client's security team can already audit it, and it cannot be quietly edited away without +branch protection noticing. `deploy-promote` refuses to run against a target environment that has +no approver configured, so "we forgot to set that up" fails loudly instead of silently promoting +to production unattended. + +Both rules above are enforced mechanically, not by convention. Promotion is rejected unless the +named CI run succeeded, ran on the protected branch, *and* the source environment has already run +those exact bytes — which is what stops someone promoting a green build straight to production +having skipped test entirely. The pipeline would otherwise happily oblige, and nothing would say +so. + --- ## 6. The infrastructure pipeline @@ -458,6 +488,10 @@ deliberately and caught: exact line, record the override label, watch it go green, close it unmerged. A blocking gate is only proven when both its block and its escape have been seen to work. - A known-bad deploy proves the **pipeline restores** the last good version. +- An attempted promotion proves the **go/no-go actually holds**: the run must pause for a named + approver, and a build that has only reached dev must be *refused* a promotion straight to prod. + A promotion path that sails through unapproved, or that lets you skip an environment, is not a + gate — and both failures look exactly like success until someone tries them. - A probe PR touching a guarded path proves the **security gate** fires — a throwaway change opened solely to confirm the gate triggers, then closed unmerged. @@ -495,6 +529,13 @@ the provenance trail is what makes the rails auditable rather than merely automa - **The rollback that was only ever written.** Documented, reviewed, never run — then executed for the first time during an incident, where every surprise costs downtime. Rehearse it in test, before it is needed. +- **The promotion gate nobody configured.** The promote pipeline exists, the environment exists, + and the environment has no approver on it — so every promotion sails straight through to + production with a green tick and no human in the loop. It looks identical to a working gate + right up until it matters, which is why the pipeline refuses to run rather than assuming. +- **The skipped environment.** A green build promoted from dev directly to prod because the + operator picked the wrong target and nothing checked. "The same build that passed test" is only + true if something enforces that it actually passed test. - **A secret in the repo.** The one unrecoverable foundation mistake. The client's vault from day one — never in code, never in CLAUDE.md, never in a spec. - **The unattended destructive apply.** An agent runs an infrastructure `apply` that deletes or diff --git a/docs/whats-installed.md b/docs/whats-installed.md index fad93ec..4fc32e6 100644 --- a/docs/whats-installed.md +++ b/docs/whats-installed.md @@ -131,8 +131,19 @@ override. Branch protection tops it off: blocking checks are mandatory and a per didn't write the change must approve it. In one sentence: *machines verify the facts; a human makes the call.* +Getting a change into an environment is two workflows, deliberately not one. Merging ships to +**dev** automatically, with no human in the loop, because the change already cleared the bar +above. Going any further — to test, then to production — is a separate workflow that cannot +start on its own and stops until a **named person approves**. Neither one rebuilds anything: both +ship the exact package the build produced, so what reaches production is the thing that was +tested rather than a fresh copy nobody has seen. And a promotion is refused unless the previous +environment has already run those same bytes, so nobody can skip test by accident. If a deploy +goes wrong, the previous good version is restored automatically; the decision to undo a deploy +that went wrong *later* is written down in advance, in `ROLLBACK.md`, rather than improvised at +3 a.m. + Before trusting any gate, run the drills in `RAILS.md`: break the build on purpose, plant a -defect, commit a fake secret — and watch each gate catch it. +defect, commit a fake secret, attempt a promotion nobody approved — and watch each gate catch it. ## Where the pieces come from (custody chain) diff --git a/kit/README.md b/kit/README.md index 30094ca..3a3c2d7 100644 --- a/kit/README.md +++ b/kit/README.md @@ -50,6 +50,9 @@ so install here unless you also repoint the references. | `CLAUDE.md.template` | `./CLAUDE.md` | Replace every `{{TOKEN}}`; delete guidance comments. | | `spec-template.md` | `./specs/spec-template.md` | Copy per feature to `specs/NNNN-name.md`. | | `spike-template.md` | `./spikes/spike-template.md` | Copy per unknown to `spikes/NNNN-name.md`. The finding is committed; the `spike/` branch never merges. | +| `rollback-template.md` | `./ROLLBACK.md` | Phase 8. The **written** trigger ("roll back if X"), the procedure, what a rollback does NOT undo, and the rehearsal record. Proven by the client's operators executing deploy → roll back → redeploy in test. | +| `alert-definitions-template.md` | `./ALERTS.md` | Phase 9. One entry per alert: measured baseline, warning/critical thresholds, who is woken, the drill record, and the standing fatigue review. | +| `incident-playbook-template.md` | `./INCIDENT-PLAYBOOK.md` | Phase 9. Detect → diagnose → escalate → communicate, per alert, plus the user-facing message templates. The RUNBOOK resolves; this detects and communicates. | | `settings.json` | `./.claude/settings.json` | Shared, committed. Leans on `deny` (see below). | | `mcp.json` | `./.mcp.json` | Team MCP servers (context7, sequential-thinking, playwright); packs merge additions (dotnet → microsoft-learn, github → github, azure-devops → azure-devops). npx-launched servers are version-pinned; the HTTP-hosted ones (context7, microsoft-learn, github) run server-side and cannot be pinned. No secrets — auth is always per-developer. Each developer approves the set once on first open. | | `HARNESS.md` | `./docs/harness.md` | The developer-facing tour: what each installed piece does and why, per layer. Point new team members here first. | @@ -58,6 +61,7 @@ so install here unless you also repoint the references. | `agents/*` | `./.claude/agents/` | `planner`, `architect`, `grader`, `security-reviewer`, `build-error-resolver`, `debugger` — model-tiered; see `agents/README.md`. | | `skills/*` | `./.claude/skills/` | `spec-writer`, `test-writer`, `api-pattern`, `pr-writer`, `eval-builder`, `diagnose`. | | `workflows/{ci,grader,correctness,security,deploy-dev,eval-regression,eval-suite}.yml` | `./.github/workflows/` | The five rails + the two eval workflows. | +| `workflows/deploy-promote.yml` | `./.github/workflows/` | The second half of the deploy rail: dev→test→prod. Manual trigger only — the target Environment's **required reviewers** are the human go/no-go, and the workflow refuses to run against an environment that has none. | | `workflows/RAILS.md` | `./.github/RAILS.md` | Operator's guide + shakedown drills. | | `profile/rubrics/*` | `./.github/profile/rubrics/` | Workflows read these by this path. | | `profile/eval-bypasses.md` | `./.github/eval-bypasses.md` | Override/bypass ledger. | diff --git a/kit/alert-definitions-template.md b/kit/alert-definitions-template.md new file mode 100644 index 0000000..af75620 --- /dev/null +++ b/kit/alert-definitions-template.md @@ -0,0 +1,120 @@ + + +# ALERTS — {{system name}} + +- **Owner:** {{Setup Owner during the engagement → client operations at transfer}} +- **Where these live:** {{Azure Monitor / Grafana / the client's tooling — and the config path}} +- **Paging channel:** {{how a critical alert reaches a human at 3 a.m.}} +- **Last fatigue review:** {{YYYY-MM-DD}} — see §Fatigue review +- **Related:** `INCIDENT-PLAYBOOK.md` (what to DO) · `RUNBOOK.md` (how to FIX) · `ROLLBACK.md` + +--- + +## Severity, and what it means for a human + +| Level | Means | Response | +| --- | --- | --- | +| **Critical** | {{someone is woken, now}} | {{acknowledge within N minutes, from the playbook}} | +| **Warning** | {{investigate during working hours}} | {{picked up next working day at the latest}} | + +The difference is **who suffers if it waits until morning** — not how alarming the number looks. + +--- + +## Coverage check + + + +| Top-priority feature / journey | Alert(s) covering it | Gap? | +| --- | --- | --- | +| {{journey}} | {{alert name(s)}} | {{none \| named gap + owner + date}} | + +--- + +## Alert: {{name}} + + + +- **Failure mode it detects:** {{the thing that is actually going wrong, in plain words — + "the claims API is rejecting valid submissions", not "HTTP 500 rate elevated"}} +- **Who is woken:** {{named rota, not a team alias — an alias is how everyone assumes someone + else has it}} +- **Signal:** {{the metric/query, exactly as configured}} +- **Evaluation window:** {{e.g. 5 minutes, 3 consecutive periods — the window is half the + threshold; a spiky metric with a short window is a pager that cries wolf}} + +| | Value | Where it came from | +| --- | --- | --- | +| **Baseline (normal)** | {{value}} | {{measured {{period}} of real production traffic \| MODELED — revisit by {{date}}}} | +| **Warning threshold** | {{value}} | {{how derived from the baseline}} | +| **Critical threshold** | {{value}} | {{how derived from the baseline}} | + +- **First response:** `INCIDENT-PLAYBOOK.md` → {{entry name}} +- **Does this trigger a rollback?** {{no \| yes — it is trigger «N» in `ROLLBACK.md` §1}} +- **Known false-positive causes:** {{e.g. the nightly batch window — and what was done about + it, because "we know it fires then" is not a mitigation}} + +### Drill record + + + +| Date | Fired how | Answered by | From the playbook alone? | What we fixed afterwards | +| --- | --- | --- | --- | --- | +| {{YYYY-MM-DD}} | {{how it was triggered}} | {{name, client on-call}} | {{yes / no}} | {{playbook gap, threshold change, or alert deleted}} | + +--- + +## Fatigue review + + + +- **Cadence:** {{e.g. monthly, at the operations review}} +- **Rule applied:** fires more than once a week without action → raise the threshold or delete it. + +| Alert | Fires / week | Acted on? | Decision | Date | +| --- | --- | --- | --- | --- | +| {{name}} | {{n}} | {{yes / no}} | {{kept \| threshold raised to X \| DELETED}} | {{YYYY-MM-DD}} | + +--- + +## Gate checklist + +- [ ] Every critical failure mode from the RUNBOOK's failure scenarios has an alert here +- [ ] Every threshold is derived from a baseline, and every modeled value carries a revisit date +- [ ] Every alert names a **person or rota**, confirmed by the people actually being paged +- [ ] Every critical alert has been fired in the drill and answered from the playbook +- [ ] Every alert has a first-response entry in `INCIDENT-PLAYBOOK.md` +- [ ] No alert on this list fires routinely without anyone acting on it diff --git a/kit/incident-playbook-template.md b/kit/incident-playbook-template.md new file mode 100644 index 0000000..44347cf --- /dev/null +++ b/kit/incident-playbook-template.md @@ -0,0 +1,136 @@ + + +# INCIDENT PLAYBOOK — {{system name}} + +- **Owner:** {{Pod Lead during the engagement → client operations at transfer}} +- **On-call rota:** {{where it lives}} +- **Status page / user comms channel:** {{where users are told}} +- **Related:** `ALERTS.md` (what fires) · `RUNBOOK.md` (how to fix) · `ROLLBACK.md` (how to undo) + +--- + +## The first five minutes + + + +1. **Acknowledge** the page — {{how}}. This stops it escalating and tells everyone it is owned. +2. **Say you have it** — post in {{channel}}: *"Investigating {{alert}}. Next update in 15 min."* +3. **Assess severity** using the table below. When torn between two levels, take the higher one. +4. **Find the alert's entry** in this file and work it. +5. **Set a 15-minute update clock.** Silence during an incident reads as nothing happening. + +**You are allowed to roll back before you understand the cause.** If a trigger in +`ROLLBACK.md` §1 is met, execute it — diagnosis can happen once service is restored. Understanding +the failure is not a prerequisite for stopping it. + +--- + +## Severity + +| Level | Looks like | Tell users? | Escalate | +| --- | --- | --- | --- | +| **SEV1** | {{users cannot do the core journey; data at risk}} | {{yes, immediately}} | {{who, straight away}} | +| **SEV2** | {{degraded — slow, partial, one feature down}} | {{yes if > N minutes}} | {{who, after N minutes}} | +| **SEV3** | {{noticeable internally, users unaffected}} | {{no}} | {{next working day}} | + +--- + +## Playbook: {{alert name}} + + + +**Detect** — {{what the responder actually sees: the alert text, and what it looks like on the +dashboard. Include what a FALSE positive looks like, if there is a known one.}} + +**Diagnose** — first three checks, in order. Stop when one explains it. + +1. {{check}} → `{{exact command or dashboard link}}` → {{what a normal answer looks like}} +2. {{check}} → `{{...}}` → {{...}} +3. {{check}} → `{{...}}` → {{...}} + +*If none of these explain it:* {{where to look next, or who to wake — do not leave this blank, +"keep digging" is not a step}} + +**Likely causes, most common first** + +| Cause | Confirms it | Fix | +| --- | --- | --- | +| {{cause}} | {{the observation that confirms}} | `RUNBOOK.md` → {{procedure name}} | +| {{cause}} | {{...}} | {{`ROLLBACK.md` if the fix is to undo the release}} | + +**Escalate** — to {{named role}} at {{contact}}. If no response in {{N}} minutes, {{second +contact}}. Escalating early is not a failure; a quiet incident that runs long is worse than a +noisy one that ends. + +**Communicate** — {{which template below}} · audience {{who}} · cadence {{every N minutes}} + +--- + +## What to tell users + + + +**Initial (within {{N}} minutes of a SEV1/SEV2)** + +> We're aware of an issue affecting {{plain-language description of what users cannot do}}. +> We're investigating and will update by {{time}}. +> {{Workaround, if there is one — otherwise delete this line rather than inventing one.}} + +**Holding update (every {{N}} minutes, even with no news)** + +> We're still working on {{issue}}. {{What is known, in plain terms.}} Next update by {{time}}. + +**Resolved** + +> {{Issue}} was resolved at {{time}}. {{What users should do now — re-submit, refresh, nothing.}} +> {{If data was affected, say so plainly and say what happens next. Do not bury it.}} + +--- + +## After it ends + + + +- [ ] Timeline captured while it is fresh — detection time, response time, resolution time +- [ ] **Which check should have caught this?** → {{the concrete harness/alert/test change}} +- [ ] Was there an alert? {{yes — it worked \| yes — too late, threshold changed to X \| NO — new alert added to `ALERTS.md`}} +- [ ] Did this playbook entry hold up? {{gaps found → edit THIS file now, not later}} +- [ ] `RUNBOOK.md` procedure accurate? {{corrections applied}} +- [ ] Escaped-bug entry raised for Retro+ (docs/build-loop.md) +- [ ] User comms sent and accurate in hindsight + +--- + +## Drill record + + + +| Date | Alert drilled | Responder | Worked from this file alone? | What we rewrote | +| --- | --- | --- | --- | --- | +| {{YYYY-MM-DD}} | {{alert}} | {{name, client on-call}} | {{yes / no}} | {{the gap — a drill that found nothing usually means it was narrated}} | diff --git a/kit/packs/cicd/azure-devops/RAILS.md b/kit/packs/cicd/azure-devops/RAILS.md index 93317ac..7fef765 100644 --- a/kit/packs/cicd/azure-devops/RAILS.md +++ b/kit/packs/cicd/azure-devops/RAILS.md @@ -21,6 +21,7 @@ and — the part most teams skip — how to **prove they actually catch things** | **correctness-review** | `correctness.yml` | every PR; reviews when source changed | **Blocks** on a high-confidence defect | | **security-review** | `security.yml` | every PR; reviews on gated paths / `risk:high` | **Blocks** on HIGH | | **deploy-dev** | `deploy-dev.yml` | successful CI on main (merge) | n/a — it ships; rolls back on failure | +| **deploy-promote** | `deploy-promote.yml` | **manual only** — never a trigger | n/a — it ships to test/prod once the target Environment's approval check is signed; rolls back on failure | | **eval-regression** | `eval-regression.yml` | PRs touching the agentic surface | **Blocks** on degradation | | **eval-suite** | `eval-suite.yml` | manual + scheduled | **Advises** — never gates PRs | @@ -78,9 +79,19 @@ security" without leaving low-risk PRs stuck. ``` This is the only sanctioned way to change branch policy — edit the JSON, re-run the script. Do not hand-edit policies in the Azure DevOps UI. -7. **Wire and rehearse `deploy-dev`.** It ships as a STARTER that fails until its placeholder - deploy/rollback steps are adapted. Create a dev **Environment** (attach approvals/checks for - promotion beyond dev), wire the steps, then rehearse the rollback (§9 shakedown) before trusting it. +7. **Wire and rehearse `deploy-dev`.** It ships as a STARTER whose placeholder deploy/rollback steps + must be adapted. Until they are, the job runs, warns that deploy is not wired, and stops — a job + red on every merge by design teaches the team that red is normal. Create a dev **Environment**, + wire the steps, set the pipeline variable `DEPLOY_WIRED = true`, then rehearse the rollback + (§9 shakedown) before trusting it. +8. **Put an approval check on every promotion target, then wire `deploy-promote`.** Pipelines → + Environments → `test` / `prod` → Approvals and checks → Approvals. That approval **is** the + go/no-go; without it, promotion beyond dev is automatic. Unlike `deploy-dev`, `deploy-promote` + **fails** rather than warns when unwired — a human asked for the promotion and is waiting. + Note the coupling: `deploy-dev` tags the CI build `deployed-dev` on success, and + `deploy-promote` refuses to promote a build that does not carry its source environment's tag. + Remove that tagging step and dev becomes a dead end — nothing will ever be promotable. + `System.AccessToken` therefore needs build **tag write** as well as build read. ## Solo-repo accommodation (read this) @@ -136,6 +147,16 @@ when **both its block and its escape** have been seen to work. at a failing build). The deployment must fail and the `on.failure` steps must **restore the last known-good version** — the rollback the rails rehearse. Run deploy → roll back → redeploy against the dev Environment, with the rollback trigger condition written down in advance, not invented mid-incident. +- **deploy-promote** — three drills, and the first two are the ones people skip: + 1. **The gate holds.** Run a promotion. It must **pause** on the target Environment's approval + check and not proceed until a named person signs. If it sails through, no approval is + configured and promotion is automatic — the standard's most protected stop, silently absent. + 2. **You cannot skip an environment.** Try to promote a build straight to `prod` that carries only + the `deployed-dev` tag. The preflight must **refuse** it. This also proves the tagging step in + `deploy-dev` actually ran — if nothing is promotable at all, that step failed silently. + 3. **The rollback still works up here.** Repeat the known-bad deploy against **test** via + `deploy-promote`, executed by the client's own operators with their own permissions — the + Phase 8 rehearsal, run before prod is ever a target. - **eval-regression** — open a PR touching `prompts/**` that degrades a key metric past the trip-wire (or point the runner at a fixture that regresses). The `eval-regression` build validation must go red. - **secret scan** — open a throwaway PR that commits a **fake but realistic credential** (e.g. an invented diff --git a/kit/packs/cicd/azure-devops/README.md b/kit/packs/cicd/azure-devops/README.md index 5624f53..f6e8f96 100644 --- a/kit/packs/cicd/azure-devops/README.md +++ b/kit/packs/cicd/azure-devops/README.md @@ -19,6 +19,7 @@ that replaces GitHub branch protection, and the scripts that stand in for GitHub | **Correctness** | `azure-pipelines/correctness.yml` | every PR (reviews when source changed) | **BLOCKS** on a high-confidence defect (label override) | | **Security** | `azure-pipelines/security.yml` | every PR (reviews on gated paths / `risk:high`) | **BLOCKS** on HIGH | | **Deploy Dev** | `azure-pipelines/deploy-dev.yml` | successful CI on main (merge) | ships; **rolls back** on failure — **STARTER** | +| **Deploy Promote** | `azure-pipelines/deploy-promote.yml` | **manual only** (Run pipeline) | ships to test/prod once the target Environment's approval check is signed; **rolls back** on failure — **STARTER** | | **Eval regression** | `azure-pipelines/eval-regression.yml` | PRs touching the agentic surface | **BLOCKS** on degradation | | **Eval suite** | `azure-pipelines/eval-suite.yml` | manual + scheduled | **ADVISORY** — never gates PRs | diff --git a/kit/packs/cicd/azure-devops/azure-pipelines/deploy-dev.yml b/kit/packs/cicd/azure-devops/azure-pipelines/deploy-dev.yml index be804b9..a67ecba 100644 --- a/kit/packs/cicd/azure-devops/azure-pipelines/deploy-dev.yml +++ b/kit/packs/cicd/azure-devops/azure-pipelines/deploy-dev.yml @@ -29,6 +29,22 @@ # <> / <> / <> / <> the real deploy, # health probe, live-version capture, and rollback for your platform. # ─────────────────────────────────────────────────────────────────────────────── +# +# ── WHY THIS DOES NOT FAIL UNTIL ADAPTED ─────────────────────────────────────── +# This starter used to `exit 1` at the deploy step, so a fresh install had a permanently-red +# post-merge job until someone adapted it. That was the wrong place to apply the pressure: a job +# that is red on every merge by design teaches the team that red is normal, which degrades the +# signal on every OTHER job too. +# +# The adaptation is enforced where it belongs — Phase 8's Step 0 go/no-go and its Step 1 rollback +# verification ("deploy, then roll back, then redeploy"), both human-gated. That is a stronger +# check than a red tick nobody reads. +# +# So until you wire it: this job runs, says loudly that deploy is not wired, and stops. Set the +# pipeline variable DEPLOY_WIRED = true (Pipeline → Edit → Variables) once <>, +# <> and the last-known-good steps are real. An unset variable reads as empty and +# every real step below stays skipped. +# ─────────────────────────────────────────────────────────────────────────────── trigger: none pr: none @@ -59,10 +75,18 @@ jobs: runOnce: deploy: steps: + # ── Is the deploy actually wired yet? (see the header note) ────────────────── + - bash: | + set -euo pipefail + echo "##vso[task.logissue type=warning]This is the starter deploy-dev pipeline. Its deploy, health-check and rollback steps are still placeholders, so nothing was deployed. Wire them (the-rails.md §5), rehearse deploy -> roll back -> redeploy (Phase 8 Step 1), then set the pipeline variable DEPLOY_WIRED = true." + displayName: 'Deploy is not wired — nothing shipped' + condition: and(succeeded(), ne(variables['DEPLOY_WIRED'], 'true')) + # ── Promote, never rebuild: fetch the artifact CI already produced, from the TRIGGERING run. - download: ci # <> artifact: '<>' # the proven bytes CI published (CI must publish a deployable, not just coverage) displayName: 'Download the CI-built artifact' + condition: and(succeeded(), eq(variables['DEPLOY_WIRED'], 'true')) # ── Record the current live version so we can put it back on failure. - bash: | @@ -75,6 +99,7 @@ jobs: echo "##vso[task.setvariable variable=previousVersion]$PREV" echo "Last known-good version: $PREV" displayName: 'Capture last known-good version' + condition: and(succeeded(), eq(variables['DEPLOY_WIRED'], 'true')) # ── Deploy the promoted artifact. - bash: | @@ -83,9 +108,13 @@ jobs: # Replace with AzureWebApp@1 / AzureCLI@2 (`az deployment` / `az containerapp update`) / # kubectl. Must exit non-zero on a failed deploy so on.failure fires. echo "Deploying $(Pipeline.Workspace)/ci/<> to dev..." - echo "##vso[task.logissue type=error]<> not yet wired — this starter intentionally fails until adapted." + # Reaching this line means DEPLOY_WIRED=true was set while <> is still a + # placeholder. Failing is right here: the repo has DECLARED the deploy wired, so a silent + # no-op would be the pipeline lying about having shipped something. + echo "##vso[task.logissue type=error]DEPLOY_WIRED is set, but <> is still a placeholder. Either wire the real deploy or unset DEPLOY_WIRED." exit 1 displayName: 'Deploy to dev' + condition: and(succeeded(), eq(variables['DEPLOY_WIRED'], 'true')) # ── Prove the new version actually serves. - bash: | @@ -95,6 +124,36 @@ jobs: # curl --fail --retry 5 --retry-delay 10 "https://<>/health" echo "Health check placeholder — wire <>." displayName: 'Health check' + condition: and(succeeded(), eq(variables['DEPLOY_WIRED'], 'true')) + + # ── Record, on the CI BUILD ITSELF, that dev now runs these bytes. + # deploy-promote.yml refuses to promote a build that does not carry the tag for its + # source environment — "you cannot promote what the previous environment never ran". + # This step is what earns that tag, so REMOVING IT MAKES DEV A DEAD END: nothing would + # ever be promotable to test. + # + # Deliberately the REST API, NOT `##vso[build.addbuildtag]`. That logging command tags + # the RUN THAT EMITS IT — this deploy run — whereas the fact being recorded is about the + # CI build whose artifact went live. Tagging the wrong build fails silently: the tag + # exists, just never where the promotion looks for it. + # + # Last on the happy path, and gated on succeeded(): a build that failed its deploy or + # health check must NOT be marked as running in dev. + - bash: | + set -euo pipefail + API="${SYSTEM_COLLECTIONURI}${SYSTEM_TEAMPROJECT}/_apis" + if ! curl -sf -X PUT -H "Authorization: Bearer ${SYSTEM_ACCESSTOKEN}" \ + "${API}/build/builds/${CI_RUN_ID}/tags/deployed-dev?api-version=7.1" \ + -o /dev/null; then + echo "##vso[task.logissue type=error]Deployed to dev, but could not tag build ${CI_RUN_ID} as deployed-dev. The deploy LANDED; the record of it did not, so this build cannot be promoted to test until fixed — check System.AccessToken has build tag write access." + exit 1 + fi + echo "Tagged build ${CI_RUN_ID} as deployed-dev." + displayName: 'Record the deployment on the CI build' + condition: and(succeeded(), eq(variables['DEPLOY_WIRED'], 'true')) + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + CI_RUN_ID: $(resources.pipeline.ci.runID) # ── Rollback: restore the last known-good version on ANY failure above. # A rollback that has never run is a wish. This path is REHEARSED in the §9 shakedown with a # deliberately known-bad deploy, before it is ever needed live. @@ -112,3 +171,13 @@ jobs: # rolled-back deploy is still a deploy that did not land. exit 1 displayName: 'Restore last known-good version (rollback)' + # Never fire for a run that never deployed — a 'rollback' of nothing is noise that + # would make the not-wired path look like a real incident. + # + # DELIBERATELY NOT `and(succeeded(), ...)`, unlike every condition above. This step + # lives in the `on.failure` hook, which by definition only runs when the deploy hook + # has ALREADY failed. Composing succeeded() here risks evaluating false against that + # failed state and skipping the rollback entirely — turning the one path that must + # never be skipped into the one path that silently is. An explicit condition replaces + # the default succeeded(), and here that replacement is the whole point. + condition: eq(variables['DEPLOY_WIRED'], 'true') diff --git a/kit/packs/cicd/azure-devops/azure-pipelines/deploy-promote.yml b/kit/packs/cicd/azure-devops/azure-pipelines/deploy-promote.yml new file mode 100644 index 0000000..c316d59 --- /dev/null +++ b/kit/packs/cicd/azure-devops/azure-pipelines/deploy-promote.yml @@ -0,0 +1,312 @@ +# ⚠️ STARTER PIPELINE — adapt before relying on it. ⚠️ +# The second half of the deploy rail. `deploy-dev.yml` owns merge → dev, which is automatic and has +# no human in the loop. THIS pipeline owns everything beyond dev — dev → test and test → prod — +# which is deliberate, human-gated, and never automatic (the-rails.md §5). +# +# Like its sibling, the deploy/rollback steps are placeholders you MUST wire to your platform and +# then REHEARSE (deploy → roll back → redeploy) in the TEST environment, by the hands that would run +# it at 2 a.m., before it is ever pointed at production (Phase 8's rehearsal week). +# +# ── PACK NOTE (azure-devops CI/CD pack) ──────────────────────────────────────── +# NO stack seam. Per §5 this pipeline does NOT build — it PROMOTES the exact artifact a named CI run +# already built, so no ci-profile command is consumed here. Its placeholders are deploy-platform / +# repo level, not stack level. Pairs with this pack's deploy-dev.yml; both halves of the deploy rail +# share the DEPLOY_WIRED pipeline variable. +# +# ── THE THREE §5 RULES THIS FILE EXISTS TO ENFORCE — do not undo them when adapting ───────── +# +# 1. PROMOTE THE ARTIFACT; NEVER REBUILD IT. This pipeline runs no build. It downloads the EXACT +# artifact a named CI run produced — the bytes dev, and then test, actually exercised — and +# ships those onward. "Rebuild it real quick for prod" puts something no environment ever +# verified into the one environment where that matters. +# +# 2. A HUMAN SAYS GO, EVERY TIME. Not just the first promotion. The go/no-go is the target +# Environment's APPROVALS AND CHECKS, not anything hand-rolled here: the run pauses at the +# `promote` deployment job until a named approver signs, and the approval is recorded in the +# environment's own deployment history. Using the platform's mechanism matters — the client's +# security team can already audit it. +# Configure it: Pipelines → Environments → → Approvals and checks → Approvals. +# An environment WITHOUT an approval check turns this pipeline into an automatic promotion, +# which is the standard's single most protected stop, silently removed. +# +# 3. A ROLLBACK THAT HAS NEVER RUN IS A WISH. The restore path is identical in shape to +# deploy-dev's, and it is rehearsed in TEST during Phase 8's rehearsal week — deploy, roll back, +# redeploy — with the trigger condition ("roll back if X") written down in advance in the repo's +# ROLLBACK.md, not invented mid-incident. +# +# ── YOU CANNOT PROMOTE WHAT THE PREVIOUS ENVIRONMENT NEVER RAN ────────────────────────────── +# The preflight refuses a promotion unless the named CI run succeeded, ran on the protected branch, +# AND carries the build tag its source-environment deploy stamped on it. +# +# WHY A BUILD TAG, WHERE THE GITHUB PACK QUERIES AN API: on GitHub, putting `environment:` on a job +# creates a first-class Deployment record queryable BY ENVIRONMENT NAME, so that pack simply asks +# "was this commit successfully deployed to dev?". Azure DevOps records deployments against an +# environment ID that must first be resolved by name, through a different service area. Rather than +# depend on that lookup, this pack has the deploy stamp an explicit, durable fact on the build +# itself: `deployed-`. Same governance rule, realized in the idiom each platform makes +# cheap and reliable — which is the entire point of the two-axis pack model. +# THIS REQUIRES the source-environment deploy to add the tag on success. deploy-dev.yml in this pack +# does exactly that; any promotion target you add must do the same or nothing can promote out of it. +# +# ── WHY THIS FAILS WHEN UNWIRED, WHERE deploy-dev ONLY WARNS ──────────────────────────────── +# deploy-dev fires automatically on every merge, so a red run there would be red constantly by design +# and would teach the team that red is normal. This pipeline only ever runs because a human +# deliberately asked for a promotion and is waiting on the result. Reporting success while having +# shipped nothing would be the pipeline lying to the one person who most needs the truth. So: +# unwired here is an ERROR, not a warning. +# +# ── PLACEHOLDERS (every one is adapt-per-repo) ───────────────────────────────── +# <> the pipeline resource alias for the CI pipeline. +# <> the CI pipeline's name in this project (must match exactly). +# <> the protected branch whose builds may be promoted (reference: main). +# <> the deployable artifact CI published. +# <> the Environment names in the `values:` lists below (reference: dev, +# test, prod) — must match your Azure DevOps Environment names exactly. +# <> / <> / <> / <> / <> +# the real deploy, health probe, journey smoke test, live-version capture, +# and rollback for your platform. +# ─────────────────────────────────────────────────────────────────────────────── + +trigger: none +pr: none + +# Runtime parameters — the "Run pipeline" dialog is the operator's entry point. A promotion is always +# somebody deciding to promote; there is no trigger here by design. +parameters: + - name: sourceEnvironment + displayName: 'Promote FROM (must already be running these bytes)' + type: string + default: dev + values: [dev, test] # <> + - name: targetEnvironment + displayName: 'Promote TO (its approval check is the go/no-go)' + type: string + default: test + values: [test, prod] # <> + - name: ciRunId + displayName: 'CI run id whose artifact is promoted — the proven bytes, not a rebuild' + type: string + +pool: + vmImage: 'ubuntu-latest' + +# The CI pipeline as a resource so its artifact can be downloaded by run id. NOTE the absence of a +# `trigger:` block — unlike deploy-dev.yml, nothing about this pipeline is automatic. +resources: + pipelines: + - pipeline: ci # <> + source: 'CI' # <> — must match the CI pipeline's name exactly + +stages: + # ── Everything checkable BEFORE a human is asked to approve ──────────────────────────────── + # Deliberately its own stage with no environment, so it runs immediately. Asking a named person to + # approve and only then discovering the run id was wrong wastes the scarcest thing in the ceremony + # — their attention — and trains people to approve on autopilot. + - stage: preflight + displayName: 'Promotion preflight' + jobs: + - job: promotion_preflight + displayName: 'promotion-preflight' + steps: + - checkout: none + + - bash: | + set -euo pipefail + + if [ "${SOURCE_ENV}" = "${TARGET_ENV}" ]; then + echo "##vso[task.logissue type=error]Source and target are both '${SOURCE_ENV}'. A promotion moves bytes between environments." + exit 1 + fi + + API="${SYSTEM_COLLECTIONURI}${SYSTEM_TEAMPROJECT}/_apis" + AUTH="Authorization: Bearer ${SYSTEM_ACCESSTOKEN}" + + # ── The build being promoted must be a SUCCESSFUL CI build of the protected branch. + # A manually supplied run id is filtered by nothing else, so restate here the same gate + # deploy-dev gets from its completion trigger. + if ! BUILD="$(curl -sf -H "$AUTH" "${API}/build/builds/${CI_RUN_ID}?api-version=7.1")"; then + echo "##vso[task.logissue type=error]Could not read build ${CI_RUN_ID}. Check the run id and that System.AccessToken has build read access." + exit 1 + fi + + RESULT="$(echo "$BUILD" | jq -r '.result')" + BRANCH="$(echo "$BUILD" | jq -r '.sourceBranch')" + PIPELINE="$(echo "$BUILD" | jq -r '.definition.name')" + COMMIT="$(echo "$BUILD" | jq -r '.sourceVersion')" + + if [ "$PIPELINE" != "CI" ]; then # <> + echo "##vso[task.logissue type=error]Build ${CI_RUN_ID} belongs to '${PIPELINE}', not the CI pipeline. Only a CI build publishes the deployable artifact." + exit 1 + fi + if [ "$RESULT" != "succeeded" ]; then + echo "##vso[task.logissue type=error]CI build ${CI_RUN_ID} result is '${RESULT}'. Only a successful build may be promoted." + exit 1 + fi + if [ "$BRANCH" != "refs/heads/main" ]; then # <> + echo "##vso[task.logissue type=error]CI build ${CI_RUN_ID} ran on '${BRANCH}', not the protected branch. Only the protected branch is promoted." + exit 1 + fi + + # ── The source environment must have actually deployed these bytes. See the header for + # why this is a build tag here and a Deployments API query in the github pack. + if ! TAGS="$(curl -sf -H "$AUTH" "${API}/build/builds/${CI_RUN_ID}/tags?api-version=7.1")"; then + echo "##vso[task.logissue type=error]Could not read the tags on build ${CI_RUN_ID}, so this pipeline cannot confirm ${SOURCE_ENV} ever ran it. Refusing to promote unverified." + exit 1 + fi + + WANTED="deployed-${SOURCE_ENV}" + if ! echo "$TAGS" | jq -e --arg t "$WANTED" '.value | index($t)' >/dev/null; then + echo "##vso[task.logissue type=error]Build ${CI_RUN_ID} carries no '${WANTED}' tag, so it has never been deployed to ${SOURCE_ENV}. Promote it there first — you cannot promote what the previous environment never ran." + exit 1 + fi + + echo "Verified: build ${CI_RUN_ID} (commit ${COMMIT}) succeeded on the protected branch and is deployed to ${SOURCE_ENV}." + echo "Promoting ${SOURCE_ENV} -> ${TARGET_ENV}." + displayName: 'Validate the build being promoted' + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + CI_RUN_ID: ${{ parameters.ciRunId }} + SOURCE_ENV: ${{ parameters.sourceEnvironment }} + TARGET_ENV: ${{ parameters.targetEnvironment }} + + # ── The promotion itself. PAUSES for the target Environment's approval check. ─────────────── + - stage: promote + displayName: 'Promote' + dependsOn: preflight + condition: succeeded() + jobs: + - deployment: promote + displayName: 'promote' + # THE GO/NO-GO. Azure DevOps holds the run here until the environment's approval check is + # signed, and records who signed it. This one line is the most important in the file — see + # rule 2 in the header before changing it. + environment: ${{ parameters.targetEnvironment }} + strategy: + runOnce: + deploy: + steps: + # ── Unwired is an ERROR here, not a warning — see the header for why this differs + # from deploy-dev. A human asked for this promotion and is waiting on the answer. + - bash: | + set -euo pipefail + if [ "${DEPLOY_WIRED:-}" != "true" ]; then + echo "##vso[task.logissue type=error]This is the starter deploy-promote pipeline — its deploy, health-check, smoke and rollback steps are still placeholders, so nothing was promoted. Wire them (the-rails.md §5), rehearse deploy -> roll back -> redeploy in test (Phase 8), then set the pipeline variable DEPLOY_WIRED = true." + exit 1 + fi + displayName: 'Check the deploy is wired' + env: + DEPLOY_WIRED: $(DEPLOY_WIRED) + + # ── Promote, never rebuild: the exact bytes the source environment ran. + - task: DownloadPipelineArtifact@2 + displayName: 'Download the promoted artifact' + inputs: + buildType: 'specific' + project: '$(System.TeamProject)' + definition: '$(resources.pipeline.ci.pipelineID)' + buildVersionToDownload: 'specific' + pipelineId: ${{ parameters.ciRunId }} + artifactName: '<>' + targetPath: '$(Pipeline.Workspace)/artifact' + + # ── Record the current live version so we can put it back on failure. + - bash: | + set -euo pipefail + # <> — record what is live in ${TARGET_ENV} RIGHT NOW (slot + # name, current revision, deployed image tag, release id...). Emit it so the + # rollback step can read it. Reference (Azure Web App slot model): + # PREV="$(az webapp deployment slot list ... --query '...' -o tsv)" + PREV="REPLACE_ME" + echo "##vso[task.setvariable variable=previousVersion]$PREV" + echo "Last known-good version in ${TARGET_ENV}: $PREV" + displayName: 'Capture last known-good version' + env: + TARGET_ENV: ${{ parameters.targetEnvironment }} + + # ── Deploy the promoted artifact. + - bash: | + set -euo pipefail + # <> — ship $(Pipeline.Workspace)/artifact (the promoted bytes) to + # ${TARGET_ENV}. Same command as deploy-dev's, parameterised by environment: the + # pipeline moves the proven thing up, it does not make a new thing at each step. + # Must exit non-zero on a failed deploy so on.failure fires. + echo "Promoting $(Pipeline.Workspace)/artifact to ${TARGET_ENV}..." + echo "##vso[task.logissue type=error]DEPLOY_WIRED is set, but <> is still a placeholder. Either wire the real deploy or unset DEPLOY_WIRED." + exit 1 + displayName: 'Deploy to the target environment' + env: + TARGET_ENV: ${{ parameters.targetEnvironment }} + + # ── Prove the new version actually serves. + - bash: | + set -euo pipefail + # <> — hit the target's health endpoint and FAIL (non-zero) if the + # new version is not serving. Reference: + # curl --fail --retry 5 --retry-delay 10 "https://<>/health" + echo "Health check placeholder — wire <>." + displayName: 'Health check' + + # ── Prove the journeys still work, not merely that the process is up. + - bash: | + set -euo pipefail + # <> — one fast, NON-DESTRUCTIVE check per top-priority user journey. + # "Serving" and "working" are different claims; the health check makes the first + # and this makes the second. Reads and harmless writes only: this same step runs + # against production, and a smoke test that can corrupt data is not a smoke test + # (docs/phase-8-deployment.md). Must exit non-zero to trigger the rollback. + echo "Smoke test placeholder — wire <>." + displayName: 'Smoke test' + + # ── Stamp the durable fact that this build now runs in the target environment. + # This is what lets the NEXT promotion out of this environment verify itself — see + # the header. Last step on the happy path: it must not run unless everything above + # actually passed, or it would licence promoting something that never worked. + - bash: | + set -euo pipefail + # NOTE: deliberately the REST API, NOT `##vso[build.addbuildtag]`. That logging + # command tags the RUN THAT EMITS IT — this promotion run — whereas the fact we + # need to record is about the CI BUILD whose artifact just went live. Tagging the + # wrong build would leave every future promotion unable to verify its source, and + # would do so silently: the tag would exist, just never where anything looks. + API="${SYSTEM_COLLECTIONURI}${SYSTEM_TEAMPROJECT}/_apis" + if ! curl -sf -X PUT -H "Authorization: Bearer ${SYSTEM_ACCESSTOKEN}" \ + "${API}/build/builds/${CI_RUN_ID}/tags/deployed-${TARGET_ENV}?api-version=7.1" \ + -o /dev/null; then + echo "##vso[task.logissue type=error]Deployed to ${TARGET_ENV}, but could not tag build ${CI_RUN_ID} as deployed-${TARGET_ENV}. The promotion LANDED; the record of it did not. Nothing will be promotable out of ${TARGET_ENV} until this is fixed — check System.AccessToken has build tag write access." + exit 1 + fi + echo "Tagged build ${CI_RUN_ID} as deployed-${TARGET_ENV}." + displayName: 'Record the deployment on the promoted build' + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + TARGET_ENV: ${{ parameters.targetEnvironment }} + CI_RUN_ID: ${{ parameters.ciRunId }} + + # ── Rollback: restore the last known-good version on ANY failure above. + # Rehearsed in TEST during Phase 8's rehearsal week, by the client's own operators, + # before it is ever pointed at production. A rollback that has never run is a wish. + on: + failure: + steps: + - bash: | + set -euo pipefail + echo "##vso[task.logissue type=warning]Promotion to ${TARGET_ENV} failed — restoring last known-good version: $(previousVersion)" + # <> — put $(previousVersion) back (swap the slot back, + # route to the previous revision, redeploy the previous image tag, roll back + # the release). The trigger condition for doing this by hand instead lives in + # ROLLBACK.md, written down in advance. Reference: + # az webapp deployment slot swap ... + echo "Rollback placeholder — wire <>." + # Keep the job RED so the failed promotion is visible even though service was + # restored — a rolled-back promotion is still a promotion that did not land, + # and Phase 8's record must show it. + exit 1 + displayName: 'Restore last known-good version (rollback)' + env: + TARGET_ENV: ${{ parameters.targetEnvironment }} + # DELIBERATELY no `and(succeeded(), ...)` — this step lives in the `on.failure` + # hook, which by definition only runs when the deploy hook has ALREADY failed. + # Composing succeeded() here risks evaluating false against that failed state and + # skipping the rollback entirely, turning the one path that must never be skipped + # into the one path that silently is. + condition: eq(variables['DEPLOY_WIRED'], 'true') diff --git a/kit/packs/cicd/azure-devops/pack.yaml b/kit/packs/cicd/azure-devops/pack.yaml index ac6d1fd..b144ee9 100644 --- a/kit/packs/cicd/azure-devops/pack.yaml +++ b/kit/packs/cicd/azure-devops/pack.yaml @@ -19,6 +19,8 @@ provides: - azure-pipelines/correctness.yml # correctness-review — BLOCKS on a high-confidence defect - azure-pipelines/security.yml # security-review — BLOCKS on HIGH - azure-pipelines/deploy-dev.yml # deployment job + environment + rollback (STARTER) + - azure-pipelines/deploy-promote.yml # dev→test→prod — manual only; the target Environment's + # approval check IS the human go/no-go (STARTER) - azure-pipelines/eval-regression.yml # per-PR eval-regression gate — BLOCKS - azure-pipelines/eval-suite.yml # full benchmark — manual + scheduled, ADVISORY templates: @@ -96,6 +98,7 @@ overlays: - { src: azure-pipelines/correctness.yml, dest: .azuredevops/pipelines/correctness.yml } - { src: azure-pipelines/security.yml, dest: .azuredevops/pipelines/security.yml } - { src: azure-pipelines/deploy-dev.yml, dest: .azuredevops/pipelines/deploy-dev.yml } + - { src: azure-pipelines/deploy-promote.yml, dest: .azuredevops/pipelines/deploy-promote.yml } - { src: azure-pipelines/eval-regression.yml, dest: .azuredevops/pipelines/eval-regression.yml } - { src: azure-pipelines/eval-suite.yml, dest: .azuredevops/pipelines/eval-suite.yml } - { src: azure-pipelines/templates/setup-toolchain.yml, dest: .azuredevops/pipelines/templates/setup-toolchain.yml } diff --git a/kit/packs/cicd/github/RAILS.md b/kit/packs/cicd/github/RAILS.md index b293998..cb8fd89 100644 --- a/kit/packs/cicd/github/RAILS.md +++ b/kit/packs/cicd/github/RAILS.md @@ -42,6 +42,7 @@ the selected stack's value. To swap stacks by hand today, replace the value on e | **correctness-review** | `correctness.yml` | every PR; reviews when source changed | **Blocks** on a high-confidence defect | | **security-review** | `security.yml` | every PR; reviews on gated paths / `risk:high` | **Blocks** on HIGH | | **deploy-dev** | `deploy-dev.yml` | successful CI on `main` (merge) | n/a — it ships; rolls back on failure | +| **deploy-promote** | `deploy-promote.yml` | **manual only** — never a trigger | n/a — it ships to test/prod once a named approver signs; rolls back on failure | | **Stop gate** | `.claude/hooks/stop-gate.ps1` | agent tries to finish locally | **Blocks** a red build or red tests (tests opt-out: `RAILS_STOP_RUN_TESTS=0`) | The two `eval-*` workflows (`eval-regression.yml`, `eval-suite.yml`) implement §11: @@ -106,10 +107,17 @@ These are deliberate, outward-facing actions. Nothing in the kit performs them. ``` This is the only sanctioned way to change branch protection — edit the JSON, re-run the script. Do not hand-edit rules in the GitHub UI. -6. **Wire and rehearse `deploy-dev`.** It ships as a STARTER that fails until its - placeholder deploy/rollback steps are adapted to the client platform. Wire them, - point it at a real dev environment, then rehearse the rollback (§9 shakedown) - before trusting it. +6. **Wire and rehearse `deploy-dev`.** It ships as a STARTER whose placeholder + deploy/rollback steps must be adapted to the client platform. Until they are, the job + runs, warns that deploy is not wired, and stops — a job red on every merge by design + teaches the team that red is normal. Wire the steps, point it at a real dev + environment, set the repository variable `DEPLOY_WIRED=true`, then rehearse the + rollback (§9 shakedown) before trusting it. +7. **Put required reviewers on every promotion target, then wire `deploy-promote`.** + Settings → Environments → `test` / `prod` → Required reviewers. That approval **is** + the go/no-go; without it, promotion beyond dev is automatic. `deploy-promote` refuses + to run against a target with none configured, and unlike `deploy-dev` it **fails** + rather than warns when unwired — a human asked for the promotion and is waiting. ## Required status checks @@ -195,6 +203,16 @@ gate is only proven when **both its block and its escape** have been seen to wor **restore the last known-good version** — the rollback the rails rehearse. Run deploy → roll back → redeploy in test, with the rollback trigger condition written down in advance, not invented mid-incident. +- **deploy-promote** — three drills, and the first two are the ones people skip: + 1. **The gate holds.** Run a promotion. It must **pause** for a reviewer and not + proceed until a named person approves. If it sails through, the environment has no + required reviewers and promotion is automatic — the standard's most protected stop, + silently absent. + 2. **You cannot skip an environment.** Try to promote a CI run straight to `prod` that + has only ever reached `dev`. The preflight must **refuse** it. + 3. **The rollback still works up here.** Repeat the known-bad deploy against **test** + via `deploy-promote`, executed by the client's own operators with their own + permissions — the Phase 8 rehearsal, run before prod is ever a target. - **security** — open a **probe PR touching a guarded path** (e.g. add a comment in a file under `**/Auth/`) with a planted HIGH issue. The check must go red. Close it unmerged. diff --git a/kit/packs/cicd/github/README.md b/kit/packs/cicd/github/README.md index 5d36dca..f03e681 100644 --- a/kit/packs/cicd/github/README.md +++ b/kit/packs/cicd/github/README.md @@ -20,6 +20,7 @@ Seven workflows plus the rails guide: | **Correctness Review** | `workflows/correctness.yml` | every PR (reviews when source changed) | **BLOCKS** on a high-confidence defect (override label) | | **Security Review** | `workflows/security.yml` | every PR (reviews on gated paths / `risk:high`) | **BLOCKS** on HIGH | | **Deploy Dev** | `workflows/deploy-dev.yml` | successful CI on `main` (merge) | ships; **rolls back** on failure — starter, adapt per client | +| **Deploy Promote** | `workflows/deploy-promote.yml` | **manual only** (`workflow_dispatch`) | ships to test/prod after a named approver signs; **rolls back** on failure — starter, adapt per client | | **Eval Regression Gate** | `workflows/eval-regression.yml` | PRs touching the HIGH-risk agentic surface | **BLOCKS** on a metric regression past the trip-wire (§11) | | **Eval Suite** | `workflows/eval-suite.yml` | manual + scheduled | **ADVISES** — periodic full benchmark, off by default | @@ -39,6 +40,7 @@ when consciously accepted. | `workflows/correctness.yml` | `.github/workflows/correctness.yml` | | `workflows/security.yml` | `.github/workflows/security.yml` | | `workflows/deploy-dev.yml` | `.github/workflows/deploy-dev.yml` | +| `workflows/deploy-promote.yml` | `.github/workflows/deploy-promote.yml` | | `workflows/eval-regression.yml` | `.github/workflows/eval-regression.yml` | | `workflows/eval-suite.yml` | `.github/workflows/eval-suite.yml` | | `RAILS.md` | `.github/RAILS.md` | @@ -76,7 +78,7 @@ each `«stack pack: …»` line with that stack's value. | `ci.yml` | `toolchain.id` → setup action, `toolchain.version`, `commands.restore`, `commands.build`, `commands.test`, `coverage.floor_percent` (enforced by the `Enforce coverage floor` step, `COVERAGE_FLOOR` env), `eval_gate.command` (eval-gate job). `commands.lint` is declared but left unwired to preserve the reference rail's gate semantics — add a lint step from it if desired. The `spec-gate` job has no stack seam (pure git + jq). | | `eval-regression.yml` | `toolchain` (the "Setup runtime" step + the `DOTNET_*` env defaults) | | `eval-suite.yml` | `toolchain` (the "Setup runtime" step + the `DOTNET_*` env defaults) | -| `grader.yml`, `correctness.yml`, `security.yml`, `deploy-dev.yml` | **none** — these run no stack build/test commands. Their placeholders are methodology/repo/deploy-platform level, and the Claude invocation is carried verbatim. | +| `grader.yml`, `correctness.yml`, `security.yml`, `deploy-dev.yml`, `deploy-promote.yml` | **none** — these run no stack build/test commands. Their placeholders are methodology/repo/deploy-platform level, and the Claude invocation is carried verbatim. | `{{SOLUTION_OR_PROJECT}}` is a distinct, repo-adaptation (Phase 3) token — the same token `ci-profile.yaml` uses — that the product repo fills with its solution/workspace diff --git a/kit/packs/cicd/github/pack.yaml b/kit/packs/cicd/github/pack.yaml index 36e2db7..4d3c338 100644 --- a/kit/packs/cicd/github/pack.yaml +++ b/kit/packs/cicd/github/pack.yaml @@ -17,6 +17,7 @@ provides: - workflows/correctness.yml # blocks on a high-confidence defect (override label) - workflows/security.yml # blocks on a HIGH finding, path-triggered - workflows/deploy-dev.yml # promote-never-rebuild; rolls back on failure (starter) + - workflows/deploy-promote.yml # dev→test→prod; manual only, human go/no-go via Environments - workflows/eval-regression.yml # §11 per-PR blocking eval-regression gate - workflows/eval-suite.yml # §11 periodic advisory full benchmark rails_guide: RAILS.md # operator's guide + shakedown drills @@ -72,6 +73,7 @@ overlays: - { src: workflows/correctness.yml, dest: .github/workflows/correctness.yml } - { src: workflows/security.yml, dest: .github/workflows/security.yml } - { src: workflows/deploy-dev.yml, dest: .github/workflows/deploy-dev.yml } + - { src: workflows/deploy-promote.yml, dest: .github/workflows/deploy-promote.yml } - { src: workflows/eval-regression.yml, dest: .github/workflows/eval-regression.yml } - { src: workflows/eval-suite.yml, dest: .github/workflows/eval-suite.yml } - { src: RAILS.md, dest: .github/RAILS.md } diff --git a/kit/packs/cicd/github/workflows/deploy-dev.yml b/kit/packs/cicd/github/workflows/deploy-dev.yml index 079e66a..afd128a 100644 --- a/kit/packs/cicd/github/workflows/deploy-dev.yml +++ b/kit/packs/cicd/github/workflows/deploy-dev.yml @@ -76,11 +76,42 @@ jobs: github.event.workflow_run.head_branch == 'main' steps: + # ── Is the deploy actually wired yet? ─────────────────────────────────────── + # This starter used to `exit 1` here, so a fresh install had a permanently-red + # post-merge job until someone adapted it. That was the wrong place to apply the + # pressure: a job that is red on every merge by design teaches the team that red + # is normal, which degrades the signal on every OTHER job too. + # + # The adaptation is enforced where it belongs — Phase 8's Step 0 go/no-go and its + # Step 1 rollback verification ("deploy, then roll back, then redeploy"), both + # human-gated. That is a stronger check than a red tick nobody reads. + # + # So until you wire it: this job runs, says loudly that deploy is not wired, and + # stops. Set the repository variable DEPLOY_WIRED=true once <>, + # <> and the last-known-good steps are real. + - name: Check the deploy is wired + id: wiring + run: | + set -euo pipefail + if [ "${DEPLOY_WIRED:-}" = "true" ]; then + echo "wired=true" >> "$GITHUB_OUTPUT" + else + echo "wired=false" >> "$GITHUB_OUTPUT" + echo "::warning title=Deploy not wired::This is the starter deploy-dev workflow. \ + Its deploy, health-check and rollback steps are still placeholders, so nothing was \ + deployed. Wire them (the-rails.md §5), rehearse deploy -> roll back -> redeploy \ + (Phase 8 Step 1), then set the repository variable DEPLOY_WIRED=true." + fi + env: + DEPLOY_WIRED: ${{ vars.DEPLOY_WIRED }} + - name: Checkout (deploy scripts only — NOT for building) + if: steps.wiring.outputs.wired == 'true' uses: actions/checkout@v7 # ── Promote, never rebuild: fetch the artifact CI already produced ────────── - name: Download the CI-built artifact + if: steps.wiring.outputs.wired == 'true' uses: actions/download-artifact@v8 with: name: <> @@ -91,6 +122,7 @@ jobs: # ── Record the current live version so we can put it back on failure ──────── - name: Capture last known-good version + if: steps.wiring.outputs.wired == 'true' id: lastgood run: | set -euo pipefail @@ -104,6 +136,7 @@ jobs: # ── Deploy the promoted artifact ──────────────────────────────────────────── - name: Deploy to dev + if: steps.wiring.outputs.wired == 'true' id: deploy run: | set -euo pipefail @@ -112,11 +145,15 @@ jobs: # kubectl / release command. Must exit non-zero on a failed deploy so the # rollback below fires. echo "Deploying ./artifact to dev..." - echo "::error::<> not yet wired — this starter intentionally fails until adapted." + # Reaching this line means DEPLOY_WIRED=true was set while <> is still + # a placeholder. Failing is right here: the repo has DECLARED the deploy wired, so a + # silent no-op would be the pipeline lying about having shipped something. + echo "::error::DEPLOY_WIRED is set, but <> is still a placeholder. Either wire the real deploy or unset DEPLOY_WIRED." exit 1 # ── Prove the new version actually serves ─────────────────────────────────── - name: Health check + if: steps.wiring.outputs.wired == 'true' id: health run: | set -euo pipefail @@ -131,7 +168,9 @@ jobs: # A rollback that has never run is a wish. This path is REHEARSED in the §9 # shakedown with a deliberately known-bad deploy, before it is ever needed live. - name: Restore last known-good version (rollback) - if: failure() + # Never fire for a run that never deployed — a 'rollback' of nothing is noise + # that would make the not-wired path look like a real incident. + if: failure() && steps.wiring.outputs.wired == 'true' env: PREVIOUS: ${{ steps.lastgood.outputs.previous }} run: | diff --git a/kit/packs/cicd/github/workflows/deploy-promote.yml b/kit/packs/cicd/github/workflows/deploy-promote.yml new file mode 100644 index 0000000..9e4241e --- /dev/null +++ b/kit/packs/cicd/github/workflows/deploy-promote.yml @@ -0,0 +1,332 @@ +name: Deploy Promote + +# ⚠️ STARTER WORKFLOW — adapt before relying on it. ⚠️ +# The second half of the deploy rail. `deploy-dev.yml` handles merge → dev, which is automatic +# and has no human in the loop. THIS file handles everything beyond dev — dev → test and +# test → prod — which is deliberate, human-gated, and never automatic (the-rails.md §5). +# +# Like its sibling, the deploy/rollback steps are placeholders you MUST wire to your client's +# platform and then REHEARSE (deploy → roll back → redeploy) in the TEST environment, by the +# hands that would run it at 2 a.m., before it is ever pointed at production (Phase 8's +# rehearsal week; the-rails.md §5, §9). +# +# ── PACK NOTE (github CI/CD pack) ────────────────────────────────────────────── +# NO stack seam. Per §5 this workflow does NOT build — it PROMOTES the exact artifact a named CI +# run already built, so no ci-profile command is consumed here. Its placeholders +# (<>, <>, <>, <>, +# <>, <>) are deploy-platform/repo level, not stack level. +# <> must equal THIS pack's ci.yml `name:` ("CI"). +# Pairs with this pack's deploy-dev.yml: that one owns merge -> dev, this one owns everything +# beyond it. Both halves of the deploy rail share the DEPLOY_WIRED repository variable. +# +# ── THE THREE §5 RULES THIS FILE EXISTS TO ENFORCE — do not undo them when adapting ───────── +# +# 1. PROMOTE THE ARTIFACT; NEVER REBUILD IT. This workflow runs no build. It downloads the +# EXACT artifact a named CI run produced — the bytes that dev, and then test, actually +# exercised — and ships those onward. "Rebuild it real quick for prod" puts something no +# environment ever verified into the one environment where that matters. +# +# 2. A HUMAN SAYS GO, EVERY TIME. Not just the first promotion. The go/no-go is enforced by +# the target GitHub Environment's REQUIRED REVIEWERS, not by anything hand-rolled here: +# the run pauses at the `promote` job until a named person approves, and the approval is +# recorded in the deployment's own audit trail. Using the platform's mechanism rather +# than inventing one matters — the client's security team can already audit it, and it +# cannot be edited away in a workflow file without branch protection noticing. +# Configure it: Settings → Environments → → Required reviewers. +# An environment WITHOUT required reviewers turns this file into an automatic promotion, +# which is precisely the stop the standard calls its most protected. The preflight below +# refuses to run against a target that has none. +# +# 3. A ROLLBACK THAT HAS NEVER RUN IS A WISH. The restore path is identical in shape to +# deploy-dev's, and it is rehearsed in TEST during Phase 8's rehearsal week — deploy, +# roll back, redeploy — with the trigger condition ("roll back if X") written down in +# advance in the repo's ROLLBACK.md (kit/rollback-template.md), not invented mid-incident. +# +# ── YOU CANNOT PROMOTE WHAT THE PREVIOUS ENVIRONMENT NEVER RAN ────────────────────────────── +# The preflight job below refuses a promotion unless the named CI run succeeded, ran on the +# protected branch, AND has a successful deployment recorded to the SOURCE environment for the +# same commit. That is the mechanical form of "the same build that passed test goes to +# production": without it, an operator could promote a green CI run straight to prod having +# skipped test entirely, and the pipeline would happily oblige. +# +# ── WHY THIS FAILS WHEN UNWIRED, WHERE deploy-dev ONLY WARNS ──────────────────────────────── +# deploy-dev fires automatically on every merge, so a red tick there would be red constantly by +# design and would teach the team that red is normal. This workflow only ever runs because a +# human deliberately asked for a promotion and is waiting on the result. Reporting green while +# having shipped nothing would be the pipeline lying to the one person who most needs the +# truth. So: unwired here is an ERROR, not a warning. +# +# ── PLACEHOLDERS (every one is adapt-per-client) ─────────────────────────────── +# <> The `name:` of the CI workflow that built the artifact +# (reference: "CI", from ci.yml). Must match exactly. +# <> The build artifact CI uploaded that this promotes. +# <> The real deploy, parameterised by target environment. +# <> How you confirm the new version is serving in the target. +# <> One non-destructive check per top-priority user journey. Reads and +# harmless writes ONLY — if it can corrupt data it is not a smoke +# test, and this one runs against production. +# <> / <> +# How you record the currently-live version BEFORE deploy and put it +# back on failure (slot swap, previous revision, image tag, release). +# <> The environment names in the `choice` options below (reference: +# dev, test, prod) — must match your GitHub Environment names exactly. +# ─────────────────────────────────────────────────────────────────────────────── + +on: + workflow_dispatch: + inputs: + source_environment: + description: 'Promote FROM (must already be running these bytes)' + required: true + type: choice + options: [dev, test] # <> + default: dev + target_environment: + description: 'Promote TO (its required reviewers are the go/no-go)' + required: true + type: choice + options: [test, prod] # <> + default: test + ci_run_id: + description: 'The CI run whose artifact is promoted — the proven bytes, not a rebuild' + required: true + type: string + +# Least privilege. `actions: read` pulls the artifact from the named CI run; `deployments: read` +# lets the preflight verify the source environment actually ran it. +permissions: + contents: read + actions: read + deployments: read + +concurrency: + # One promotion per target environment at a time. NOT cancel-in-progress: cancelling a + # half-finished promotion mid-deploy is how you end up with an environment in a state nobody + # recorded — the opposite of what the rollback path is for. + group: deploy-promote-${{ github.event.inputs.target_environment }} + cancel-in-progress: false + +jobs: + # ── Everything that can be checked BEFORE a human is asked to approve ────────────────────── + # Deliberately a separate job with no `environment:`, so it runs immediately. Asking a named + # person to approve a promotion and only then discovering the run id was wrong wastes the + # scarcest thing in the ceremony — their attention — and trains people to approve on autopilot. + promotion-preflight: + name: promotion-preflight + runs-on: ubuntu-latest + outputs: + sha: ${{ steps.resolve.outputs.sha }} + steps: + - name: Resolve and validate the CI run being promoted + id: resolve + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + RUN_ID: ${{ github.event.inputs.ci_run_id }} + SOURCE_ENV: ${{ github.event.inputs.source_environment }} + TARGET_ENV: ${{ github.event.inputs.target_environment }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + run: | + set -euo pipefail + + if [ "$SOURCE_ENV" = "$TARGET_ENV" ]; then + echo "::error::Source and target are both '$SOURCE_ENV'. A promotion moves bytes between environments." + exit 1 + fi + + RUN_JSON="$(gh api "repos/${REPO}/actions/runs/${RUN_ID}")" + CONCLUSION="$(echo "$RUN_JSON" | jq -r '.conclusion')" + BRANCH="$(echo "$RUN_JSON" | jq -r '.head_branch')" + NAME="$(echo "$RUN_JSON" | jq -r '.name')" + SHA="$(echo "$RUN_JSON" | jq -r '.head_sha')" + + # Promote a PROVEN build: a failed run, or one from a feature branch, never reaches an + # environment. This is the same gate deploy-dev applies to its trigger, restated here + # because a manually-supplied run id is not filtered by anything else. + if [ "$NAME" != "CI" ]; then # <> + echo "::error::Run ${RUN_ID} is '${NAME}', not the CI workflow. Only a CI run publishes the deployable artifact." + exit 1 + fi + if [ "$CONCLUSION" != "success" ]; then + echo "::error::CI run ${RUN_ID} concluded '${CONCLUSION}'. Only a successful CI run may be promoted." + exit 1 + fi + if [ "$BRANCH" != "$DEFAULT_BRANCH" ]; then + echo "::error::CI run ${RUN_ID} ran on '${BRANCH}', not '${DEFAULT_BRANCH}'. Only the protected branch is promoted." + exit 1 + fi + + echo "sha=${SHA}" >> "$GITHUB_OUTPUT" + echo "Promoting commit ${SHA} (CI run ${RUN_ID}) from ${SOURCE_ENV} to ${TARGET_ENV}." + + - name: Verify the source environment actually ran these bytes + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + SOURCE_ENV: ${{ github.event.inputs.source_environment }} + SHA: ${{ steps.resolve.outputs.sha }} + run: | + set -euo pipefail + # "The same build that passed test goes to production." Without this check an operator + # could promote a green CI run straight to prod, having skipped test entirely — the + # pipeline would oblige and nothing would say so. + DEPLOYMENTS="$(gh api "repos/${REPO}/deployments?environment=${SOURCE_ENV}&sha=${SHA}")" + COUNT="$(echo "$DEPLOYMENTS" | jq 'length')" + if [ "$COUNT" -eq 0 ]; then + echo "::error::Commit ${SHA} has never been deployed to '${SOURCE_ENV}'. Promote it there first — you cannot promote what the previous environment never ran." + exit 1 + fi + + DEPLOYMENT_ID="$(echo "$DEPLOYMENTS" | jq -r '.[0].id')" + STATE="$(gh api "repos/${REPO}/deployments/${DEPLOYMENT_ID}/statuses" | jq -r '.[0].state // "none"')" + if [ "$STATE" != "success" ]; then + echo "::error::The most recent '${SOURCE_ENV}' deployment of ${SHA} is in state '${STATE}', not 'success'. A failed or rolled-back deploy is not a promotion candidate." + exit 1 + fi + echo "Verified: ${SHA} is deployed and healthy in ${SOURCE_ENV}." + + - name: Verify the target environment has a human gate + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + # Rule 2, made mechanical. The `environment:` key on the promote job below only pauses + # for approval if that environment HAS required reviewers configured. An environment + # without them turns this workflow into an automatic promotion — the standard's single + # most protected stop, silently removed, with nothing on screen to say so. + # + # Fail closed rather than promote unapproved. If this fires, the fix is to configure the + # reviewers, never to delete this check. + # + # The two failure modes are DELIBERATELY distinguished below. "No reviewers configured" + # and "I could not read the environment" need different fixes, and collapsing them into + # one message sends the operator to the wrong place — the classic way a fail-closed check + # gets deleted in frustration instead of satisfied. + # + # PERMISSION NOTE: reading an environment's protection rules may require more than the + # `contents: read` this workflow grants, depending on repo visibility and the + # organisation's default GITHUB_TOKEN permissions. Verify this step passes during the + # Phase 8 rehearsal — that is what the rehearsal is for. If it 403s, grant the workflow + # the permission rather than dropping the check. + if ! ENV_JSON="$(gh api "repos/${REPO}/environments/${TARGET_ENV}" 2>&1)"; then + echo "::error title=Could not verify the human gate::Reading environment '${TARGET_ENV}' failed, so this workflow cannot confirm a required reviewer would be asked. Refusing to promote unverified. Either the environment does not exist, or GITHUB_TOKEN lacks permission to read it. API said: ${ENV_JSON}" + exit 1 + fi + + REVIEWERS="$(echo "$ENV_JSON" | jq '[.protection_rules[]? | select(.type == "required_reviewers")] | length')" + if [ "$REVIEWERS" -eq 0 ]; then + echo "::error title=No human gate on ${TARGET_ENV}::Environment '${TARGET_ENV}' has no required reviewers, so this promotion would proceed with nobody approving it. Configure Settings -> Environments -> ${TARGET_ENV} -> Required reviewers. A promotion beyond dev is human-gated every time (the-rails.md §5)." + exit 1 + fi + echo "Target '${TARGET_ENV}' requires reviewer approval — the go/no-go is in place." + + # ── The promotion itself. PAUSES HERE for the target environment's required reviewers. ───── + promote: + name: promote + needs: promotion-preflight + runs-on: ubuntu-latest + # THE GO/NO-GO. GitHub holds the run at this job until a required reviewer approves, and + # records who approved it against the deployment. This one line is the most important in + # the file — see rule 2 in the header before changing it. + environment: ${{ github.event.inputs.target_environment }} + + steps: + # ── Unwired is an ERROR here, not a warning — see the header for why this differs from + # deploy-dev. A human asked for this promotion and is waiting on the answer. + - name: Check the deploy is wired + env: + DEPLOY_WIRED: ${{ vars.DEPLOY_WIRED }} + run: | + set -euo pipefail + if [ "${DEPLOY_WIRED:-}" != "true" ]; then + echo "::error title=Deploy not wired::This is the starter deploy-promote workflow — its deploy, health-check, smoke and rollback steps are still placeholders, so nothing was promoted. Wire them (the-rails.md §5), rehearse deploy -> roll back -> redeploy in test (Phase 8), then set the repository variable DEPLOY_WIRED=true." + exit 1 + fi + + - name: Checkout (deploy scripts only — NOT for building) + uses: actions/checkout@v7 + + # ── Promote, never rebuild: the exact bytes the source environment ran ────────────────── + - name: Download the promoted artifact + uses: actions/download-artifact@v8 + with: + name: <> + path: ./artifact + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ github.event.inputs.ci_run_id }} + + # ── Record the current live version so we can put it back on failure ──────────────────── + - name: Capture last known-good version + id: lastgood + env: + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + # <> — record what is live in ${TARGET_ENV} RIGHT NOW (slot name, + # current revision id, deployed image tag, release id...). Emit it as an output the + # rollback step reads. Reference example (Azure slot model): + # PREV="$(az webapp deployment slot list ... --query '...' -o tsv)" + PREV="REPLACE_ME" + echo "previous=${PREV}" >> "$GITHUB_OUTPUT" + echo "Last known-good version in ${TARGET_ENV}: ${PREV}" + + # ── Deploy the promoted artifact ──────────────────────────────────────────────────────── + - name: Deploy to the target environment + id: deploy + env: + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + # <> — ship ./artifact (the promoted bytes) to ${TARGET_ENV}. Same command + # as deploy-dev's, parameterised by environment: the pipeline moves the proven thing up, + # it does not make a new thing at each step. Must exit non-zero on a failed deploy so + # the rollback below fires. + echo "Promoting ./artifact to ${TARGET_ENV}..." + echo "::error::DEPLOY_WIRED is set, but <> is still a placeholder. Either wire the real deploy or unset DEPLOY_WIRED." + exit 1 + + # ── Prove the new version actually serves ─────────────────────────────────────────────── + - name: Health check + id: health + run: | + set -euo pipefail + # <> — hit the target's health endpoint and FAIL (non-zero) if the new + # version is not serving. Reference: + # curl --fail --retry 5 --retry-delay 10 "https://<>/health" + echo "Health check placeholder — wire <>." + + # ── Prove the journeys still work, not merely that the process is up ──────────────────── + - name: Smoke test + id: smoke + run: | + set -euo pipefail + # <> — one fast, NON-DESTRUCTIVE check per top-priority user journey. + # "Serving" and "working" are different claims; the health check makes the first and + # this makes the second. Reads and harmless writes only: this same step runs against + # production, and a smoke test that can corrupt data is not a smoke test + # (docs/phase-8-deployment.md). Must exit non-zero to trigger the rollback. + echo "Smoke test placeholder — wire <>." + + # ── Rollback: restore the last known-good version on any failure above ────────────────── + # Rehearsed in TEST during Phase 8's rehearsal week, by the client's own operators, before + # it is ever pointed at production. A rollback that has never run is a wish. + - name: Restore last known-good version (rollback) + if: failure() + env: + PREVIOUS: ${{ steps.lastgood.outputs.previous }} + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + echo "::warning::Promotion to ${TARGET_ENV} failed — restoring last known-good version: ${PREVIOUS}" + # <> — put ${PREVIOUS} back (swap the slot back, route to the + # previous revision, redeploy the previous image tag, roll back the release). The + # trigger condition for doing this by hand instead lives in ROLLBACK.md, written down + # in advance. Reference: + # az webapp deployment slot swap ... + echo "Rollback placeholder — wire <>." + # Keep the job RED so the failed promotion is visible even though service was restored — + # a rolled-back promotion is still a promotion that did not land, and Phase 8's record + # must show it. + exit 1 diff --git a/kit/rollback-template.md b/kit/rollback-template.md new file mode 100644 index 0000000..80d4905 --- /dev/null +++ b/kit/rollback-template.md @@ -0,0 +1,159 @@ + + +# ROLLBACK — {{system / deployable name}} + +- **Environment(s) this covers:** {{prod | test + prod — list each, they may differ}} +- **Owner:** {{name}} — Setup Owner owns this file's truth +- **Last rehearsed:** {{YYYY-MM-DD}} by {{who actually typed the commands}} +- **Automatic restore:** {{yes — deploy-promote.yml restores on a failed deploy | no}} +- **Related:** `RUNBOOK.md` {{§ or heading}} · `docs/phase-9-monitoring.md` alerts that trigger this + +--- + +## 1. The trigger — roll back if… + + + +Roll back **without further discussion** if any of these hold after {{N}} minutes: + +- {{e.g. error rate above X% sustained for N minutes — the alert that fires is «alert name»}} +- {{e.g. the P95 latency of «journey» exceeds Nms}} +- {{e.g. «top-priority journey» fails its smoke check in production}} +- {{e.g. any data-integrity alarm, regardless of volume}} + +Do **not** roll back for: + +- {{e.g. a single failed request, a known-noisy alert, anything the ledger below records as accepted}} + +**Who can call it:** {{named role — e.g. the on-call engineer, alone, at any hour}}. Explicitly +does NOT require {{e.g. waking the Pod Lead first}} — needing permission is how a two-minute +rollback becomes a forty-minute outage. + +--- + +## 2. What "the last known-good version" means here + + + +- **Identified by:** {{deployment slot | revision id | image tag | release number}} +- **Where to look it up:** {{exact command or exact UI path}} +- **How far back can we go:** {{how many prior versions are retained, and for how long}} + +--- + +## 3. The procedure + + + +1. **{{Announce}}** — {{exact channel, exact message; see §6}} + - *Check:* {{the message is visible in «channel»}} +2. **{{Capture the current broken state before destroying it}}** — {{exact command}} + - *Check:* {{logs/snapshot saved where; you will need this for the incident review}} +3. **{{Execute the restore}}** — `{{exact command}}` + - *Check:* {{what output confirms it ran}} +4. **{{Verify service}}** — `{{exact health/smoke command}}` + - *Check:* {{what a healthy response looks like — the actual expected output}} +5. **{{Confirm the journeys}}** — {{the same smoke checks the promotion runs}} + - *Check:* {{all green}} +6. **{{Stand down / escalate}}** — {{if the check at step 4 or 5 fails, go to §5}} + +**Expected wall-clock:** {{measured in rehearsal, not estimated}} + +--- + +## 4. What rolling back does NOT undo + + + +| Concern | This release | If it cannot be undone, the plan is | +| --- | --- | --- | +| Schema migration | {{none \| additive-only, backward compatible \| destructive}} | {{forward-fix only — say so explicitly}} | +| Data transformed in place | {{none \| describe}} | {{restore-from-backup procedure + RPO}} | +| Messages published / webhooks sent | {{none \| describe}} | {{consumer tolerance; replay or ignore}} | +| Third-party state changed | {{none \| describe}} | {{compensating action}} | + +**Point of no return:** {{the step in the deploy after which rollback stops being possible, and +what to do instead. If there is none, write "none — rollback is safe at any point" and prove it +in rehearsal.}} + +--- + +## 5. When the rollback itself fails + + + +1. {{Immediate containment — e.g. take the service out of rotation, serve maintenance page}} +2. **Escalate to:** {{named person}} at {{contact}}, then {{second contact}} after {{N}} minutes +3. {{The break-glass path — restore from backup, redeploy from scratch, failover region}} + +--- + +## 6. Communication + +| When | Who is told | Where | What | +| --- | --- | --- | --- | +| Deciding to roll back | {{internal}} | {{channel}} | {{one line: what, when, expected duration}} | +| Rolled back, service restored | {{internal + client contact}} | {{channel}} | {{what happened, current state, what is NOT yet fixed}} | +| Users affected | {{who writes it}} | {{status page / in-app / email}} | {{the template message in `INCIDENT-PLAYBOOK.md`}} | + +--- + +## 7. Rehearsal record + + + +| Date | Environment | Executed by (their hands) | Deploy → roll back → redeploy all green? | What broke / what we changed in this file | +| --- | --- | --- | --- | --- | +| {{YYYY-MM-DD}} | {{test}} | {{name, client ops}} | {{yes / no}} | {{the gaps found — a rehearsal that found nothing usually means it was watched, not run}} | + +- [ ] Executed by the client's own operators, with their own permissions +- [ ] The trigger conditions in §1 are observable — each maps to something on a dashboard or an alert +- [ ] §4 answered for THIS release, not in general +- [ ] Wall-clock in §3 is a measured number +- [ ] Re-rehearsed after any change to the deploy pipeline or the restore mechanism diff --git a/kit/workflows/RAILS.md b/kit/workflows/RAILS.md index 685dcb9..62fb516 100644 --- a/kit/workflows/RAILS.md +++ b/kit/workflows/RAILS.md @@ -25,6 +25,7 @@ live, and — the part most teams skip — how to **prove they actually catch th | **correctness-review** | `correctness.yml` | every PR; reviews when source changed | **Blocks** on a high-confidence defect | | **security-review** | `security.yml` | every PR; reviews on gated paths / `risk:high` | **Blocks** on HIGH | | **deploy-dev** | `deploy-dev.yml` | successful CI on `main` (merge) | n/a — it ships; rolls back on failure | +| **deploy-promote** | `deploy-promote.yml` | **manual only** — never a trigger | n/a — it ships to test/prod once a named approver signs; rolls back on failure | | **Stop gate** | `.claude/hooks/stop-gate.ps1` | agent tries to finish locally | **Blocks** a red build or red tests (tests opt-out: `RAILS_STOP_RUN_TESTS=0`) | Branch protection (`profile/rulesets/branch-protection.json`) makes the blocking @@ -77,10 +78,21 @@ These are deliberate, outward-facing actions. Nothing in the kit performs them. ``` This is the only sanctioned way to change branch protection — edit the JSON, re-run the script. Do not hand-edit rules in the GitHub UI. -5. **Wire and rehearse `deploy-dev`.** It ships as a STARTER that fails until its - placeholder deploy/rollback steps are adapted to the client platform. Wire them, - point it at a real dev environment, then rehearse the rollback (§9 shakedown) - before trusting it. +5. **Wire and rehearse `deploy-dev`.** It ships as a STARTER whose placeholder + deploy/rollback steps must be adapted to the client platform. Until they are, the + job runs, warns loudly that deploy is not wired, and stops — deliberately, because a + job that is red on every merge by design teaches the team that red is normal. Wire + the steps, point it at a real dev environment, set the repository variable + `DEPLOY_WIRED=true`, then rehearse the rollback (§9 shakedown) before trusting it. +6. **Configure required reviewers on every promotion target, then wire + `deploy-promote`.** Settings → Environments → `test` / `prod` → Required reviewers. + This is the go/no-go: without it, promotion beyond dev becomes automatic. The + workflow **refuses to run** against a target with no reviewers configured, so this + is not optional — but configure it deliberately rather than discovering it as an + error. `deploy-promote` shares `DEPLOY_WIRED` with `deploy-dev`, and unlike its + sibling it **fails** rather than warns when unwired: a human asked for the promotion + and is waiting on the answer, so reporting green having shipped nothing would be a + lie to the one person who most needs the truth. ## Required status checks @@ -164,6 +176,17 @@ gate is only proven when **both its block and its escape** have been seen to wor **restore the last known-good version** — the rollback the rails rehearse. Run deploy → roll back → redeploy in test, with the rollback trigger condition written down in advance, not invented mid-incident. +- **deploy-promote** — three drills, and the first two are the ones people skip: + 1. **The gate holds.** Run a promotion to a target environment. It must **pause** + waiting for a reviewer, and must not proceed until a named person approves. If it + sails through, the environment has no required reviewers and the promotion is + automatic — the standard's most protected stop, silently absent. + 2. **You cannot skip an environment.** Try to promote a CI run straight to `prod` + that has only ever been deployed to `dev`. The preflight must **refuse** it. + A promotion path that lets you jump test is not a promotion path. + 3. **The rollback still works up here.** Repeat the known-bad deploy against **test** + via `deploy-promote` — the rollback rehearsal Phase 8 requires, run by the + client's own operators with their own permissions, before prod is ever a target. - **security** — open a **probe PR touching a guarded path** (e.g. add a comment in a file under `**/Auth/`) with a planted HIGH issue. The check must go red. Close it unmerged. diff --git a/kit/workflows/README.md b/kit/workflows/README.md index 09d74b3..a28b543 100644 --- a/kit/workflows/README.md +++ b/kit/workflows/README.md @@ -11,6 +11,11 @@ safely — read [`RAILS.md`](./RAILS.md). `ci.yml` carries two blocking gates — `build-and-test` and `spec-gate` — so five workflow files yield six gate rows here. +`deploy-promote.yml` is listed below but is **not a sixth rail**: it is the second half of +the deploy rail. `deploy-dev` owns merge → dev (automatic, unattended); `deploy-promote` +owns everything beyond it (manual, human-gated). Neither carries a merge gate — they run +after the merge bar has already been cleared. + | Workflow | File | Fires on | Block or advise | Source | | --- | --- | --- | --- | --- | | **CI** | `ci.yml` | every PR + push to main | **BLOCKS** (secret scan/build/test + enforced coverage floor; optional eval-gate) | generalized from source `ci.yml` | @@ -19,6 +24,7 @@ workflow files yield six gate rows here. | **Correctness Review** | `correctness.yml` | every PR (reviews when source changed) | **BLOCKS** on a high-confidence defect (override label) | generalized from source `correctness-review.yml` | | **Security Review** | `security.yml` | every PR (reviews on gated paths / `risk:high`) | **BLOCKS** on HIGH | generalized from source `security-review.yml` | | **Deploy Dev** | `deploy-dev.yml` | successful CI on `main` (merge) | ships; **rolls back** on failure | **BUILT FRESH** — starter, adapt per client | +| **Deploy Promote** | `deploy-promote.yml` | **manual only** (`workflow_dispatch`) | ships to test/prod after a named approver signs; **rolls back** on failure | **BUILT FRESH** — starter, adapt per client | Why each block-vs-advise choice exists (the-rails.md §3): mechanical truth (CI) blocks; the grader **advises** because a confident AI verdict is exactly how an agent @@ -51,11 +57,13 @@ adapting. | `<>` | grader / security / correctness | reviewer model (sonnet / opus) | | `<>` | security + CODEOWNERS header | slash-anchored guarded-dir regex — keep both, and the security rubric's prose path list, in sync | | `<>` | correctness | source root pathspec (reference: `src/`) | -| `<>` | deploy-dev | must equal `ci.yml`'s `name:` | -| `<>` | deploy-dev | the deployable artifact CI uploads (CI must upload it) | -| `<>` / `<>` | deploy-dev | real deploy + health probe (azure/webapps-deploy, `az deployment`, kubectl…) | -| `<>` / `<>` | deploy-dev | record live version + rollback mechanism | +| `<>` | deploy-dev + deploy-promote | must equal `ci.yml`'s `name:` | +| `<>` | deploy-dev + deploy-promote | the deployable artifact CI uploads (CI must upload it) | +| `<>` / `<>` | deploy-dev + deploy-promote | real deploy + health probe (azure/webapps-deploy, `az deployment`, kubectl…) | +| `<>` / `<>` | deploy-dev + deploy-promote | record live version + rollback mechanism | | `<>` | deploy-dev | GitHub Environment name (reference: `dev`) | +| `<>` | deploy-promote | one non-destructive check per top-priority journey — it runs against **prod** | +| `<>` | deploy-promote | the `choice` options for source/target (reference: `dev`, `test`, `prod`) — must match your GitHub Environment names | | `<>` | CODEOWNERS | owning user/team handle | | `<>` | apply-branch-protection.sh | ruleset JSON path if layout differs | diff --git a/kit/workflows/deploy-promote.yml b/kit/workflows/deploy-promote.yml new file mode 100644 index 0000000..67f6c77 --- /dev/null +++ b/kit/workflows/deploy-promote.yml @@ -0,0 +1,323 @@ +name: Deploy Promote + +# ⚠️ STARTER WORKFLOW — adapt before relying on it. ⚠️ +# The second half of the deploy rail. `deploy-dev.yml` handles merge → dev, which is automatic +# and has no human in the loop. THIS file handles everything beyond dev — dev → test and +# test → prod — which is deliberate, human-gated, and never automatic (the-rails.md §5). +# +# Like its sibling, the deploy/rollback steps are placeholders you MUST wire to your client's +# platform and then REHEARSE (deploy → roll back → redeploy) in the TEST environment, by the +# hands that would run it at 2 a.m., before it is ever pointed at production (Phase 8's +# rehearsal week; the-rails.md §5, §9). +# +# ── THE THREE §5 RULES THIS FILE EXISTS TO ENFORCE — do not undo them when adapting ───────── +# +# 1. PROMOTE THE ARTIFACT; NEVER REBUILD IT. This workflow runs no build. It downloads the +# EXACT artifact a named CI run produced — the bytes that dev, and then test, actually +# exercised — and ships those onward. "Rebuild it real quick for prod" puts something no +# environment ever verified into the one environment where that matters. +# +# 2. A HUMAN SAYS GO, EVERY TIME. Not just the first promotion. The go/no-go is enforced by +# the target GitHub Environment's REQUIRED REVIEWERS, not by anything hand-rolled here: +# the run pauses at the `promote` job until a named person approves, and the approval is +# recorded in the deployment's own audit trail. Using the platform's mechanism rather +# than inventing one matters — the client's security team can already audit it, and it +# cannot be edited away in a workflow file without branch protection noticing. +# Configure it: Settings → Environments → → Required reviewers. +# An environment WITHOUT required reviewers turns this file into an automatic promotion, +# which is precisely the stop the standard calls its most protected. The preflight below +# refuses to run against a target that has none. +# +# 3. A ROLLBACK THAT HAS NEVER RUN IS A WISH. The restore path is identical in shape to +# deploy-dev's, and it is rehearsed in TEST during Phase 8's rehearsal week — deploy, +# roll back, redeploy — with the trigger condition ("roll back if X") written down in +# advance in the repo's ROLLBACK.md (kit/rollback-template.md), not invented mid-incident. +# +# ── YOU CANNOT PROMOTE WHAT THE PREVIOUS ENVIRONMENT NEVER RAN ────────────────────────────── +# The preflight job below refuses a promotion unless the named CI run succeeded, ran on the +# protected branch, AND has a successful deployment recorded to the SOURCE environment for the +# same commit. That is the mechanical form of "the same build that passed test goes to +# production": without it, an operator could promote a green CI run straight to prod having +# skipped test entirely, and the pipeline would happily oblige. +# +# ── WHY THIS FAILS WHEN UNWIRED, WHERE deploy-dev ONLY WARNS ──────────────────────────────── +# deploy-dev fires automatically on every merge, so a red tick there would be red constantly by +# design and would teach the team that red is normal. This workflow only ever runs because a +# human deliberately asked for a promotion and is waiting on the result. Reporting green while +# having shipped nothing would be the pipeline lying to the one person who most needs the +# truth. So: unwired here is an ERROR, not a warning. +# +# ── PLACEHOLDERS (every one is adapt-per-client) ─────────────────────────────── +# <> The `name:` of the CI workflow that built the artifact +# (reference: "CI", from ci.yml). Must match exactly. +# <> The build artifact CI uploaded that this promotes. +# <> The real deploy, parameterised by target environment. +# <> How you confirm the new version is serving in the target. +# <> One non-destructive check per top-priority user journey. Reads and +# harmless writes ONLY — if it can corrupt data it is not a smoke +# test, and this one runs against production. +# <> / <> +# How you record the currently-live version BEFORE deploy and put it +# back on failure (slot swap, previous revision, image tag, release). +# <> The environment names in the `choice` options below (reference: +# dev, test, prod) — must match your GitHub Environment names exactly. +# ─────────────────────────────────────────────────────────────────────────────── + +on: + workflow_dispatch: + inputs: + source_environment: + description: 'Promote FROM (must already be running these bytes)' + required: true + type: choice + options: [dev, test] # <> + default: dev + target_environment: + description: 'Promote TO (its required reviewers are the go/no-go)' + required: true + type: choice + options: [test, prod] # <> + default: test + ci_run_id: + description: 'The CI run whose artifact is promoted — the proven bytes, not a rebuild' + required: true + type: string + +# Least privilege. `actions: read` pulls the artifact from the named CI run; `deployments: read` +# lets the preflight verify the source environment actually ran it. +permissions: + contents: read + actions: read + deployments: read + +concurrency: + # One promotion per target environment at a time. NOT cancel-in-progress: cancelling a + # half-finished promotion mid-deploy is how you end up with an environment in a state nobody + # recorded — the opposite of what the rollback path is for. + group: deploy-promote-${{ github.event.inputs.target_environment }} + cancel-in-progress: false + +jobs: + # ── Everything that can be checked BEFORE a human is asked to approve ────────────────────── + # Deliberately a separate job with no `environment:`, so it runs immediately. Asking a named + # person to approve a promotion and only then discovering the run id was wrong wastes the + # scarcest thing in the ceremony — their attention — and trains people to approve on autopilot. + promotion-preflight: + name: promotion-preflight + runs-on: ubuntu-latest + outputs: + sha: ${{ steps.resolve.outputs.sha }} + steps: + - name: Resolve and validate the CI run being promoted + id: resolve + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + RUN_ID: ${{ github.event.inputs.ci_run_id }} + SOURCE_ENV: ${{ github.event.inputs.source_environment }} + TARGET_ENV: ${{ github.event.inputs.target_environment }} + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + run: | + set -euo pipefail + + if [ "$SOURCE_ENV" = "$TARGET_ENV" ]; then + echo "::error::Source and target are both '$SOURCE_ENV'. A promotion moves bytes between environments." + exit 1 + fi + + RUN_JSON="$(gh api "repos/${REPO}/actions/runs/${RUN_ID}")" + CONCLUSION="$(echo "$RUN_JSON" | jq -r '.conclusion')" + BRANCH="$(echo "$RUN_JSON" | jq -r '.head_branch')" + NAME="$(echo "$RUN_JSON" | jq -r '.name')" + SHA="$(echo "$RUN_JSON" | jq -r '.head_sha')" + + # Promote a PROVEN build: a failed run, or one from a feature branch, never reaches an + # environment. This is the same gate deploy-dev applies to its trigger, restated here + # because a manually-supplied run id is not filtered by anything else. + if [ "$NAME" != "CI" ]; then # <> + echo "::error::Run ${RUN_ID} is '${NAME}', not the CI workflow. Only a CI run publishes the deployable artifact." + exit 1 + fi + if [ "$CONCLUSION" != "success" ]; then + echo "::error::CI run ${RUN_ID} concluded '${CONCLUSION}'. Only a successful CI run may be promoted." + exit 1 + fi + if [ "$BRANCH" != "$DEFAULT_BRANCH" ]; then + echo "::error::CI run ${RUN_ID} ran on '${BRANCH}', not '${DEFAULT_BRANCH}'. Only the protected branch is promoted." + exit 1 + fi + + echo "sha=${SHA}" >> "$GITHUB_OUTPUT" + echo "Promoting commit ${SHA} (CI run ${RUN_ID}) from ${SOURCE_ENV} to ${TARGET_ENV}." + + - name: Verify the source environment actually ran these bytes + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + SOURCE_ENV: ${{ github.event.inputs.source_environment }} + SHA: ${{ steps.resolve.outputs.sha }} + run: | + set -euo pipefail + # "The same build that passed test goes to production." Without this check an operator + # could promote a green CI run straight to prod, having skipped test entirely — the + # pipeline would oblige and nothing would say so. + DEPLOYMENTS="$(gh api "repos/${REPO}/deployments?environment=${SOURCE_ENV}&sha=${SHA}")" + COUNT="$(echo "$DEPLOYMENTS" | jq 'length')" + if [ "$COUNT" -eq 0 ]; then + echo "::error::Commit ${SHA} has never been deployed to '${SOURCE_ENV}'. Promote it there first — you cannot promote what the previous environment never ran." + exit 1 + fi + + DEPLOYMENT_ID="$(echo "$DEPLOYMENTS" | jq -r '.[0].id')" + STATE="$(gh api "repos/${REPO}/deployments/${DEPLOYMENT_ID}/statuses" | jq -r '.[0].state // "none"')" + if [ "$STATE" != "success" ]; then + echo "::error::The most recent '${SOURCE_ENV}' deployment of ${SHA} is in state '${STATE}', not 'success'. A failed or rolled-back deploy is not a promotion candidate." + exit 1 + fi + echo "Verified: ${SHA} is deployed and healthy in ${SOURCE_ENV}." + + - name: Verify the target environment has a human gate + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + REPO: ${{ github.repository }} + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + # Rule 2, made mechanical. The `environment:` key on the promote job below only pauses + # for approval if that environment HAS required reviewers configured. An environment + # without them turns this workflow into an automatic promotion — the standard's single + # most protected stop, silently removed, with nothing on screen to say so. + # + # Fail closed rather than promote unapproved. If this fires, the fix is to configure the + # reviewers, never to delete this check. + # + # The two failure modes are DELIBERATELY distinguished below. "No reviewers configured" + # and "I could not read the environment" need different fixes, and collapsing them into + # one message sends the operator to the wrong place — the classic way a fail-closed check + # gets deleted in frustration instead of satisfied. + # + # PERMISSION NOTE: reading an environment's protection rules may require more than the + # `contents: read` this workflow grants, depending on repo visibility and the + # organisation's default GITHUB_TOKEN permissions. Verify this step passes during the + # Phase 8 rehearsal — that is what the rehearsal is for. If it 403s, grant the workflow + # the permission rather than dropping the check. + if ! ENV_JSON="$(gh api "repos/${REPO}/environments/${TARGET_ENV}" 2>&1)"; then + echo "::error title=Could not verify the human gate::Reading environment '${TARGET_ENV}' failed, so this workflow cannot confirm a required reviewer would be asked. Refusing to promote unverified. Either the environment does not exist, or GITHUB_TOKEN lacks permission to read it. API said: ${ENV_JSON}" + exit 1 + fi + + REVIEWERS="$(echo "$ENV_JSON" | jq '[.protection_rules[]? | select(.type == "required_reviewers")] | length')" + if [ "$REVIEWERS" -eq 0 ]; then + echo "::error title=No human gate on ${TARGET_ENV}::Environment '${TARGET_ENV}' has no required reviewers, so this promotion would proceed with nobody approving it. Configure Settings -> Environments -> ${TARGET_ENV} -> Required reviewers. A promotion beyond dev is human-gated every time (the-rails.md §5)." + exit 1 + fi + echo "Target '${TARGET_ENV}' requires reviewer approval — the go/no-go is in place." + + # ── The promotion itself. PAUSES HERE for the target environment's required reviewers. ───── + promote: + name: promote + needs: promotion-preflight + runs-on: ubuntu-latest + # THE GO/NO-GO. GitHub holds the run at this job until a required reviewer approves, and + # records who approved it against the deployment. This one line is the most important in + # the file — see rule 2 in the header before changing it. + environment: ${{ github.event.inputs.target_environment }} + + steps: + # ── Unwired is an ERROR here, not a warning — see the header for why this differs from + # deploy-dev. A human asked for this promotion and is waiting on the answer. + - name: Check the deploy is wired + env: + DEPLOY_WIRED: ${{ vars.DEPLOY_WIRED }} + run: | + set -euo pipefail + if [ "${DEPLOY_WIRED:-}" != "true" ]; then + echo "::error title=Deploy not wired::This is the starter deploy-promote workflow — its deploy, health-check, smoke and rollback steps are still placeholders, so nothing was promoted. Wire them (the-rails.md §5), rehearse deploy -> roll back -> redeploy in test (Phase 8), then set the repository variable DEPLOY_WIRED=true." + exit 1 + fi + + - name: Checkout (deploy scripts only — NOT for building) + uses: actions/checkout@v7 + + # ── Promote, never rebuild: the exact bytes the source environment ran ────────────────── + - name: Download the promoted artifact + uses: actions/download-artifact@v8 + with: + name: <> + path: ./artifact + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ github.event.inputs.ci_run_id }} + + # ── Record the current live version so we can put it back on failure ──────────────────── + - name: Capture last known-good version + id: lastgood + env: + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + # <> — record what is live in ${TARGET_ENV} RIGHT NOW (slot name, + # current revision id, deployed image tag, release id...). Emit it as an output the + # rollback step reads. Reference example (Azure slot model): + # PREV="$(az webapp deployment slot list ... --query '...' -o tsv)" + PREV="REPLACE_ME" + echo "previous=${PREV}" >> "$GITHUB_OUTPUT" + echo "Last known-good version in ${TARGET_ENV}: ${PREV}" + + # ── Deploy the promoted artifact ──────────────────────────────────────────────────────── + - name: Deploy to the target environment + id: deploy + env: + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + # <> — ship ./artifact (the promoted bytes) to ${TARGET_ENV}. Same command + # as deploy-dev's, parameterised by environment: the pipeline moves the proven thing up, + # it does not make a new thing at each step. Must exit non-zero on a failed deploy so + # the rollback below fires. + echo "Promoting ./artifact to ${TARGET_ENV}..." + echo "::error::DEPLOY_WIRED is set, but <> is still a placeholder. Either wire the real deploy or unset DEPLOY_WIRED." + exit 1 + + # ── Prove the new version actually serves ─────────────────────────────────────────────── + - name: Health check + id: health + run: | + set -euo pipefail + # <> — hit the target's health endpoint and FAIL (non-zero) if the new + # version is not serving. Reference: + # curl --fail --retry 5 --retry-delay 10 "https://<>/health" + echo "Health check placeholder — wire <>." + + # ── Prove the journeys still work, not merely that the process is up ──────────────────── + - name: Smoke test + id: smoke + run: | + set -euo pipefail + # <> — one fast, NON-DESTRUCTIVE check per top-priority user journey. + # "Serving" and "working" are different claims; the health check makes the first and + # this makes the second. Reads and harmless writes only: this same step runs against + # production, and a smoke test that can corrupt data is not a smoke test + # (docs/phase-8-deployment.md). Must exit non-zero to trigger the rollback. + echo "Smoke test placeholder — wire <>." + + # ── Rollback: restore the last known-good version on any failure above ────────────────── + # Rehearsed in TEST during Phase 8's rehearsal week, by the client's own operators, before + # it is ever pointed at production. A rollback that has never run is a wish. + - name: Restore last known-good version (rollback) + if: failure() + env: + PREVIOUS: ${{ steps.lastgood.outputs.previous }} + TARGET_ENV: ${{ github.event.inputs.target_environment }} + run: | + set -euo pipefail + echo "::warning::Promotion to ${TARGET_ENV} failed — restoring last known-good version: ${PREVIOUS}" + # <> — put ${PREVIOUS} back (swap the slot back, route to the + # previous revision, redeploy the previous image tag, roll back the release). The + # trigger condition for doing this by hand instead lives in ROLLBACK.md, written down + # in advance. Reference: + # az webapp deployment slot swap ... + echo "Rollback placeholder — wire <>." + # Keep the job RED so the failed promotion is visible even though service was restored — + # a rolled-back promotion is still a promotion that did not land, and Phase 8's record + # must show it. + exit 1