diff --git a/MINER_VALIDATOR.md b/MINER_VALIDATOR.md index 7e4cfc3..3ae47de 100644 --- a/MINER_VALIDATOR.md +++ b/MINER_VALIDATOR.md @@ -1,77 +1,94 @@ # What a miner and a validator run on SN39 -SN39 (Cathedral) rewards **two lanes of verified work**, composed into one weight vector -each tempo — **70% Intel-TDX compute** / **30% CyberGym vulnerability solving**, with any -unfilled share burned. This page is the whole mental model. The deeper contracts are in -[VALIDATOR.md](VALIDATOR.md), [deploy/MECHANISM_ROUTER_CONTRACT.md](deploy/MECHANISM_ROUTER_CONTRACT.md), -and the reward-path docs. +This page describes what the shipped validator and publisher **actually do**. +If a sentence here disagrees with a flag default, the flag default is the +product. See [docs/CYBERGYM_LANE.md](docs/CYBERGYM_LANE.md) for the CyberGym +switches and the v3 gate. + +## What is live today + +SN39 currently pays **one lane**: attested Intel TDX compute, composed as +`validated_supply_v1` (v2 vector: 90% TDX / 10% burn). CyberGym is implemented +in the publisher, and it is **switched off** in what we ship. Four flags are +all off. No CyberGym weight and no CyberGym money move today. + +That is intentional until cathedral-distill refuses public-catalog `task_id`s +(`arvo:`, `oss-fuzz:`). Do not set `CATHEDRAL_ALLOCATION_CONTRACT=v3` +before that fix is deployed. Flipping v3 first is the one non-survivable +move: a miner who sees those ids can pull `n132/arvo:-vul` and read +`/tmp/poc`. ## A miner -One neuron, one lane at a time: +One neuron, one live lane: ``` -register a hotkey on SN39 (btcli subnet register --netuid 39 …) - └─ do verified work in a lane: - • compute : run the approved Intel-TDX workload → a Cathedral attestation receipt - • cybergym : solve the dispatched sealed PoC → a proof-of-crash + trace - └─ submit it: an on-chain COMMIT of H(work_hash ‖ your_hotkey) + an off-chain pointer - └─ the artifact is graded after a short reveal delay +register a hotkey on SN39 + └─ run the approved Intel TDX workload + └─ Cathedral issues a customer receipt over that work + └─ the producer exports a signed score-class report + public evidence ``` -- The receipt / PoC lives **off-chain**; only the hotkey-bound hash + pointer go on-chain, - so a copier who points at your artifact commits a hash that will not validate under their - hotkey. -- You earn **proportional to verified work** (not a fixed slot): more verified solves / more - attested compute → more of your lane's share. +CyberGym solving is not a paying lane on the shipped defaults. Miners for +that lane come after the task-id fix and a coordinated v3 cutover. ## A validator -One neuron, `cathedral-validator serve`. Each tempo it does exactly this: +One neuron, `cathedral-validator serve`. Each tempo: ``` -1. COMPUTE lane — for each miner, fetch its Intel-TDX receipt and DCAP-verify it - *itself*, then score. (Publicly verifiable → every validator agrees.) -2. CYBERGYM lane — ingest the corpus holder's signed score report and independently - verify its attestation receipt (the spot-check), then score. -3. COMPOSE — 70% compute / 30% cybergym; a lane that fails to verify BURNS its - share rather than handing it to the other lane. -4. SET WEIGHTS — one path: smooth, normalize, apply the chain limits + commit-reveal, - zero any replaced hotkey, and confirm the extrinsic landed. +1. COMPUTE lane — ingest Cathedral's signed compute feed (not a per-validator + Intel DCAP replay of every miner). The public evidence + chain is independently verifiable. Validator-self-scored + compute is still building (see Status). +2. CYBERGYM lane — only if v3 is selected AND the mechanism is enabled. + Ingest the corpus holder's HMAC-authenticated score report. + Optional spot-check: verify Cathedral's Ed25519 signature + on one carried receipt. Default: record failure, pay anyway. +3. COMPOSE — v2: 90% compute / 10% burn. + v3 (not shipped): 70% compute / 30% CyberGym; an + unfilled CyberGym share burns. A v3 compose that cannot + build the CyberGym lane fails the whole vector. +4. SET WEIGHTS — one path: smooth, normalize, chain limits, commit-reveal. ``` -It **verifies before it trusts**: it never relays a score it has not checked. Run it in -**shadow** first (`--dry-run`, composes but writes nothing), then broadcast for real from -the staged install (`deploy/sn39`). +### What "attestation" means on this path -## Why the two lanes are scored differently +- Offline verifier success on a Cathedral receipt proves Cathedral signed + those exact assertions with the pinned Ed25519 key. +- It does **not** independently replay vendor evidence. +- It does **not** prove AMD SEV host attestation. +- It is **not** "every validator independently DCAP-verifies CyberGym." + That sentence was wrong. The code checks Cathedral's signature. -| Lane | Scored by | Because | -|---|---|---| -| **Compute / TDX** | **each validator, independently** | an Intel-TDX (DCAP) receipt is publicly verifiable — no secret needed, so it is fully decentralized. | -| **CyberGym** | the **corpus holder**, centrally, then relayed | the vulnerability holdout is a **sealed private corpus** — if every validator held it, so could miners, and the "solve an unseen bug" mechanism collapses. It stays central, but **provably honest**: the signed report carries one Intel-signed attestation receipt that every validator verifies binds to *this epoch's chain-named miner*. | +`CATHEDRAL_CYBERGYM_REQUIRE_ATTESTATION_RECEIPT` is off by default. When off, +a missing or invalid carried receipt is recorded and the lane still pays. +Once an audience has ingested a receipt-bearing report, a later report +without one is refused by the ingest ratchet even while this flag stays +off. Real Intel DCAP quote verification is separate work. -That single asymmetry — decentralize what is verifiable, attest what cannot be — is the -design. A validator on this network is never a rubber stamp: the compute lane it computes, -and the CyberGym lane it *proves*. +## Why the two lanes are scored differently (design, not today's defaults) -## Getting started - -- **Run a validator:** [deploy/sn39/docker/](deploy/sn39/docker/) (Docker, 3 commands) or - the native quickstart in the [README](README.md#quickstart). Register + stake a hotkey - first (an unstaked validator's weights are ignored by consensus). -- **Run a miner:** register a hotkey, pick a lane, submit. (Miner tooling: `cathedral-miner`.) - -## Status (what's shipped vs. building) +| Lane | Who scores it today | Why | +|---|---|---| +| **Compute / TDX** | Cathedral producer, relayed. Each validator checks the signed vector + public evidence, not a local DCAP loop over miners. | An Intel TDX receipt is publicly verifiable; the live path still uses the signed feed. | +| **CyberGym** | Corpus holder, HMAC-relayed, then optional Cathedral-signature spot-check | The holdout is a sealed private corpus. If every validator held it, so could miners. | -This page is the **launch design**. Some of it ships today; some is the refactor that gets us -there (tracked in the launch refactor plan): +## Getting started -- **Shipped:** the validator that composes lanes + sets weights (`scaffold/publisher/mechanism_router.py`, - `scaffold/validator_thin.py`); the CyberGym lane ingest + tournament + **attestation spot-check** - (`mechanism_cybergym_adapter.py`, `cybergym_attestation.py`); onboarding (`deploy/sn39`). -- **Building:** moving the **compute lane from relayed to validator-self-scored** (DCAP-verified in - the validator), and the **miner commit-reveal** submission path. Until then the compute lane is - relayed from Cathedral's signed feed, and the miner path is the compute/`cathedral-miner` attestation - flow. The end state is one validator + two lane modules + one miner — no separate relay. - +- **Run a validator:** [deploy/sn39/docker/](deploy/sn39/docker/) or the native + quickstart in the [README](README.md#quickstart). Register + stake a hotkey + first. +- **Run a miner:** register a hotkey and run the compute/`cathedral-miner` + attestation flow. + +## Status (shipped vs building) + +- **Shipped:** compose + set weights; CyberGym ingest + tournament + Cathedral + Ed25519 spot-check (default advisory, mechanism default off); SN39 onboarding. +- **Shipped defaults:** `require_policy = validated_supply_v1`; + `CATHEDRAL_ALLOCATION_CONTRACT` unset/v2; `CATHEDRAL_CYBERGYM_MECHANISM_ENABLED` + false; `CATHEDRAL_CYBERGYM_WEIGHT_FRACTION` 0.0. +- **Building:** validator-self-scored compute (local DCAP of miner receipts); + miner commit-reveal; real Intel DCAP verification of CyberGym receipts; + distill public-catalog `task_id` refusal (blocks any v3 flip). diff --git a/config/validator-selfcompose-sn39.toml b/config/validator-selfcompose-sn39.toml index a316983..af249f3 100644 --- a/config/validator-selfcompose-sn39.toml +++ b/config/validator-selfcompose-sn39.toml @@ -61,9 +61,8 @@ url = "http://127.0.0.1:8010" [weight_policy] public_key_hex = "10890a66aa752479cb3b634f366d7bd27c374324d83f88d2d6b69ab066f25e26" # pragma: allowlist secret key_id = "cathedral-weight-policy" -# v2 (90% Intel TDX / 10% burn) — the byte-identical default. Flipped to -# validated_supply_v3 only in the coordinated Phase 5 cutover, paired with the -# publisher-side v3 flip, and only after CyberGym is confirmed healthy. +# v2 (90% Intel TDX / 10% burn). Do not flip to validated_supply_v3 until +# cathedral-distill refuses public-catalog task_ids. See docs/CYBERGYM_LANE.md. require_policy = "validated_supply_v1" state_file = "/var/lib/cathedral-validator/thin-state.json" diff --git a/deploy/publisher/cathedral-publisher.env.example b/deploy/publisher/cathedral-publisher.env.example index a2897bb..7ac611a 100644 --- a/deploy/publisher/cathedral-publisher.env.example +++ b/deploy/publisher/cathedral-publisher.env.example @@ -16,10 +16,28 @@ CATHEDRAL_DB_PATH=/var/lib/cathedral-publisher/publisher.db # --- allocation contract (v2 default) --------------------------------------- # The consolidated validator's config/validator-selfcompose-sn39.toml pins the -# validator to require_policy = "validated_supply_v1" (v2). Keep this side on v2 -# too until the paired Phase 5 flip. See deploy/publisher/README.md. -# CATHEDRAL_ALLOCATION_CONTRACT=v3 +# validator to require_policy = "validated_supply_v1" (v2). Keep this side on v2. +# +# HARD RULE: do not set CATHEDRAL_ALLOCATION_CONTRACT=v3 until cathedral-distill +# refuses public-catalog task_ids (arvo:, oss-fuzz:). Those ids name the +# public n132/arvo image. v3 is the only path that puts weight through that +# hole. See docs/CYBERGYM_LANE.md. +# +# v2 (what this file ships): 10% fixed burn. Leave the next block commented. # CATHEDRAL_WEIGHT_POLICY_FORCED_BURN_PERCENTAGE_V2=10 +# +# v3-only stanza. Copy as a set. A partial flip fails the whole vector. +# CATHEDRAL_VALIDATED_SUPPLY_ENABLED=1 +# CATHEDRAL_ALLOCATION_CONTRACT=v3 +# CATHEDRAL_CYBERGYM_MECHANISM_ENABLED=1 +# CATHEDRAL_CYBERGYM_WEIGHT_FRACTION=0.30 +# CATHEDRAL_WEIGHT_POLICY_FORCED_BURN_PERCENTAGE_V2=0 +# CATHEDRAL_WEIGHT_POLICY_BURN_UID= +# CATHEDRAL_WEIGHT_POLICY_BURN_HOTKEY= +# CATHEDRAL_EXTERNAL_SCORES_ENABLED=1 +# CATHEDRAL_EXTERNAL_SCORES_SOURCE=cathedral_confidential_tdx +# CATHEDRAL_EXTERNAL_SCORES_MODE=confidential_primary +# CATHEDRAL_EXTERNAL_SCORES_PRIMARY_CONFIRM=true # --- serving ---------------------------------------------------------------- # This role binds the local address the validator's [publisher] url points at diff --git a/deploy/sn39/cathedral-validator-hardening.conf b/deploy/sn39/cathedral-validator-hardening.conf index de36eec..3fe9759 100644 --- a/deploy/sn39/cathedral-validator-hardening.conf +++ b/deploy/sn39/cathedral-validator-hardening.conf @@ -12,8 +12,9 @@ # against that instruction. Restart=always and a poison exit are a contradiction. # Exit 3 is reserved for "this will not succeed by trying again". # -# StartLimit: even a retryable failure should stop hammering. Five failures in ten -# minutes means something is wrong that another restart will not fix. +# Do not put StartLimitBurst in this drop-in. Pending-receipt recovery exits 1 +# until archive proof is available. A generic start cap disables that retry +# and leaves the fenced signed attempt unrecovered. [Service] MemoryMax=2G @@ -21,7 +22,3 @@ MemoryHigh=1500M Restart=on-failure RestartSec=15s RestartPreventExitStatus=3 - -[Unit] -StartLimitIntervalSec=600 -StartLimitBurst=5 diff --git a/deploy/sn39/cathedral-validator-sn39-relay.service b/deploy/sn39/cathedral-validator-sn39-relay.service index d16ed02..579c92b 100644 --- a/deploy/sn39/cathedral-validator-sn39-relay.service +++ b/deploy/sn39/cathedral-validator-sn39-relay.service @@ -85,6 +85,10 @@ ExecStartPre=/bin/sh -c '! systemctl is-active --quiet cathedral-validator-sn39. ExecStart=/usr/bin/python3.12 -I -E -s /usr/local/libexec/cathedral-sn39-release continuous Restart=on-failure RestartSec=15s +# Same stay-down as the origin unit. A post-signed contradiction exits 3 +# and must not be restarted. Do not add StartLimitBurst here: pending +# receipt recovery exits 1 and must keep retrying. +RestartPreventExitStatus=3 TimeoutStopSec=30s StateDirectory=cathedral-validator StateDirectoryMode=0700 diff --git a/deploy/sn39/cathedral-validator-sn39.service b/deploy/sn39/cathedral-validator-sn39.service index 0919274..461ecf4 100644 --- a/deploy/sn39/cathedral-validator-sn39.service +++ b/deploy/sn39/cathedral-validator-sn39.service @@ -45,6 +45,12 @@ ExecStartPre=/bin/sh -c '! systemctl is-active --quiet cathedral-confidential-va ExecStart=/usr/bin/python3.12 -I -E -s /usr/local/libexec/cathedral-sn39-release continuous Restart=on-failure RestartSec=15s +# #114: contradiction exits 3 ("will not succeed by trying again"). Without +# this, Restart=on-failure restarted the writer 66 times against its own +# stay-stopped remediation. See cathedral-validator-hardening.conf. +RestartPreventExitStatus=3 +MemoryMax=2G +MemoryHigh=1500M TimeoutStopSec=30s StateDirectory=cathedral-validator StateDirectoryMode=0700 diff --git a/deploy/sn39/docker/docker-compose.yml b/deploy/sn39/docker/docker-compose.yml index ba65294..02b0390 100644 --- a/deploy/sn39/docker/docker-compose.yml +++ b/deploy/sn39/docker/docker-compose.yml @@ -9,7 +9,9 @@ services: dockerfile: deploy/sn39/docker/Dockerfile env_file: .env command: ["validator"] - restart: unless-stopped + # on-failure restarts exit 1 (pending-receipt retry). Exit 3 is translated + # to a stay-stopped sleep in the entrypoint so this policy never loops it. + restart: on-failure volumes: # Wallet is READ-ONLY: the validator only reads the hotkey to sign, never writes it. - ${HOST_WALLET_PATH:-~/.bittensor/wallets}:/root/.bittensor/wallets:ro diff --git a/deploy/sn39/docker/entrypoint.sh b/deploy/sn39/docker/entrypoint.sh index 7303b86..3b90a7c 100755 --- a/deploy/sn39/docker/entrypoint.sh +++ b/deploy/sn39/docker/entrypoint.sh @@ -37,6 +37,10 @@ case "$ROLE" in ;; validator) + if [ -f /state/stay-stopped ]; then + echo ">> stay-stopped: previous post-signed contradiction. Not starting." >&2 + exec sleep infinity + fi if ! _detect; then echo ">> No valid validator candidate for this wallet — see guidance above. Not starting." >&2 exit 2 @@ -50,11 +54,20 @@ case "$ROLE" in FLAG=--dry-run echo ">> SHADOW (default): reads chain, composes, writes NOTHING (dry-run)." >&2 fi - exec cathedral-validator serve --config "$CFG" \ + set +e + cathedral-validator serve --config "$CFG" \ --runtime-root /state \ --state-file /state/thin-state.json \ --jsonl /state/validator-events.jsonl \ "$FLAG" "$@" + status=$? + set -e + if [ "$status" -eq 3 ]; then + echo ">> stay-stopped: post-signed contradiction (exit 3)." >&2 + date -u +"stay-stopped %Y-%m-%dT%H:%M:%SZ" > /state/stay-stopped + exec sleep infinity + fi + exit "$status" ;; shell) exec /bin/bash "$@" ;; diff --git a/docs/CYBERGYM_LANE.md b/docs/CYBERGYM_LANE.md new file mode 100644 index 0000000..5e0760c --- /dev/null +++ b/docs/CYBERGYM_LANE.md @@ -0,0 +1,120 @@ +# CyberGym lane: what the code does, what is on + +Generated from the 2026-08 audit of the shipped composer, not from the +design we wished we had shipped. If this page and a flag default disagree, +the flag is the product. + +## Hard rule + +**Do not set `CATHEDRAL_ALLOCATION_CONTRACT=v3` until cathedral-distill +refuses public-catalog `task_id`s (`arvo:`, `oss-fuzz:`).** + +Those ids map 1:1 to public images (`n132/arvo:-vul`). A miner who sees +the id can `docker run --rm --entrypoint cat n132/arvo:-vul /tmp/poc`. +Admission today fingerprints disclosed *context*, not the catalog id +(distill #127 / #131). v3 is the only path that puts weight and money +through that hole. Leave v3 off until that refuse ships and is deployed. + +Everything else in the audit is survivable. That flip is not. + +## What the composer actually does + +Pipeline, when someone turns the flags on: + +1. Distill grades PoCs and HMAC-posts a score report. +2. Validator publisher authenticates the HMAC (this part works). +3. Only if the allocation contract is v3, the publisher puts those scores + in a 30% CyberGym lane. +4. The validator never calls `verify_poc`. It does not re-run the crash. + +The tournament is deterministic. Two validators with the same reports +agree. The HMAC path authenticates. v3 fails closed if the lane cannot +compose. Those bits are done. Do not one-shot-rewrite the validator to +"fix the docs." + +## The four switches (all off in what we ship) + +| Switch | Shipped default | On means | +|---|---|---| +| `require_policy` in `config/validator-selfcompose-sn39.toml` | `validated_supply_v1` | Validator accepts a v3 vector | +| `CATHEDRAL_ALLOCATION_CONTRACT` | unset / v2 | Publisher composes 70/30 | +| `CATHEDRAL_CYBERGYM_MECHANISM_ENABLED` | false | Lane is allowed to contribute | +| `CATHEDRAL_CYBERGYM_WEIGHT_FRACTION` | 0.0 | Lane share (v3 uses 0.30) | + +v3 without a posting producer, or without the mechanism flags, fails the +**entire** weight vector. That is worse than CyberGym off. Do not flip +`require_policy` or `CATHEDRAL_ALLOCATION_CONTRACT` from a repo PR alone. + +## What the CyberGym "DCAP" check actually is + +`scaffold/publisher/cybergym_attestation.py` verifies Cathedral's Ed25519 +signature on a `cathedral_customer_receipt_v1`. It does not verify an Intel +quote. `CATHEDRAL_CYBERGYM_REQUIRE_ATTESTATION_RECEIPT` is off: a missing or +**invalid carried** receipt is recorded and the lane still pays. + +That is not the whole rule. Once an audience has ingested a report that +carries a receipt, a later report **without** one is refused by the ingest +ratchet (`cybergym_ingest.py`), even while the require flag stays off. +After that refused report expires, a v3 compose fails the entire vector. + +Do not write "every validator independently DCAP-verifies CyberGym." That +sentence is how the pitch collapses in public. + +Real Intel DCAP quote verification is its own piece of work. Build it +deliberately. Do not pretend this module is it. + +## Distill issues that block a honest v3 (note, do not flip past them) + +These live in cathedral-distill. This repo cannot close them. + +- **Public catalog `task_id` leak (the one that would actually hurt).** + Dispatched `arvo:` / `oss-fuzz:` names the public image. Distill + #131 is the seal-time genericisation; the smallest close is refuse those + ids at admission. Until that ships, v3 stays off. +- **Unauthenticated task/dispatch HTTP (live now).** + Distill #33: `require_authentication` defaults false on + `cybergym_http.make_handler`. Anyone on the internet can ask the task + endpoint for task ids. Timeouts and the `authenticated_caller` seam exist; + the default is still open. This is more exposed than the leak because it + is on today and needs no subnet registration. +- **Trace bonus computed, never paid.** Distill #116. +- **Commit-then-draw is decorative.** Distill #136. +- **Crash differential not always deterministic.** Distill #153. + +Miners for CyberGym come after the task-id fix, auth default-on, and a +coordinated v3 cutover. Not before. + +## How to turn the lane on later + +Only after distill refuses public catalog ids **and** a producer is posting +HMAC reports. These are the publisher settings that actually compose v3. +Omitting any of them fails closed: no `CATHEDRAL_VALIDATED_SUPPLY_ENABLED` +falls back to a flat vector; a leftover 10% burn fails because v3 requires +zero fixed burn. + +``` +CATHEDRAL_VALIDATED_SUPPLY_ENABLED=1 +CATHEDRAL_ALLOCATION_CONTRACT=v3 +CATHEDRAL_CYBERGYM_MECHANISM_ENABLED=1 +CATHEDRAL_CYBERGYM_WEIGHT_FRACTION=0.30 +CATHEDRAL_WEIGHT_POLICY_FORCED_BURN_PERCENTAGE_V2=0 +CATHEDRAL_WEIGHT_POLICY_BURN_UID= +CATHEDRAL_WEIGHT_POLICY_BURN_HOTKEY= +CATHEDRAL_EXTERNAL_SCORES_ENABLED=1 +CATHEDRAL_EXTERNAL_SCORES_SOURCE=cathedral_confidential_tdx +CATHEDRAL_EXTERNAL_SCORES_MODE=confidential_primary +CATHEDRAL_EXTERNAL_SCORES_PRIMARY_CONFIRM=true +``` + +Then: + +1. Deploy that distill revision. +2. Apply the publisher stanza above. +3. Set `require_policy = "validated_supply_v3"` on the validator. +4. Flip publisher and validator in one window. See + [SN39_V3_PUBLISHER_CUTOVER.md](SN39_V3_PUBLISHER_CUTOVER.md). + +Optional, still not DCAP: `CATHEDRAL_CYBERGYM_REQUIRE_ATTESTATION_RECEIPT=1` +makes an invalid carried receipt fail closed. It does not replace the +ingest ratchet. Once an audience has carried a receipt, omission is +already refused. diff --git a/docs/PROVENANCE_CATCHUP.md b/docs/PROVENANCE_CATCHUP.md index 59f8e3a..45cb631 100644 --- a/docs/PROVENANCE_CATCHUP.md +++ b/docs/PROVENANCE_CATCHUP.md @@ -60,12 +60,21 @@ fences. Stop first. sudo python3 -c "import json; d=json.load(open('/var/lib/cathedral-validator/thin-state.json')); print(d.get('provenance_last_source_epoch'), d.get('provenance_last_report_id'))" ``` -3. **Stop the validator:** +3. **Stop the writer that is actually running.** Confirm first. The + catch-up edits `thin-state.json`; doing that under a live process risks + a fence write. ```bash - sudo systemctl stop cathedral-validator-passive.service + systemctl is-active cathedral-validator-sn39.service \ + cathedral-validator-sn39-relay.service \ + cathedral-validator-passive.service + sudo systemctl stop cathedral-validator-sn39.service \ + cathedral-validator-sn39-relay.service \ + cathedral-validator-passive.service ``` + On a Docker relay, stop the compose service instead of systemd. + 4. **Back up, then clear both keys.** Clear both. Clearing only the epoch leaves `provenance_last_report_id` set, and the bridge still triggers on it: @@ -86,16 +95,18 @@ fences. Stop first. are what stop a double write; rewinding them is a far worse failure than the one being repaired. -5. **Start the validator:** +5. **Start the same unit you stopped:** ```bash - sudo systemctl start cathedral-validator-passive.service + sudo systemctl start cathedral-validator-sn39.service + # or cathedral-validator-sn39-relay.service + # or cathedral-validator-passive.service on a host that still uses that name ``` 6. **Verify it actually recovered.** A restart proves nothing on its own: ```bash - sudo journalctl -u cathedral-validator-passive.service -f | grep -i provenance + sudo journalctl -u cathedral-validator-sn39.service -f | grep -i provenance ``` Expect `PROVENANCE_AUDIT_PASS` within a couple of ticks, and confirm the tip diff --git a/scaffold/publisher/cybergym_attestation.py b/scaffold/publisher/cybergym_attestation.py index fe31779..a5e91b9 100644 --- a/scaffold/publisher/cybergym_attestation.py +++ b/scaffold/publisher/cybergym_attestation.py @@ -1,17 +1,20 @@ -"""Independent DCAP spot-check of the CyberGym attestation receipt (distill #115). - -cathedral-validator #103 makes the validator *carry* the one representative Intel-TDX -receipt the report attaches and *ratchet its presence*. This module *verifies* it: -that it is a genuine Cathedral-signed Intel-TDX receipt whose committed ``(nonce, -miner)`` is the one the CHAIN named this epoch — turning "a receipt is present" into -"a real Intel-signed receipt existed for the miner the chain named, this epoch". - -It depends only on ``cryptography`` and the operator's pinned trusted-keys file; it does -NOT import ``cathedral_distill`` (the adapter sits on the weight path and must stay off -that default-off dependency). The signature check is the same trustless shape as the -producer-side gate (cathedral-distill #113): the receipt's Ed25519 signature must verify -against Cathedral's PUBLISHED key, the key must be ``active``, and ``issued_at`` must fall -inside its validity window. Everything fails closed to ``(False, )``. +"""Cathedral Ed25519 spot-check of the CyberGym attestation receipt (distill #115). + +This is not Intel DCAP quote verification. The check verifies Cathedral's own +Ed25519 signature over a ``cathedral_customer_receipt_v1`` document. Offline +success proves Cathedral signed those assertions. It does not replay vendor +evidence and it does not prove an Intel TDX quote. + +cathedral-validator #103 makes the validator carry the one representative receipt +the report attaches and ratchet its presence. This module checks that the +receipt verifies under the pinned Cathedral key, and that the committed +``(nonce, miner)`` is the one the chain named this epoch. + +``CATHEDRAL_CYBERGYM_REQUIRE_ATTESTATION_RECEIPT`` is off by default: a failed +or missing *carried* receipt is recorded and the lane still pays. After an +audience has adopted receipt carriage, ingest refuses a later report that +omits the field regardless of this flag. Real Intel DCAP quote verification +is separate work. Do not describe this module as DCAP. """ from __future__ import annotations diff --git a/scaffold/publisher/cybergym_contract.py b/scaffold/publisher/cybergym_contract.py index 1efc766..2a931e1 100644 --- a/scaffold/publisher/cybergym_contract.py +++ b/scaffold/publisher/cybergym_contract.py @@ -65,13 +65,12 @@ # Absent them, the tournament composer forfeits the lane (see the adapter) rather # than inventing a ranking, so the lane simply burns until the producer emits them. # -# attestation_receipt — {receipt, result_b64}: one representative Intel-TDX receipt -# the chain nonce named (cathedral-distill #115), for the validator -# to independently DCAP-verify. Carried VERBATIM so the receipt bytes -# Cathedral signed survive the round-trip; folded into the digest so a -# producer cannot add or swap it after signing. Its presence is also -# ratcheted per audience in cybergym_ingest (once seen, never optional -# again), so a compromised producer cannot silently drop it. +# attestation_receipt — {receipt, result_b64}: one representative Cathedral +# customer receipt the chain nonce named (distill #115), +# carried VERBATIM so the bytes Cathedral signed survive +# the round-trip. The validator verifies Cathedral's +# Ed25519 signature, not an Intel DCAP quote. Presence is +# ratcheted per audience in cybergym_ingest. OPTIONAL_SEMANTIC_KEYS = ( "nonce", "dispatched_units", diff --git a/scaffold/publisher/mechanism_cybergym_adapter.py b/scaffold/publisher/mechanism_cybergym_adapter.py index e9e8447..30d79bf 100644 --- a/scaffold/publisher/mechanism_cybergym_adapter.py +++ b/scaffold/publisher/mechanism_cybergym_adapter.py @@ -623,14 +623,11 @@ def _empty(reason: str, *, signed_at_ms: int = 0, sig_ok: bool = False): # A report without them keeps the original behaviour exactly (below). newest_doc = verified["document"] - # DCAP spot-check (distill #115): independently verify the carried representative - # Intel-TDX receipt — a real Cathedral signature over a receipt whose committed - # (nonce, miner) is the one the CHAIN named this epoch. #103 requires the receipt's - # PRESENCE; this verifies it. Advisory by default (records the outcome in `info`); - # under CATHEDRAL_CYBERGYM_REQUIRE_ATTESTATION_RECEIPT, a receipt that is absent, - # unbindable (present but nonce-less), or invalid burns the lane rather than paying - # on an unproven claim. Gating here - # (before the branch) covers the tournament and legacy paths alike. + # Cathedral Ed25519 spot-check (distill #115), not Intel DCAP. #103 requires + # the receipt's PRESENCE; this verifies Cathedral's signature over it. + # Advisory by default (records the outcome in `info`); under + # CATHEDRAL_CYBERGYM_REQUIRE_ATTESTATION_RECEIPT a missing or invalid + # receipt burns the lane. Gating here covers tournament and legacy alike. att_receipt = newest_doc.get("attestation_receipt") att_nonce = newest_doc.get("nonce") if att_nonce is None and att_receipt is None: diff --git a/scaffold/publisher/tests/test_cybergym_attestation.py b/scaffold/publisher/tests/test_cybergym_attestation.py index 307f760..72e26bf 100644 --- a/scaffold/publisher/tests/test_cybergym_attestation.py +++ b/scaffold/publisher/tests/test_cybergym_attestation.py @@ -1,4 +1,4 @@ -"""Independent DCAP spot-check of the carried CyberGym attestation receipt. +"""Cathedral Ed25519 spot-check of the carried CyberGym attestation receipt. Self-contained: it mints an Ed25519 "Cathedral" key, builds an internally consistent receipt + result envelope, signs it, and points the module at a trusted-keys file for diff --git a/scaffold/publisher/tests/test_mechanism_cybergym_adapter.py b/scaffold/publisher/tests/test_mechanism_cybergym_adapter.py index 0ff224d..2a9318c 100644 --- a/scaffold/publisher/tests/test_mechanism_cybergym_adapter.py +++ b/scaffold/publisher/tests/test_mechanism_cybergym_adapter.py @@ -819,7 +819,7 @@ def test_vendored_tournament_constants_match_the_mechanism(tmp_path): assert [str(s) for s in T.TOURNAMENT_SHARES] == ["0.65", "0.14", "0.10", "0.07", "0.04"] -# --- DCAP attestation gate (distill #115 follow-on) -------------------------- +# --- Cathedral Ed25519 attestation gate (distill #115 follow-on) ------------- def test_attestation_is_advisory_by_default_and_does_not_burn(tmp_path, monkeypatch): """With the require flag unset, a report whose receipt cannot be verified (here, absent) still contributes — the outcome is only recorded, so turning the check on diff --git a/scaffold/publisher/tests/test_sn39_public_status.py b/scaffold/publisher/tests/test_sn39_public_status.py index a82dff7..77fcfc1 100644 --- a/scaffold/publisher/tests/test_sn39_public_status.py +++ b/scaffold/publisher/tests/test_sn39_public_status.py @@ -252,6 +252,27 @@ def test_event_status_mismatch_is_dropped() -> None: assert status.clean_event(_event("WEIGHTS_DRY_RUN", "FAIL"))["status"] == "FAIL" +def test_aged_out_provenance_fail_gets_catchup_remediation() -> None: + aged = status.clean_event( + _event( + "PROVENANCE_AUDIT_FAIL", + "FAIL", + detail="recorded chain tip (epoch 12) has aged out of the signed index", + ) + ) + assert aged is not None + assert "PROVENANCE_CATCHUP.md" in aged["remediation"] + assert "cathedral-validator-sn39.service" in aged["remediation"] + + +def test_signature_provenance_fail_does_not_get_catchup_remediation() -> None: + failed = status.clean_event( + _event("PROVENANCE_AUDIT_FAIL", "FAIL", detail="index signature is invalid") + ) + assert failed is not None + assert "remediation" not in failed + + def test_public_status_is_time_bounded() -> None: stale = status.clean_event( _event( diff --git a/scaffold/publisher/tests/test_validator_lifecycle.py b/scaffold/publisher/tests/test_validator_lifecycle.py index 68ad952..a66d695 100644 --- a/scaffold/publisher/tests/test_validator_lifecycle.py +++ b/scaffold/publisher/tests/test_validator_lifecycle.py @@ -558,7 +558,7 @@ def test_recurring_loop_exits_on_post_signed_contradiction( ), ) - assert validator_thin.run(args) == 1 + assert validator_thin.run(args) == validator_thin.STAY_STOPPED_EXIT names = [name for name, _fields in events.rows] assert names.count("PENDING_RECEIPT_CONTRADICTION") == 1 assert "TICK_FAILED" not in names diff --git a/scaffold/validator_thin.py b/scaffold/validator_thin.py index 265454b..a6c9065 100644 --- a/scaffold/validator_thin.py +++ b/scaffold/validator_thin.py @@ -149,6 +149,13 @@ class _PostSignedSubmissionMismatch(wire.VectorError): """A signed attempt has a positive receipt or execution contradiction.""" +# systemd RestartPreventExitStatus=3. A contradiction will not succeed by +# trying again; returning 1 with Restart=on-failure/always is how #114 +# restarted the writer 66 times against its own "keep every writer stopped" +# remediation. +STAY_STOPPED_EXIT = 3 + + class _NothingToScoreYet(wire.VectorError): """No paid hotkey was independently proven this epoch. This is NOT a failure: the validator verified there is nothing to submit and should idle until the @@ -10542,7 +10549,7 @@ def run(args) -> int: "and named transaction; never submit a replacement." ), ) - return 1 + return STAY_STOPPED_EXIT except _PendingReceiptNotProven as exc: render.outcome(False, f"pending receipt not proven: {stable_error(exc)}") _get_events(args).event( @@ -10671,7 +10678,7 @@ def run(args) -> int: "a replacement." ), ) - return 1 + return STAY_STOPPED_EXIT except _PendingReceiptNotProven as e: render.outcome(False, f"pending receipt not proven: {stable_error(e)}") _get_events(args).event( diff --git a/scripts/publish_sn39_validator_status.py b/scripts/publish_sn39_validator_status.py index 6ce5091..9740992 100644 --- a/scripts/publish_sn39_validator_status.py +++ b/scripts/publish_sn39_validator_status.py @@ -112,8 +112,14 @@ "WEIGHTS_DRY_RUN": "PASS", "WEIGHTS_SUBMITTED": "PASS", } +AGED_OUT_REMEDIATION = ( + "the recorded provenance tip aged out of the signed index; stop the " + "writer (cathedral-validator-sn39.service, " + "cathedral-validator-sn39-relay.service, or " + "cathedral-validator-passive.service) and run docs/PROVENANCE_CATCHUP.md; " + "the audit will not self-heal. Thin authority is unaffected" +) EVENT_REMEDIATION = { - "PROVENANCE_AUDIT_FAIL": "inspect the validator-local audit log; thin authority is unaffected", "PROVENANCE_AUDIT_NOT_PROVEN": "keep thin authority until every anchored outcome has replayable evidence", "PROVENANCE_AUDIT_UNRESOLVED": "inspect the validator-local audit log and evidence endpoint", "PROVENANCE_RESERVATION_REFUSED": "inspect the validator-local state fence; nothing was submitted", @@ -320,6 +326,11 @@ def public_detail(event: str, raw: Any) -> str | None: "names; the verified evidence has since advanced" ) if event == "PROVENANCE_AUDIT_FAIL": + if "aged out" in detail.lower(): + return ( + "the recorded provenance tip aged out of the signed index; " + "run docs/PROVENANCE_CATCHUP.md; this does not self-heal" + ) return "the provenance audit failed" if event == "PROVENANCE_HEALTH_GATE_FAILED": return "the current provenance health gate failed" @@ -532,9 +543,14 @@ def clean_event(document: Any) -> dict[str, Any] | None: boundary = parse_weight_boundary(document.get("detail")) if boundary is not None: clean.update(boundary) - remediation = EVENT_REMEDIATION.get(event) - if remediation: - clean["remediation"] = remediation[: TEXT_LIMITS["remediation"]] + if event == "PROVENANCE_AUDIT_FAIL": + raw_detail = document.get("detail") + if isinstance(raw_detail, str) and "aged out" in raw_detail.lower(): + clean["remediation"] = AGED_OUT_REMEDIATION[: TEXT_LIMITS["remediation"]] + else: + remediation = EVENT_REMEDIATION.get(event) + if remediation: + clean["remediation"] = remediation[: TEXT_LIMITS["remediation"]] if not clean.get("event") or not clean.get("ts"): return None return clean diff --git a/tests/thin/test_relay_release_install.py b/tests/thin/test_relay_release_install.py index 08d0edb..b1101cb 100644 --- a/tests/thin/test_relay_release_install.py +++ b/tests/thin/test_relay_release_install.py @@ -127,7 +127,12 @@ def _release_checkout(base: pathlib.Path) -> tuple[pathlib.Path, str]: _git(release, "add", "--all") _git(release, "commit", "--quiet", "--message=fixture") for path in [release, *release.rglob("*")]: - path.chmod(0o755 if path.is_dir() else 0o644) + if ".git" in path.parts: + continue + try: + path.chmod(0o755 if path.is_dir() else 0o644) + except FileNotFoundError: + continue sha = subprocess.check_output( ["git", "-c", f"safe.directory={release}", "rev-parse", "HEAD"], cwd=release,