Provenance audit has been failing since 2026-08-14: recorded tip aged out of the signed index

[wallscaler]

What happens

The provenance audit fails on every tick:

ProvenanceAuditError: recorded chain tip (epoch 1786716215) has aged out of the
signed index's recent window (oldest retained epoch 1786717150); the export chain
back to the tip can no longer be walked -- operator catch-up required
(see the archived cathedral_provenance_catchup_v1 procedure)

PROVENANCE_AUDIT_FAIL has fired 350 times and is currently firing continuously. The last PROVENANCE_AUDIT_PASS was 2026-08-14T14:31:22Z.

How it got here

During the 2026-08-14 writer outage the recorded tip stopped advancing. It was first observed lagging ("86 links behind the signed index") and has since passed out of the index's retention window entirely. It degraded from lagging to unwalkable while nobody was looking at it.

Why it is not an outage, and why that is the problem

The validator runs provenance.mode = "shadow". A failed audit records a bad write rather than preventing one, so weights keep landing and nothing alerts. The practical effect is that the provenance audit has been non-functional since the outage, and the only sign is a line in the tick output.

This will not self-heal. The retention window only moves further away from the recorded tip.

Fix

Run the cathedral_provenance_catchup_v1 re-anchor. I re-anchored the tip once already during the incident by clearing it in thin-state.json; that was a live workaround under time pressure, not the documented procedure, and it has since drifted out again, which suggests the workaround does not durably re-establish the chain.

Worth deciding separately

A gate that can fail silently for over a day is a monitoring gap regardless of this instance. Shadow mode is the right posture for now, but "audit failed" should be visible without reading tick output. There is already an allowlisted public status projection that could carry it.

[wallscaler]

Half done: the procedure now exists

The remediation this issue points at was a dead reference. cathedral_provenance_catchup_v1 was never written down anywhere: not in this repo, not in docs/history/, not on the validator host, not in git history. #67 introduced the name as a string and nothing else.

Written up in #127 as docs/PROVENANCE_CATCHUP.md, and the error message now names a file that exists.

What the investigation changed about this issue

It is a deadlock, not a lag. The tip only advances on a successful audit, and the audit cannot succeed while the tip sits behind the retention window. The gap only widens. There is no self-recovery path, by construction:

• _write_state rejects a backwards tip (stale reservation: source epoch N < reserved M)
• _assert_anchor_not_rewound rejects a rewound anchor

So clearing provenance_last_source_epoch and provenance_last_report_id with the process stopped is the only exit. Both keys: the bridge arms on last_report is not None, so clearing only the epoch leaves it live.

Live state, unchanged

provenance_last_source_epoch = 1786716215
provenance_last_report_id    = sha256:4a234873...
provenance_index_epoch       = 1786716215
provenance_candidate_block   = 8843286

Still pinned to the 2026-08-14 outage. PROVENANCE_AUDIT_FAIL fired 53 times in the last 3 hours. Last PROVENANCE_AUDIT_PASS was 2026-08-14T14:31:22Z.

Why I did not execute it

It requires stopping the validator, and the standing instruction is that the writer being up beats every other consideration. The writer is currently healthy (last write 6 minutes ago). This needs a maintenance window outside an epoch boundary, not an unattended edit to the state file that also holds the weight fences.

Also worth noting

What made this expensive was not the failure, it was the silence. Shadow mode means a failed audit records a bad write rather than preventing one, so this ran dead for over a day with the only symptom being a line in tick output. The audit's health should reach the allowlisted public status projection.

[wallscaler]

Not self-healing. Catch-up is still the operator action in docs/PROVENANCE_CATCHUP.md.

#144 makes the public status text name the aged-out tip and point at that procedure. It does not re-anchor production. The recorded tip has to be cleared offline, both keys, with the validator stopped.

Shadow mode still writes weights while this fails. After catch-up, confirm a PROVENANCE_AUDIT_PASS lands; the workaround of clearing only one key drifts out again.