feat(release-service): add operations and recovery

[ascorbic]

What does this PR do?

Adds the sharded identity directory, per-publisher abuse limits, encrypted R2 archive and fail-safe restore, encryption rotation controls, Analytics Engine metrics, archive Workflow, and the operational runbook. The follow-up commit makes approval expiry authoritative and restore preparation replay-safe after partial progress.

This is PR 6 of 7. The implementation and recovery-safety follow-up remain separate commits. The stack merges as one unit and this branch is not deployable by itself.

Discussion: #1590

Type of change

• Bug fix
• Feature (requires maintainer-approved Discussion)
• Refactor (no behavior change)
• Translation
• Documentation
• Performance improvement
• Tests
• Chore (dependencies, CI, tooling)

Checklist

• I have read CONTRIBUTING.md
• pnpm typecheck passes
• pnpm lint passes
• pnpm test passes (or targeted tests for my change)
• pnpm format has been run
• I have added/updated tests for my changes (if applicable)
• User-visible strings use Lingui and logical RTL-safe layout
• I have added and reviewed the user-facing changeset for the typed operator client
• New features link to an approved Discussion: RFC: Attested Automated Publishing #1590

AI-generated code disclosure

• This PR includes AI-generated code — model/tool: OpenAI Codex (GPT-5)

Screenshots / test output

Directory, rate-limit, archive/restore, encryption operations, metrics, alarm expiry, replay safety, Workflow, and typed-client tests pass.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 837561a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 20 packages

Name	Type
@emdash-cms/registry-client	Minor
@emdash-cms/plugin-cli	Minor
@emdash-cms/admin	Patch
emdash	Patch
@emdash-cms/release-action	Patch
@emdash-cms/release-service	Patch
@emdash-cms/perf-demo-site	Patch
@emdash-cms/cache-demo-site	Patch
@emdash-cms/do-demo-site	Patch
@emdash-cms/do-solo-demo-site	Patch
@emdash-cms/cloudflare	Patch
@emdash-cms/sandbox-workerd	Patch
@emdash-cms/fixture-perf-site	Patch
@emdash-cms/auth	Patch
@emdash-cms/blocks	Patch
@emdash-cms/gutenberg-to-portable-text	Patch
@emdash-cms/x402	Patch
create-emdash	Patch
@emdash-cms/auth-atproto	Patch
@emdash-cms/plugin-embeds	Patch

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	docs	837561a	Aug 29 2026, 07:33 AM

[cloudflare-workers-and-pages]

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://feat-drs-review-06-operations.try.emdashcms.com, https://feat-drs-review-06-operations-emdash-playground.emdash-cms.workers.dev (commit 837561a)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status	Deployment URL	Commit	Updated (UTC)	See this deployment's details
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://64917772.try.emdashcms.com, https://64917772-emdash-playground.emdash-cms.workers.dev	837561a	2026-08-29T07:33:28.201Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://1afb4e70.try.emdashcms.com, https://1afb4e70-emdash-playground.emdash-cms.workers.dev	467c150	2026-08-29T07:15:07.169Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://7c824a2b.try.emdashcms.com, https://7c824a2b-emdash-playground.emdash-cms.workers.dev	61cb7fa	2026-08-28T23:21:29.154Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://2d3f9223.try.emdashcms.com, https://2d3f9223-emdash-playground.emdash-cms.workers.dev	1e3150b	2026-08-28T23:06:09.443Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://74cf0b35.try.emdashcms.com, https://74cf0b35-emdash-playground.emdash-cms.workers.dev	6fca979	2026-08-28T22:43:16.846Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://92d9fe58.try.emdashcms.com, https://92d9fe58-emdash-playground.emdash-cms.workers.dev	4073859	2026-08-28T22:18:51.291Z	Visit the dashboard ↗
Build: In progress 🔵 View logs ↗		738349c	2026-08-28T22:10:51.392Z	View logs ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://e9d893c8.try.emdashcms.com, https://e9d893c8-emdash-playground.emdash-cms.workers.dev	47fbbb1	2026-08-28T21:34:40.801Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://2752c8f6.try.emdashcms.com, https://2752c8f6-emdash-playground.emdash-cms.workers.dev	4a1afe7	2026-08-28T15:24:00.186Z	Visit the dashboard ↗
Build: Success ✅ Deployment: Success ✅ View logs ↗	https://b3b53834.try.emdashcms.com, https://b3b53834-emdash-playground.emdash-cms.workers.dev	20a63de	2026-08-28T13:43:03.111Z	Visit the dashboard ↗

View all previews: View all previews ↗

[github-actions]

Scope check

This PR changes 5,600 lines across 47 files. Large PRs are harder to review and more likely to be closed without review.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

[github-actions]

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

• feat(release-service): complete delegated publishing #2749 (29 shared files)
• feat(release-service): add publication product flow #2747 (24 shared files)
• feat(release-service): add passkey approvals #2745 (12 shared files)
• feat(release-service): add delegated release foundations #2743 (8 shared files)
• feat(registry): verify delegated releases independently #2746 (6 shared files)
• feat(release-service): add publisher authority and admission #2744 (12 shared files)

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

[pkg-pr-new]

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/admin@2748

@emdash-cms/auth

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/auth@2748

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/auth-atproto@2748

@emdash-cms/blocks

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/blocks@2748

@emdash-cms/cloudflare

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/cloudflare@2748

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/contentful-to-portable-text@2748

emdash

npm i https://pkg.pr.new/emdash-cms/emdash@2748

create-emdash

npm i https://pkg.pr.new/emdash-cms/emdash/create-emdash@2748

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/gutenberg-to-portable-text@2748

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-cli@2748

@emdash-cms/plugin-types

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-types@2748

@emdash-cms/registry-client

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-client@2748

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-lexicons@2748

@emdash-cms/registry-moderation

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-moderation@2748

@emdash-cms/registry-verification

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-verification@2748

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/sandbox-workerd@2748

@emdash-cms/x402

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/x402@2748

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-ai-moderation@2748

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-atproto@2748

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-audit-log@2748

@emdash-cms/plugin-color

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-color@2748

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-embeds@2748

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-field-kit@2748

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-forms@2748

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-webhook-notifier@2748

commit: 837561a

[ascorbic]

🔍 Adversarial review — stack #2766, layer 7 of 8

Automated deep review of this layer's diff (feat/drs-review-05-publication-product...feat/drs-review-06-operations). Findings ranked most-severe first; confidence is CONFIRMED (full code path read) or PLAUSIBLE (strong suspicion, path partially read). Nothing was auto-fixed.

What the PR actually does + verdict

Layer 06 adds, on top of the layer-05 release service: a 256-shard non-authoritative identity directory DO registered from OAuth callbacks and listed via an Access-viewer route; a per-publisher-DO fixed-window rate limiter (workload/repository/publisher scopes) consumed in the submit-intent route after token verification and policy evaluation; encrypted R2 publisher archives (JWE pages context-bound to deploymentId/objectClass/table/archiveId:page/ownerDid, create-only R2 writes with decrypt-and-compare replay) plus plaintext "sanitized" audit export objects; a prepare-then-page-by-page restore state machine in the publisher DO (atomic wipe+marker, digest-bound page replay, forced suspension, delegation stripped to reauthorization_required, non-terminal intents forced to failed); CAS-based encryption-record rotation endpoints for publisher and approver ciphertexts; Secrets-Store-aware config loading; Analytics Engine metrics; an archive Workflow; operator UI panels; a runbook; and typed operator client methods with a changeset. The two headline recovery claims hold: intent/approval expiry is enforced at transition time inside the authoritative DO (intent-state.ts:523 rejects any transition of an expired intent except to expired; the new alarm sweep is only cleanup), and restore preparation/pages are replay-safe (wipe + operations_restore marker in one transactionSync; page data + operations_restore_pages digest marker in one transaction). Overall quality is high — validation is obsessive, CAS discipline is consistent, and authorization on every new route is admin/viewer-gated with audience-scoped Access JWTs — but there are real gaps between the runbook's privacy claims and the audit-export code, and the restore state machine has unrecoverable dead-ends.

Findings

• [medium] apps/release-service/src/backup/routes.ts:427-460 — Audit export objects are written to R2 unencrypted and contain the full audit event rows plus the raw publisherDid, contradicting the runbook's invariant ("Audit export objects contain only the sanitized audit_events.public_payload contract"). writeAuditObject serializes {version, publisherDid, events} where each event carries actorIdentity (Access operator subjects/emails from #appendAudit), subject, and reasonCode — the plaintext DID also defeats the ownerHash pseudonymization used for the R2 key. The test only asserts the ciphertext/private-payload strings are absent, not the DID or actor identities (it asserts DID absence for snapshots but not audit objects). Failure scenario: anyone with R2 bucket read access reads operator identities and publisher DIDs the encrypted-snapshot design was built to protect. CONFIRMED.
• [medium] apps/release-service/src/publisher-do/publisher-do.ts:1096-1122 (prepareOperationsRestore) + operations-restore.ts — a restore that cannot complete is a permanent dead-end. Once operations_restore is prepared/restoring, prepare for any other archive returns RESTORE_CONFLICT, and there is no abort/reset operation anywhere. If an archive page object is corrupted or deleted after the manifest check (manifest decrypts, page N doesn't — readEncryptedObject 404/409 forever), the shard can never be restored from any archive, contradicting the runbook's "Runbooks never require direct Durable Object SQLite edits." Relatedly, a completed restore of an archive can never be re-applied (prepare replays by archiveId+pages; encryption context pins objects to the archiveId), so each archive is one-shot — undocumented. CONFIRMED mechanism; PLAUSIBLE trigger.
• [medium] apps/release-service/src/backup/routes.ts:447-459 — after restoring into genuinely lost DO storage, audit sequences restart at 1 (fresh sqlite_sequence; AUTOINCREMENT only protects the same storage), so the next archive of that publisher writes audit keys like audit/{hash}/…0001-…0100.json that collide with the pre-loss export objects with different content → ARCHIVE_OPERATION_FAILED 409, permanently, once page boundaries align (guaranteed at ≥100 events since the first page is always 1–100). The restored publisher becomes un-archivable — and therefore un-restorable next time. PLAUSIBLE (only after true storage loss, which is exactly the scenario this feature exists for).
• [low/medium] apps/release-service/src/backup/routes.ts:279 (INTENT_PAGE_SIZE = 4) vs crypto/encryption.ts MAX_PLAINTEXT_BYTES = 1MB — four maximal intents (16KB identity + 64KB release + 64KB stateData each, doubled by JSON string escaping when re-embedded in the snapshot) can exceed the 1MB encrypt cap; encrypt throws → deterministic ARCHIVE_OPERATION_FAILED on that page on every retry → that publisher can never be archived. PLAUSIBLE.
• [low] apps/release-service/src/workflows/release-intent.ts:417-425 — writeOperationsMetric throws TypeError for any outcome not matching DIMENSION_PATTERN, and it is called with the verifier's error code before the failure report is persisted; an out-of-pattern or empty code would turn a terminal "verification failed" into workflow step retries and an errored workflow. Today the verifier emits only VERIFIER_INPUT_INVALID/VERIFIER_INTERNAL_ERROR (both pass), so this is a fragility, not a live bug — but the RPC boundary types the code as arbitrary string. PLAUSIBLE.
• [low] apps/release-service/src/operations/encryption-routes.ts:1993 / OperatorPage.tsx — the rotation complete flag is per-page (nextCursor === null && raced === 0); races on earlier pages are forgotten once the cursor advances, so the final page (and the UI "Verified" badge, which keeps only the last result) can read complete while earlier raced records remain under the old key. The runbook's mandatory second full scan mitigates this, but an operator trusting the badge and executing step 8 (retire old key) strands retained OAuth ciphertext under a removed key → ENCRYPTION_OPERATION_FAILED / forced reauthorization. CONFIRMED behavior, procedural mitigation exists.
• [low] apps/release-service/src/publisher-do/publisher-do.ts — #scheduleNextAlarm includes oauth_states, publisher_sessions, and intent_rate_idempotency expiries, but nothing arms the alarm when only those rows are written (putOAuthState, createPublisherSession, consumeIntentRateLimit don't schedule); a publisher DO that only ever does OAuth flows accumulates expired rows until an intent or publication event first schedules an alarm. Correctness is protected by lazy expiry checks at read; housekeeping only. CONFIRMED.
• [low] apps/release-service/src/publisher-do/publisher-do.ts:456-467 and approver-do/schema.ts — the layer changes CREATE TABLE IF NOT EXISTS DDL in place (adds encryption_purpose NOT NULL to oauth_states; adds 'access' to the approver audit_events CHECK). IF NOT EXISTS never migrates existing DO storage, so any pre-existing DO would fail inserts. Benign only because the runbook explicitly states the service must not be deployed before the full stack lands; if any earlier layer was ever deployed, this breaks OAuth on existing shards. CONFIRMED mechanism, moot under the stated deployment constraint.

Verified non-findings worth relaying: rate-limit attribution is the authenticated OIDC identity (repository.id/workflow.ref from the verified token) and quota is consumed only after the victim publisher's own policy authorizes the workload, so a third party cannot burn a victim's budget; limiter errors fail closed (500); replays don't double-count. Restore cannot silently roll back newer live state — it requires admin role, service-control suspension, exact DID confirmation, and an explicit destructive prepare, and restored authority is neutralized (empty delegation ciphertext, reauthorization_required, workloads inactive, non-terminal intents failed; restored reservations for terminal intents are lazily reaped by create()). Archive objects are tamper/substitution-proof via the JWE context digest (deploymentId, table, archiveId:page, ownerDid, key version in AAD-equivalent header). Rotation is CAS-per-record with fail-closed decryption on missing retained keys, and every ciphertext class currently stored (delegation, publisher oauth_states, approver identity_transactions) is enumerated by the rotation cursors. Metrics writes are non-blocking, dimension-validated, and use hashes not DIDs.

PR description vs code

• The runbook's "Audit export objects contain only the sanitized audit_events.public_payload contract" is false — full rows plus plaintext publisherDid are exported (finding 1).
• "Fail-safe restore" is accurate for authority handling but overstated for recoverability: no abort path for a wedged restore, and archives are one-shot restorable — neither is documented.
• The changeset exists and accurately describes the operator-client additions.
• Everything else claimed (sharded directory, per-publisher limits, rotation, workflow, metrics, runbook, typed client) is present and matches.

Test-coverage gaps

• No negative restore tests at all: page-digest conflict (RESTORE_CONFLICT), out-of-order pages, RESTORE_NOT_EMPTY, missing/corrupt page mid-restore (the dead-end), or a tampered/substituted R2 object failing decrypt.
• Audit export objects are never asserted free of publisherDid/actor identities (they aren't free of them).
• No test of the archive Workflow's retry-after-content-changed 409 path, nor of archive page size vs the 1MB encryption cap.
• Rotation tests don't cover a race on a non-final page followed by a "complete" final page.
• Rate-limit tests don't cover fixed-window boundary bursts or limiter-error fail-closed behavior.
• The new admin routes are all tested by calling handlers directly, so handleRequest-level enforcement (Access audience + CSRF headers) for these specific paths is untested at this layer.

~ 🤖 Fable