Skip to content

feat(registry): host release artifacts as publisher blobs - #2765

Open
ascorbic wants to merge 13 commits into
mainfrom
feat/blob-hosted-registry-artifacts
Open

feat(registry): host release artifacts as publisher blobs#2765
ascorbic wants to merge 13 commits into
mainfrom
feat/blob-hosted-registry-artifacts

Conversation

@ascorbic

Copy link
Copy Markdown
Collaborator

What does this PR do?

Makes the publisher's PDS the default host for sandboxed plugin release artifacts. emdash-plugin publish now builds and validates the bundle, uploads the gzip package and declared PNG/JPEG/WebP listing images through com.atproto.repo.uploadBlob, and writes CID-bound checksums into the release record. --url remains available for externally hosted package bundles; the HTTP PUT --artifact-base-url flow is removed.

The release Lexicon accepts blob or URL artifact sources, applies slot-specific MIME and size constraints, and changes auth to an open union. Shared verification rejects source-less artifacts, blob/checksum mismatches, gated artifacts without a supported auth method, unsafe fetch targets, and corrupt bytes. Resolution tries typed record-scoped artifact caches, the publisher PDS, then the explicit URL.

Aggregator release envelopes now expose typed artifactCaches instead of mirrors. Raw and image cache routes bind both the exact release-record CID and blob CID. The admin image proxy uses Cumulus image presets, while package installation and labeler assessment use checksum-verifiable raw bytes.

Related work:

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change)
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable). No new admin UI strings are introduced.
  • I have added and reviewed the user-facing changeset (if this PR changes a published package)
  • New features link to an approved Discussion: Marketplace Discussion #296

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: OpenAI Codex (GPT-5)

Screenshots / test output

No visual UI changes.

Verified locally:

  • pnpm typecheck
  • pnpm lint:json | jq '.diagnostics | length'0
  • Plugin CLI: 21 files, 397 tests
  • Registry client: 5 files, 82 tests
  • Registry Lexicons: 2 files, 30 tests
  • Registry verification: 7 files, 105 Node tests; 3 files, 37 Workerd tests; packed-output check
  • Core registry handlers/proxy: 2 files, 38 tests
  • Aggregator read API: 36 tests under Workerd
  • Labeler canonical assessment: 11 tests under Workerd
  • Admin registry artifact extraction: 13 tests
  • pnpm --dir docs build

@changeset-bot

changeset-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 01d1f98

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 21 packages
Name Type
@emdash-cms/admin Minor
@emdash-cms/plugin-cli Minor
@emdash-cms/registry-client Minor
@emdash-cms/registry-lexicons Minor
@emdash-cms/registry-verification Minor
emdash Minor
@emdash-cms/perf-demo-site Patch
@emdash-cms/cache-demo-site Patch
@emdash-cms/do-demo-site Patch
@emdash-cms/do-solo-demo-site Patch
@emdash-cms/labeler Patch
@emdash-cms/cloudflare Minor
@emdash-cms/sandbox-workerd Patch
@emdash-cms/fixture-perf-site Patch
@emdash-cms/auth Minor
@emdash-cms/blocks Minor
@emdash-cms/gutenberg-to-portable-text Minor
@emdash-cms/x402 Minor
create-emdash Minor
@emdash-cms/auth-atproto Patch
@emdash-cms/plugin-embeds Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 28, 2026

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://feat-blob-hosted-registry-artifacts.try.emdashcms.com, https://feat-blob-hosted-registry-artifacts-emdash-playground.emdash-cms.workers.dev (commit 01d1f98)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://c4f8ebdf.try.emdashcms.com, https://c4f8ebdf-emdash-playground.emdash-cms.workers.dev 01d1f98 2026-08-28T23:16:24.215Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://722cd0a9.try.emdashcms.com, https://722cd0a9-emdash-playground.emdash-cms.workers.dev a3a150a 2026-08-28T21:30:04.284Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://645c3b52.try.emdashcms.com, https://645c3b52-emdash-playground.emdash-cms.workers.dev cb0de84 2026-08-28T10:59:39.392Z Visit the dashboard ↗
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://b6066c41.try.emdashcms.com, https://b6066c41-emdash-playground.emdash-cms.workers.dev 36ea6c1 2026-08-28T10:37:08.234Z Visit the dashboard ↗

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 28, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
🔵 In progress
View logs
docs f0715ab Aug 28 2026, 09:16 PM

@github-actions

Copy link
Copy Markdown
Contributor

Scope check

This PR changes 3,250 lines across 65 files. Large PRs are harder to review and more likely to be closed without review.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

@emdashbot emdashbot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right architectural change: it moves sandboxed plugin artifacts from publisher-hosted tarballs and PUT-uploaded images to atproto PDS blobs and record-scoped caches, which matches the RFC direction and keeps checksum verification end-to-end. The diff is large but the critical paths (CLI publish, server install/update, artifact proxy, labeler canonical input, aggregator view mappers) are consistent with the new lexicon and most edge cases are handled.

I checked the core and CLI handlers, the new registry-verification artifact fetcher, the aggregator view mappers, the admin proxy route, labeler assessment, the lexicon/generated types, the changeset, and the updated docs. Tests cover the new blob path, CID/checksum mismatch rejection, OAuth scope gating, and the proxy routes.

Two things stand out:

  1. emdash-plugin info now calls client.getLatestRelease() to print hosting metadata, but it doesn't guard against a missing/withdrawn release or a network failure. Because this command previously worked without that round-trip, a transient XRPC error or an inconsistent latestVersion pointer can now crash the CLI.
  2. The labeler hardcodes https://cdn.em-da.sh for media URLs even though the aggregator now advertises typed artifactCaches. For now the constant matches the canonical deployment, but any staging/alternative cache origin will produce stale URLs without warning.

Fixing the first one is the only blocking concern; the second is architectural debt to address before the cache origin is allowed to vary.

Comment thread packages/plugin-cli/src/commands/info.ts Outdated
@github-actions github-actions Bot added the review/awaiting-author Reviewed; waiting on the author to respond label Aug 28, 2026
@pkg-pr-new

pkg-pr-new Bot commented Aug 28, 2026

Copy link
Copy Markdown

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/admin@2765

@emdash-cms/auth

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/auth@2765

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/auth-atproto@2765

@emdash-cms/blocks

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/blocks@2765

@emdash-cms/cloudflare

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/cloudflare@2765

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/contentful-to-portable-text@2765

emdash

npm i https://pkg.pr.new/emdash-cms/emdash@2765

create-emdash

npm i https://pkg.pr.new/emdash-cms/emdash/create-emdash@2765

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/gutenberg-to-portable-text@2765

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-cli@2765

@emdash-cms/plugin-types

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-types@2765

@emdash-cms/registry-client

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-client@2765

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-lexicons@2765

@emdash-cms/registry-moderation

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-moderation@2765

@emdash-cms/registry-verification

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/registry-verification@2765

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/sandbox-workerd@2765

@emdash-cms/x402

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/x402@2765

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-ai-moderation@2765

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-atproto@2765

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-audit-log@2765

@emdash-cms/plugin-color

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-color@2765

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-embeds@2765

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-field-kit@2765

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-forms@2765

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/emdash-cms/emdash/@emdash-cms/plugin-webhook-notifier@2765

commit: 01d1f98

@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/awaiting-author Reviewed; waiting on the author to respond labels Aug 28, 2026
@ascorbic

Copy link
Copy Markdown
Collaborator Author

🔍 Adversarial review — stack #2766, layer 1 of 8

Automated deep review of this layer's diff (origin/main...feat/blob-hosted-registry-artifacts). Findings ranked most-severe first; confidence is CONFIRMED (full code path read) or PLAUSIBLE (strong suspicion, path partially read). Nothing was auto-fixed.

What the PR actually does + verdict

The PR moves plugin release artifacts from publisher-hosted URLs to atproto blobs on the publisher's PDS. Verified against code: emdash-plugin publish now builds the bundle itself, checks OAuth blob scopes, uploads gzip + PNG/JPEG/WebP images via uploadBlob (with client-side caps 256 KiB / 1 MiB and blob-CID-vs-bytes cross-checks), and writes CID-bound checksums; --url remains and --artifact-base-url is gone. A new fetchReleaseArtifact in registry-verification tries record-scoped caches → publisher PDS (PLC/did:web resolution, https-only, no creds) → declared URL, sha256-verifying fetched bytes against the checksum at every raw source, and derives/enforces checksum-from-blob-CID equality before any fetch. Record-level verification (verifyPackageReleaseRecords) rejects source-less artifacts, blob/checksum mismatches, and fails closed on any auth value or requiresAuth artifact (the open union with zero variants means every auth is unknown). The core install/update handlers delegate fetching to this machinery over the existing DNS-pinned SSRF transport; decompression stays capped (256 KiB / 32 entries). The aggregator envelope renames mirrors → required artifactCaches and always advertises https://cdn.em-da.sh. Overall quality is high — the checksum/CID binding, fail-closed auth, SSRF layering, and CLI validation are genuinely thorough — but the admin image proxy quietly drops its byte-integrity guarantee for blob-carrying artifacts, and there are several compat sharp edges.

Findings

  • [high] packages/core/src/astro/routes/api/admin/plugins/registry/artifact.ts:305,415-439 — Checksum verification of fetched bytes is skipped for any artifact that carries a blob ref (!descriptor.blobCid && !(await verifyChecksum(...))), including when the bytes come from the publisher's arbitrary external url. Scenario: a malicious publisher declares an icon with a self-consistent blob+checksum plus a url pointing at their server; against an aggregator that advertises no artifactCaches (any honest aggregator not running a Cumulus deployment — the field is unsigned operational data), resolveArtifact leaves descriptor.url as the external URL, and the proxy serves whatever PNG/JPEG/WebP bytes come back — different per request, never matching the approved record — with no ARTIFACT_CHECKSUM_MISMATCH. The skip is justifiable only for cache-transformed variants; for raw URL fetches verification was possible and enforced before this PR. Residual mitigations: content-type allowlist, CSP sandbox/nosniff/attachment, admin-auth, no-store. CONFIRMED.
  • [medium] packages/plugin-cli/src/commands/publish.ts:292-305 + src/oauth.ts:96,620-627 — Publishers whose AS rejects granular scopes are permanently bricked with a misleading loop: login falls back to atproto transition:generic (which does permit blob uploads), missingBlobScopes then reports both blob scopes missing, and the remediation ("logout, login again") reproduces the same legacy grant forever. Any manifest with listing images (even with --url) hits this. CONFIRMED code path.
  • [medium] packages/registry-lexicons/src/generated/.../aggregator/defs.ts (artifactCaches non-optional) + @atcute/client output validation — atcute runtime-validates XRPC outputs (ClientValidationError('output'), surfaced as AGGREGATOR_RESPONSE_INVALID in handlers/registry.ts:996-1002), so a CMS on this release pointed at a not-yet-upgraded aggregator (returns mirrors, no artifactCaches) fails every release lookup — browse/install/update all hard-broken rather than degrading. Making the field optional client-side would have given a graceful window. The changeset notes the rename but not the lockstep/hard-fail. CONFIRMED (test fixtures had to add artifactCaches: [] to keep passing).
  • [medium] packages/registry-verification/src/records.ts:387-392, src/artifact.ts (unsupportedAuth) → packages/core/src/api/handlers/registry.ts:1037/1523 — Publisher-controlled auth.hint is used verbatim, unbounded, as the error message (Artifact retrieval failed (AUTH_METHOD_UNSUPPORTED): <hint>) which the install/update handlers return to the admin client — an attacker-text injection channel into trusted UI error surfaces; details.hintUrl is never scheme-validated (a javascript: URL passes through the report contract; nothing renders it at this layer, but later layers inherit it unvalidated). CONFIRMED.
  • [medium-low] lexicon blob constraints vs runtime — maxSize/accept validate only the publisher-declared size/mimeType fields (atcute blobSize/blobAccept); fetched bytes are never compared to the declared size. A publisher declaring size: 6 on a 500 KiB gzip blob installs fine (real cap is the generic 512 KiB MAX_ARTIFACT_BYTES); images pass up to the proxy's 5 MiB despite the lexicon's 1 MiB. Bounded, but the "slot-specific size constraints" are effectively honest-CLI-only. CONFIRMED.
  • [low] artifact.ts route (resolveArtifact) — Blob-backed images have no publisher-PDS fallback: blob-only artifact + cache-less aggregator → 404; cache advertised but down → 502 (the declared url, if any, is overwritten by the cache URL). Install path falls back to the PDS; the image proxy never does. CONFIRMED.
  • [low] packages/registry-verification/src/records.ts:324-360validateArtifacts/unsupportedAuth inspect only the four known slots; x--prefixed custom artifacts (explicitly allowed by the lexicon) pass record verification with missing sources, mismatched blob CIDs, or requiresAuth set. Nothing fetches them today. CONFIRMED at this layer.
  • [low] handlers/registry.ts fetchArtifact + registry-verification fetch.ts — Dev regression: install/update artifact fetches now go through fetchVerifiedResource, which rejects http: and localhost/IP-literal hosts unconditionally; the old path allowed http://localhost artifacts in dev (assertSafeArtifactUrl DEV branch, still used only by the image proxy). Local E2E installs from a localhost-hosted tarball/PDS break. CONFIRMED.
  • [low] packages/plugin-cli/src/commands/publish.ts--artifact-base-url was removed without an explicit rejection; citty does not error on unknown flags, so old publish scripts silently switch to PDS-hosted images instead of failing loudly (the changeset/docs do cover the migration). PLAUSIBLE (citty default behavior not exercised in-repo).

PR description vs code

  • "slot-specific MIME and size constraints" — enforced against self-declared blob metadata and in the CLI only; no verifier compares fetched bytes to the declared blob size or MIME (the proxy checks the response content-type, the install path never checks MIME).
  • "Resolution order: caches → publisher PDS → explicit URL" — true for install/update only; the admin image proxy skips the PDS entirely and drops the URL fallback once a cache is advertised.
  • "Raw and image cache routes bind both the exact release-record CID and blob CID" — the binding lives only in the request URL; for image routes the response bytes are accepted unverified (finding 1).
  • The lexicon checksum doc says clients SHOULD support sha2-512 and blake3; the implementation supports sha2-256 only and rejects the rest (correctly fail-closed, but the SHOULD is unimplemented).
  • Changeset says nothing about old EmDash sites being unable to install blob-only releases ("Release record is missing artifact url or checksum" on pre-upgrade cores), nor about the client/aggregator lockstep hard-fail.

Test-coverage gaps

  • No test that a cache returning 200 with wrong bytes falls through to the PDS — only HTTP-503 fallthrough is pinned; the poisoned-cache-with-content case is the security-relevant one.
  • Nothing pins the proxy's checksum-skip semantics for blob artifacts, and the blob+url/no-cache unverified path (finding 1) is untested — proxy fixtures always advertise the Cumulus cache.
  • No oversized-artifact test through fetchReleaseArtifact (maxBytes), and no declared-blob-size-vs-actual-bytes test anywhere.
  • The aggregator test "does not represent the cache descriptor as admission for a gated blob" asserts the descriptor is advertised for a requiresAuth blob — the name claims the opposite of its assertion; it pins nothing.
  • No fetch-layer test for requiresAuth: true with auth absent (only the combined case), and no test that a transition:generic grant can(not) publish — which would have caught finding 2.

~ 🤖 Fable

@github-actions

Copy link
Copy Markdown
Contributor

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

@ascorbic
ascorbic force-pushed the feat/blob-hosted-registry-artifacts branch from cb0de84 to f0715ab Compare August 28, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant