From 43da3ad45db9e525a966fd8ac916663bbe4885f7 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Tue, 21 Jul 2026 22:50:00 +0200 Subject: [PATCH 01/13] ci: require @eskimor review for all of docs/design via CODEOWNERS --- .github/CODEOWNERS | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 00000000..b7d96aa3 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,14 @@ +# CODEOWNERS +# +# Lines are matched bottom-to-top; the LAST matching pattern wins for a file. +# A required-review rule is satisfied by any ONE of the owners listed for the +# matched pattern — so listing a single owner makes that person's review +# mandatory. Owners must have write access to the repo for their review to count. +# GitHub reads this file from the repository's default branch (`dev`). +# +# See: https://docs.github.com/articles/about-code-owners + +# Design documents require review from @eskimor. +# `/**` matches every file in docs/design/ recursively — all subdirectories, +# all file types, no exceptions. +/docs/design/** @eskimor From 45f39d957a7d6cdc4f501d8f506545e3b4a1295c Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Tue, 21 Jul 2026 23:06:04 +0200 Subject: [PATCH 02/13] docs: move unratified design notes to docs/drafts, refresh README --- docs/README.md | 53 ++++++++++--------- .../scalable-web3-storage-implementation.md | 2 +- .../{design => drafts}/CHECKPOINT_PROTOCOL.md | 0 .../CLIENT_SIDE_ENCRYPTION.md | 0 docs/{design => drafts}/EXECUTION_FLOWS.md | 0 docs/{design => drafts}/S3_METADATA_INDEX.md | 0 docs/{design => drafts}/marketplace.md | 0 .../provider-initiated-checkpoints.md | 0 docs/{design => drafts}/smart-contracts.md | 0 docs/filesystems/API_REFERENCE.md | 2 +- docs/filesystems/README.md | 2 +- docs/filesystems/USER_GUIDE.md | 4 +- docs/getting-started/LAYER1_QUICKSTART.md | 2 +- 13 files changed, 33 insertions(+), 32 deletions(-) rename docs/{design => drafts}/CHECKPOINT_PROTOCOL.md (100%) rename docs/{design => drafts}/CLIENT_SIDE_ENCRYPTION.md (100%) rename docs/{design => drafts}/EXECUTION_FLOWS.md (100%) rename docs/{design => drafts}/S3_METADATA_INDEX.md (100%) rename docs/{design => drafts}/marketplace.md (100%) rename docs/{design => drafts}/provider-initiated-checkpoints.md (100%) rename docs/{design => drafts}/smart-contracts.md (100%) diff --git a/docs/README.md b/docs/README.md index f7549355..982d6b68 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,37 +1,46 @@ # Documentation -Project structure: - ``` docs/ ├── getting-started/ — quickstart ├── reference/ — pallet API + payment math -├── design/ — architecture, economics, protocols +├── design/ — canonical system design (review-gated) +├── drafts/ — work-in-progress protocol notes (need triage) └── filesystems/ — Layer 1 file system interface ``` +> `design/` is the source of truth and requires design review before changes. +> `drafts/` holds evolving notes that have not been ratified — treat as WIP. + ## Getting started -- **[Layer 1 Quick Start](./getting-started/LAYER1_QUICKSTART.md)** — three-terminal setup (chain → provider → demo) plus the SDK examples for the file-system and S3 interfaces. The canonical entry point. -- **[`FILE_SYSTEM_QUICKSTART.md`](../FILE_SYSTEM_QUICKSTART.md)** (repo root) — short version focused on the file-system layer only. -- **[`CLAUDE.md`](../CLAUDE.md)** (repo root) — build/test/run commands and code-review guidelines for contributors and AI agents. +- **[Layer 1 Quick Start](./getting-started/LAYER1_QUICKSTART.md)** — three-terminal setup (chain → provider → demo) plus SDK examples for the file-system and S3 interfaces. The canonical entry point. +- **[`FILE_SYSTEM_QUICKSTART.md`](../FILE_SYSTEM_QUICKSTART.md)** (repo root) — short version, file-system layer only. +- **[`CLAUDE.md`](../CLAUDE.md)** (repo root) — build/test/run commands and contributor guidelines. ## Reference - **[Extrinsics Reference](./reference/EXTRINSICS_REFERENCE.md)** — every pallet extrinsic with parameters, errors, and example workflows. -- **[Payment Calculator](./reference/PAYMENT_CALCULATOR.md)** — `payment = price_per_byte × max_bytes × duration`, with worked examples and the common `PaymentExceedsMax` failure mode. +- **[Payment Calculator](./reference/PAYMENT_CALCULATOR.md)** — `payment = price_per_byte × max_bytes × duration`, with worked examples. ## Design +The canonical system design. Changes require review (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)). + - **[Scalable Web3 Storage](./design/scalable-web3-storage.md)** — architecture, economic model, comparisons with Filecoin/IPFS/Arweave, rebuttals to common review concerns. - **[Implementation Details](./design/scalable-web3-storage-implementation.md)** — pallet extrinsics, provider HTTP API, MMR layout, challenge mechanism, replica sync. -- **[Smart Contracts](./design/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow. -- **[Execution Flows](./design/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs for the main flows. -- **[Marketplace](./design/marketplace.md)** — provider capacity, discovery, and matching. -- **[Checkpoint Protocol](./design/CHECKPOINT_PROTOCOL.md)** — multi-provider checkpoint coordination. -- **[Provider-Initiated Checkpoints](./design/provider-initiated-checkpoints.md)** — extension where providers proactively commit state. -- **[Client-Side Encryption](./design/CLIENT_SIDE_ENCRYPTION.md)** — wire format, cipher choice. -- **[S3 Metadata Index](./design/S3_METADATA_INDEX.md)** — how prefix/delimiter queries are served. + +## Drafts + +Work-in-progress notes, not yet ratified. **These need triage** — each should be reviewed and either promoted into `design/`, folded into an existing doc, or dropped. + +- **[Smart Contracts](./drafts/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow. +- **[Execution Flows](./drafts/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs for the main flows. +- **[Marketplace](./drafts/marketplace.md)** — provider capacity, discovery, and matching. +- **[Checkpoint Protocol](./drafts/CHECKPOINT_PROTOCOL.md)** — multi-provider checkpoint coordination. +- **[Provider-Initiated Checkpoints](./drafts/provider-initiated-checkpoints.md)** — extension where providers proactively commit state. +- **[Client-Side Encryption](./drafts/CLIENT_SIDE_ENCRYPTION.md)** — wire format, cipher choice. +- **[S3 Metadata Index](./drafts/S3_METADATA_INDEX.md)** — how prefix/delimiter queries are served. ## Layer 1 — file system interface @@ -41,24 +50,16 @@ docs/ - **[Admin Guide](./filesystems/ADMIN_GUIDE.md)** — operations, monitoring, dispute resolution. - **[API Reference](./filesystems/API_REFERENCE.md)** — extrinsics, SDK methods, primitives, events, errors. -## Other clients +## Clients - **[Storage Client SDK](../client/README.md)** — Layer-0 Rust client. - **[S3 Interface](../storage-interfaces/s3/README.md)** — Layer-1 S3-compatible Rust client. -- **TypeScript SDK** — `@web3-storage/sdk` at `packages/sdk` (umbrella over `packages/{core,layer0,layer1}`; `./fs`, `./s3`, and `./revive` subpaths). - -## Scripts - -- `scripts/build-chain-spec.sh` — build runtime + emit chain spec (used by `just generate-chain-spec`). -- `scripts/check-chain.sh` — relay + parachain health probe. -- `scripts/quick-test.sh` — curl-based smoke test of the Layer-0 provider HTTP API. +- **TypeScript SDK** — `@web3-storage/sdk` at `packages/sdk` (`./fs`, `./s3`, `./revive` subpaths). ## External -- [Polkadot SDK](https://paritytech.github.io/polkadot-sdk/) — FRAME, Cumulus, networking. -- [Substrate Docs](https://docs.substrate.io/). -- [Polkadot.js Apps](https://polkadot.js.org/apps/). +- [Polkadot SDK](https://paritytech.github.io/polkadot-sdk/) · [Substrate Docs](https://docs.substrate.io/) · [Polkadot.js Apps](https://polkadot.js.org/apps/) ## License -See the repository root [README](../README.md#license) for license details. +See the repository root [README](../README.md#license). diff --git a/docs/design/scalable-web3-storage-implementation.md b/docs/design/scalable-web3-storage-implementation.md index 56536f9a..8be3d8df 100644 --- a/docs/design/scalable-web3-storage-implementation.md +++ b/docs/design/scalable-web3-storage-implementation.md @@ -2178,7 +2178,7 @@ Response: ### Layer 1 — File System & S3 Interfaces The provider node also serves the Layer 1 interfaces described in -`docs/filesystems/` and `docs/design/marketplace.md`. These mount on top of +`docs/filesystems/` and `docs/drafts/marketplace.md`. These mount on top of the Layer 0 blob primitives and require Writer/Admin authorization for mutating routes. diff --git a/docs/design/CHECKPOINT_PROTOCOL.md b/docs/drafts/CHECKPOINT_PROTOCOL.md similarity index 100% rename from docs/design/CHECKPOINT_PROTOCOL.md rename to docs/drafts/CHECKPOINT_PROTOCOL.md diff --git a/docs/design/CLIENT_SIDE_ENCRYPTION.md b/docs/drafts/CLIENT_SIDE_ENCRYPTION.md similarity index 100% rename from docs/design/CLIENT_SIDE_ENCRYPTION.md rename to docs/drafts/CLIENT_SIDE_ENCRYPTION.md diff --git a/docs/design/EXECUTION_FLOWS.md b/docs/drafts/EXECUTION_FLOWS.md similarity index 100% rename from docs/design/EXECUTION_FLOWS.md rename to docs/drafts/EXECUTION_FLOWS.md diff --git a/docs/design/S3_METADATA_INDEX.md b/docs/drafts/S3_METADATA_INDEX.md similarity index 100% rename from docs/design/S3_METADATA_INDEX.md rename to docs/drafts/S3_METADATA_INDEX.md diff --git a/docs/design/marketplace.md b/docs/drafts/marketplace.md similarity index 100% rename from docs/design/marketplace.md rename to docs/drafts/marketplace.md diff --git a/docs/design/provider-initiated-checkpoints.md b/docs/drafts/provider-initiated-checkpoints.md similarity index 100% rename from docs/design/provider-initiated-checkpoints.md rename to docs/drafts/provider-initiated-checkpoints.md diff --git a/docs/design/smart-contracts.md b/docs/drafts/smart-contracts.md similarity index 100% rename from docs/design/smart-contracts.md rename to docs/drafts/smart-contracts.md diff --git a/docs/filesystems/API_REFERENCE.md b/docs/filesystems/API_REFERENCE.md index c1cc4dbc..8fb04d18 100644 --- a/docs/filesystems/API_REFERENCE.md +++ b/docs/filesystems/API_REFERENCE.md @@ -644,7 +644,7 @@ pub struct DirectoryEntry { ### Checkpoint Operations -Layer 1 checkpoint methods delegate to Layer 0's `CheckpointManager` for multi-provider coordination and consensus verification. See [Checkpoint Protocol Design](../design/CHECKPOINT_PROTOCOL.md) for details. +Layer 1 checkpoint methods delegate to Layer 0's `CheckpointManager` for multi-provider coordination and consensus verification. See [Checkpoint Protocol Design](../drafts/CHECKPOINT_PROTOCOL.md) for details. **Key Concepts:** - Layer 1 maps `drive_id` → `bucket_id` automatically diff --git a/docs/filesystems/README.md b/docs/filesystems/README.md index d0762a29..01c03e49 100644 --- a/docs/filesystems/README.md +++ b/docs/filesystems/README.md @@ -38,7 +38,7 @@ cargo run -p file-system-client --example basic_usage - [Layer 0 Design](../design/scalable-web3-storage.md) — the underlying storage system. - [Layer 0 Implementation](../design/scalable-web3-storage-implementation.md) — pallet, provider, MMR, challenges. -- [Checkpoint Protocol](../design/CHECKPOINT_PROTOCOL.md) — multi-provider checkpoint coordination used by Layer 1's commit strategies. +- [Checkpoint Protocol](../drafts/CHECKPOINT_PROTOCOL.md) — multi-provider checkpoint coordination used by Layer 1's commit strategies. ## License diff --git a/docs/filesystems/USER_GUIDE.md b/docs/filesystems/USER_GUIDE.md index 7a43355a..09cfe90a 100644 --- a/docs/filesystems/USER_GUIDE.md +++ b/docs/filesystems/USER_GUIDE.md @@ -548,7 +548,7 @@ let drive_id = fs_client.create_drive_with_providers( - Known reliable providers from past experience - Testing with specific provider configurations -For more details, see [Storage Marketplace Design](../design/marketplace.md). +For more details, see [Storage Marketplace Design](../drafts/marketplace.md). --- @@ -625,7 +625,7 @@ If providers disagree on the data state, the system detects and handles it: // 4. Continues with majority consensus ``` -For more details, see [Checkpoint Protocol Design](../design/CHECKPOINT_PROTOCOL.md). +For more details, see [Checkpoint Protocol Design](../drafts/CHECKPOINT_PROTOCOL.md). --- diff --git a/docs/getting-started/LAYER1_QUICKSTART.md b/docs/getting-started/LAYER1_QUICKSTART.md index c3669d4d..3060274c 100644 --- a/docs/getting-started/LAYER1_QUICKSTART.md +++ b/docs/getting-started/LAYER1_QUICKSTART.md @@ -392,7 +392,7 @@ Once infrastructure is running: ### Architecture - [System Design](../design/scalable-web3-storage.md) -- [Checkpoint Protocol](../design/CHECKPOINT_PROTOCOL.md) +- [Checkpoint Protocol](../drafts/CHECKPOINT_PROTOCOL.md) ## Quick Reference Card From b29db01d61baaa865d9f76e6e5a72d6fe411e991 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 10:26:52 +0200 Subject: [PATCH 03/13] docs: split Layer 1 design out of Layer 0 impl doc into drafts Move the Layer 1 (file-system & S3) interface design into docs/drafts/L1_design_implementation.md, remove it from the Layer 0 implementation doc (including the provider-API endpoint group and the auth reference), and flag it for triage against #51. Add a triage note to the root README. --- README.md | 6 +++ docs/README.md | 7 +++- .../scalable-web3-storage-implementation.md | 40 ++----------------- docs/drafts/L1_design_implementation.md | 36 +++++++++++++++++ 4 files changed, 50 insertions(+), 39 deletions(-) create mode 100644 docs/drafts/L1_design_implementation.md diff --git a/README.md b/README.md index 7313906d..efc7d016 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,12 @@ just demo ## File System Interface (Layer 1) +> [!NOTE] +> **Layer 1 design is under triage.** Its design/implementation notes have been +> split into a draft — [`docs/drafts/L1_design_implementation.md`](./docs/drafts/L1_design_implementation.md) — +> pending a decision on the layered architecture (what belongs on-chain vs. +> provider-only). See [#51](https://github.com/paritytech/web3-storage/issues/51). + The Layer 1 File System Interface provides a familiar file/folder abstraction over Layer 0's raw blob storage. diff --git a/docs/README.md b/docs/README.md index 982d6b68..34f1d819 100644 --- a/docs/README.md +++ b/docs/README.md @@ -34,7 +34,8 @@ The canonical system design. Changes require review (see [`.github/CODEOWNERS`]( Work-in-progress notes, not yet ratified. **These need triage** — each should be reviewed and either promoted into `design/`, folded into an existing doc, or dropped. -- **[Smart Contracts](./drafts/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow. +- **[Layer 1 Design / Implementation](./drafts/L1_design_implementation.md)** — file-system & S3 provider interfaces on top of Layer 0 (split out of the Layer 0 implementation doc); triage tracked in [#51](https://github.com/paritytech/web3-storage/issues/51). +- **[Smart Contracts](./drafts/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow. Candidate for promotion to `reference/` (it's API-reference material, not a draft design). - **[Execution Flows](./drafts/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs for the main flows. - **[Marketplace](./drafts/marketplace.md)** — provider capacity, discovery, and matching. - **[Checkpoint Protocol](./drafts/CHECKPOINT_PROTOCOL.md)** — multi-provider checkpoint coordination. @@ -58,7 +59,9 @@ Work-in-progress notes, not yet ratified. **These need triage** — each should ## External -- [Polkadot SDK](https://paritytech.github.io/polkadot-sdk/) · [Substrate Docs](https://docs.substrate.io/) · [Polkadot.js Apps](https://polkadot.js.org/apps/) +- [Polkadot SDK](https://paritytech.github.io/polkadot-sdk/) — FRAME, Cumulus, networking. +- [Substrate Docs](https://docs.substrate.io/). +- [Polkadot.js Apps](https://polkadot.js.org/apps/). ## License diff --git a/docs/design/scalable-web3-storage-implementation.md b/docs/design/scalable-web3-storage-implementation.md index 8be3d8df..59fbb6d4 100644 --- a/docs/design/scalable-web3-storage-implementation.md +++ b/docs/design/scalable-web3-storage-implementation.md @@ -1773,7 +1773,7 @@ pub enum ChallengeResponse { ## Off-Chain: Provider Node API The provider node exposes a JSON-over-HTTP API (axum) on, by default, -`http://localhost:3333`. Endpoints fall into five groups: +`http://localhost:3333`. Endpoints fall into four groups: 1. **Health & info** — public, unauthenticated. 2. **Layer-0 blob storage** — content-addressed node upload, existence check, @@ -1782,14 +1782,11 @@ The provider node exposes a JSON-over-HTTP API (axum) on, by default, replica providers; read-only. 4. **Provider-initiated checkpoint coordination** — proposal signing, leader duty, force trigger. Used by the autonomous checkpoint coordinator. -5. **Layer-1 interfaces** — S3-compatible (`/s3/...`) and POSIX-like - (`/fs/...`). See `docs/filesystems/` for the full spec. ### Authentication & RBAC -Mutating Layer-0 endpoints (`PUT /node`, `POST /commit`, `POST /delete`, the -Layer-1 mutating endpoints) and authenticated read endpoints require an -`Authorization` header. The provider node verifies an sr25519 signature +Mutating Layer-0 endpoints (`PUT /node`, `POST /commit`, `POST /delete`) and +authenticated read endpoints require an `Authorization` header. The provider node verifies an sr25519 signature locally and resolves the caller's role via a TTL-cached query against the chain's `Buckets` storage (`bucket.members`). @@ -2175,37 +2172,6 @@ Response: } ``` -### Layer 1 — File System & S3 Interfaces - -The provider node also serves the Layer 1 interfaces described in -`docs/filesystems/` and `docs/drafts/marketplace.md`. These mount on top of -the Layer 0 blob primitives and require Writer/Admin authorization for -mutating routes. - -``` -S3-compatible object storage -──────────────────────────── -PUT /s3/:bucket_id/object?key=path/to/object -GET /s3/:bucket_id/object?key=path/to/object -HEAD /s3/:bucket_id/object?key=path/to/object -DELETE /s3/:bucket_id/object?key=path/to/object -GET /s3/:bucket_id/objects # list -GET /s3/:bucket_id/index_root # current S3 index CID - -File system interface -───────────────────── -PUT /fs/:bucket_id/file?path=/dir/file.txt -GET /fs/:bucket_id/file?path=/dir/file.txt -DELETE /fs/:bucket_id/file?path=/dir/file.txt -POST /fs/:bucket_id/mkdir # body: { path: ... } -GET /fs/:bucket_id/ls?path=/dir -GET /fs/:bucket_id/index_root # current drive root CID -``` - -See [`docs/filesystems/API_REFERENCE.md`](../filesystems/API_REFERENCE.md) for -the full Layer 1 contract (request/response shapes, error codes, manifest -formats). - ### Replica Sync API Replicas sync data autonomously from primaries or other replicas using a diff --git a/docs/drafts/L1_design_implementation.md b/docs/drafts/L1_design_implementation.md new file mode 100644 index 00000000..89a880fa --- /dev/null +++ b/docs/drafts/L1_design_implementation.md @@ -0,0 +1,36 @@ +# Layer 1 — File System & S3 Interfaces (Design / Implementation) + +> **Draft — needs triage.** Extracted from the Layer 0 implementation design +> doc. Promote into `design/`, fold into `docs/filesystems/`, or drop — pending +> the layered-architecture decision in +> [#51](https://github.com/paritytech/web3-storage/issues/51) (what belongs +> on-chain vs. provider-only). + +The provider node also serves the Layer 1 interfaces described in +`docs/filesystems/` and [`./marketplace.md`](./marketplace.md). These mount on +top of the Layer 0 blob primitives and require Writer/Admin authorization for +mutating routes. + +``` +S3-compatible object storage +──────────────────────────── +PUT /s3/:bucket_id/object?key=path/to/object +GET /s3/:bucket_id/object?key=path/to/object +HEAD /s3/:bucket_id/object?key=path/to/object +DELETE /s3/:bucket_id/object?key=path/to/object +GET /s3/:bucket_id/objects # list +GET /s3/:bucket_id/index_root # current S3 index CID + +File system interface +───────────────────── +PUT /fs/:bucket_id/file?path=/dir/file.txt +GET /fs/:bucket_id/file?path=/dir/file.txt +DELETE /fs/:bucket_id/file?path=/dir/file.txt +POST /fs/:bucket_id/mkdir # body: { path: ... } +GET /fs/:bucket_id/ls?path=/dir +GET /fs/:bucket_id/index_root # current drive root CID +``` + +See [`docs/filesystems/API_REFERENCE.md`](../filesystems/API_REFERENCE.md) for +the full Layer 1 contract (request/response shapes, error codes, manifest +formats). From 865d90dec302344d2f9b214a0f28994d9b7639cb Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 11:53:05 +0200 Subject: [PATCH 04/13] docs: extract provider-initiated checkpoint coordination API to drafts Move the off-chain Provider-Initiated Checkpoint Coordination endpoint section out of the review-gated Layer 0 implementation doc into docs/drafts/provider-initiated-checkpoints.md (as the shipped API, flagging drift from that draft's proposed API), drop the now-orphaned provider-node endpoint group, and fix the group count. --- .../scalable-web3-storage-implementation.md | 67 +---------------- docs/drafts/provider-initiated-checkpoints.md | 71 +++++++++++++++++++ 2 files changed, 72 insertions(+), 66 deletions(-) diff --git a/docs/design/scalable-web3-storage-implementation.md b/docs/design/scalable-web3-storage-implementation.md index 83fdc6cb..2e673f77 100644 --- a/docs/design/scalable-web3-storage-implementation.md +++ b/docs/design/scalable-web3-storage-implementation.md @@ -1773,15 +1773,13 @@ pub enum ChallengeResponse { ## Off-Chain: Provider Node API The provider node exposes a JSON-over-HTTP API (axum) on, by default, -`http://localhost:3333`. Endpoints fall into four groups: +`http://localhost:3333`. Endpoints fall into three groups: 1. **Health & info** — public, unauthenticated. 2. **Layer-0 blob storage** — content-addressed node upload, existence check, commit, read, proofs, deletion. Mutating endpoints require auth. 3. **Replica sync** — peaks, subtree, bulk node fetch, sync status. Used by replica providers; read-only. -4. **Provider-initiated checkpoint coordination** — proposal signing, leader - duty, force trigger. Used by the autonomous checkpoint coordinator. ### Authentication & RBAC @@ -2071,69 +2069,6 @@ Note: Public observability endpoint. Useful for operators and the Prometheus/Grafana setup in `docs/`. ``` -### Provider-Initiated Checkpoint Coordination - -These endpoints back the autonomous checkpoint coordinator -(`checkpoint_coordinator.rs`). Primary providers exchange signed -`CheckpointProposal`s over HTTP, and one of them submits the -`provider_checkpoint` extrinsic on-chain. - -``` -Sign a Checkpoint Proposal -────────────────────────── -POST /checkpoint/sign - -Request: -{ - "bucket_id": 1234, - "mmr_root": "0xfed...", - "start_seq": 0, - "leaf_count": 42, - "window": 7 -} - -Response (200 OK): -{ - "signer": "5G...", // this provider's SS58 address - "signature": "0x...", // sr25519 over CheckpointProposal - "agreed": true, // false if local state diverges - "local_mmr_root": "0xfed..." // included for divergence diagnostics -} - -Note: If `agreed: false`, the signature is empty — the local view of the -bucket doesn't match the proposal. Callers compare `local_mmr_root` to -investigate divergence (e.g. one provider is behind). - -Get Checkpoint Duty -─────────────────── -GET /checkpoint/duty?bucket_id=1234 - -Response: -{ - "bucket_id": 1234, - "mmr_root": "0xfed...", - "start_seq": 0, - "leaf_count": 42, - "ready": true // false if leaf_count == 0 -} - -Trigger Checkpoint (operator-only) -────────────────────────────────── -POST /checkpoint/trigger?bucket_id=1234 -Authorization: Web3Storage <...> # Admin - -Response: -{ - "bucket_id": 1234, - "triggered": true, - "message": "Checkpoint triggered for bucket 1234 with 42 leaves..." -} - -Note: Sends a `ForceCheckpoint` command to the coordinator task. Requires the -provider to have been launched with `--enable-checkpoint-coordinator`, -otherwise returns 500. Mostly used in tests and manual recovery. -``` - ### Replica Sync Status ``` diff --git a/docs/drafts/provider-initiated-checkpoints.md b/docs/drafts/provider-initiated-checkpoints.md index 428b61b3..f1c48557 100644 --- a/docs/drafts/provider-initiated-checkpoints.md +++ b/docs/drafts/provider-initiated-checkpoints.md @@ -688,6 +688,77 @@ async fn discover_peers(&self, bucket_id: BucketId) -> Vec { --- +## Implemented HTTP API (as shipped) + +> ⚠️ **Drift from the proposal above.** These are the endpoints the provider +> node actually ships today (backing the autonomous checkpoint coordinator, +> `checkpoint_coordinator.rs`). They differ from the proposed +> `/checkpoint/propose` + `/checkpoint/status` in +> [Provider Node Changes](#2-provider-to-provider-communication) — reconcile the +> two during triage. Extracted from the Layer 0 implementation doc, which is +> review-gated and should not carry unratified design. + +Primary providers exchange signed `CheckpointProposal`s over HTTP, and one of +them submits the `provider_checkpoint` extrinsic on-chain. + +``` +Sign a Checkpoint Proposal +────────────────────────── +POST /checkpoint/sign + +Request: +{ + "bucket_id": 1234, + "mmr_root": "0xfed...", + "start_seq": 0, + "leaf_count": 42, + "window": 7 +} + +Response (200 OK): +{ + "signer": "5G...", // this provider's SS58 address + "signature": "0x...", // sr25519 over CheckpointProposal + "agreed": true, // false if local state diverges + "local_mmr_root": "0xfed..." // included for divergence diagnostics +} + +Note: If `agreed: false`, the signature is empty — the local view of the +bucket doesn't match the proposal. Callers compare `local_mmr_root` to +investigate divergence (e.g. one provider is behind). + +Get Checkpoint Duty +─────────────────── +GET /checkpoint/duty?bucket_id=1234 + +Response: +{ + "bucket_id": 1234, + "mmr_root": "0xfed...", + "start_seq": 0, + "leaf_count": 42, + "ready": true // false if leaf_count == 0 +} + +Trigger Checkpoint (operator-only) +────────────────────────────────── +POST /checkpoint/trigger?bucket_id=1234 +Authorization: Web3Storage <...> # Admin + +Response: +{ + "bucket_id": 1234, + "triggered": true, + "message": "Checkpoint triggered for bucket 1234 with 42 leaves..." +} + +Note: Sends a `ForceCheckpoint` command to the coordinator task. Requires the +provider to have been launched with `--enable-checkpoint-coordinator`, +otherwise returns 500. Mostly used in tests and manual recovery. +``` + +--- + ## Security Analysis ### Attack: Malicious Leader Submits Wrong Root From f8125b7998cd9688769acd927cdeb4bd85b2bf00 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 12:20:00 +0200 Subject: [PATCH 05/13] docs: remove provider-initiated checkpoint design from canonical impl doc Pull the entire provider-initiated checkpoint surface (config constants, storage, events, signed payload, on-chain extrinsics) out of the review-gated Layer 0 implementation doc, pending triage on whether the feature is wanted at all (see follow-up issue). Client-initiated checkpoint / extend_checkpoint / challenge_checkpoint are unchanged. The removed content remains recoverable from git for a clean revert. --- .../scalable-web3-storage-implementation.md | 221 +----------------- 1 file changed, 4 insertions(+), 217 deletions(-) diff --git a/docs/design/scalable-web3-storage-implementation.md b/docs/design/scalable-web3-storage-implementation.md index 2e673f77..22cae763 100644 --- a/docs/design/scalable-web3-storage-implementation.md +++ b/docs/design/scalable-web3-storage-implementation.md @@ -169,24 +169,6 @@ pub trait Config: frame_system::Config>> { #[pallet::constant] type RequestTimeout: Get>; - /// Default interval between provider-initiated checkpoint windows. - #[pallet::constant] - type DefaultCheckpointInterval: Get>; - - /// Default grace period (in blocks) for the elected leader before any - /// primary provider may submit the checkpoint as a fallback. - #[pallet::constant] - type DefaultCheckpointGrace: Get>; - - /// Reward paid (from the per-bucket checkpoint pool) to the submitter - /// of a provider-initiated checkpoint. - #[pallet::constant] - type CheckpointReward: Get>; - - /// Penalty slashed from a leader's stake if they miss their window. - #[pallet::constant] - type CheckpointMissPenalty: Get>; - /// Maximum number of buckets a single account can be a member of /// (bounds the per-account reverse index). #[pallet::constant] @@ -235,10 +217,6 @@ parachain `HOURS`: | `ChallengeTimeout` | `48 * RC_HOURS` | | `SettlementTimeout` | `24 * RC_HOURS` | | `RequestTimeout` | `6 * RC_HOURS` | -| `DefaultCheckpointInterval` | `100` anchor blocks (~10 min) | -| `DefaultCheckpointGrace` | `20` anchor blocks (~2 min) | -| `CheckpointReward` | `1_000_000_000_000` (1 token) | -| `CheckpointMissPenalty` | `500_000_000_000` (0.5 token) | | `MaxBucketsPerMember` | `1_000` | | `DeregisterAnnouncementPeriod` | `54 * RC_HOURS` (48h challenge window + 6h grace) | | `MaxChallengesPerDeadline` | `1_000` | @@ -574,54 +552,6 @@ pub struct Challenge { pub deposit: BalanceOf, } -// ───────────────────────────────────────────────────────────────────────────── -// Provider-initiated checkpoint storage -// ───────────────────────────────────────────────────────────────────────────── - -/// Per-bucket checkpoint window configuration. -/// `None` means the bucket uses `T::DefaultCheckpointInterval` / -/// `T::DefaultCheckpointGrace` and provider-initiated checkpoints are off. -#[pallet::storage] -pub type CheckpointConfigs = StorageMap< - _, - Blake2_128Concat, - BucketId, - CheckpointWindowConfig>, ->; - -pub struct CheckpointWindowConfig { - /// Blocks between checkpoint windows (e.g., 100 ≈ 10 minutes) - pub interval: BlockNumber, - /// Grace period reserved for the elected leader (e.g., 20 blocks) - pub grace_period: BlockNumber, - /// Whether provider-initiated checkpoints are enabled for this bucket - pub enabled: bool, -} - -/// Last successful checkpoint window per bucket. `None` = none yet. -/// Prevents both re-submission and re-reporting of the same window. -#[pallet::storage] -pub type LastCheckpointWindow = - StorageMap<_, Blake2_128Concat, BucketId, u64, OptionQuery>; - -/// Pending checkpoint rewards per (provider, bucket). -/// Provider-first key order so a provider's pending rewards can be drained -/// on `complete_deregister` without scanning every bucket. -#[pallet::storage] -pub type CheckpointRewards = StorageDoubleMap< - _, - Blake2_128Concat, T::AccountId, - Blake2_128Concat, BucketId, - BalanceOf, - ValueQuery, ->; - -/// Per-bucket reward pool used to pay providers for submitting checkpoints. -/// Funded permissionlessly via `fund_checkpoint_pool`. -#[pallet::storage] -pub type CheckpointPool = - StorageMap<_, Blake2_128Concat, BucketId, BalanceOf, ValueQuery>; - /// Reverse index: account → bucket IDs they are a member of. /// Bounded by `T::MaxBucketsPerMember` to keep iteration costs predictable. #[pallet::storage] @@ -641,7 +571,7 @@ Sr25519/Ed25519, 33 bytes for compressed Ecdsa). All on-chain signature verification uses `sp_runtime::MultiSignature` against this key, so a single provider can use any of the supported schemes. -Three on-chain signed payloads exist (all SCALE-encoded, all carry an explicit +Two on-chain signed payloads exist (all SCALE-encoded, all carry an explicit `version: u8` so the protocol can evolve without breaking existing signatures): - `CommitmentPayload { version, bucket_id, commitment, nonce }` — what @@ -650,9 +580,6 @@ Three on-chain signed payloads exist (all SCALE-encoded, all carry an explicit Structures](#data-structures)). For `challenge_offchain` the challenger passes the signed `commitment` through unchanged so the pallet's payload reconstruction matches the signature. -- `CheckpointProposal { version, bucket_id, commitment, window }` — - what providers sign for `provider_checkpoint`. The `window` field prevents - cross-window replay. - The replica sync `roots` array (`[Option; 7]`) — signed for `confirm_replica_sync` to attest which roots the replica actually has. @@ -865,47 +792,6 @@ pub enum Event { slashed_amount: BalanceOf, }, - // ───────────────────────────────────────────────────────────── - // Provider-initiated checkpoint events - // ───────────────────────────────────────────────────────────── - - /// A primary provider successfully submitted a checkpoint for a window. - /// `signers` lists every primary that contributed a valid signature; - /// `reward` is what was actually drawn from the pool (zero if empty). - ProviderCheckpointSubmitted { - bucket_id: BucketId, - mmr_root: H256, - window: u64, - leader: T::AccountId, - signers: Vec, - reward: BalanceOf, - }, - /// Admin updated this bucket's checkpoint window config. - CheckpointConfigUpdated { - bucket_id: BucketId, - interval: BlockNumberFor, - grace_period: BlockNumberFor, - enabled: bool, - }, - /// A leader was penalised for missing their window. Reporter received 10%. - CheckpointMissPenalized { - bucket_id: BucketId, - provider: T::AccountId, - window: u64, - penalty: BalanceOf, - }, - /// Provider drained accumulated rewards for a bucket. - CheckpointRewardClaimed { - bucket_id: BucketId, - provider: T::AccountId, - amount: BalanceOf, - }, - /// Funder added balance to a bucket's checkpoint reward pool. - CheckpointPoolFunded { - bucket_id: BucketId, - funder: T::AccountId, - amount: BalanceOf, - }, } ``` @@ -1030,9 +916,8 @@ impl Pallet { /// Finalise a previously-announced deregistration (step 2 of 2). /// /// Callable once `T::DeregisterAnnouncementPeriod` has elapsed since - /// `deregister_provider`. Drains pending `CheckpointRewards` for this - /// provider into their free balance, unreserves the remaining stake, - /// and removes the provider record. Still requires `committed_bytes == 0`. + /// `deregister_provider`. Unreserves the remaining stake and removes the + /// provider record. Still requires `committed_bytes == 0`. #[pallet::weight(...)] pub fn complete_deregister(origin: OriginFor) -> DispatchResult; @@ -1440,96 +1325,6 @@ impl Pallet { additional_signatures: BoundedVec<(T::AccountId, Signature), T::MaxPrimaryProviders>, ) -> DispatchResult; - // ───────────────────────────────────────────────────────────── - // Provider-initiated checkpoints (window-based) - // ───────────────────────────────────────────────────────────── - // - // Lets primary providers autonomously checkpoint on a schedule without - // requiring the client to be online. Blocks are divided into windows of - // length `config.interval`; for each window a deterministic leader is - // elected from `primary_providers` by hashing `(bucket_id, window)`. - // - // window n window n+1 - // ├── grace ──┤───── fallback ─────┤ - // - // - During the grace period (`config.grace_period` blocks at the start), - // ONLY the elected leader may submit via `provider_checkpoint`. - // - After the grace period, any primary provider may submit (fallback). - // - If the window closes with no submission, anyone can call - // `report_missed_checkpoint` to slash the leader's stake. - - /// Submit a checkpoint for the current window (provider-initiated). - /// - /// Caller must be the elected leader during the grace period, or any - /// primary provider afterwards. Signatures cover a `CheckpointProposal` - /// (including the `window` field, which prevents cross-window replay). - /// Must collect at least `bucket.min_providers` valid signatures. - /// - /// On success: snapshot is updated, `LastCheckpointWindow` is set to - /// `window`, historical roots are rotated, and the submitter accrues - /// `T::CheckpointReward` from the bucket's `CheckpointPool` (or zero if - /// the pool is empty — the checkpoint is still valid). - #[pallet::weight(...)] - pub fn provider_checkpoint( - origin: OriginFor, - bucket_id: BucketId, - commitment: Commitment, - window: u64, - signatures: BoundedVec< - (T::AccountId, MultiSignature), - T::MaxPrimaryProviders, - >, - ) -> DispatchResult; - - /// Configure provider-initiated checkpoint windows for a bucket (admin only). - /// - /// Setting `enabled: false` disables provider-initiated checkpoints - /// (client-initiated `checkpoint` still works). When unset, the bucket - /// uses `T::DefaultCheckpointInterval` / `T::DefaultCheckpointGrace`. - #[pallet::weight(...)] - pub fn configure_checkpoint_window( - origin: OriginFor, - bucket_id: BucketId, - interval: BlockNumberFor, - grace_period: BlockNumberFor, - enabled: bool, - ) -> DispatchResult; - - /// Report a missed checkpoint window and penalise the leader. - /// - /// Permissionless. Callable only after the window has fully passed (past - /// its grace period) and no checkpoint was submitted. The leader for - /// `window` is slashed by `T::CheckpointMissPenalty` (from reserved - /// stake), the reporter receives 10% of the actual slash, and - /// `LastCheckpointWindow` is bumped to `window` to prevent re-reporting. - #[pallet::weight(...)] - pub fn report_missed_checkpoint( - origin: OriginFor, - bucket_id: BucketId, - window: u64, - ) -> DispatchResult; - - /// Claim accumulated checkpoint rewards for a bucket. - /// - /// Drains `CheckpointRewards[(caller, bucket_id)]` into the caller's free - /// balance. Provider-keyed map layout means a provider can claim across - /// many buckets cheaply, and `complete_deregister` drains everything in - /// one pass. - #[pallet::weight(...)] - pub fn claim_checkpoint_rewards( - origin: OriginFor, - bucket_id: BucketId, - ) -> DispatchResult; - - /// Fund a bucket's checkpoint reward pool. Permissionless — anyone can - /// top up. Reserves `amount` from the caller into `CheckpointPool`. - #[pallet::weight(...)] - pub fn fund_checkpoint_pool( - origin: OriginFor, - bucket_id: BucketId, - amount: BalanceOf, - ) -> DispatchResult; - // ───────────────────────────────────────────────────────────── // Challenges // ───────────────────────────────────────────────────────────── @@ -2230,7 +2025,7 @@ confirm using an older historical root they successfully synced to. `Commitment` groups the `(mmr_root, start_seq, leaf_count)` triplet that identifies an MMR commitment over a contiguous range of leaves. It is a field -group inside `CommitmentPayload`, `CheckpointProposal`, and `BucketSnapshot`, +group inside `CommitmentPayload` and `BucketSnapshot`, and the single argument the checkpoint/challenge extrinsics take in place of three loose fields. @@ -2278,14 +2073,6 @@ pub struct CommitmentPayload { /// time the signer signed. Checked against `current_anchor_block()`. pub nonce: u64, } - -pub struct CheckpointProposal { - pub version: u8, // CURRENT_VERSION = 1 - pub bucket_id: BucketId, - pub commitment: Commitment, - /// Window number this proposal is for — prevents cross-window replay. - pub window: u64, -} ``` ### MMR Leaf From 572231f8fdc4dcd4322ae13118814f857dfea044 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 12:41:38 +0200 Subject: [PATCH 06/13] docs: link provider-initiated checkpoints draft to triage issue #306 --- docs/drafts/provider-initiated-checkpoints.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/drafts/provider-initiated-checkpoints.md b/docs/drafts/provider-initiated-checkpoints.md index f1c48557..388d7745 100644 --- a/docs/drafts/provider-initiated-checkpoints.md +++ b/docs/drafts/provider-initiated-checkpoints.md @@ -1,5 +1,13 @@ # Provider-Initiated Checkpoints Design +> ⚠️ **Needs triage — [#306](https://github.com/paritytech/web3-storage/issues/306).** +> This design was added and implemented without review, and is now questioned +> (it may be redundant with replica nodes + client-held signatures). Its design +> was removed from the review-gated `docs/design/` in #305. Pending #306, this +> doc is either reviewed and canonicalized, or removed together with the related +> code. The "Problem Statement" / "Why" reasoning below has **not** been +> validated — treat with skepticism. + ## Problem Statement The current checkpoint system requires the data owner's client to: From af5ef29c66abca414da743633fc83cdf425350d1 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 13:19:14 +0200 Subject: [PATCH 07/13] docs: cross-reference signature (#274) and auth/RBAC (#304) open issues --- .../scalable-web3-storage-implementation.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/design/scalable-web3-storage-implementation.md b/docs/design/scalable-web3-storage-implementation.md index 22cae763..01ecfc86 100644 --- a/docs/design/scalable-web3-storage-implementation.md +++ b/docs/design/scalable-web3-storage-implementation.md @@ -571,6 +571,14 @@ Sr25519/Ed25519, 33 bytes for compressed Ecdsa). All on-chain signature verification uses `sp_runtime::MultiSignature` against this key, so a single provider can use any of the supported schemes. +> **⚠️ Scheme asymmetry — [#274](https://github.com/paritytech/web3-storage/issues/274).** +> On-chain verification is multi-scheme (`MultiSignature`: +> Sr25519/Ed25519/Ecdsa), but the provider node's off-chain HTTP auth (see +> [Authentication & RBAC](#authentication--rbac)) verifies **sr25519 only**, and +> only sr25519 is exercised end-to-end. The supported matrix (incl. the reserved +> 64-byte / `Eth` shapes the pallet accepts but can't verify) is **unratified** — +> this doc should be updated once #274 decides it. + Two on-chain signed payloads exist (all SCALE-encoded, all carry an explicit `version: u8` so the protocol can evolve without breaking existing signatures): @@ -1583,6 +1591,15 @@ authenticated read endpoints require an `Authorization` header. The provider nod locally and resolves the caller's role via a TTL-cached query against the chain's `Buckets` storage (`bucket.members`). +> **⚠️ Under-specified — [#304](https://github.com/paritytech/web3-storage/issues/304).** +> This scheme grew organically across several crates and needs one source of +> truth: the wire format is currently defined twice (Rust `provider-negotiation` +> + TS `core`) and hand-synced; the provider also accepts a ``-wrapped +> form (what wallets / PAPI `signBytes` send) not documented below; and the +> signed message binds only method + bucket + timestamp — **no body or provider +> binding**, leaving a replay window (default 5 min skew). #304 tracks the +> canonical definition + the binding/replay fix. + ``` Authorization: Web3Storage :: From ab31799f8bbbf413105fa222a8a3880993ae80d9 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 13:32:35 +0200 Subject: [PATCH 08/13] docs: extract 'Capped Split' to drafts; record challenge two-tier design/code gap --- docs/README.md | 1 + docs/design/scalable-web3-storage.md | 11 ------ docs/drafts/challenge-economics-extensions.md | 39 +++++++++++++++++++ 3 files changed, 40 insertions(+), 11 deletions(-) create mode 100644 docs/drafts/challenge-economics-extensions.md diff --git a/docs/README.md b/docs/README.md index 34f1d819..8679230e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -42,6 +42,7 @@ Work-in-progress notes, not yet ratified. **These need triage** — each should - **[Provider-Initiated Checkpoints](./drafts/provider-initiated-checkpoints.md)** — extension where providers proactively commit state. - **[Client-Side Encryption](./drafts/CLIENT_SIDE_ENCRYPTION.md)** — wire format, cipher choice. - **[S3 Metadata Index](./drafts/S3_METADATA_INDEX.md)** — how prefix/delimiter queries are served. +- **[Challenge Economics — Extensions](./drafts/challenge-economics-extensions.md)** — speculative "Capped Split for the general public"; also records that the design's two-tier challenger split isn't implemented yet. ## Layer 1 — file system interface diff --git a/docs/design/scalable-web3-storage.md b/docs/design/scalable-web3-storage.md index fa86d721..cc87c258 100644 --- a/docs/design/scalable-web3-storage.md +++ b/docs/design/scalable-web3-storage.md @@ -1102,17 +1102,6 @@ replicas (which lack natural client verification) and fire-and-forget archival periodic proofs similar to Filecoin's PDP could be added as a premium feature. This can be layered on later without changing the core protocol. -### Capped Split for the General Public - -Today the general public gets no cost split (challengers pay the response in -full) to close the DDoS hole. A capped version could give the public *some* -leverage without reopening that hole: apply the split to anonymous challenges -too, but only up to a per-provider budget over a rolling window of X blocks. -Once the budget is spent, further public challenges revert to full pay until the -window resets. The budget caps the total a crowd can extract per window, so the -DDoS attack is bounded rather than open-ended. Left out of the initial design; -addable later without changing the core mechanism. - ### Isolation Mode Admins can instruct providers to temporarily refuse serving non-members, then challenge a specific provider. If that diff --git a/docs/drafts/challenge-economics-extensions.md b/docs/drafts/challenge-economics-extensions.md new file mode 100644 index 00000000..fd354bf4 --- /dev/null +++ b/docs/drafts/challenge-economics-extensions.md @@ -0,0 +1,39 @@ +# Challenge Economics — Extensions (Draft) + +> **Draft — needs triage.** Speculative extension extracted from the canonical +> design's "Future Directions". **Not implemented**, and it presupposes a base +> mechanism that is *also* not implemented yet — see the gap note below. + +## Capped Split for the General Public + +Today the general public gets no cost split (challengers pay the response in +full) to close the DDoS hole. A capped version could give the public *some* +leverage without reopening that hole: apply the split to anonymous challenges +too, but only up to a per-provider budget over a rolling window of X blocks. +Once the budget is spent, further public challenges revert to full pay until the +window resets. The budget caps the total a crowd can extract per window, so the +DDoS attack is bounded rather than open-ended. Left out of the initial design; +addable later without changing the core mechanism. + +## ⚠️ Prerequisite gap: the two-tier split isn't implemented + +Capped Split refines the **authorized-vs-public** distinction — but that +distinction does not exist in code today: + +- The design specifies two tiers (authorized bucket members/owners get a cost + split where the provider bears a fraction; the general public pays in full, + for anti-DoS). +- The pallet (`crates/pallets/storage-provider/src/lib.rs`, challenge + resolution) applies a purely **response-time-based** split + (challenger 90→50%) to **every** challenger, with no authorized/public check + anywhere. So the design's anti-DoS premise is currently unenforced, and a + public challenger already receives the split. +- Possible further divergence to verify: on a *valid* defense the code slashes + the provider's stake for `provider_cost`, whereas the design states a valid + response never touches stake (the fraction should come from the challenger's + deposit). + +**Triage:** decide the challenge cost model end-to-end — implement the two tiers +(then Capped Split becomes a coherent add-on) or amend the design to match the +flat time-based split that shipped. Capped Split should not be revisited until +the base tier is settled. From 0274079cfed10d30749cddbb2fd8ce4c008d0b2b Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 15:34:29 +0200 Subject: [PATCH 09/13] docs: fix EXECUTION_FLOWS link to impl doc after move to drafts/ --- docs/drafts/EXECUTION_FLOWS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/drafts/EXECUTION_FLOWS.md b/docs/drafts/EXECUTION_FLOWS.md index 6d2316c9..4bc08167 100644 --- a/docs/drafts/EXECUTION_FLOWS.md +++ b/docs/drafts/EXECUTION_FLOWS.md @@ -7,7 +7,7 @@ timeouts, `current_block`) is denominated on the **anchor clock** — the relay chain in production, read via `current_anchor_block()`. Pseudocode showing `frame_system::block_number()` or `current_block` is illustrative; the implementation reads the anchor. See the anchor-clock section in -[scalable-web3-storage-implementation.md](scalable-web3-storage-implementation.md). +[scalable-web3-storage-implementation.md](../design/scalable-web3-storage-implementation.md). ## Table of Contents From d06190ce9f081c1b2917940551a023615af9db27 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 15:43:38 +0200 Subject: [PATCH 10/13] docs: move EXECUTION_FLOWS to reference/ (derived, ungated); clarify design/reference/drafts in README --- docs/README.md | 13 +++++++------ docs/drafts/CHECKPOINT_PROTOCOL.md | 2 +- docs/{drafts => reference}/EXECUTION_FLOWS.md | 0 3 files changed, 8 insertions(+), 7 deletions(-) rename docs/{drafts => reference}/EXECUTION_FLOWS.md (100%) diff --git a/docs/README.md b/docs/README.md index 8679230e..59cc2ed0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,14 +3,15 @@ ``` docs/ ├── getting-started/ — quickstart -├── reference/ — pallet API + payment math +├── reference/ — derived / how-it-works docs (accurate, not gated) ├── design/ — canonical system design (review-gated) -├── drafts/ — work-in-progress protocol notes (need triage) +├── drafts/ — unratified / WIP notes (need triage) └── filesystems/ — Layer 1 file system interface ``` -> `design/` is the source of truth and requires design review before changes. -> `drafts/` holds evolving notes that have not been ratified — treat as WIP. +> - **`design/`** — the source of truth; changes require design-owner review (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)). +> - **`reference/`** — accurate but *derived* material (flow walkthroughs, API refs). Kept in sync with the design, but not itself design-of-record, so it is **not** review-gated. +> - **`drafts/`** — unratified / WIP notes; treat as provisional. Each needs triage: promote to `design/` or `reference/`, fold into an existing doc, or drop. ## Getting started @@ -22,6 +23,7 @@ docs/ - **[Extrinsics Reference](./reference/EXTRINSICS_REFERENCE.md)** — every pallet extrinsic with parameters, errors, and example workflows. - **[Payment Calculator](./reference/PAYMENT_CALCULATOR.md)** — `payment = price_per_byte × max_bytes × duration`, with worked examples. +- **[Execution Flows](./reference/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs of the main flows (derived from the design; no design-owner sign-off needed). ## Design @@ -32,11 +34,10 @@ The canonical system design. Changes require review (see [`.github/CODEOWNERS`]( ## Drafts -Work-in-progress notes, not yet ratified. **These need triage** — each should be reviewed and either promoted into `design/`, folded into an existing doc, or dropped. +Unratified / WIP notes. **These need triage** ([#308](https://github.com/paritytech/web3-storage/issues/308)) — each should be reviewed and either promoted into `design/` (design-of-record) or `reference/` (accurate derived material), folded into an existing doc, or dropped. - **[Layer 1 Design / Implementation](./drafts/L1_design_implementation.md)** — file-system & S3 provider interfaces on top of Layer 0 (split out of the Layer 0 implementation doc); triage tracked in [#51](https://github.com/paritytech/web3-storage/issues/51). - **[Smart Contracts](./drafts/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow. Candidate for promotion to `reference/` (it's API-reference material, not a draft design). -- **[Execution Flows](./drafts/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs for the main flows. - **[Marketplace](./drafts/marketplace.md)** — provider capacity, discovery, and matching. - **[Checkpoint Protocol](./drafts/CHECKPOINT_PROTOCOL.md)** — multi-provider checkpoint coordination. - **[Provider-Initiated Checkpoints](./drafts/provider-initiated-checkpoints.md)** — extension where providers proactively commit state. diff --git a/docs/drafts/CHECKPOINT_PROTOCOL.md b/docs/drafts/CHECKPOINT_PROTOCOL.md index b91ae818..32dde639 100644 --- a/docs/drafts/CHECKPOINT_PROTOCOL.md +++ b/docs/drafts/CHECKPOINT_PROTOCOL.md @@ -984,6 +984,6 @@ fs_client.disable_auto_checkpoints().await?; ## Related Documents -- [Execution Flows](./EXECUTION_FLOWS.md) - Detailed sequence diagrams +- [Execution Flows](../reference/EXECUTION_FLOWS.md) - Detailed sequence diagrams - [Architecture](../filesystems/ARCHITECTURE.md) - System architecture - [API Reference](../filesystems/API_REFERENCE.md) - Complete API docs diff --git a/docs/drafts/EXECUTION_FLOWS.md b/docs/reference/EXECUTION_FLOWS.md similarity index 100% rename from docs/drafts/EXECUTION_FLOWS.md rename to docs/reference/EXECUTION_FLOWS.md From 40572dd6038a0861b29ac2197b15e13ba1438cfa Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 15:54:05 +0200 Subject: [PATCH 11/13] docs: add design/spec discipline rules to CLAUDE.md; repoint moved-doc links --- CLAUDE.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 057c202a..7aa75447 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,6 +23,14 @@ - ALWAYS run `/format` before creating any git commit - This ensures all code follows project formatting standards (Rust, TOML, feature propagation) and passes clippy +**Design & spec discipline:** +- `docs/design/` is the **canonical, review-gated source of truth** (enforced by `.github/CODEOWNERS`). Reason and implement *from* it; treat it as the spec. +- **Validate code against the design.** When writing or changing code, check it against `docs/design/`. On any divergence, **stop and flag** — don't proceed on assumptions. +- **Prefer flagging over quietly editing the design to match the code.** If implementation and design disagree, treat it as a *finding*: open or reference an issue and discuss before changing the spec, rather than silently reconciling the gap. +- If something in the design looks **wrong or vulnerable**, **flag and discuss** (open an issue and ping the design owner) — don't just fix it. Changes to the design itself go through a PR reviewed per `.github/CODEOWNERS`. +- **`docs/reference/`** is *derived* documentation and **not** gated. When you change behavior, **update the relevant `reference/` doc** so it keeps reflecting the implementation. +- **`docs/drafts/`** is unratified / WIP — don't treat it as authoritative or reason from it as if it were the spec. + ## Project Overview Scalable Web3 Storage is a decentralized storage system built on Substrate with game-theoretic guarantees. Storage providers lock stake and face slashing for data loss, while the chain acts as a credible threat rather than the hot path. @@ -298,7 +306,7 @@ web3-storage/ **Primitives (`primitives/`)**: Shared types used across pallet, provider node, and client. -**Smart Contracts (`precompiles/`, `examples/contracts/`)**: `pallet_revive` (PolkaVM-based smart contracts) is wired into both runtimes. Two custom precompiles expose the client-side bucket lifecycle and drive registry to Solidity contracts. The `StorageMarketplace.sol` example shows how a dApp buys storage on behalf of its users; `just sc-demo` runs the end-to-end PAPI test. Full design in [docs/design/smart-contracts.md](docs/design/smart-contracts.md). +**Smart Contracts (`precompiles/`, `examples/contracts/`)**: `pallet_revive` (PolkaVM-based smart contracts) is wired into both runtimes. Two custom precompiles expose the client-side bucket lifecycle and drive registry to Solidity contracts. The `StorageMarketplace.sol` example shows how a dApp buys storage on behalf of its users; `just sc-demo` runs the end-to-end PAPI test. Full design in [docs/drafts/smart-contracts.md](docs/drafts/smart-contracts.md). #### Layer 1 (File System Interface) @@ -530,7 +538,7 @@ let recommendations = client.suggest_providers(bytes, duration, budget).await?; - Price (too high = -30 points) - Duration (mismatch = -20 points) -See [Storage Marketplace Design](docs/design/marketplace.md) for details. +See [Storage Marketplace Design](docs/drafts/marketplace.md) for details. ### Checkpoint Management @@ -564,7 +572,7 @@ handle.stop().await?; // Stop background loop - `EventSubscriber`: Real-time blockchain event monitoring (checkpoints, challenges) - `ProviderHealthHistory`: Tracks provider reliability and response times -See [Checkpoint Protocol Design](docs/design/CHECKPOINT_PROTOCOL.md) for details. +See [Checkpoint Protocol Design](docs/drafts/CHECKPOINT_PROTOCOL.md) for details. ### Event Subscription @@ -649,9 +657,9 @@ For the full review criteria (Parity Standards), see the `/review` skill. The re | [Payment Calculator](docs/reference/PAYMENT_CALCULATOR.md) | Calculate agreement costs | | [Architecture Design](docs/design/scalable-web3-storage.md) | System design, economics, common concerns | | [Implementation Details](docs/design/scalable-web3-storage-implementation.md) | Technical specs | -| [Execution Flows](docs/design/EXECUTION_FLOWS.md) | Sequence diagrams for all extrinsics | -| [Storage Marketplace](docs/design/marketplace.md) | Provider capacity & discovery | -| [Checkpoint Protocol](docs/design/CHECKPOINT_PROTOCOL.md) | Automated checkpoint management | +| [Execution Flows](docs/reference/EXECUTION_FLOWS.md) | Sequence diagrams for all extrinsics | +| [Storage Marketplace](docs/drafts/marketplace.md) | Provider capacity & discovery | +| [Checkpoint Protocol](docs/drafts/CHECKPOINT_PROTOCOL.md) | Automated checkpoint management | | [File System Architecture](docs/filesystems/ARCHITECTURE.md) | Layer 1 encoding, security, blockchain details | ## Common Issues & Solutions From c67b6e69f03ee382d7461bc2266bb00cdf322b37 Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 16:02:45 +0200 Subject: [PATCH 12/13] docs: move FILE_SYSTEM_QUICKSTART.md into docs/getting-started/, fix all links --- CLAUDE.md | 3 +-- README.md | 6 +++--- docs/README.md | 2 +- .../getting-started/FILE_SYSTEM_QUICKSTART.md | 10 +++++----- 4 files changed, 10 insertions(+), 11 deletions(-) rename FILE_SYSTEM_QUICKSTART.md => docs/getting-started/FILE_SYSTEM_QUICKSTART.md (87%) diff --git a/CLAUDE.md b/CLAUDE.md index 7aa75447..47841306 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -148,7 +148,7 @@ just fs-demo-ci cargo run -p file-system-client --example basic_usage ``` -**Quick Start Guide**: [FILE_SYSTEM_QUICKSTART.md](./FILE_SYSTEM_QUICKSTART.md) +**Quick Start Guide**: [FILE_SYSTEM_QUICKSTART.md](docs/getting-started/FILE_SYSTEM_QUICKSTART.md) **Complete Documentation**: [docs/filesystems/README.md](./docs/filesystems/README.md) @@ -280,7 +280,6 @@ web3-storage/ │ ├── USER_GUIDE.md # User guide │ ├── API_REFERENCE.md # API documentation │ └── ADMIN_GUIDE.md # Admin guide -├── FILE_SYSTEM_QUICKSTART.md # Quick start for file system └── justfile # Development commands ``` diff --git a/README.md b/README.md index 811a4d3b..694e397f 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ just demo status), `just health` (provider node health), `just stats` (provider stats). 3. **Build something on top**: see [Client Documentation](./client/README.md) - for the Layer-0 SDK, or [`FILE_SYSTEM_QUICKSTART.md`](./FILE_SYSTEM_QUICKSTART.md) + for the Layer-0 SDK, or [`FILE_SYSTEM_QUICKSTART.md`](./docs/getting-started/FILE_SYSTEM_QUICKSTART.md) for the Layer-1 file-system interface. ## File System Interface (Layer 1) @@ -99,7 +99,7 @@ just fs-test-all # Run all unit tests across primitives, pallet, clie just fs-demo-ci # Integration example against a running chain + provider ``` -**Complete guide**: [FILE_SYSTEM_QUICKSTART.md](./FILE_SYSTEM_QUICKSTART.md) +**Complete guide**: [FILE_SYSTEM_QUICKSTART.md](./docs/getting-started/FILE_SYSTEM_QUICKSTART.md) ### When to Use Layer 0 vs Layer 1 @@ -149,7 +149,7 @@ just s3-test-all # S3 layer only | Document | Description | |----------|-------------| | **[Layer 1 Quick Start](./docs/getting-started/LAYER1_QUICKSTART.md)** | **Three-terminal setup + SDK examples (recommended)** | -| [File System Quick Start](./FILE_SYSTEM_QUICKSTART.md) | File-system-only quickstart | +| [File System Quick Start](./docs/getting-started/FILE_SYSTEM_QUICKSTART.md) | File-system-only quickstart | | [File System Docs](./docs/filesystems/README.md) | Complete Layer 1 documentation | | [Extrinsics Reference](./docs/reference/EXTRINSICS_REFERENCE.md) | Complete blockchain API | | [Payment Calculator](./docs/reference/PAYMENT_CALCULATOR.md) | Calculate agreement costs | diff --git a/docs/README.md b/docs/README.md index 59cc2ed0..9d494aca 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,7 +16,7 @@ docs/ ## Getting started - **[Layer 1 Quick Start](./getting-started/LAYER1_QUICKSTART.md)** — three-terminal setup (chain → provider → demo) plus SDK examples for the file-system and S3 interfaces. The canonical entry point. -- **[`FILE_SYSTEM_QUICKSTART.md`](../FILE_SYSTEM_QUICKSTART.md)** (repo root) — short version, file-system layer only. +- **[`FILE_SYSTEM_QUICKSTART.md`](./getting-started/FILE_SYSTEM_QUICKSTART.md)** — short version, file-system layer only. - **[`CLAUDE.md`](../CLAUDE.md)** (repo root) — build/test/run commands and contributor guidelines. ## Reference diff --git a/FILE_SYSTEM_QUICKSTART.md b/docs/getting-started/FILE_SYSTEM_QUICKSTART.md similarity index 87% rename from FILE_SYSTEM_QUICKSTART.md rename to docs/getting-started/FILE_SYSTEM_QUICKSTART.md index e6d96956..da7e965c 100644 --- a/FILE_SYSTEM_QUICKSTART.md +++ b/docs/getting-started/FILE_SYSTEM_QUICKSTART.md @@ -91,8 +91,8 @@ just build ## Further reading -- [User Guide](docs/filesystems/USER_GUIDE.md) — complete user workflows -- [Architecture](docs/filesystems/ARCHITECTURE.md) — encoding, security, chain integration -- [API Reference](docs/filesystems/API_REFERENCE.md) — complete API docs -- [Client README](storage-interfaces/file-system/client/README.md) — SDK docs -- [Architecture](docs/filesystems/ARCHITECTURE.md) — encoding, security, chain integration +- [User Guide](../filesystems/USER_GUIDE.md) — complete user workflows +- [Architecture](../filesystems/ARCHITECTURE.md) — encoding, security, chain integration +- [API Reference](../filesystems/API_REFERENCE.md) — complete API docs +- [Client README](../../storage-interfaces/file-system/client/README.md) — SDK docs +- [Architecture](../filesystems/ARCHITECTURE.md) — encoding, security, chain integration From 7025bb90ec14c5cdae81b8e881e906305ec8f96e Mon Sep 17 00:00:00 2001 From: Branislav Kontur Date: Wed, 22 Jul 2026 16:23:32 +0200 Subject: [PATCH 13/13] docs: fix stale sc-flow.js link (now sc-flow.ts) in smart-contracts draft --- docs/drafts/smart-contracts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/drafts/smart-contracts.md b/docs/drafts/smart-contracts.md index 00b7619b..6b6bacf4 100644 --- a/docs/drafts/smart-contracts.md +++ b/docs/drafts/smart-contracts.md @@ -2,7 +2,7 @@ Web3-storage exposes its client-side surface to Solidity / PolkaVM contracts via `pallet_revive` and two custom precompiles. A dApp can buy storage on behalf of its users, end agreements, or stitch the storage pallets into larger on-chain workflows — without ever signing a substrate-native extrinsic. -This document covers the **on-chain shape** of that integration. For the example dApp, see [`examples/contracts/`](../../examples/contracts/README.md); for the test driver, see [`examples/papi/sc-flow.js`](../../examples/papi/sc-flow.js). +This document covers the **on-chain shape** of that integration. For the example dApp, see [`examples/contracts/`](../../examples/contracts/README.md); for the test driver, see [`examples/papi/sc-flow.ts`](../../examples/papi/sc-flow.ts). ## Architecture @@ -120,7 +120,7 @@ For extrinsics with parameterized weights (e.g. `end_agreement(a: u32)`), we pas Four scripts cover the surface end-to-end: -- **`just sc-demo`** (`examples/papi/sc-flow.js`) — full marketplace dApp story via `StorageMarketplace.sol`: deploy, `buyStorage` with `msg.value`, off-chain upload/challenge round-trip, `endMyAgreement`. Asserts provider earned tokens + contract events fired. +- **`just sc-demo`** (`examples/papi/sc-flow.ts`) — full marketplace dApp story via `StorageMarketplace.sol`: deploy, `buyStorage` with `msg.value`, off-chain upload/challenge round-trip, `endMyAgreement`. Asserts provider earned tokens + contract events fired. - **`just sc-coverage`** (`examples/papi/sc-coverage.js`) — direct precompile invocations (no intermediate contract) for every selector across all three precompiles. Each call submits as a signed substrate tx whose `dest` is the precompile address; on success, the script asserts the underlying pallet's storage / event was updated. - **`just sc-team-drive`** (`examples/papi/sc-team-drive.js`) — drive-registry dApp via `SharedTeamDrive.sol`: deploy, `createTeam`, `invite` / `kick`, `disband`. - **`just sc-token-gated`** (`examples/papi/sc-token-gated.js`) — s3-registry dApp via `TokenGatedDrive.sol`: deploy, `initialize`, `mint` an NFT-shaped access token per S3 object, `transfer`, `burn` (deletes object metadata), `shutdown`.