Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# CODEOWNERS
#
# Lines are matched bottom-to-top; the LAST matching pattern wins for a file.
# A required-review rule is satisfied by any ONE of the owners listed for the
# matched pattern — so listing a single owner makes that person's review
# mandatory. Owners must have write access to the repo for their review to count.
# GitHub reads this file from the repository's default branch (`dev`).
#
# See: https://docs.github.com/articles/about-code-owners

# Design documents require review from @eskimor.
# `/**` matches every file in docs/design/ recursively — all subdirectories,
# all file types, no exceptions.
/docs/design/** @eskimor
23 changes: 15 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@
- ALWAYS run `/format` before creating any git commit
- This ensures all code follows project formatting standards (Rust, TOML, feature propagation) and passes clippy

**Design & spec discipline:**
- `docs/design/` is the **canonical, review-gated source of truth** (enforced by `.github/CODEOWNERS`). Reason and implement *from* it; treat it as the spec.
- **Validate code against the design.** When writing or changing code, check it against `docs/design/`. On any divergence, **stop and flag** — don't proceed on assumptions.
- **Prefer flagging over quietly editing the design to match the code.** If implementation and design disagree, treat it as a *finding*: open or reference an issue and discuss before changing the spec, rather than silently reconciling the gap.
- If something in the design looks **wrong or vulnerable**, **flag and discuss** (open an issue and ping the design owner) — don't just fix it. Changes to the design itself go through a PR reviewed per `.github/CODEOWNERS`.
- **`docs/reference/`** is *derived* documentation and **not** gated. When you change behavior, **update the relevant `reference/` doc** so it keeps reflecting the implementation.
- **`docs/drafts/`** is unratified / WIP — don't treat it as authoritative or reason from it as if it were the spec.

## Project Overview

Scalable Web3 Storage is a decentralized storage system built on Substrate with game-theoretic guarantees. Storage providers lock stake and face slashing for data loss, while the chain acts as a credible threat rather than the hot path.
Expand Down Expand Up @@ -140,7 +148,7 @@ just fs-demo-ci
cargo run -p file-system-client --example basic_usage
```

**Quick Start Guide**: [FILE_SYSTEM_QUICKSTART.md](./FILE_SYSTEM_QUICKSTART.md)
**Quick Start Guide**: [FILE_SYSTEM_QUICKSTART.md](docs/getting-started/FILE_SYSTEM_QUICKSTART.md)

**Complete Documentation**: [docs/filesystems/README.md](./docs/filesystems/README.md)

Expand Down Expand Up @@ -272,7 +280,6 @@ web3-storage/
│ ├── USER_GUIDE.md # User guide
│ ├── API_REFERENCE.md # API documentation
│ └── ADMIN_GUIDE.md # Admin guide
├── FILE_SYSTEM_QUICKSTART.md # Quick start for file system
└── justfile # Development commands
```

Expand All @@ -298,7 +305,7 @@ web3-storage/

**Primitives (`primitives/`)**: Shared types used across pallet, provider node, and client.

**Smart Contracts (`precompiles/`, `examples/contracts/`)**: `pallet_revive` (PolkaVM-based smart contracts) is wired into both runtimes. Two custom precompiles expose the client-side bucket lifecycle and drive registry to Solidity contracts. The `StorageMarketplace.sol` example shows how a dApp buys storage on behalf of its users; `just sc-demo` runs the end-to-end PAPI test. Full design in [docs/design/smart-contracts.md](docs/design/smart-contracts.md).
**Smart Contracts (`precompiles/`, `examples/contracts/`)**: `pallet_revive` (PolkaVM-based smart contracts) is wired into both runtimes. Two custom precompiles expose the client-side bucket lifecycle and drive registry to Solidity contracts. The `StorageMarketplace.sol` example shows how a dApp buys storage on behalf of its users; `just sc-demo` runs the end-to-end PAPI test. Full design in [docs/drafts/smart-contracts.md](docs/drafts/smart-contracts.md).

#### Layer 1 (File System Interface)

Expand Down Expand Up @@ -530,7 +537,7 @@ let recommendations = client.suggest_providers(bytes, duration, budget).await?;
- Price (too high = -30 points)
- Duration (mismatch = -20 points)

See [Storage Marketplace Design](docs/design/marketplace.md) for details.
See [Storage Marketplace Design](docs/drafts/marketplace.md) for details.

### Checkpoint Management

Expand Down Expand Up @@ -564,7 +571,7 @@ handle.stop().await?; // Stop background loop
- `EventSubscriber`: Real-time blockchain event monitoring (checkpoints, challenges)
- `ProviderHealthHistory`: Tracks provider reliability and response times

See [Checkpoint Protocol Design](docs/design/CHECKPOINT_PROTOCOL.md) for details.
See [Checkpoint Protocol Design](docs/drafts/CHECKPOINT_PROTOCOL.md) for details.

### Event Subscription

Expand Down Expand Up @@ -649,9 +656,9 @@ For the full review criteria (Parity Standards), see the `/review` skill. The re
| [Payment Calculator](docs/reference/PAYMENT_CALCULATOR.md) | Calculate agreement costs |
| [Architecture Design](docs/design/scalable-web3-storage.md) | System design, economics, common concerns |
| [Implementation Details](docs/design/scalable-web3-storage-implementation.md) | Technical specs |
| [Execution Flows](docs/design/EXECUTION_FLOWS.md) | Sequence diagrams for all extrinsics |
| [Storage Marketplace](docs/design/marketplace.md) | Provider capacity & discovery |
| [Checkpoint Protocol](docs/design/CHECKPOINT_PROTOCOL.md) | Automated checkpoint management |
| [Execution Flows](docs/reference/EXECUTION_FLOWS.md) | Sequence diagrams for all extrinsics |
| [Storage Marketplace](docs/drafts/marketplace.md) | Provider capacity & discovery |
| [Checkpoint Protocol](docs/drafts/CHECKPOINT_PROTOCOL.md) | Automated checkpoint management |
| [File System Architecture](docs/filesystems/ARCHITECTURE.md) | Layer 1 encoding, security, blockchain details |

## Common Issues & Solutions
Expand Down
12 changes: 9 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,17 @@ just demo
status), `just health` (provider node health), `just stats` (provider stats).

3. **Build something on top**: see [Client Documentation](./client/README.md)
for the Layer-0 SDK, or [`FILE_SYSTEM_QUICKSTART.md`](./FILE_SYSTEM_QUICKSTART.md)
for the Layer-0 SDK, or [`FILE_SYSTEM_QUICKSTART.md`](./docs/getting-started/FILE_SYSTEM_QUICKSTART.md)
for the Layer-1 file-system interface.

## File System Interface (Layer 1)

> [!NOTE]
> **Layer 1 design is under triage.** Its design/implementation notes have been
> split into a draft — [`docs/drafts/L1_design_implementation.md`](./docs/drafts/L1_design_implementation.md) —
> pending a decision on the layered architecture (what belongs on-chain vs.
> provider-only). See [#51](https://github.com/paritytech/web3-storage/issues/51).

The Layer 1 File System Interface provides a familiar file/folder abstraction
over Layer 0's raw blob storage.

Expand All @@ -93,7 +99,7 @@ just fs-test-all # Run all unit tests across primitives, pallet, clie
just fs-demo-ci # Integration example against a running chain + provider
```

**Complete guide**: [FILE_SYSTEM_QUICKSTART.md](./FILE_SYSTEM_QUICKSTART.md)
**Complete guide**: [FILE_SYSTEM_QUICKSTART.md](./docs/getting-started/FILE_SYSTEM_QUICKSTART.md)

### When to Use Layer 0 vs Layer 1

Expand Down Expand Up @@ -143,7 +149,7 @@ just s3-test-all # S3 layer only
| Document | Description |
|----------|-------------|
| **[Layer 1 Quick Start](./docs/getting-started/LAYER1_QUICKSTART.md)** | **Three-terminal setup + SDK examples (recommended)** |
| [File System Quick Start](./FILE_SYSTEM_QUICKSTART.md) | File-system-only quickstart |
| [File System Quick Start](./docs/getting-started/FILE_SYSTEM_QUICKSTART.md) | File-system-only quickstart |
| [File System Docs](./docs/filesystems/README.md) | Complete Layer 1 documentation |
| [Extrinsics Reference](./docs/reference/EXTRINSICS_REFERENCE.md) | Complete blockchain API |
| [Payment Calculator](./docs/reference/PAYMENT_CALCULATOR.md) | Calculate agreement costs |
Expand Down
54 changes: 30 additions & 24 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,49 @@
# Documentation

Project structure:

```
docs/
├── getting-started/ — quickstart
├── reference/ — pallet API + payment math
├── design/ — architecture, economics, protocols
├── reference/ — derived / how-it-works docs (accurate, not gated)
├── design/ — canonical system design (review-gated)
├── drafts/ — unratified / WIP notes (need triage)
└── filesystems/ — Layer 1 file system interface
```

> - **`design/`** — the source of truth; changes require design-owner review (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)).
> - **`reference/`** — accurate but *derived* material (flow walkthroughs, API refs). Kept in sync with the design, but not itself design-of-record, so it is **not** review-gated.
> - **`drafts/`** — unratified / WIP notes; treat as provisional. Each needs triage: promote to `design/` or `reference/`, fold into an existing doc, or drop.

## Getting started

- **[Layer 1 Quick Start](./getting-started/LAYER1_QUICKSTART.md)** — three-terminal setup (chain → provider → demo) plus the SDK examples for the file-system and S3 interfaces. The canonical entry point.
- **[`FILE_SYSTEM_QUICKSTART.md`](../FILE_SYSTEM_QUICKSTART.md)** (repo root) — short version focused on the file-system layer only.
- **[`CLAUDE.md`](../CLAUDE.md)** (repo root) — build/test/run commands and code-review guidelines for contributors and AI agents.
- **[Layer 1 Quick Start](./getting-started/LAYER1_QUICKSTART.md)** — three-terminal setup (chain → provider → demo) plus SDK examples for the file-system and S3 interfaces. The canonical entry point.
- **[`FILE_SYSTEM_QUICKSTART.md`](./getting-started/FILE_SYSTEM_QUICKSTART.md)** — short version, file-system layer only.
- **[`CLAUDE.md`](../CLAUDE.md)** (repo root) — build/test/run commands and contributor guidelines.

## Reference

- **[Extrinsics Reference](./reference/EXTRINSICS_REFERENCE.md)** — every pallet extrinsic with parameters, errors, and example workflows.
- **[Payment Calculator](./reference/PAYMENT_CALCULATOR.md)** — `payment = price_per_byte × max_bytes × duration`, with worked examples and the common `PaymentExceedsMax` failure mode.
- **[Payment Calculator](./reference/PAYMENT_CALCULATOR.md)** — `payment = price_per_byte × max_bytes × duration`, with worked examples.
- **[Execution Flows](./reference/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs of the main flows (derived from the design; no design-owner sign-off needed).

## Design

The canonical system design. Changes require review (see [`.github/CODEOWNERS`](../.github/CODEOWNERS)).

- **[Scalable Web3 Storage](./design/scalable-web3-storage.md)** — architecture, economic model, comparisons with Filecoin/IPFS/Arweave, rebuttals to common review concerns.
- **[Implementation Details](./design/scalable-web3-storage-implementation.md)** — pallet extrinsics, provider HTTP API, MMR layout, challenge mechanism, replica sync.
- **[Smart Contracts](./design/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow.
- **[Execution Flows](./design/EXECUTION_FLOWS.md)** — sequence-by-sequence walkthroughs for the main flows.
- **[Marketplace](./design/marketplace.md)** — provider capacity, discovery, and matching.
- **[Checkpoint Protocol](./design/CHECKPOINT_PROTOCOL.md)** — multi-provider checkpoint coordination.
- **[Provider-Initiated Checkpoints](./design/provider-initiated-checkpoints.md)** — extension where providers proactively commit state.
- **[Client-Side Encryption](./design/CLIENT_SIDE_ENCRYPTION.md)** — wire format, cipher choice.
- **[S3 Metadata Index](./design/S3_METADATA_INDEX.md)** — how prefix/delimiter queries are served.

## Drafts

Unratified / WIP notes. **These need triage** ([#308](https://github.com/paritytech/web3-storage/issues/308)) — each should be reviewed and either promoted into `design/` (design-of-record) or `reference/` (accurate derived material), folded into an existing doc, or dropped.

- **[Layer 1 Design / Implementation](./drafts/L1_design_implementation.md)** — file-system & S3 provider interfaces on top of Layer 0 (split out of the Layer 0 implementation doc); triage tracked in [#51](https://github.com/paritytech/web3-storage/issues/51).
- **[Smart Contracts](./drafts/smart-contracts.md)** — `pallet_revive` integration, custom precompile ABI, address mapping, payment flow. Candidate for promotion to `reference/` (it's API-reference material, not a draft design).
- **[Marketplace](./drafts/marketplace.md)** — provider capacity, discovery, and matching.
- **[Checkpoint Protocol](./drafts/CHECKPOINT_PROTOCOL.md)** — multi-provider checkpoint coordination.
- **[Provider-Initiated Checkpoints](./drafts/provider-initiated-checkpoints.md)** — extension where providers proactively commit state.
- **[Client-Side Encryption](./drafts/CLIENT_SIDE_ENCRYPTION.md)** — wire format, cipher choice.
- **[S3 Metadata Index](./drafts/S3_METADATA_INDEX.md)** — how prefix/delimiter queries are served.
- **[Challenge Economics — Extensions](./drafts/challenge-economics-extensions.md)** — speculative "Capped Split for the general public"; also records that the design's two-tier challenger split isn't implemented yet.

## Layer 1 — file system interface

Expand All @@ -41,17 +53,11 @@ docs/
- **[Admin Guide](./filesystems/ADMIN_GUIDE.md)** — operations, monitoring, dispute resolution.
- **[API Reference](./filesystems/API_REFERENCE.md)** — extrinsics, SDK methods, primitives, events, errors.

## Other clients
## Clients

- **[Storage Client SDK](../client/README.md)** — Layer-0 Rust client.
- **[S3 Interface](../storage-interfaces/s3/README.md)** — Layer-1 S3-compatible Rust client.
- **TypeScript SDK** — `@web3-storage/sdk` at `packages/sdk` (umbrella over `packages/{core,layer0,layer1}`; `./fs`, `./s3`, and `./revive` subpaths).

## Scripts

- `scripts/build-chain-spec.sh` — build runtime + emit chain spec (used by `just generate-chain-spec`).
- `scripts/check-chain.sh` — relay + parachain health probe.
- `scripts/quick-test.sh` — curl-based smoke test of the Layer-0 provider HTTP API.
- **TypeScript SDK** — `@web3-storage/sdk` at `packages/sdk` (`./fs`, `./s3`, `./revive` subpaths).

## External

Expand All @@ -61,4 +67,4 @@ docs/

## License

See the repository root [README](../README.md#license) for license details.
See the repository root [README](../README.md#license).
Loading
Loading