Move user-facing docs to the docs site; reorganize developer-docs/

.github/workflows/typescript-test.yml

@@ -53,8 +53,8 @@ jobs:
         run: npm ci
         working-directory: web
 
-      - name: Run MCP tests
-        run: make fe-test-mcp
+      - name: Run unit tests
+        run: make fe-test-unit
 
       - name: Run API integration tests
         run: make fe-test-integration

AGENTS.md

@@ -1,56 +1,38 @@
 # AGENTS.md
 
+## Documentation
+
+User-, operator-, and admin-facing documentation — installing a watcher, adding an instrument, managing tokens, deploying the web app and AWS infrastructure, security/permissions — lives on the docs site at https://arcadia-data-hub-docs.vercel.app/, **not in this repository**. Search there first for "how do I use/deploy Data Hub" questions; don't rely on training data or guess at UI flows, since the site's `/llms.txt` and `/llms-full.txt` routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to fetch.
+
+This repo's `developer-docs/` only covers contributing to Data Hub itself: architecture internals, local dev setup (`getting-started.md`, `local-development.md`), conventions, CI/deployment, and per-package references (`lambda.md`, `watcher.md`, `shared-library.md`). See `developer-docs/README.md` for the full index.
+
 ## Cursor Cloud specific instructions
 
-Data Hub is a multi-component repo (see `README.md`). The component you can run end-to-end
-locally with zero external credentials is the **Next.js web app + REST API + PostgreSQL**
-(`web/`). The `lambda/`, `watcher/`, and `packages/shared/` Python packages are exercised
-via tests and a local S3 mirror — no real AWS is needed for local work.
+Data Hub is a multi-component repo (see `README.md`). The component you can run end-to-end locally with zero external credentials is the **Next.js web app + REST API + PostgreSQL** (`web/`). The `lambda/`, `watcher/`, and `packages/shared/` Python packages are exercised via tests and a local S3 mirror — no real AWS is needed for local work.
 
-Standard commands live in the `Makefile`, `web/package.json`, `developer-docs/getting-started.md`, and
-`developer-docs/local-development.md`. The notes below are the non-obvious caveats that those docs don't
-make obvious for a fresh cloud VM (where the update script has already installed deps).
+Standard commands live in the `Makefile`, `web/package.json`, `developer-docs/getting-started.md`, and `developer-docs/local-development.md`. The notes below are the non-obvious caveats that those docs don't make obvious for a fresh cloud VM (where the update script has already installed deps).
 
 ### Starting services (not handled by the update script)
 
-- **PostgreSQL must be started on every fresh VM** — the cluster is installed and the data
-  (roles + databases) persist in the snapshot, but the server process is not running at boot:
-  `sudo pg_ctlcluster 16 main start` (or `sudo service postgresql start`).
-- Postgres is reachable at `postgres://postgres:postgres@127.0.0.1:5432`. Databases
-  `data-hub-local` (dev) and `data_hub_test` (integration tests) already exist. The integration
-  harness (`web/tests/integration/global-setup.ts`) hardcodes these same credentials and creates
-  `data_hub_test` itself if missing.
-- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at
-  `/login` with the "Sign in (dev)" button using email `dev@local` (workspace admin; no password).
+- **PostgreSQL must be started on every fresh VM** — the cluster is installed and the data (roles + databases) persist in the snapshot, but the server process is not running at boot: `sudo pg_ctlcluster 16 main start` (or `sudo service postgresql start`).
+- Postgres is reachable at `postgres://postgres:postgres@127.0.0.1:5432`. Databases `data-hub-local` (dev) and `data_hub_test` (integration tests) already exist. The integration harness (`web/tests/integration/global-setup.ts`) hardcodes these same credentials and creates `data_hub_test` itself if missing.
+- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at `/login` with the "Sign in (dev)" button using email `dev@local` (workspace admin; no password).
 
 ### Environment file
 
-`web/.env` is gitignored and required for `make dev` / seeding. If it is missing on a fresh VM,
-recreate it from the "Minimal `.env`" block in `developer-docs/local-development.md` (the key lines are
-`DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/data-hub-local`, a 32+ char
-`AUTH_SECRET`, dummy `AWS_*` values, and `LOCAL_S3_MIRROR=../lambda/.local-s3`).
+`web/.env` is gitignored and required for `make dev` / seeding. If it is missing on a fresh VM, recreate it from the "Minimal `.env`" block in `developer-docs/local-development.md` (the key lines are `DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/data-hub-local`, a 32+ char `AUTH_SECRET`, dummy `AWS_*` values, and `LOCAL_S3_MIRROR=../lambda/.local-s3`).
 
 ### Node / Python toolchain
 
-- Use **Node 24 (npm 11)** — it is the nvm default and is what CI uses. `npm ci` against the
-  committed `web/package-lock.json` **fails under npm 10** ("Missing: esbuild@… from lock file"),
-  so don't downgrade. A clean login shell already selects Node 24 via nvm.
-- Python is managed by `uv` (Python 3.13, pinned in `.python-version`). Run Python tools through
-  `uv run …` (e.g. `uv run pytest`); the Makefile targets already do this.
+- Use **Node 24 (npm 11)** — it is the nvm default and is what CI uses. `npm ci` against the committed `web/package-lock.json` **fails under npm 10** ("Missing: esbuild@… from lock file"), so don't downgrade. A clean login shell already selects Node 24 via nvm.
+- Python is managed by `uv` (Python 3.13, pinned in `.python-version`). Run Python tools through `uv run …` (e.g. `uv run pytest`); the Makefile targets already do this.
 
 ### Seeding and local file bytes
 
-- `make db-reseed` resets + pushes the Drizzle schema + seeds deterministic data. It prints a
-  personal access token (`dhub_…`) for the dev user — use it for `Authorization: Bearer` API calls.
-- The seed's fixture-processing step is **skipped if the dev server isn't running** (it prints a
-  hint). To populate processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata), start
-  `make dev` first, then run `npm run db:process-fixtures` from `web/`.
+- `make db-reseed` resets + pushes the Drizzle schema + seeds deterministic data. It prints a personal access token (`dhub_…`) for the dev user — use it for `Authorization: Bearer` API calls.
+- The seed's fixture-processing step is **skipped if the dev server isn't running** (it prints a hint). To populate processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata), start `make dev` first, then run `npm run db:process-fixtures` from `web/`.
 
 ### Testing caveat
 
-- `make fe-test-integration` and `make py-test-integration` run `next build` + `next start`, which
-  writes to `web/.next` and **contends with a running `make dev`** (also using `.next`). Stop the
-  dev server before running integration tests, then restart it afterward.
-- Lint/format/typecheck: `make check-all` (note: `py-format`/`fe-format` auto-rewrite files; use
-  `uv run ruff check .`, `npm run lint:check` (Biome formatter + linter, read-only), and
-  `npm run typecheck` for read-only checks).
+- `make fe-test-integration` and `make py-test-integration` run `next build` + `next start`, which writes to `web/.next` and **contends with a running `make dev`** (also using `.next`). Stop the dev server before running integration tests, then restart it afterward.
+- Lint/format/typecheck: `make check-all` (note: `py-format`/`fe-format` auto-rewrite files; use `uv run ruff check .`, `npm run lint:check` (Biome formatter + linter, read-only), and `npm run typecheck` for read-only checks).

Makefile

@@ -59,17 +59,17 @@ fe-lint:
 fe-typecheck:
 	cd web && npm run typecheck
 
-.PHONY: fe-test-mcp
-fe-test-mcp:
-	cd web && npm run test:mcp
+.PHONY: fe-test-unit
+fe-test-unit:
+	cd web && npm run test:unit
 
 .PHONY: fe-test-integration
 fe-test-integration:
 	cd web && npm run test:integration
 
 .PHONY: fe-test
 fe-test:
-	cd web && npm run test:mcp && npm run test:integration
+	cd web && npm run test:unit && npm run test:integration
 
 .PHONY: dev
 dev:

README.md

@@ -14,15 +14,15 @@ flowchart LR
 
 ## Repository structure
 
-| Directory | Description | Docs |
-| --- | --- | --- |
-| `web/` | Next.js web application and REST API (Vercel) | [API reference](developer-docs/reference/api.md) |
-| `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](developer-docs/reference/lambda.md) |
-| `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](developer-docs/reference/watcher.md) |
-| `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](developer-docs/reference/shared-library.md) |
-| `developer-docs/` | Project documentation | — |
+| Directory | Description |
+| --- | --- |
+| `web/` | Next.js web application and REST API (Vercel) |
+| `lambda/` | AWS Lambda function for instrument data processing |
+| `watcher/` | CLI agent for lab instrument PCs |
+| `packages/shared/` | Shared Python library (S3, enums, test infra) |
+| `developer-docs/` | Project documentation |
 
-## Quick start
+## Getting started
 
 ```sh
 # Install Python packages (all workspace members).
@@ -41,33 +41,14 @@ cd web && createdb data-hub-local && npm run db:push && cd ..
 make dev
 ```
 
-See the full [Getting Started guide](developer-docs/getting-started.md) for prerequisites and details.
+See the full [Getting started guide](developer-docs/getting-started.md) for prerequisites and details. Don't have AWS/Google credentials? [Local development](developer-docs/local-development.md) covers a zero-credential setup for the web app + API + database alone (no watcher or Lambda needed).
 
-## Documentation
+Developer docs live in [developer-docs/](developer-docs/README.md). You can find user documentation (self-hosted deployment, watcher installation, adding an instrument, managing tokens) on the [docs site](https://arcadia-data-hub-docs.vercel.app/).
 
-### Guides
-
-- [Local development](developer-docs/local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed)
-- [Adding an instrument](developer-docs/guides/adding-an-instrument.md) — end-to-end: watcher setup, activation, optional Lambda preprocessing
-- [Installing a watcher](developer-docs/guides/installing-a-watcher.md) — lab operator focused: init, watch, troubleshooting
-- [Managing tokens](developer-docs/guides/managing-tokens.md) — creating, using, and revoking API tokens
-
-### Reference
-
-- [Architecture](developer-docs/architecture.md) — system overview, data flow, and design decisions
-- [Getting started](developer-docs/getting-started.md) — development setup, environment variables, running locally
-- [Watcher](developer-docs/reference/watcher.md) — CLI commands, configuration, run detection, upload modes
-- [Lambda](developer-docs/reference/lambda.md) — processing pipeline, supported instruments, adding new instruments
-- [REST API](developer-docs/reference/api.md) — endpoint reference and authentication
-- [MCP server](developer-docs/reference/mcp.md) — tools, resources, prompts, and installation for Claude Desktop / Cursor
-- [Shared library](developer-docs/reference/shared-library.md) — module reference for `data-hub-shared`
-- [CI and deployment](developer-docs/ops/ci-and-deployment.md) — GitHub Actions, Vercel, Render, Lambda deployment
-- [Conventions](developer-docs/conventions.md) — S3 key layout, instrument IDs, code style, environments
-
-## Development
+## Checks and tests
 
 ```sh
-# Run formatting, linting, and type checking.
+# Format, lint, and type-check everything.
 make check-all
 
 # Run all Python tests.
@@ -79,6 +60,9 @@ make py-test-unit
 # Run Python integration tests (requires Postgres).
 make py-test-integration
 
+# Run web app unit tests.
+make fe-test-unit
+
 # Run API integration tests (requires Postgres).
 make fe-test-integration
 ```
@@ -87,6 +71,4 @@ make fe-test-integration
 
 Data Hub is released under the [MIT License](LICENSE). Copyright (c) 2026 Arcadia Science.
 
-"Data Hub" and "Arcadia Science", along with related names and logos, are marks of
-Arcadia Science. The MIT License covers the source code only and does not grant any
-right to use these names or logos.
+"Data Hub" and "Arcadia Science", along with related names and logos, are marks of Arcadia Science. The MIT License covers the source code only and does not grant any right to use these names or logos.

developer-docs/README.md

@@ -0,0 +1,17 @@
+# Developer docs
+
+Documentation for developing Data Hub itself. User, operator, and admin
+documentation (installing a watcher, adding an instrument, managing tokens,
+deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/)
+instead — see the root [README](../README.md#getting-started) for that split.
+
+- [Getting started](getting-started.md) — development setup, environment variables, running locally
+- [Local development](local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed)
+- [Architecture](architecture.md) — system overview, data flow, and design decisions
+- [Testing](testing.md) — per-package test frameworks, the shared test-server fixture, S3 mocking
+- [Watcher](watcher.md) — CLI commands, configuration, run detection, upload modes
+- [Lambda](lambda.md) — processing pipeline, supported instruments, adding new instruments
+- [Shared library](shared-library.md) — module reference for `data-hub-shared`
+- [CI and deployment](ci-and-deployment.md) — GitHub Actions, Vercel, Render, Lambda deployment
+- [Run archives](run-archives.md) — "Download all" flow, cache/dedup model, and on-call runbook
+- [Conventions](conventions.md) — S3 key layout, instrument IDs, code style, environments

developer-docs/architecture.md

@@ -55,8 +55,9 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead:
 - **S3 as the integration boundary.** The watcher and Lambda never communicate directly. S3 acts as a durable hand-off point: the watcher writes, Lambda reads.
 - **API-driven coordination.** The watcher registers with the API, syncs its YAML config, and sends periodic heartbeats. This lets the web dashboard show watcher health and manage upload queues.
 - **Presigned URLs from the API.** The web app generates presigned S3 upload and download URLs so watchers and browsers can transfer files directly to/from S3 without routing data through the API. On Vercel, the app assumes an IAM role via OIDC federation (no long-lived AWS credentials).
-- **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](ops/run-archives.md) for the full flow and runbook.
+- **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](run-archives.md) for the full flow and runbook.
 - **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `<head>` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows all generic crawlers (with an allow-list for unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on every response.
 - **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code.
+- **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security & permissions](https://arcadia-data-hub-docs.vercel.app/docs/security#token-scopes) for the scope vocabulary.
 - **MCP for AI access.** The web app includes a [Model Context Protocol](https://modelcontextprotocol.io/) server at `/api/v1/mcp` that exposes read-only tools, resources, and prompts. AI clients (e.g. Claude Desktop, Cursor) can query instruments, runs, and system status using a personal access token.
 - **Integration tests against a real server.** The shared `testing.py` module spins up a real Next.js server backed by a Postgres database, so Lambda and watcher integration tests exercise the actual API surface.