diff --git a/AGENTS.md b/AGENTS.md index 9f7b5a89..1489dedf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,7 +2,7 @@ ## Documentation -User-, operator-, and admin-facing documentation — installing a watcher, adding an instrument, managing tokens, security/permissions — lives on the docs site at https://datahub.arcadiascience.com/docs, **not in this repository**. Search there first for "how do I use Data Hub" questions; don't rely on training data or guess at UI flows, since the site's `/docs/llms.txt` and `/docs/llms-full.txt` routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to fetch. +User-, operator-, and admin-facing documentation — installing a watcher, setting up an instrument, managing tokens, security/permissions — lives on the docs site at https://datahub.arcadiascience.com/docs, **not in this repository**. Search there first for "how do I use Data Hub" questions; don't rely on training data or guess at UI flows, since the site's `/docs/llms.txt` and `/docs/llms-full.txt` routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to fetch. This repo's `developer-docs/` covers contributing to and self-hosting Data Hub itself: architecture internals, local dev setup (`getting-started.md`, `local-development.md`), conventions, the step-by-step self-hosting guide for the web app and AWS infrastructure (`first-time-deployment.md`) plus CI/ongoing-deploy reference (`ci-and-deployment.md`), and per-package references (`lambda.md`, `watcher.md`, `shared-library.md`). See `developer-docs/README.md` for the full index. diff --git a/README.md b/README.md index ab977400..b6ad27e7 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ make dev See the full [Getting started guide](developer-docs/getting-started.md) for prerequisites and details. Don't have AWS/Google credentials? [Local development](developer-docs/local-development.md) covers a zero-credential setup for the web app + API + database alone (no watcher or Lambda needed). -Developer docs live in [developer-docs/](developer-docs/README.md), including the step-by-step guide to self-hosting Data Hub (deploying the web app and AWS infrastructure — see [First-time deployment](developer-docs/first-time-deployment.md)). You can find user documentation (watcher installation, adding an instrument, managing tokens) on the [docs site](https://datahub.arcadiascience.com/docs). +Developer docs live in [developer-docs/](developer-docs/README.md), including the step-by-step guide to self-hosting Data Hub (deploying the web app and AWS infrastructure — see [First-time deployment](developer-docs/first-time-deployment.md)). You can find user documentation (watcher installation, setting up an instrument, managing tokens) on the [docs site](https://datahub.arcadiascience.com/docs). ## Checks and tests diff --git a/developer-docs/architecture.md b/developer-docs/architecture.md index dc34060e..cb0c5a96 100644 --- a/developer-docs/architecture.md +++ b/developer-docs/architecture.md @@ -58,6 +58,6 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead: - **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](run-archives.md) for the full flow and runbook. - **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `
` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows all generic crawlers (with an allow-list for unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on every response. - **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code. -- **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security & permissions](https://datahub.arcadiascience.com/docs/security#token-scopes) for the scope vocabulary. -- **MCP for AI access.** The web app includes a [Model Context Protocol](https://modelcontextprotocol.io/) server at `/api/v1/mcp` that exposes tools (read + scoped writes), resources, and prompts. AI clients (e.g. Claude Desktop, Cursor) authenticate with a personal access token and can browse instruments/runs, download files, claim runs, comment, reprocess, and inspect watcher health. See the [MCP server docs](https://datahub.arcadiascience.com/docs/mcp-server). +- **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security and permissions](https://datahub.arcadiascience.com/docs/security#token-scopes) for the scope vocabulary. +- **MCP for AI access.** The web app includes a [Model Context Protocol](https://modelcontextprotocol.io/) server at `/api/v1/mcp` that exposes tools (read + scoped writes), resources, and prompts. AI clients (e.g. Claude Desktop, Cursor) authenticate with a personal access token and can browse instruments/runs, download files, claim runs, comment, reprocess, and inspect watcher health. See the [MCP overview](https://datahub.arcadiascience.com/docs/mcp). - **Integration tests against a real server.** The shared `testing.py` module spins up a real Next.js server backed by a Postgres database, so Lambda and watcher integration tests exercise the actual API surface. diff --git a/developer-docs/ci-and-deployment.md b/developer-docs/ci-and-deployment.md index 5def2ba1..e62e02a2 100644 --- a/developer-docs/ci-and-deployment.md +++ b/developer-docs/ci-and-deployment.md @@ -164,7 +164,7 @@ The CI deploy role has permission to roll new triggers out, so the trigger goes ### Watcher (PyPI) -The `data-hub-watcher` Python package is published to [PyPI](https://pypi.org/project/data-hub-watcher/) so lab PCs can install and self-update via `uv tool install data-hub-watcher`. The full release flow — version bump, tag, approval, env-var roll-out, mandatory updates, and rollback — is documented in the admin-facing [Managing watchers → Releases and fleet updates](https://datahub.arcadiascience.com/docs/managing-watchers#releases-and-fleet-updates) guide; this section is intentionally a pointer rather than a second source of truth so the two can't drift. +The `data-hub-watcher` Python package is published to [PyPI](https://pypi.org/project/data-hub-watcher/) so lab PCs can install and self-update via `uv tool install data-hub-watcher`. The full release flow — version bump, tag, approval, env-var roll-out, mandatory updates, and rollback — is documented in the admin-facing [Roll out watcher releases](https://datahub.arcadiascience.com/docs/watcher-releases) guide; this section is intentionally a pointer rather than a second source of truth so the two can't drift. Trusted publishing is configured under **Project → Publishing** on PyPI for `Arcadia-Science/data-hub` and the workflow `publish-watcher.yml`; no API token lives in repo secrets. If trust is ever revoked or rotated, update it there and re-run the workflow. diff --git a/developer-docs/first-time-deployment.md b/developer-docs/first-time-deployment.md index bda939e4..54a23391 100644 --- a/developer-docs/first-time-deployment.md +++ b/developer-docs/first-time-deployment.md @@ -82,7 +82,7 @@ npm run db:migrate ### Create an API key for the Lambda -Sign in with an account listed in `ADMIN_EMAILS`, then create a personal access token under Settings. The AWS stack and the Lambda use this token as `DATA_HUB_API_KEY` to call the Data Hub API, so create it now and keep it for [step 4](#4-deploy-the-aws-infrastructure). See [Managing tokens](https://datahub.arcadiascience.com/docs/managing-tokens) for the token UI. +Sign in with an account listed in `ADMIN_EMAILS`, then create a personal access token under Settings. The AWS stack and the Lambda use this token as `DATA_HUB_API_KEY` to call the Data Hub API, so create it now and keep it for [step 4](#4-deploy-the-aws-infrastructure). See [Issue and revoke tokens](https://datahub.arcadiascience.com/docs/manage-tokens) for the token UI. ## 3. Bootstrap AWS (once per account) @@ -180,4 +180,4 @@ The S3 bucket names default to `arcadia-data-hub-raw-