From 1e10742067a7696ebcbb902c1fc2cca6d722a343 Mon Sep 17 00:00:00 2001 From: Wasim Sandhu Date: Mon, 6 Jul 2026 17:52:13 -0700 Subject: [PATCH 1/4] Move user-facing docs to the docs site; keep only contributor docs here MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit developer-docs/ had a mix of user/operator/admin guides (installing a watcher, adding an instrument, managing tokens, upgrading the watcher) and reference docs (REST API, MCP server) that duplicated — or were superseded by — the polished docs site at arcadia-data-hub-docs.vercel.app. - Delete developer-docs/guides/ (adding-an-instrument, installing-a-watcher, managing-tokens, upgrading-the-watcher) and developer-docs/reference/ {api,mcp}.md — all fully covered by the docs site now. - Preserve the two pieces of dev-only content those files carried that weren't duplicated anywhere else: the S3 trigger + deploy step for adding a Lambda processor (folded into the docs site's deploying-aws-infrastructure page) and the 0022_pat_scopes wildcard-backfill rationale (added to architecture.md's design decisions). - Repoint every remaining cross-reference (READMEs, reference/*.md, watcher/pyproject.toml) at the docs site instead of the deleted files. - Add a Documentation section to AGENTS.md directing agents to the docs site for user/operator/admin questions, since developer-docs/ now only covers contributing to Data Hub itself. Co-authored-by: Cursor --- AGENTS.md | 17 ++ README.md | 12 +- developer-docs/architecture.md | 1 + developer-docs/guides/adding-an-instrument.md | 145 ---------- developer-docs/guides/installing-a-watcher.md | 253 ------------------ developer-docs/guides/managing-tokens.md | 142 ---------- .../guides/upgrading-the-watcher.md | 187 ------------- developer-docs/local-development.md | 6 +- developer-docs/ops/ci-and-deployment.md | 2 +- developer-docs/ops/run-archives.md | 2 +- developer-docs/reference/api.md | 164 ------------ developer-docs/reference/lambda.md | 2 + developer-docs/reference/mcp.md | 154 ----------- developer-docs/reference/watcher.md | 6 +- watcher/README.md | 2 +- watcher/pyproject.toml | 2 +- 16 files changed, 36 insertions(+), 1061 deletions(-) delete mode 100644 developer-docs/guides/adding-an-instrument.md delete mode 100644 developer-docs/guides/installing-a-watcher.md delete mode 100644 developer-docs/guides/managing-tokens.md delete mode 100644 developer-docs/guides/upgrading-the-watcher.md delete mode 100644 developer-docs/reference/api.md delete mode 100644 developer-docs/reference/mcp.md diff --git a/AGENTS.md b/AGENTS.md index 3cefedc6..16877a2e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,22 @@ # AGENTS.md +## Documentation + +User-, operator-, and admin-facing documentation — installing a watcher, adding +an instrument, managing tokens, deploying the web app and AWS infrastructure, +security/permissions — lives on the docs site at +https://arcadia-data-hub-docs.vercel.app/, **not in this repository**. Search +there first for "how do I use/deploy Data Hub" questions; don't rely on training +data or guess at UI flows, since the site's `/llms.txt` and `/llms-full.txt` +routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to +fetch. + +This repo's `developer-docs/` only covers contributing to Data Hub itself: +architecture internals, local dev setup (`getting-started.md`, +`local-development.md`), conventions, CI/ops, and per-package references +(`reference/lambda.md`, `reference/watcher.md`, `reference/shared-library.md`). +See `README.md`'s Documentation section for the full index. + ## Cursor Cloud specific instructions Data Hub is a multi-component repo (see `README.md`). The component you can run end-to-end diff --git a/README.md b/README.md index eb0a3da8..1263d075 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ flowchart LR | Directory | Description | Docs | | --- | --- | --- | -| `web/` | Next.js web application and REST API (Vercel) | [API reference](developer-docs/reference/api.md) | +| `web/` | Next.js web application and REST API (Vercel) | [API reference](https://arcadia-data-hub-docs.vercel.app/docs/api-reference) | | `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](developer-docs/reference/lambda.md) | | `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](developer-docs/reference/watcher.md) | | `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](developer-docs/reference/shared-library.md) | @@ -45,12 +45,14 @@ See the full [Getting Started guide](developer-docs/getting-started.md) for prer ## Documentation +User, operator, and admin documentation (installing a watcher, adding an +instrument, managing tokens, deployment) lives on the +[docs site](https://arcadia-data-hub-docs.vercel.app/), not in this repository. +The docs below are for people developing Data Hub itself. + ### Guides - [Local development](developer-docs/local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed) -- [Adding an instrument](developer-docs/guides/adding-an-instrument.md) — end-to-end: watcher setup, activation, optional Lambda preprocessing -- [Installing a watcher](developer-docs/guides/installing-a-watcher.md) — lab operator focused: init, watch, troubleshooting -- [Managing tokens](developer-docs/guides/managing-tokens.md) — creating, using, and revoking API tokens ### Reference @@ -58,8 +60,6 @@ See the full [Getting Started guide](developer-docs/getting-started.md) for prer - [Getting started](developer-docs/getting-started.md) — development setup, environment variables, running locally - [Watcher](developer-docs/reference/watcher.md) — CLI commands, configuration, run detection, upload modes - [Lambda](developer-docs/reference/lambda.md) — processing pipeline, supported instruments, adding new instruments -- [REST API](developer-docs/reference/api.md) — endpoint reference and authentication -- [MCP server](developer-docs/reference/mcp.md) — tools, resources, prompts, and installation for Claude Desktop / Cursor - [Shared library](developer-docs/reference/shared-library.md) — module reference for `data-hub-shared` - [CI and deployment](developer-docs/ops/ci-and-deployment.md) — GitHub Actions, Vercel, Render, Lambda deployment - [Conventions](developer-docs/conventions.md) — S3 key layout, instrument IDs, code style, environments diff --git a/developer-docs/architecture.md b/developer-docs/architecture.md index a4fe74d6..d99f7120 100644 --- a/developer-docs/architecture.md +++ b/developer-docs/architecture.md @@ -58,5 +58,6 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead: - **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](ops/run-archives.md) for the full flow and runbook. - **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows all generic crawlers (with an allow-list for unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on every response. - **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code. +- **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security & permissions](https://arcadia-data-hub-docs.vercel.app/docs/security#token-scopes) for the scope vocabulary. - **MCP for AI access.** The web app includes a [Model Context Protocol](https://modelcontextprotocol.io/) server at `/api/v1/mcp` that exposes read-only tools, resources, and prompts. AI clients (e.g. Claude Desktop, Cursor) can query instruments, runs, and system status using a personal access token. - **Integration tests against a real server.** The shared `testing.py` module spins up a real Next.js server backed by a Postgres database, so Lambda and watcher integration tests exercise the actual API surface. diff --git a/developer-docs/guides/adding-an-instrument.md b/developer-docs/guides/adding-an-instrument.md deleted file mode 100644 index 1e62ed3f..00000000 --- a/developer-docs/guides/adding-an-instrument.md +++ /dev/null @@ -1,145 +0,0 @@ -# Adding a new instrument - -This guide walks through the end-to-end process of adding a new lab instrument to Data Hub: creating the instrument, installing a watcher, and optionally adding Lambda preprocessing. - -## Prerequisites - -- A personal access token (see [Managing tokens](managing-tokens.md)) -- Access to the instrument PC where files are generated -- Python >= 3.12 and [uv](https://docs.astral.sh/uv/) installed on the instrument PC - -## Step 1: Install the watcher on the instrument PC - -Follow the [Installing a watcher](installing-a-watcher.md) guide. During `data-hub-watcher init`, choose "Register a new instrument" and provide: - -- **Instrument ID** — a kebab-case identifier (e.g., `bio-rad-cfx96`). This becomes the S3 key prefix and the permanent identifier across the system. -- **Display name** — a human-readable name (e.g., "Bio-Rad CFX96"). Defaults to a title-cased version of the ID. - -The instrument is created with status `pending`. - -## Step 2: Activate the instrument - -An admin must confirm the instrument before the watcher can start uploading. - -1. Open the Data Hub web app. -2. Navigate to the **Instruments** page. -3. Find the new instrument — it will have a yellow `pending` badge. -4. Click the **Confirm** button next to it. - -The instrument status changes to `active` and the watcher can now run. - -## Step 3: Start watching - -On the instrument PC: - -```sh -data-hub-watcher watch -``` - -The watcher will detect new files, group them into runs, upload them to S3, and report everything to the API. Files become viewable in the web dashboard immediately after upload. - -See the [watcher reference](../reference/watcher.md) for details on upload modes, run detection, and configuration options. - -## Step 4 (optional): Add Lambda preprocessing - -If the new instrument needs automated preprocessing (metadata extraction, image processing, etc.), you'll need to add a processor to the Lambda function. This requires changes to the codebase. - -### 4.1 Register the instrument in the shared library - -Add the instrument to `packages/shared/src/data_hub_shared/enums.py`: - -```python -class Instrument(Enum): - # ... existing instruments ... - BIO_RAD_CFX96 = "bio-rad-cfx96" -``` - -Add a display name mapping in `packages/shared/src/data_hub_shared/constants.py`: - -```python -INSTRUMENT_ID_TO_NAME_MAP: dict[str, str] = { - # ... existing entries ... - Instrument.BIO_RAD_CFX96.value: "Bio-Rad CFX96", -} -``` - -### 4.2 Create a processor module - -Create a new directory and `process_file.py` under `lambda/src/data_hub_lambda/`: - -``` -lambda/src/data_hub_lambda/bio_rad_cfx96/ -├── __init__.py -└── process_file.py -``` - -The processor must expose a `process_file` function: - -```python -def process_file(run_id: str, filename: str) -> None: - """Preprocess a file, reporting progress through the Data Hub API.""" - ... -``` - -A typical processor: - -1. Gets an API client and the S3 bucket from config. -2. Calls `client.ensure_run()` to create or find the run. -3. Calls `client.create_file()` to register the raw file. -4. Downloads the raw file from S3. -5. Performs instrument-specific preprocessing (parsing, metadata extraction, etc.). -6. Optionally uploads processed artifacts (CSV, images) to the S3 processed bucket and registers them via `client.create_file(..., category="processed")`. This is the pattern used by the SpectraMax plate reader (processed CSV) and Azure 600 Gel Doc (contrast-enhanced PNG). -7. Updates the raw file status to `completed`. - -See any existing processor (e.g., `lambda/src/data_hub_lambda/azure_cielo_qpcr/process_file.py` for simple metadata extraction, or `lambda/src/data_hub_lambda/spectramax_plate_reader/process_file.py` for the processed-artifact pattern) for complete examples. - -### 4.3 Register the dispatch - -Add an `elif` branch in the `lambda_handler` function in `lambda/src/data_hub_lambda/handler.py`: - -```python -elif instrument_id == Instrument.BIO_RAD_CFX96.value: - bio_rad_cfx96.process_file( - run_id=event_info.run_id, - filename=event_info.filename, - ) -``` - -Don't forget to add the import at the top of `handler.py`. - -### 4.4 Add tests - -Add unit tests in `lambda/tests/` for the new processor. Integration tests will automatically cover the new instrument if it's registered in the shared library. - -For a quick end-to-end smoke against your local web app — without S3, AWS credentials, or LocalStack — use `data-hub-process handler` to drive `lambda_handler` against a gitignored local mirror. See [Testing the Lambda end-to-end](../local-development.md#testing-the-lambda-end-to-end). - -If your instrument has a representative fixture (small, redistributable) you can also wire it into the dev seed: drop the file under `lambda/tests/fixtures/`, then add paired entries to `INSTRUMENT_FIXTURES` and `CANONICAL_INSTRUMENT_ID` in [web/lib/db/seed.ts](../../web/lib/db/seed.ts). Every `make db-reseed` will then drive your `process_file` end-to-end so the new instrument's runs render real bytes (and processed artifacts) in the dashboard immediately. - -### 4.5 Configure the S3 trigger - -Add a `LambdaConfiguration` entry to the `RawDataBucket` resource's `NotificationConfiguration` in `infra/template.yaml`. Each entry specifies a prefix (the instrument ID) and a suffix (the file extension): - -```yaml -- Event: s3:ObjectCreated:* - Filter: - S3Key: - Rules: - - Name: prefix - Value: bio-rad-cfx96/ - - Name: suffix - Value: .csv - Function: !GetAtt DataHubFunction.Arn -``` - -The trigger is created automatically on the next `sam deploy` (or when the deploy workflow runs after merge). - -## What you get without Lambda - -Even without Step 4, you get a fully functional instrument in Data Hub: - -- The watcher uploads raw files to S3. -- Runs and files appear in the web dashboard. -- Files are downloadable via pre-signed S3 URLs. -- Watcher health monitoring (heartbeats, events) works in the dashboard. - -Lambda preprocessing adds automated metadata extraction on top of that. (Slack notifications fire from the web app whenever a new run is created, regardless of whether a Lambda processor exists for the instrument.) diff --git a/developer-docs/guides/installing-a-watcher.md b/developer-docs/guides/installing-a-watcher.md deleted file mode 100644 index dcd7d271..00000000 --- a/developer-docs/guides/installing-a-watcher.md +++ /dev/null @@ -1,253 +0,0 @@ -# Installing a watcher - -This guide is for lab operators setting up the Data Hub Watcher on an instrument PC. The watcher monitors a directory for new files, uploads them to S3, and reports run data to the Data Hub web app. - -## Prerequisites - -- **[uv](https://docs.astral.sh/uv/getting-started/installation/)** — the recommended Python package manager. Install it if you don't have it yet; it will handle Python for you. -- **A personal access token** — ask your Data Hub admin to create one for you, or create one yourself at **Settings > Access Tokens** in the web app (see [Managing tokens](managing-tokens.md)) -- **The watch directory** — the folder where the instrument writes its output files -- **File patterns** — the file extensions you want to upload (e.g., `*.csv`, `*.xlsx`, `*.tiff`) - -## Installation - -The watcher is published as a versioned package on PyPI. For use on lab PCs, install it directly from PyPI: - -```sh -uv tool install data-hub-watcher -``` - -This installs the `data-hub-watcher` CLI into an isolated venv managed by `uv`, on PATH for any shell. - -After installation, every example below that says `data-hub-watcher …` runs the installed CLI directly. - -### Windows service support - -The watcher can run as a Windows service. Install it with the `windows-service` extra so the `pywin32` dependency is included: - -```sh -uv tool install "data-hub-watcher[windows-service]" -``` - -### Developer install (from a checkout) - -If you're modifying the watcher itself, install in editable mode from the repo so your local edits are reflected immediately: - -```sh -git clone https://github.com/Arcadia-Science/data-hub -cd data-hub -uv sync --all-packages --extra windows-service -uv run data-hub-watcher --help -``` - -When working from the checkout, prefix every example below with `uv run` (e.g. `uv run data-hub-watcher init`) so the editable install in `.venv/` is used. - -## Setup - -Run the interactive setup wizard: - -```sh -data-hub-watcher init -``` - -The wizard will walk you through: - -1. **Environment** — choose `staging` (for testing), `production`, or `preview` (for testing against a Vercel preview deployment). If you choose `preview`, you'll be prompted for the deployment's API base URL (e.g. `https://data-hub-git-my-branch.vercel.app/api/v1`). -2. **API key** — paste the personal access token. The key is saved to `~/.data-hub/.env.` (e.g. `~/.data-hub/.env.staging`), so each environment keeps its own key and you can switch between them by re-running `init` without re-entering credentials. You can also set the `DATA_HUB_API_KEY` environment variable before running `init` to skip this prompt. -3. **Instrument** — select an existing instrument from the list, or register a new one by choosing the last option. New instruments start as `pending` and must be activated by an admin in the web app before the watcher can start. -4. **Watch directory** — the absolute path to the folder the instrument writes to. -5. **File patterns** — comma-separated glob patterns (e.g., `*.csv,*.xlsx`). Only files matching these patterns will be uploaded. -6. **Run detection method**: - - `**prefix`** — extracts a run ID from each filename using a regex. The default pattern `^([^_]+)` captures everything before the first underscore (e.g., `RUN001_data.csv` → run ID `RUN001`). - - `**directory**` — each subdirectory under the watch directory is treated as a separate run. -7. **Stability period** — how many seconds a file must remain unchanged (size + modification time) before it's considered fully written. Increase this for instruments that produce large files slowly. Default is 5 seconds. -8. **Upload mode**: - - `**auto`** — files are uploaded to S3 immediately after detection. - - `**manual**` — files are reported to the server but not uploaded until an admin approves them via the upload queue. - -The wizard saves configuration to `~/.data-hub/config.yaml`, the API key to `~/.data-hub/.env.`, and syncs the config to the server. - -> **Heads-up on the existing backlog.** On `staging` and `preview`, the watcher does *not* upload files that are already in the watch directory when you first run `init` — it records them as a baseline and uploads only files created afterwards. This keeps test environments from ingesting a PC's entire history. `production` uploads the existing backlog as normal. To force a full upload on staging/preview, set `initial_scan: full` in `~/.data-hub/config.yaml` before starting the watcher (see the [watcher reference](../reference/watcher.md#initial-scan-and-the-backlog)). - -## Starting the watcher - -First, verify your setup with a dry run: - -```sh -data-hub-watcher watch --dry-run -``` - -This validates the config, checks that the API is reachable and the instrument is active, and previews what files would be uploaded — without actually starting the monitor. - -When you're ready: - -```sh -data-hub-watcher watch -``` - -The watcher will now: - -- Monitor the watch directory for new and modified files. -- Wait for files to stabilize before processing them. -- Group files into runs and report them to the API. -- Upload files to S3 (in auto mode) or wait for server approval (in manual mode). -- Send heartbeats every 60 seconds so the web dashboard shows watcher health. - -Press `Ctrl+C` to stop. - -## Running as a Windows service - -On Windows, you can install the watcher as a service so it starts automatically: - -```sh -data-hub-watcher service install -data-hub-watcher service start -``` - -Other service commands: - -```sh -data-hub-watcher service status # Check if the service is running -data-hub-watcher service stop # Stop the service -data-hub-watcher service uninstall # Remove the service -data-hub-watcher service reinstall # Stop, uninstall, install, and start (e.g. after a manual wheel upgrade) -``` - -## Changing configuration - -To re-prompt each config field with current values as defaults: - -```sh -data-hub-watcher config edit -``` - -To open the YAML file directly in your editor: - -```sh -data-hub-watcher config open -``` - -To view the current config: - -```sh -data-hub-watcher config show -``` - -Changes are automatically synced to the server after editing. - -## Upgrading the watcher - -The watcher can upgrade itself in place, either on demand via `data-hub-watcher self-update` or — when running as a Windows service — automatically on an hourly background tick. The full release flow, mandatory-update behavior, rollback steps, and operator troubleshooting all live in [Upgrading the watcher](upgrading-the-watcher.md). - -If you just want the quick command: - -```sh -data-hub-watcher self-update # check + upgrade if needed -data-hub-watcher self-update --check # report status only, no upgrade -data-hub-watcher self-update --force # re-run the upgrade subprocess even if the version already matches -``` - -After a successful upgrade you must restart the watcher (or the Windows service) for the new code to take effect — `self-update` does not restart the running process. Lab PCs running the Windows service get auto-restart for free via the SCM's failure-actions policy; see the [upgrade guide](./upgrading-the-watcher.md) for details. - -## Manual uploads - -To upload a specific file outside the normal watch loop: - -```sh -data-hub-watcher upload --file /path/to/file.csv --run-id RUN001 -``` - -To process the server-side upload queue (manual mode): - -```sh -data-hub-watcher upload -``` - -Add `--dry-run` to preview without uploading. - -## Troubleshooting - -### "Instrument is still pending activation" - -The instrument was registered but hasn't been confirmed by an admin yet. Ask your admin to click "Confirm" on the instrument in the web app's Instruments page. - -### "Instrument already has an active watcher" - -Each instrument can have at most one active watcher at a time. If `init` fails with this error, an earlier install (often on a different PC, or before a reimage) is still registered against the instrument. The CLI prints the existing watcher's id; deregister it before re-running `init`: - -- **Web UI** — go to **Watchers**, open the existing watcher, and click **Deregister**. -- **API** — `curl -X DELETE -H "Authorization: Bearer $DATA_HUB_API_KEY" https:///api/v1/watchers/` - -Deregistration is a soft-delete: heartbeats, events, and runs reported by the old watcher remain visible in **Watchers > Deregistered** (in the web app) for auditing. - -### "Connection error" or "Request timed out" - -The watcher can't reach the Data Hub API. Check: - -- Your internet connection. -- That the correct environment is set in the config (`staging`, `production`, or `preview`). -- That the API URL is reachable. - -### Files aren't being detected - -- Verify the watch directory is correct: `data-hub-watcher config show` -- Check that file patterns match your files. The watcher uses glob matching (e.g., `*.csv` matches `data.csv` but not `data.CSV` on case-sensitive systems). -- Run `data-hub-watcher watch --dry-run` to see what files the watcher would pick up. - -### Files are detected but not uploading - -- In **manual mode**, files are not uploaded until approved via the upload queue. Check the web dashboard. -- Check the log file at `C:\ProgramData\DataHubWatcher\watcher.log` (Windows) or `~/.data-hub/watcher.log` (macOS/Linux) for error details. -- Verify your API token hasn't expired. - -### Logs - -The watcher writes rotating logs to: - -- **Windows**: `C:\ProgramData\DataHubWatcher\watcher.log` -- **macOS / Linux**: `~/.data-hub/watcher.log` - -Files are 10 MB each with 5 backups kept (`watcher.log.1` through `watcher.log.5`). Both the CLI `watch` command and the Windows service write to the same path so a single `Get-Content -Wait` (or `tail -F`) shows everything regardless of which entrypoint is running. You can also pass `--verbose` for debug-level console output on the CLI: - -```sh -data-hub-watcher --verbose watch -``` - -#### Don't run `watch` alongside the service - -Running `data-hub-watcher watch` interactively while the Windows service is also running is not supported — both processes would race on the same rotating log file (and on the same watch directory). Stop the service first if you need to run the CLI for debugging: - -```powershell -data-hub-watcher service stop -data-hub-watcher --verbose watch -data-hub-watcher service start # when you're done -``` - -#### Bootstrap log for pre-dispatcher crashes - -The service writes a separate `service-bootstrap.log` next to `watcher.log` that captures crashes happening before the service control dispatcher takes over — for example, a missing `pywin32`, a moved virtualenv, or a corrupt install. If `watcher.log` is empty after a crash, check `C:\ProgramData\DataHubWatcher\service-bootstrap.log` for the traceback. - -#### Turning on debug logging for the service - -Add `DATA_HUB_WATCHER_LOG_LEVEL=DEBUG` to the env file the service is registered against (typically `~/.data-hub/.env.`) and restart the service. No redeploy or `service reinstall` is needed. - -#### Triaging a service that crashes immediately - -If the service exits before writing anything to `watcher.log`, two read-only commands will surface the failure: - -1. Query the Windows Application event log for entries from the watcher or the underlying Python service host: - - ```powershell - Get-WinEvent -FilterHashtable @{LogName='Application'; StartTime=(Get-Date).AddHours(-1)} | - Where-Object { $_.ProviderName -match 'Python|DataHubWatcher' -or $_.Message -match 'DataHubWatcher' } | - Format-List TimeCreated, ProviderName, Id, LevelDisplayName, Message - ``` - -2. Run the service in the foreground from the venv `pywin32` itself ships: - - ```powershell - & "C:\path\to\venv\Scripts\python.exe" -m win32serviceutil debug DataHubWatcher - ``` - - This bypasses the SCM, runs the same startup path the service uses, and prints the full traceback to the console — the fastest way to see why a phase-A/B crash is happening. - diff --git a/developer-docs/guides/managing-tokens.md b/developer-docs/guides/managing-tokens.md deleted file mode 100644 index 63d00340..00000000 --- a/developer-docs/guides/managing-tokens.md +++ /dev/null @@ -1,142 +0,0 @@ -# Managing tokens - -Personal access tokens authenticate the watcher and other API clients with the Data Hub API. This guide covers creating, using, and revoking tokens. - -> **Admin-only.** Creating and revoking tokens requires the workspace admin role. Regular members can view the workspace token audit list at **Settings > Access Tokens** but cannot mint or delete tokens. Ask an existing admin (or someone with their email listed in `ADMIN_EMAILS`) if you need a token issued. - -## Creating a token - -### In the web app - -1. Sign in to the Data Hub web app as an admin. -2. Go to **Settings > Access Tokens**. -3. Click **Create Token**. -4. Enter a descriptive name (e.g., "FPLC watcher - Lab 201"). -5. Optionally set an expiration date. -6. Click **Create**. - -The plaintext token is displayed once — **copy it immediately**. It cannot be retrieved again. The token starts with `dhub_` followed by a 64-character hex string. - -### Via the API - -```sh -curl -X POST https://datahub.example.com/api/v1/tokens \ - -H "Cookie: " \ - -H "Content-Type: application/json" \ - -d '{"name": "FPLC watcher", "expires_at": "2027-01-01T00:00:00Z"}' -``` - -The `expires_at` field is optional. If omitted, the token never expires. - -The response includes the plaintext token in the `token` field — this is the only time it's returned. - -## Using a token - -### With the watcher - -During `data-hub-watcher init`, paste the token when prompted for the API key. Alternatively, set it as an environment variable: - -```sh -export DATA_HUB_API_KEY=dhub_abc123... -data-hub-watcher init -``` - -The watcher stores the API key in its environment configuration and uses it for all subsequent API calls. - -### With an MCP client - -Add Data Hub to your MCP client configuration. For example, in Claude Desktop (`claude_desktop_config.json`): - -```json -{ - "mcpServers": { - "data-hub": { - "url": "https://datahub.example.com/api/v1/mcp", - "headers": { - "Authorization": "Bearer dhub_abc123..." - } - } - } -} -``` - -Or in Cursor (`.cursor/mcp.json`): - -```json -{ - "mcpServers": { - "data-hub": { - "url": "https://datahub.example.com/api/v1/mcp", - "headers": { - "Authorization": "Bearer dhub_abc123..." - } - } - } -} -``` - -See the [MCP server docs](../reference/mcp.md) for the full list of tools, resources, prompts, and client-specific setup details. - -### With the API directly - -Pass the token in the `Authorization` header: - -```sh -curl https://datahub.example.com/api/v1/instruments \ - -H "Authorization: Bearer dhub_abc123..." -``` - -## Viewing tokens - -Go to **Settings > Access Tokens** in the web app. The table shows: - - -| Column | Description | -| ------------- | ----------------------------------------------------------------------- | -| **Name** | The label you gave the token | -| **Token** | The prefix only (e.g., `dhub_a1b2...`) — the full token is never stored | -| **Last used** | When the token was last used to authenticate an API request | -| **Expires** | Expiration date, or "Never" | -| **Created** | When the token was created | - - -## Revoking a token - -### In the web app - -1. Go to **Settings > Access Tokens**. -2. Click the delete button next to the token you want to revoke. -3. Confirm the deletion. - -The token is immediately invalidated. Any watcher or client using it will start receiving `401 Unauthorized` errors. - -### Via the API - -```sh -curl -X DELETE https://datahub.example.com/api/v1/tokens/ \ - -H "Cookie: " -``` - -## Security notes - -- Tokens are hashed with SHA-256 before storage. The plaintext is never persisted. -- Token create and delete are admin-only operations. Admins can revoke any user's token to support off-boarding and incident response. -- Use descriptive names so you can identify which watcher or client each token belongs to. -- Set expiration dates for tokens used in temporary setups. -- Revoke tokens immediately when a watcher is decommissioned or a token may have been exposed. - -## After revoking a token - -If you revoke a token that a running watcher is using, the watcher will start failing on its next heartbeat or API call. To fix it: - -1. Create a new token. -2. On the instrument PC, re-run the setup wizard: - ```sh - data-hub-watcher init - ``` -3. Enter the new token when prompted. -4. Restart the watcher: - ```sh - data-hub-watcher watch - ``` - diff --git a/developer-docs/guides/upgrading-the-watcher.md b/developer-docs/guides/upgrading-the-watcher.md deleted file mode 100644 index 8eb0873c..00000000 --- a/developer-docs/guides/upgrading-the-watcher.md +++ /dev/null @@ -1,187 +0,0 @@ -# Upgrading the watcher - -This guide covers what happens between a new `data-hub-watcher` release landing on PyPI and the upgraded code running on a lab instrument PC. It's written for two audiences: **lab operators** keeping a single PC up to date (start at [How upgrades reach a lab PC](#how-upgrades-reach-a-lab-pc)), and **release admins** cutting a new version (jump to [Cutting a new release](#cutting-a-new-release)). - -If you're installing the watcher for the first time, see [Installing a watcher](installing-a-watcher.md) first; this guide assumes the watcher is already registered. - -## How upgrades reach a lab PC - -Three paths feed an upgraded wheel into the running watcher. Lab PCs running as a Windows service get the auto-update path for free; everything else upgrades on demand via the CLI. - -### Background auto-update (Windows service, recommended) - -When the watcher runs as a Windows service in `staging` or `production`, the in-process updater polls the API roughly once an hour and applies new releases on its own — no operator action and no manual Task Scheduler setup required. - -On each tick the service: - -1. Calls `GET /api/v1/watchers//update-check` and compares the server's `latest_version` against its own. -2. Only attempts an upgrade if **all** of these are true: a newer version is available, no files have been uploaded for several heartbeats in a row, and no run has been reported within roughly 5× the configured `stability_period_seconds`. The activity-window guard exists so the watcher never takes itself down mid-acquisition. Releases flagged as **mandatory** on the server skip this guard — see [Mandatory updates](#mandatory-updates). -3. Drives the upgrade subprocess: Windows uv-tool installs route through the `DataHubWatcherUpgrade` Scheduled Task (registered by `service install`); other installs run `uv tool install --reinstall` (or `pip install -U`) inline and exit non-zero so the SCM restarts into the new wheel. -4. Emits an `update_started` event before the subprocess runs, then `update_succeeded` or `update_failed` from the next process startup. Windows uv-tool events also carry `worker_returncode`, `worker_stdout_tail`, and `worker_stderr_tail` for dashboard-side debugging. - -Auto-update is **disabled** in the `preview` environment so PR preview deployments can never push code to production lab PCs. - -### Manual operator-driven upgrade (`self-update`) - -Use this when you want an upgrade immediately, or on a non-Windows / non-service install. - -```sh -data-hub-watcher self-update # check + upgrade if needed -data-hub-watcher self-update --check # report status only, no upgrade -data-hub-watcher self-update --force # re-run the upgrade subprocess - # even if the version already matches -``` - -The command asks the API for the latest published version, compares it to the locally installed version, and runs the appropriate flow for your install method: - -| Install method | Upgrade flow | -| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Windows + `uv tool install` | Routes through the `DataHubWatcherUpgrade` Scheduled Task — same out-of-process worker the auto-updater uses. CLI exits as soon as the task accepts the request; service restart is automatic. Tail `~/.data-hub/upgrade-worker.log` to watch progress. | -| POSIX + `uv tool install` | `uv tool install --reinstall data-hub-watcher==` inline. **Restart the watcher** to pick up the new code. | -| Plain venv `pip install` | ` -m pip install -U data-hub-watcher==` inline. **Restart the watcher.** | -| Editable / `uv sync` checkout | Refused; upgrade manually with `git pull && uv sync`. | - -The Windows uv-tool path **requires** that `data-hub-watcher service install` (or `service reinstall`) has been run from an Administrator shell at least once on the machine — that's what registers the `DataHubWatcherUpgrade` Scheduled Task. If the task is missing, `self-update` fails fast with an actionable error. Fleet PCs auto-updating into a worker-aware build for the first time must be reinstalled once with `service reinstall` to pick up the new task. - -To run upgrades unattended on a non-service install, schedule the CLI via Windows Task Scheduler (e.g. weekly). - -### Editable / developer checkouts - -If the watcher is installed editable from a checkout (`uv sync --all-packages`), both the auto-updater and `self-update` refuse to act and tell you to upgrade manually with `git pull && uv sync`. The refusal is intentional — auto-upgrading would silently shadow your source tree with an index build. - -## Pinning a specific version - -To keep a specific PC on a particular release rather than tracking the server's `latest_version`, pin it explicitly: - -```sh -uv tool install data-hub-watcher== -``` - -Run `data-hub-watcher self-update --check` afterwards to confirm what the server's target is. As long as the pinned version matches, the auto-update tick is a no-op; the moment the server's target moves past your pin, the next tick will try to upgrade past it again. **To pin a whole fleet instead, use *Latest version* in [Settings → Watchers](#cutting-a-new-release).** - -## Cutting a new release - -Admins-only. The flow is tag-driven; releases publish from a `production` commit, never from a feature branch. - -1. **Bump the version.** Edit `[project].version` in `watcher/pyproject.toml` ([PEP 440](https://peps.python.org/pep-0440/)). Merge through `staging` to `production`. -2. **Tag and push** from `production`: - ```sh - git checkout production && git pull - git tag watcher-v0.3.0 && git push origin watcher-v0.3.0 - ``` -3. **Approve the `pypi` deployment** under **Actions → Publish watcher** in GitHub. -4. **Advertise the release.** Open **Settings → Watchers** in Data Hub and set **Latest version** to the new tag. Each Vercel env has its own DB, so staging and production are independent — bump them separately to roll the fleet gradually. - -> **Always tag → publish → verify → save.** Saving a **Latest version** before the wheel is on PyPI causes a wave of `update_failed` events from the fleet. - -### Release-config fields - -| Field | Purpose | -| ----------------------------- | ---------------------------------------------------------------------------------- | -| **Latest version** | Required. Blank = "no update info available". | -| **Minimum supported version** | Reserved; not enforced server-side yet. | -| **Release channel** | Defaults to `stable`. Surfaced in `self-update` output. | -| **Mandatory update** | Skips the activity-window guard — see [Mandatory updates](#mandatory-updates). | - -### Notes on the publish workflow - -- `make py-check-watcher-version` refuses to proceed if the git tag and `watcher/pyproject.toml` disagree. -- PyPI uploads use OIDC trusted publishing — no API token in repo secrets. -- The `verify` job installs from PyPI in a clean venv and runs `data-hub-watcher --version` + `import data_hub_watcher` as a smoke test. -- Re-running the workflow after a transient PyPI outage is safe (`skip-existing: true` skips already-published files). -- Manual `workflow_dispatch` is refused from any branch other than `production`. - -## Mandatory updates - -Toggling **Mandatory update** on the **Settings → Watchers** page skips the activity-window guard (see [Background auto-update](#background-auto-update-windows-service-recommended)) and fires the upgrade on the next hourly check. The server still compares the running version to `latest_version`, so a correctly-pinned PC isn't forced to anything. - -Use sparingly — a forced upgrade mid-microscopy-run loses data. Reserve it for security fixes, wire-protocol breaks, or any case where leaving the bad version running is strictly worse than restarting in flight. - -## Rolling back - -Rollback is just another release. To revert the fleet from `0.3.0` to `0.2.5`: - -1. Open **Settings → Watchers** in Data Hub and set **Latest version** to `0.2.5` for the affected environment(s). -2. Toggle **Mandatory update** on if you need the rollback to bypass the activity-window guard (most rollback scenarios qualify — you're rolling back precisely because the running version is misbehaving). -3. Wait for the next hourly tick. Lab PCs running an auto-update-capable build will downgrade themselves; PCs being upgraded manually need a `data-hub-watcher self-update` (or `uv tool install data-hub-watcher==0.2.5` if `self-update` itself is what's broken). -4. Once the fleet has converged, toggle **Mandatory update** back off. - -There's no separate "yank" step — a rolled-back release is still on PyPI and still reinstallable, just not advertised by `/update-check`. - -## Troubleshooting - -### Common - -#### `update_started` is followed by `update_failed` - -The upgrade subprocess started but didn't end up running the new version. The `details.reason` field tells you which sub-case fired: - -- `subprocess raised: …` — `subprocess.run` raised before exec. Usually `FileNotFoundError` because `uv` isn't on PATH for the service account. Check the watcher log for the full traceback. -- `subprocess exited ` — the upgrade command itself failed. Event details include the last 1000 bytes of stdout/stderr; most often a transient PyPI failure or the version not yet existing on the index. -- `expected '' after upgrade, running ''` — install succeeded, restart succeeded, but the running interpreter still imports the old version. Usually a stale `__pycache__` or a separate copy on `sys.path`. Run `uv tool uninstall data-hub-watcher && uv tool install data-hub-watcher==` as the service account. - -#### `update_failed` without a preceding `update_started` - -`details.attempted_subprocess: false` means the auto-updater refused before running the upgrade. The most common `details.reason`: - -- `install method '' not eligible for auto-update` — the watcher detected a development install and refused to shadow your source tree. Switch the host to a PyPI install (`uv tool install data-hub-watcher`); on a developer machine, ignore the event. Throttled to one emission per server target. - -#### Auto-update never fires - -Check, in order: - -- The watcher is running as a service, not in a console window. -- The environment isn't `preview` (auto-update is hard-disabled there). -- The activity-window guard isn't holding things up. Use `data-hub-watcher self-update` for an immediate upgrade, or toggle **Mandatory update** if the release warrants it. -- The dashboard's **Last Heartbeat** is recent. A stale watcher isn't ticking and won't auto-update. - -#### `self-update --check` says "(none configured)" - -**Settings → Watchers** has a blank **Latest version** for that environment. Benign — the CLI returns successfully and treats it as "no update available". - -#### "Refusing to self-update an editable / unknown install" - -You're running from a checkout (`uv sync --all-packages`). Upgrade manually with `git pull && uv sync`. To switch a developer machine onto a PyPI build, `uv tool install data-hub-watcher` from a separate shell gets you a parallel install on PATH. - -#### A failed upgrade left a stale marker - -`~/.data-hub/.upgrade-in-progress` is consumed and deleted on the next clean startup. If it persists across multiple restarts, the watcher is failing to start at all — fix the underlying startup issue and delete the marker. - -#### Where to find the raw installer transcript - -| Install path | Log file | -| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| In-process (POSIX, Windows pip) | `~/.data-hub/watcher.log` (`C:\ProgramData\DataHubWatcher\watcher.log` on Windows) — subprocess output prefixed `Upgrade subprocess stdout/stderr tail:`. | -| Windows uv-tool worker | `~/.data-hub/upgrade-worker.log` — every line of uv's output with a UTC timestamp. | - -Both files are append-only and survive service restarts. - -### Windows uv-tool specifics - -#### `update_failed` with `reason: "scheduled task could not be triggered…"` - -The watcher tried to dispatch through the `DataHubWatcherUpgrade` Scheduled Task but `schtasks /Run` refused. Two causes: - -- **Task not registered** (most common on first auto-update into a worker-aware build). Open an Administrator PowerShell and run `data-hub-watcher service reinstall` — it registers the task and restarts the service. -- **Service account can't talk to Task Scheduler** (`details.schtasks_stderr` shows "Access is denied"). Reinstall the service under `LocalSystem` (the default) via `service reinstall`. - -The watcher self-heals on the next service start by re-registering a missing task, so a single SCM-driven restart often resolves this without operator action. - -#### `update_failed` with `details.worker_result_missing: true` - -The worker dispatched but never wrote `~/.data-hub/.upgrade-result.json` — usually a PowerShell crash between `Stop-Service` and the `uv` invocation, or Task Scheduler killing the worker. Open `~/.data-hub/upgrade-worker.log` for the captured output. The service still comes back up regardless. - -#### Worker reported success but the dashboard shows `update_failed` - -The post-restart evaluator trusts the worker's `succeeded` flag over the marker's version comparison. If `uv` exited non-zero but the install moved the version forward, you still get `update_failed` with the real installer error in `worker_stderr_tail`. The marker's classification is preserved in `details.marker_succeeded` / `details.marker_reason` for debugging. - -#### `self-update` reports "Upgrade dispatched" but nothing happens - -The CLI returns as soon as the Scheduled Task accepts the request — the install runs out-of-process under SYSTEM and takes 30–60s. Track progress with: - -```powershell -Get-Service data-hub-watcher # cycles Stopped -> Start Pending -> Running -Get-Content ~/.data-hub/upgrade-worker.log -Tail 20 -Wait -``` - -The dashboard event lands once the new service process boots and reads the marker + result sentinel. diff --git a/developer-docs/local-development.md b/developer-docs/local-development.md index 19da1993..289dbd60 100644 --- a/developer-docs/local-development.md +++ b/developer-docs/local-development.md @@ -138,7 +138,7 @@ Some features depend on services that aren't running in this workflow. Each one ## Testing the Lambda end-to-end -Working on a `process_file()` module (or wiring up a brand new one — see [Adding an instrument](guides/adding-an-instrument.md)) and want to run it against the local web app without standing up real S3? The lambda CLI ships a `handler` subcommand that drives `lambda_handler` end-to-end against a gitignored directory mirroring the S3 layout. +Working on a `process_file()` module (or wiring up a brand new one — see [Lambda → Adding a new instrument](reference/lambda.md#adding-a-new-instrument)) and want to run it against the local web app without standing up real S3? The lambda CLI ships a `handler` subcommand that drives `lambda_handler` end-to-end against a gitignored directory mirroring the S3 layout. ```sh cd lambda @@ -226,5 +226,5 @@ The same builders back the integration test harness in [web/tests/integration/he - [Getting started](getting-started.md) — full setup with real Google OAuth and AWS credentials. - [Architecture](architecture.md) — system overview and data flow. -- [REST API](reference/api.md) — endpoint reference for the seeded PAT. -- [MCP server](reference/mcp.md) — Model Context Protocol tools at `/api/v1/mcp`. +- [REST API](https://arcadia-data-hub-docs.vercel.app/docs/api-reference) — endpoint reference for the seeded PAT. +- [MCP server](https://arcadia-data-hub-docs.vercel.app/docs/mcp-server) — Model Context Protocol tools at `/api/v1/mcp`. diff --git a/developer-docs/ops/ci-and-deployment.md b/developer-docs/ops/ci-and-deployment.md index 27af123e..309abdd1 100644 --- a/developer-docs/ops/ci-and-deployment.md +++ b/developer-docs/ops/ci-and-deployment.md @@ -228,7 +228,7 @@ make sam-deploy ENV=staging ### Watcher (PyPI) -The `data-hub-watcher` Python package is published to [PyPI](https://pypi.org/project/data-hub-watcher/) so lab PCs can install and self-update via `uv tool install data-hub-watcher`. The full release flow — version bump, tag, approval, env-var roll-out, mandatory updates, and rollback — is documented in the operator-facing [Upgrading the watcher](../guides/upgrading-the-watcher.md) guide; this section is intentionally a pointer rather than a second source of truth so the two can't drift. +The `data-hub-watcher` Python package is published to [PyPI](https://pypi.org/project/data-hub-watcher/) so lab PCs can install and self-update via `uv tool install data-hub-watcher`. The full release flow — version bump, tag, approval, env-var roll-out, mandatory updates, and rollback — is documented in the admin-facing [Managing watchers → Releases and fleet updates](https://arcadia-data-hub-docs.vercel.app/docs/managing-watchers#releases-and-fleet-updates) guide; this section is intentionally a pointer rather than a second source of truth so the two can't drift. Trusted publishing is configured under **Project → Publishing** on PyPI for `Arcadia-Science/data-hub` and the workflow `publish-watcher.yml`; no API token lives in repo secrets. If trust is ever revoked or rotated, update it there and re-run the workflow. diff --git a/developer-docs/ops/run-archives.md b/developer-docs/ops/run-archives.md index cb0f10c6..b2d58d67 100644 --- a/developer-docs/ops/run-archives.md +++ b/developer-docs/ops/run-archives.md @@ -4,7 +4,7 @@ The "Download all" actions on the run detail page and the runs table deliver eve Each archive can mix files from the raw bucket and the processed bucket in a single zip. This matters for instruments that produce processed artifacts via Lambda preprocessing (SpectraMax raw `.xls` → processed CSV; Hina `.nd2` → processed JPG; Azure 600 Gel Doc `.tif` → processed PNG): the run's file rows reference both buckets, and "Download all" zips them together. -This page covers the end-to-end flow, the cache + dedup model, and the on-call runbook. For the Lambda invocation contract, see [Lambda → Function URL (archive build)](../reference/lambda.md#function-url-archive-build). For the HTTP endpoints, see [REST API → Archive jobs](../reference/api.md#archive-jobs). +This page covers the end-to-end flow, the cache + dedup model, and the on-call runbook. For the Lambda invocation contract, see [Lambda → Function URL (archive build)](../reference/lambda.md#function-url-archive-build). For the HTTP endpoints, see [REST API → Archive jobs](https://arcadia-data-hub-docs.vercel.app/docs/api-reference#archive-jobs). ## Flow diff --git a/developer-docs/reference/api.md b/developer-docs/reference/api.md deleted file mode 100644 index fca2fdba..00000000 --- a/developer-docs/reference/api.md +++ /dev/null @@ -1,164 +0,0 @@ -# REST API - -The Data Hub API is served by the Next.js web application at `/api/v1/`. It is used by the watcher, the Lambda function, MCP clients, and the web dashboard. - -## Authentication - -The API supports two authentication methods: - -- **Session cookies** — used by the web dashboard (Google OAuth via NextAuth). Session-authenticated callers implicitly hold every scope; scope enforcement only applies to token-authenticated requests. -- **Bearer tokens** — used by the watcher, Lambda, and MCP clients. Tokens are created in the web dashboard under personal access tokens and sent in the `Authorization: Bearer ` header. - -Tokens are hashed with SHA-256 before storage. The plaintext token is shown once at creation time. - -Web page routes use a different gating model from the `/api/v1/*` surface: they're publicly reachable so link previews work, and the page body itself short-circuits to a sign-in CTA when there's no session. The API always requires either a session cookie or a bearer token (see [architecture](../architecture.md)). - -### Admin role - -A subset of mutations is gated on the workspace admin role in addition to (or instead of) the scope check: - -- `PATCH /api/v1/instruments/:instrumentId` — session callers must be admin; bearer-token callers continue to authenticate solely via the `instruments:write` scope, so existing watcher/Lambda automation is unaffected. -- `POST /api/v1/tokens` and `DELETE /api/v1/tokens/:id` — admin-only, session-only. Bearer tokens cannot manage other tokens. -- `GET /api/v1/users`, `PATCH /api/v1/users/:userId` — admin-only, session-only. Used by the **Settings > Members** page to toggle other users' admin flag. - -The first admin is bootstrapped from the `ADMIN_EMAILS` env var (comma-separated, case-insensitive); listed users are promoted to admin on every sign-in. Subsequent admins can be promoted in the UI by any existing admin. Admins cannot demote themselves — `PATCH /api/v1/users/:userId` with `{ is_admin: false }` on the caller's own user id returns `400 VALIDATION_ERROR`. - -### Scopes - -Every personal access token carries an array of permission scopes. A request is rejected with `403 FORBIDDEN` when the token's scopes do not include the scope required by the route. The vocabulary is: - -| Scope | Grants | -| --- | --- | -| `instruments:read` | List/read instruments | -| `instruments:write` | Create/update instruments | -| `runs:read` | List/read runs (and their comments) | -| `runs:write` | Create/update/delete runs, comments, attributions, and run-level upload/reprocess endpoints | -| `files:read` | Read file metadata, download files, download run archives | -| `files:write` | Create/update/delete file records and reprocess files | -| `watchers:read` | Read watcher state (list, heartbeats, events, config, upload queue, update-check) | -| `watchers:write` | Register/deregister watchers, post heartbeats and events, push config | -| `archive-jobs:read` | Read archive job state. No endpoints currently consume this scope — the existing run-archive download is gated on `files:read` because it returns file bytes — but it is reserved for future archive-job listing/status endpoints. | -| `archive-jobs:write` | Update archive jobs (Lambda callback). | -| `*` | Wildcard — matches every scope. Reserved for the migration backfill and the watcher/Lambda PATs until they are rotated to least-privilege scopes; `POST /api/v1/tokens` rejects `*` from API callers. | - -MCP tools enforce the same scopes their REST counterparts do: `search_runs` and `get_run` require `runs:read`, `reprocess_file` requires `files:write`, `claim_run`/`unclaim_run` require `runs:write`, and so on. There is no MCP-specific scope vocabulary — a token with `runs:read` over REST also covers the read-side run tools over MCP. - -Migration `0022_pat_scopes` backfills every pre-existing token with `["*"]`, so deployed watchers and the Lambda continue to work after deploy until their tokens are rotated to explicit scopes. - -`403 FORBIDDEN` responses use the standard error shape: - -```json -{ - "error": { - "code": "FORBIDDEN", - "message": "Token is missing required scope: runs:write" - } -} -``` - -## Endpoints - -### Instruments - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/instruments` | List all instruments | -| `POST` | `/api/v1/instruments` | Create a new instrument | -| `GET` | `/api/v1/instruments/:instrumentId` | Get instrument details (includes run and watcher counts) | - -### Runs - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/instruments/:instrumentId/runs` | List runs for an instrument | -| `POST` | `/api/v1/instruments/:instrumentId/runs` | Create a new run | -| `GET` | `/api/v1/instruments/:instrumentId/runs/:runId` | Get run details | -| `PATCH` | `/api/v1/instruments/:instrumentId/runs/:runId` | Update a run | -| `DELETE` | `/api/v1/instruments/:instrumentId/runs/:runId` | Soft-delete a run | -| `POST` | `/api/v1/instruments/:instrumentId/runs/:runId/restore` | Restore a soft-deleted run | -| `GET` | `/api/v1/instrument-runs` | List runs across all instruments | - -### Comments - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/instruments/:instrumentId/runs/:runId/comments` | List active comments on a run (oldest first) | -| `POST` | `/api/v1/instruments/:instrumentId/runs/:runId/comments` | Create a markdown comment on a run | -| `PATCH` | `/api/v1/instruments/:instrumentId/runs/:runId/comments/:commentId` | Edit a comment (author only; sets `edited_at`) | -| `DELETE` | `/api/v1/instruments/:instrumentId/runs/:runId/comments/:commentId` | Soft-delete a comment (author only) | - -Comment bodies are markdown source, capped at 10 000 characters. Author-only mutations are enforced server-side and return `403 FORBIDDEN` for cross-user edit/delete attempts. Mutations on comments whose parent run has been soft-deleted return `409 CONFLICT`. - -### Files - -| Method | Path | Description | -| --- | --- | --- | -| `POST` | `/api/v1/instruments/:instrumentId/runs/:runId/files` | Create a file record | -| `GET` | `/api/v1/files/:fileId` | Get file details | -| `PATCH` | `/api/v1/files/:fileId` | Update file metadata | -| `GET` | `/api/v1/files/:fileId/download` | Get a pre-signed S3 download URL | -| `POST` | `/api/v1/instruments/:instrumentId/runs/:runId/request-upload` | Request file upload (manual mode) | -| `POST` | `/api/v1/instruments/:instrumentId/runs/:runId/request-upload-url` | Get a pre-signed S3 upload URL for a file | - -### Watchers - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/watchers` | List all watchers | -| `POST` | `/api/v1/watchers/register` | Register a new watcher (returns `409 CONFLICT` if the instrument already has an active watcher; the existing watcher's id is included in `error.details.existing_watcher_id`) | -| `GET` | `/api/v1/watchers/:watcherId` | Get watcher details | -| `DELETE` | `/api/v1/watchers/:watcherId` | Deregister (soft-delete) a watcher | -| `POST` | `/api/v1/watchers/:watcherId/heartbeat` | Send a heartbeat | -| `GET` | `/api/v1/watchers/:watcherId/heartbeats` | Get heartbeat history | -| `POST` | `/api/v1/watchers/:watcherId/events` | Submit watcher events | -| `GET` | `/api/v1/watchers/:watcherId/config` | Get synced config YAML | -| `PUT` | `/api/v1/watchers/:watcherId/config` | Push config YAML and checksum | -| `GET` | `/api/v1/watchers/:watcherId/config-checksum` | Get the config checksum | -| `GET` | `/api/v1/watchers/:watcherId/upload-queue` | Get pending upload queue | -| `GET` | `/api/v1/watchers/:watcherId/update-check` | Get server-advertised release info (latest version, channel, mandatory flag); used by the watcher's self-update CLI and background auto-updater. See [Upgrading the watcher](../guides/upgrading-the-watcher.md). | - -### Archive jobs - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/instruments/:instrumentId/runs/:runId/download-archive` | Download the run archive. On cache hits returns `302` with a presigned S3 URL (or `200 { status: "ready", download_url, size_bytes }` if the caller sent `Accept: application/json`). On cache misses always returns `202 { status: "building", job_id }` and dispatches the build asynchronously; the same URL re-issued is the canonical poll target — every poll re-runs the S3 HEAD, so a finished build is visible the moment the multipart upload lands. Optional `?file_ids=1,2,3` narrows the archive to a subset of files (always intersected with the run's own files). | -| `PATCH` | `/api/v1/archive-jobs/:id` | Lambda callback: marks an async build as `ready` (with `archive_bucket`, `archive_key`, `size_bytes`) or `failed` (with `error_message`). Stamps `completed_at` on terminal transitions. Uses standard PAT/session auth — the Lambda calls this with its `DATA_HUB_API_KEY` PAT. The UI does not trust this row's `status` for download readiness (it polls `/download-archive`, which short-circuits on an S3 HEAD), so a tampered row at worst breaks its own download. | - -The download-archive endpoint sits in front of a Lambda-driven builder pipeline that produces zips in S3 and serves them via presigned URLs, so download bytes never travel through Vercel. See [Run archives](../ops/run-archives.md) for the full flow, cache semantics, and operator runbook. - -### Tokens - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/tokens` | List personal access tokens. Response includes each token's `scopes`. | -| `POST` | `/api/v1/tokens` | Create a new token. Admin-only; requires a non-empty `scopes` array (see [Scopes](#scopes)); the wildcard `*` is rejected. | -| `DELETE` | `/api/v1/tokens/:id` | Revoke a token. Admin-only; admins can revoke any user's PAT. | - -### Users - -| Method | Path | Description | -| --- | --- | --- | -| `GET` | `/api/v1/users` | List workspace users with their admin flag. Admin-only, session-only. | -| `PATCH` | `/api/v1/users/:userId` | Toggle the workspace `is_admin` flag for a user. Admin-only, session-only; admins cannot demote themselves. | - -### MCP (Model Context Protocol) - -| Method | Path | Description | -| --- | --- | --- | -| `GET`, `POST` | `/api/v1/mcp` | MCP server endpoint (Streamable HTTP transport) | - -The MCP server exposes Data Hub data to AI clients (e.g. Claude Desktop, Cursor) via the [Model Context Protocol](https://modelcontextprotocol.io/). It uses Bearer token authentication only — session cookies are not supported. - -See the [MCP server docs](mcp.md) for the full tool, resource, and prompt reference and installation instructions for Claude Desktop and Cursor. - -## Error responses - -Errors follow a consistent shape: - -```json -{ - "code": "NOT_FOUND", - "message": "Instrument not found.", - "details": null -} -``` diff --git a/developer-docs/reference/lambda.md b/developer-docs/reference/lambda.md index d7b04b71..11ed7181 100644 --- a/developer-docs/reference/lambda.md +++ b/developer-docs/reference/lambda.md @@ -72,6 +72,8 @@ Slack channel notifications are sent by the **web app** (`web/lib/slack.ts`), no 4. **Add tests.** Add unit tests in `lambda/tests/` for the new processor. +5. **Configure the S3 trigger and deploy.** See [Deploying AWS infrastructure → Adding a Lambda processor for a new instrument](https://arcadia-data-hub-docs.vercel.app/docs/deploying-aws-infrastructure#adding-a-lambda-processor-for-a-new-instrument) for the `infra/template.yaml` trigger entry and the deploy steps. + ## Local processing CLI The `data-hub-process` CLI lets you exercise instrument-specific parsing and processing locally. Most subcommands run a single processor in isolation against a file on disk and print the result; the `handler` subcommand drives `lambda_handler` end-to-end against a local S3 mirror and the dev API. diff --git a/developer-docs/reference/mcp.md b/developer-docs/reference/mcp.md deleted file mode 100644 index 4d0cfb96..00000000 --- a/developer-docs/reference/mcp.md +++ /dev/null @@ -1,154 +0,0 @@ -# MCP server - -The Data Hub MCP server exposes instruments, runs, files, and watcher data to AI clients — such as Claude Desktop and Cursor — through the [Model Context Protocol](https://modelcontextprotocol.io/). Clients can list instruments, search runs, fetch experimental results, generate download URLs, and re-trigger Lambda processing, all through a single Bearer-authenticated HTTP endpoint served by the web app. - -The server lives at `/api/v1/mcp` on the same Next.js deployment that serves the [REST API](api.md) and uses the MCP Streamable HTTP transport. - -## Authentication - -The MCP server accepts **Bearer tokens only** — session cookies are not supported. Create a personal access token in the web app at **Settings > Access Tokens**, then pass it in the `Authorization: Bearer ` header when configuring your client. - -See [Managing tokens](../guides/managing-tokens.md) for details on creating, using, and revoking tokens. - -## Installation - -### Claude Desktop - -Edit `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`): - -```json -{ - "mcpServers": { - "data-hub": { - "url": "https://datahub.example.com/api/v1/mcp", - "headers": { - "Authorization": "Bearer dhub_abc123..." - } - } - } -} -``` - -Restart Claude Desktop. The `data-hub` server should appear in the MCP panel and its tools become available in conversations. - -### Cursor - -Edit `.cursor/mcp.json` in your project or `~/.cursor/mcp.json` globally: - -```json -{ - "mcpServers": { - "data-hub": { - "url": "https://datahub.example.com/api/v1/mcp", - "headers": { - "Authorization": "Bearer dhub_abc123..." - } - } - } -} -``` - -Reload Cursor. Tools are invoked via the agent automatically when relevant. - -### Other clients - -The endpoint follows the MCP Streamable HTTP spec, so any compliant client works. Configure it with: - -- **URL**: `https://datahub.example.com/api/v1/mcp` -- **Transport**: Streamable HTTP (`GET` for the SSE stream, `POST` for client messages) -- **Auth header**: `Authorization: Bearer ` - -For local development, point at `http://localhost:3000/api/v1/mcp` instead. - -## Tools - -All tools return JSON encoded as a single text content block. Error cases set `isError: true` and return a plain-text message. - -### Instruments - -| Tool | Description | -| --- | --- | -| `list_instruments` | List all registered instruments with run counts, watcher status, and file patterns. Optionally filter by `status` (`pending`, `active`, `inactive`). | -| `get_instrument` | Get full detail for an instrument by its kebab-case ID, including watcher online/offline counts. | - -### Runs - -| Tool | Description | -| --- | --- | -| `search_runs` | Paginated search across runs with filtering, sorting, and date range. Supports plate-reader metadata filters (`wavelength`, `measurementMode`, `measurementType`) and attribution filtering via `ranBy` (a user id or the literal `"unattributed"`). | -| `get_run` | Get a single run by its natural key (`instrumentId` + `runId`). | -| `list_run_files` | List all files attached to a run, including processing status and metadata. Use `get_file_download_url` on processed CSV files to access experimental results. | -| `get_run_archive` | Get a downloadable ZIP archive of all uploaded files for a run. Returns a 15-minute pre-signed S3 URL on a cache hit (clickable in a browser, no auth required), or a `building` job + `retryAfterSeconds` hint on a miss — call again after the suggested wait to poll. | - -Both `get_run` and `search_runs` responses embed an `attributions` array on each run, listing the users who have claimed it (user id, display name, initials, avatar URL). No separate read tool is needed to inspect who ran a run. - -### Run attribution - -| Tool | Description | -| --- | --- | -| `claim_run` | **Write tool.** Mark a run as performed by the authenticated user. Idempotent. Only self-attribution is supported — the user id comes from the session token, never from an argument, so you cannot claim a run on behalf of another user. | -| `unclaim_run` | **Write tool.** Remove the authenticated user's attribution from a run. Idempotent. Annotated `destructiveHint: true` because removing attribution is user-visible across the dashboard and runs tables. | -| `list_run_attributors` | List distinct users who have claimed at least one run on a given instrument. Use the returned `userId` values to construct a `search_runs` call with `ranBy=`. | - -### Files - -| Tool | Description | -| --- | --- | -| `get_file` | Get detailed metadata for a single file by numeric ID. | -| `get_file_download_url` | Get a pre-signed S3 URL for downloading a file's raw contents. URLs expire after 15 minutes and can be fetched without additional authentication. | -| `reprocess_file` | **Write tool.** Re-run the Lambda processing workflow for a `failed` or `completed` file. Transitions the file back to `processing`. Annotated `destructiveHint: true` so clients can warn before invoking. | - -### Watchers and system status - -| Tool | Description | -| --- | --- | -| `get_system_status` | Dashboard-level overview of all instruments, watcher health, and pending upload counts. | -| `list_watchers` | List watcher agents with effective status, hostname, instrument assignment, and last heartbeat. Optionally filter by `instrumentId`. | -| `get_watcher_heartbeats` | Recent heartbeat history for a watcher, useful for diagnosing connectivity gaps and error trends. Configurable `hours` lookback (default 24, max 168). | - -## Resources - -Resources provide reference context that clients can attach to prompts without an explicit tool call. - -| URI | Description | -| --- | --- | -| `datahub://instruments` | List of all instrument IDs, display names, and types. Useful as grounding context when constructing tool calls. | -| `datahub://instruments/{instrumentId}/filter-options` | Available filter values for an instrument. Plate readers expose wavelengths and measurement modes/types; gel-doc instruments expose capture types, imaging modes, wavelengths, and colors. Helps build valid `search_runs` queries. | - -## Prompts - -Prompts are scripted workflows the client surfaces to the user. Each prompt assembles a multi-step instruction that the model then executes using the tools above. - -| Prompt | Args | Description | -| --- | --- | --- | -| `daily_summary` | `date` (optional, YYYY-MM-DD) | Summarize all instrument activity for a given day — run counts, failures, and system health. | -| `troubleshoot_instrument` | `instrumentId` | Diagnose connectivity or processing issues for an instrument by inspecting its status, watcher heartbeats, and recent runs. | -| `compare_runs` | `instrumentId`, `runId1`, `runId2` | Compare two runs on the same instrument side by side, highlighting differences in conditions and outcomes. | - -## Example usage - -Once installed, ask your client questions like: - -- *"What instruments are active right now?"* → `list_instruments` with `status="active"` -- *"Show me all SpectraMax runs from last Friday."* → `search_runs` with an `instrumentId` and date range -- *"The gel-doc in Lab 3 stopped uploading — what's wrong?"* → `troubleshoot_instrument` prompt, which inspects the watcher list and heartbeat history -- *"Re-run processing for file 4217, we pushed a parser fix."* → `reprocess_file`. Clients typically confirm the destructive action with the user first. -- *"Claim run `2026-03-26_experiment` on the SpectraMax — I ran it this morning."* → `claim_run`. To find runs you've already claimed, use `search_runs` with `ranBy` set to your user id (discoverable via `list_run_attributors`). - -## Troubleshooting - -### `401 Unauthorized` - -The Bearer token is missing, mistyped, revoked, or expired. Verify the token at **Settings > Access Tokens** and re-issue if necessary. - -### Tools don't appear in the client - -- Confirm the server is listed under `mcpServers` in the client config. -- Check for JSON syntax errors in the config file. -- Restart the client after editing — most clients don't hot-reload MCP server definitions. -- Hit `https://datahub.example.com/api/v1/mcp` with `curl -H "Authorization: Bearer "` to confirm the endpoint responds. - -### `get_file_download_url` vs. `get_run_archive` - -- `get_file_download_url` returns a pre-signed S3 URL for a single file. Anyone with the link can fetch it for 15 minutes — no Data Hub credentials required on the follow-up request. -- `get_run_archive` returns the same kind of pre-signed S3 URL, but for a multi-file ZIP archive of an entire run. On a cache miss the Lambda builds the archive asynchronously and the tool returns `{ status: "building", jobId, retryAfterSeconds }`; call the tool again after the suggested wait until you get back `{ status: "ready", downloadUrl }`. Like `get_file_download_url`, the URL itself carries the auth, so the resulting link is browser-clickable without the original Bearer token. diff --git a/developer-docs/reference/watcher.md b/developer-docs/reference/watcher.md index bf86aa62..71477be9 100644 --- a/developer-docs/reference/watcher.md +++ b/developer-docs/reference/watcher.md @@ -30,7 +30,7 @@ uv run data-hub-watcher init uv run data-hub-watcher watch ``` -For lab-PC installs (PyPI), see [Installing a watcher](../guides/installing-a-watcher.md). For releasing new versions and how the in-place upgrade flow works (CLI `self-update` and the Windows-service auto-updater), see [Upgrading the watcher](../guides/upgrading-the-watcher.md). +For lab-PC installs (PyPI), see [Installing a watcher](https://arcadia-data-hub-docs.vercel.app/docs/installing-a-watcher). For releasing new versions and how the in-place upgrade flow works (CLI `self-update` and the Windows-service auto-updater), see [Upgrading the watcher](https://arcadia-data-hub-docs.vercel.app/docs/upgrading-the-watcher). ## Commands @@ -71,7 +71,7 @@ While running: - **Upload worker** (manual mode only) polls the server's upload queue on its own long-lived thread every 60 seconds, decoupled from the heartbeat so a slow or large upload can't delay heartbeats and make a busy watcher look offline. On shutdown it is stopped and joined before the state DB is closed. Auto mode has no worker: uploads run on the monitor's stability-checker thread via the run detector's upload callback. - **Heartbeat loop** sends periodic heartbeats (every 60 seconds) to the API. The payload includes the watcher version, instrument ID, watch directory, upload mode, per-interval activity counters, and process uptime; a final `status="stopped"` heartbeat is sent on graceful shutdown. - **Event reporter** batches and flushes lifecycle events (started, stopped, file uploaded, errors) to the API. See [Observability](#observability) for the full taxonomy. -- **Auto-updater** runs from the same heartbeat tick on every platform — not only Windows services. It polls `GET /watchers/:id/update-check` roughly hourly and applies new releases when the watcher has been idle long enough not to clobber an in-flight run. The full activity-window guard, mandatory-update behavior, and rollback flow are documented in [Upgrading the watcher](../guides/upgrading-the-watcher.md); auto-update is hard-disabled in the `preview` environment. +- **Auto-updater** runs from the same heartbeat tick on every platform — not only Windows services. It polls `GET /watchers/:id/update-check` roughly hourly and applies new releases when the watcher has been idle long enough not to clobber an in-flight run. The full activity-window guard, mandatory-update behavior, and rollback flow are documented in [Upgrading the watcher](https://arcadia-data-hub-docs.vercel.app/docs/upgrading-the-watcher); auto-update is hard-disabled in the `preview` environment. Use `--dry-run` to validate config and preview what would happen without starting the monitor. @@ -126,7 +126,7 @@ Manage the watcher as a Windows service: ### `self-update` -Checks the API for a newer published version and runs the appropriate `uv tool install --reinstall` (or `pip install -U`) subprocess in place. See [Upgrading the watcher](../guides/upgrading-the-watcher.md) for the supported install methods, the activity-window guard, mandatory updates, and rollback flow. +Checks the API for a newer published version and runs the appropriate `uv tool install --reinstall` (or `pip install -U`) subprocess in place. See [Upgrading the watcher](https://arcadia-data-hub-docs.vercel.app/docs/upgrading-the-watcher) for the supported install methods, the activity-window guard, mandatory updates, and rollback flow. ## Configuration diff --git a/watcher/README.md b/watcher/README.md index c7084a78..450f55bb 100644 --- a/watcher/README.md +++ b/watcher/README.md @@ -22,7 +22,7 @@ data-hub-watcher self-update # check for and apply package updates data-hub-watcher service install # Windows: install as a service ``` -See [the operator guide](https://github.com/Arcadia-Science/data-hub/blob/main/developer-docs/guides/installing-a-watcher.md) for the full setup walk-through, configuration reference, and troubleshooting. +See [the operator guide](https://arcadia-data-hub-docs.vercel.app/docs/installing-a-watcher) for the full setup walk-through, configuration reference, and troubleshooting. ## License diff --git a/watcher/pyproject.toml b/watcher/pyproject.toml index 7348515c..2710fdaa 100644 --- a/watcher/pyproject.toml +++ b/watcher/pyproject.toml @@ -42,7 +42,7 @@ Homepage = "https://github.com/Arcadia-Science/data-hub" # the PyPI page actually needs. The developer-facing `developer-docs/reference/watcher.md` # describes the editable-checkout workflow and is reachable from the # repository link below. -Documentation = "https://github.com/Arcadia-Science/data-hub/blob/production/developer-docs/guides/installing-a-watcher.md" +Documentation = "https://arcadia-data-hub-docs.vercel.app/docs/installing-a-watcher" Repository = "https://github.com/Arcadia-Science/data-hub" Issues = "https://github.com/Arcadia-Science/data-hub/issues" From 3f4d984ba7b3d3af9494e1703db464ee68c90f4b Mon Sep 17 00:00:00 2001 From: Wasim Sandhu Date: Tue, 7 Jul 2026 13:00:09 -0700 Subject: [PATCH 2/4] Flatten the developer-docs index in README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Guides/Reference split no longer earned its keep once guides/ shrank to a single entry — Reference also mixed genuine reference docs (watcher, lambda, shared-library) with conceptual/onboarding ones (architecture, getting-started, conventions). Flatten into one ordered list, and add the missing link to ops/run-archives.md. Co-authored-by: Cursor --- README.md | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 1263d075..4600ce1a 100644 --- a/README.md +++ b/README.md @@ -45,23 +45,16 @@ See the full [Getting Started guide](developer-docs/getting-started.md) for prer ## Documentation -User, operator, and admin documentation (installing a watcher, adding an -instrument, managing tokens, deployment) lives on the -[docs site](https://arcadia-data-hub-docs.vercel.app/), not in this repository. -The docs below are for people developing Data Hub itself. - -### Guides +User, operator, and admin documentation (installing a watcher, adding an instrument, managing tokens, deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/), not in this repository. The docs below are for people developing Data Hub itself. +- [Getting started](developer-docs/getting-started.md) — development setup, environment variables, running locally - [Local development](developer-docs/local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed) - -### Reference - - [Architecture](developer-docs/architecture.md) — system overview, data flow, and design decisions -- [Getting started](developer-docs/getting-started.md) — development setup, environment variables, running locally - [Watcher](developer-docs/reference/watcher.md) — CLI commands, configuration, run detection, upload modes - [Lambda](developer-docs/reference/lambda.md) — processing pipeline, supported instruments, adding new instruments - [Shared library](developer-docs/reference/shared-library.md) — module reference for `data-hub-shared` - [CI and deployment](developer-docs/ops/ci-and-deployment.md) — GitHub Actions, Vercel, Render, Lambda deployment +- [Run archives](developer-docs/ops/run-archives.md) — "Download all" flow, cache/dedup model, and on-call runbook - [Conventions](developer-docs/conventions.md) — S3 key layout, instrument IDs, code style, environments ## Development @@ -87,6 +80,4 @@ make fe-test-integration Data Hub is released under the [MIT License](LICENSE). Copyright (c) 2026 Arcadia Science. -"Data Hub" and "Arcadia Science", along with related names and logos, are marks of -Arcadia Science. The MIT License covers the source code only and does not grant any -right to use these names or logos. +"Data Hub" and "Arcadia Science", along with related names and logos, are marks of Arcadia Science. The MIT License covers the source code only and does not grant any right to use these names or logos. From 02175f7159498c8a348e891070d372577a3e1371 Mon Sep 17 00:00:00 2001 From: Wasim Sandhu Date: Tue, 7 Jul 2026 13:27:15 -0700 Subject: [PATCH 3/4] docs: Reorganize documentation --- .github/workflows/typescript-test.yml | 4 +- AGENTS.md | 65 +++------ Makefile | 8 +- README.md | 23 ++-- developer-docs/README.md | 17 +++ developer-docs/architecture.md | 2 +- developer-docs/{ops => }/ci-and-deployment.md | 4 +- developer-docs/conventions.md | 9 +- developer-docs/getting-started.md | 7 +- developer-docs/{reference => }/lambda.md | 4 +- developer-docs/local-development.md | 4 +- developer-docs/{ops => }/run-archives.md | 4 +- .../{reference => }/shared-library.md | 0 developer-docs/testing.md | 126 ++++++++++++++++++ developer-docs/{reference => }/watcher.md | 0 infra/template.yaml | 2 +- watcher/pyproject.toml | 2 +- watcher/src/data_hub_watcher/constants.py | 2 +- web/package.json | 4 +- ...vitest.config.ts => vitest.unit.config.ts} | 0 20 files changed, 197 insertions(+), 90 deletions(-) create mode 100644 developer-docs/README.md rename developer-docs/{ops => }/ci-and-deployment.md (98%) rename developer-docs/{reference => }/lambda.md (98%) rename developer-docs/{ops => }/run-archives.md (97%) rename developer-docs/{reference => }/shared-library.md (100%) create mode 100644 developer-docs/testing.md rename developer-docs/{reference => }/watcher.md (100%) rename web/{vitest.config.ts => vitest.unit.config.ts} (100%) diff --git a/.github/workflows/typescript-test.yml b/.github/workflows/typescript-test.yml index ee968962..d880ac10 100644 --- a/.github/workflows/typescript-test.yml +++ b/.github/workflows/typescript-test.yml @@ -53,8 +53,8 @@ jobs: run: npm ci working-directory: web - - name: Run MCP tests - run: make fe-test-mcp + - name: Run unit tests + run: make fe-test-unit - name: Run API integration tests run: make fe-test-integration diff --git a/AGENTS.md b/AGENTS.md index 16877a2e..443e781a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,72 +2,37 @@ ## Documentation -User-, operator-, and admin-facing documentation — installing a watcher, adding -an instrument, managing tokens, deploying the web app and AWS infrastructure, -security/permissions — lives on the docs site at -https://arcadia-data-hub-docs.vercel.app/, **not in this repository**. Search -there first for "how do I use/deploy Data Hub" questions; don't rely on training -data or guess at UI flows, since the site's `/llms.txt` and `/llms-full.txt` -routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to -fetch. - -This repo's `developer-docs/` only covers contributing to Data Hub itself: -architecture internals, local dev setup (`getting-started.md`, -`local-development.md`), conventions, CI/ops, and per-package references -(`reference/lambda.md`, `reference/watcher.md`, `reference/shared-library.md`). -See `README.md`'s Documentation section for the full index. +User-, operator-, and admin-facing documentation — installing a watcher, adding an instrument, managing tokens, deploying the web app and AWS infrastructure, security/permissions — lives on the docs site at https://arcadia-data-hub-docs.vercel.app/, **not in this repository**. Search there first for "how do I use/deploy Data Hub" questions; don't rely on training data or guess at UI flows, since the site's `/llms.txt` and `/llms-full.txt` routes (and a `.md` suffix on any page URL) serve clean Markdown that's cheap to fetch. + +This repo's `developer-docs/` only covers contributing to Data Hub itself: architecture internals, local dev setup (`getting-started.md`, `local-development.md`), conventions, CI/deployment, and per-package references (`lambda.md`, `watcher.md`, `shared-library.md`). See `developer-docs/README.md` for the full index. ## Cursor Cloud specific instructions -Data Hub is a multi-component repo (see `README.md`). The component you can run end-to-end -locally with zero external credentials is the **Next.js web app + REST API + PostgreSQL** -(`web/`). The `lambda/`, `watcher/`, and `packages/shared/` Python packages are exercised -via tests and a local S3 mirror — no real AWS is needed for local work. +Data Hub is a multi-component repo (see `README.md`). The component you can run end-to-end locally with zero external credentials is the **Next.js web app + REST API + PostgreSQL** (`web/`). The `lambda/`, `watcher/`, and `packages/shared/` Python packages are exercised via tests and a local S3 mirror — no real AWS is needed for local work. -Standard commands live in the `Makefile`, `web/package.json`, `developer-docs/getting-started.md`, and -`developer-docs/local-development.md`. The notes below are the non-obvious caveats that those docs don't -make obvious for a fresh cloud VM (where the update script has already installed deps). +Standard commands live in the `Makefile`, `web/package.json`, `developer-docs/getting-started.md`, and `developer-docs/local-development.md`. The notes below are the non-obvious caveats that those docs don't make obvious for a fresh cloud VM (where the update script has already installed deps). ### Starting services (not handled by the update script) -- **PostgreSQL must be started on every fresh VM** — the cluster is installed and the data - (roles + databases) persist in the snapshot, but the server process is not running at boot: - `sudo pg_ctlcluster 16 main start` (or `sudo service postgresql start`). -- Postgres is reachable at `postgres://postgres:postgres@127.0.0.1:5432`. Databases - `data-hub-local` (dev) and `data_hub_test` (integration tests) already exist. The integration - harness (`web/tests/integration/global-setup.ts`) hardcodes these same credentials and creates - `data_hub_test` itself if missing. -- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at - `/login` with the "Sign in (dev)" button using email `dev@local` (workspace admin; no password). +- **PostgreSQL must be started on every fresh VM** — the cluster is installed and the data (roles + databases) persist in the snapshot, but the server process is not running at boot: `sudo pg_ctlcluster 16 main start` (or `sudo service postgresql start`). +- Postgres is reachable at `postgres://postgres:postgres@127.0.0.1:5432`. Databases `data-hub-local` (dev) and `data_hub_test` (integration tests) already exist. The integration harness (`web/tests/integration/global-setup.ts`) hardcodes these same credentials and creates `data_hub_test` itself if missing. +- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at `/login` with the "Sign in (dev)" button using email `dev@local` (workspace admin; no password). ### Environment file -`web/.env` is gitignored and required for `make dev` / seeding. If it is missing on a fresh VM, -recreate it from the "Minimal `.env`" block in `developer-docs/local-development.md` (the key lines are -`DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/data-hub-local`, a 32+ char -`AUTH_SECRET`, dummy `AWS_*` values, and `LOCAL_S3_MIRROR=../lambda/.local-s3`). +`web/.env` is gitignored and required for `make dev` / seeding. If it is missing on a fresh VM, recreate it from the "Minimal `.env`" block in `developer-docs/local-development.md` (the key lines are `DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/data-hub-local`, a 32+ char `AUTH_SECRET`, dummy `AWS_*` values, and `LOCAL_S3_MIRROR=../lambda/.local-s3`). ### Node / Python toolchain -- Use **Node 24 (npm 11)** — it is the nvm default and is what CI uses. `npm ci` against the - committed `web/package-lock.json` **fails under npm 10** ("Missing: esbuild@… from lock file"), - so don't downgrade. A clean login shell already selects Node 24 via nvm. -- Python is managed by `uv` (Python 3.13, pinned in `.python-version`). Run Python tools through - `uv run …` (e.g. `uv run pytest`); the Makefile targets already do this. +- Use **Node 24 (npm 11)** — it is the nvm default and is what CI uses. `npm ci` against the committed `web/package-lock.json` **fails under npm 10** ("Missing: esbuild@… from lock file"), so don't downgrade. A clean login shell already selects Node 24 via nvm. +- Python is managed by `uv` (Python 3.13, pinned in `.python-version`). Run Python tools through `uv run …` (e.g. `uv run pytest`); the Makefile targets already do this. ### Seeding and local file bytes -- `make db-reseed` resets + pushes the Drizzle schema + seeds deterministic data. It prints a - personal access token (`dhub_…`) for the dev user — use it for `Authorization: Bearer` API calls. -- The seed's fixture-processing step is **skipped if the dev server isn't running** (it prints a - hint). To populate processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata), start - `make dev` first, then run `npm run db:process-fixtures` from `web/`. +- `make db-reseed` resets + pushes the Drizzle schema + seeds deterministic data. It prints a personal access token (`dhub_…`) for the dev user — use it for `Authorization: Bearer` API calls. +- The seed's fixture-processing step is **skipped if the dev server isn't running** (it prints a hint). To populate processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata), start `make dev` first, then run `npm run db:process-fixtures` from `web/`. ### Testing caveat -- `make fe-test-integration` and `make py-test-integration` run `next build` + `next start`, which - writes to `web/.next` and **contends with a running `make dev`** (also using `.next`). Stop the - dev server before running integration tests, then restart it afterward. -- Lint/format/typecheck: `make check-all` (note: `py-format`/`fe-format` auto-rewrite files; use - `uv run ruff check .`, `npm run lint:check` (Biome formatter + linter, read-only), and - `npm run typecheck` for read-only checks). +- `make fe-test-integration` and `make py-test-integration` run `next build` + `next start`, which writes to `web/.next` and **contends with a running `make dev`** (also using `.next`). Stop the dev server before running integration tests, then restart it afterward. +- Lint/format/typecheck: `make check-all` (note: `py-format`/`fe-format` auto-rewrite files; use `uv run ruff check .`, `npm run lint:check` (Biome formatter + linter, read-only), and `npm run typecheck` for read-only checks). diff --git a/Makefile b/Makefile index 5d721663..d9f73c0c 100644 --- a/Makefile +++ b/Makefile @@ -59,9 +59,9 @@ fe-lint: fe-typecheck: cd web && npm run typecheck -.PHONY: fe-test-mcp -fe-test-mcp: - cd web && npm run test:mcp +.PHONY: fe-test-unit +fe-test-unit: + cd web && npm run test:unit .PHONY: fe-test-integration fe-test-integration: @@ -69,7 +69,7 @@ fe-test-integration: .PHONY: fe-test fe-test: - cd web && npm run test:mcp && npm run test:integration + cd web && npm run test:unit && npm run test:integration .PHONY: dev dev: diff --git a/README.md b/README.md index 4600ce1a..54925489 100644 --- a/README.md +++ b/README.md @@ -17,10 +17,10 @@ flowchart LR | Directory | Description | Docs | | --- | --- | --- | | `web/` | Next.js web application and REST API (Vercel) | [API reference](https://arcadia-data-hub-docs.vercel.app/docs/api-reference) | -| `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](developer-docs/reference/lambda.md) | -| `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](developer-docs/reference/watcher.md) | -| `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](developer-docs/reference/shared-library.md) | -| `developer-docs/` | Project documentation | — | +| `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](developer-docs/lambda.md) | +| `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](developer-docs/watcher.md) | +| `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](developer-docs/shared-library.md) | +| `developer-docs/` | Project documentation | [Index](developer-docs/README.md) | ## Quick start @@ -45,17 +45,9 @@ See the full [Getting Started guide](developer-docs/getting-started.md) for prer ## Documentation -User, operator, and admin documentation (installing a watcher, adding an instrument, managing tokens, deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/), not in this repository. The docs below are for people developing Data Hub itself. +User, operator, and admin documentation (installing a watcher, adding an instrument, managing tokens, deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/), not in this repository. -- [Getting started](developer-docs/getting-started.md) — development setup, environment variables, running locally -- [Local development](developer-docs/local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed) -- [Architecture](developer-docs/architecture.md) — system overview, data flow, and design decisions -- [Watcher](developer-docs/reference/watcher.md) — CLI commands, configuration, run detection, upload modes -- [Lambda](developer-docs/reference/lambda.md) — processing pipeline, supported instruments, adding new instruments -- [Shared library](developer-docs/reference/shared-library.md) — module reference for `data-hub-shared` -- [CI and deployment](developer-docs/ops/ci-and-deployment.md) — GitHub Actions, Vercel, Render, Lambda deployment -- [Run archives](developer-docs/ops/run-archives.md) — "Download all" flow, cache/dedup model, and on-call runbook -- [Conventions](developer-docs/conventions.md) — S3 key layout, instrument IDs, code style, environments +Documentation for developing Data Hub itself lives in [developer-docs/](developer-docs/README.md), starting with [Getting started](developer-docs/getting-started.md). ## Development @@ -72,6 +64,9 @@ make py-test-unit # Run Python integration tests (requires Postgres). make py-test-integration +# Run web app unit + in-memory MCP tests. +make fe-test-unit + # Run API integration tests (requires Postgres). make fe-test-integration ``` diff --git a/developer-docs/README.md b/developer-docs/README.md new file mode 100644 index 00000000..cf708768 --- /dev/null +++ b/developer-docs/README.md @@ -0,0 +1,17 @@ +# Developer docs + +Documentation for developing Data Hub itself. User, operator, and admin +documentation (installing a watcher, adding an instrument, managing tokens, +deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/) +instead — see the root [README](../README.md#documentation) for that split. + +- [Getting started](getting-started.md) — development setup, environment variables, running locally +- [Local development](local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed) +- [Architecture](architecture.md) — system overview, data flow, and design decisions +- [Testing](testing.md) — per-package test frameworks, the shared test-server fixture, S3 mocking +- [Watcher](watcher.md) — CLI commands, configuration, run detection, upload modes +- [Lambda](lambda.md) — processing pipeline, supported instruments, adding new instruments +- [Shared library](shared-library.md) — module reference for `data-hub-shared` +- [CI and deployment](ci-and-deployment.md) — GitHub Actions, Vercel, Render, Lambda deployment +- [Run archives](run-archives.md) — "Download all" flow, cache/dedup model, and on-call runbook +- [Conventions](conventions.md) — S3 key layout, instrument IDs, code style, environments diff --git a/developer-docs/architecture.md b/developer-docs/architecture.md index d99f7120..657e8952 100644 --- a/developer-docs/architecture.md +++ b/developer-docs/architecture.md @@ -55,7 +55,7 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead: - **S3 as the integration boundary.** The watcher and Lambda never communicate directly. S3 acts as a durable hand-off point: the watcher writes, Lambda reads. - **API-driven coordination.** The watcher registers with the API, syncs its YAML config, and sends periodic heartbeats. This lets the web dashboard show watcher health and manage upload queues. - **Presigned URLs from the API.** The web app generates presigned S3 upload and download URLs so watchers and browsers can transfer files directly to/from S3 without routing data through the API. On Vercel, the app assumes an IAM role via OIDC federation (no long-lived AWS credentials). -- **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](ops/run-archives.md) for the full flow and runbook. +- **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](run-archives.md) for the full flow and runbook. - **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows all generic crawlers (with an allow-list for unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on every response. - **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code. - **Wildcard PAT scope for the legacy backfill.** Migration `0022_pat_scopes` backfilled every pre-existing token with `["*"]` so deployed watchers and the Lambda kept working once scope enforcement shipped. `POST /api/v1/tokens` rejects `*` from API callers, so every token minted since carries explicit least-privilege scopes; see [Security & permissions](https://arcadia-data-hub-docs.vercel.app/docs/security#token-scopes) for the scope vocabulary. diff --git a/developer-docs/ops/ci-and-deployment.md b/developer-docs/ci-and-deployment.md similarity index 98% rename from developer-docs/ops/ci-and-deployment.md rename to developer-docs/ci-and-deployment.md index 309abdd1..d0ff2bd3 100644 --- a/developer-docs/ops/ci-and-deployment.md +++ b/developer-docs/ci-and-deployment.md @@ -25,7 +25,7 @@ Four workflows run on pushes to `staging`/`production` and on pull requests targ ### TypeScript tests (`typescript-test.yml`) - Starts a Postgres 17 service container and Node.js 24. -- `make fe-test-mcp` — runs in-memory MCP protocol tests (mocked data layer, no database). +- `make fe-test-unit` — runs `tests/unit/` and `tests/mcp/` (in-memory MCP protocol tests) together; no Postgres, no Next.js server, no global setup. See [Testing](testing.md). - `make fe-test-integration` — runs Vitest integration tests that test the API routes and MCP server over HTTP against a real database. ### Apply database migrations (`apply-migrations.yml`) @@ -200,7 +200,7 @@ Secrets (`DATA_HUB_API_KEY`, etc.) are stored in GitHub environment secrets scop #### Local deployment -Local deployment requires the following tools in addition to the [general prerequisites](../getting-started.md#prerequisites): +Local deployment requires the following tools in addition to the [general prerequisites](getting-started.md#prerequisites): - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) — used for bootstrap commands and ECR login. - [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) — used by `make sam-deploy` to package and deploy CloudFormation stacks. Install with `brew install aws-sam-cli` on macOS. diff --git a/developer-docs/conventions.md b/developer-docs/conventions.md index 9d572ea7..20cc02b4 100644 --- a/developer-docs/conventions.md +++ b/developer-docs/conventions.md @@ -60,7 +60,8 @@ Run `make check-all` before pushing. CI enforces the same checks. ## Testing -- Tests are co-located with each package: `lambda/tests/`, `watcher/tests/`, `packages/shared/tests/`. -- Integration tests are marked with `@pytest.mark.integration` and require Postgres + a running Next.js server. -- The shared `testing.py` module provides the `start_test_server()` context manager that handles all setup and teardown. -- Unit tests should not require any external services. +Tests are co-located with each package (`lambda/tests/`, `watcher/tests/`, +`web/tests/`); unit tests need no external services, integration tests need +Postgres + a built Next.js server. See [Testing](testing.md) for the full +per-package breakdown, the shared `start_test_server()` fixture pattern, and +how S3 is mocked in Lambda's integration suite. diff --git a/developer-docs/getting-started.md b/developer-docs/getting-started.md index c3bd785e..a5a80733 100644 --- a/developer-docs/getting-started.md +++ b/developer-docs/getting-started.md @@ -72,7 +72,7 @@ These are set in the Lambda runtime environment: ### Watcher -The watcher reads its configuration from a YAML file at `~/.data-hub/config.yaml`. See the [watcher docs](reference/watcher.md) for details. The only environment variable it uses is `DATA_HUB_API_KEY` (optional, can also be provided interactively during `init`). +The watcher reads its configuration from a YAML file at `~/.data-hub/config.yaml`. See the [watcher docs](watcher.md) for details. The only environment variable it uses is `DATA_HUB_API_KEY` (optional, can also be provided interactively during `init`). ## Database setup @@ -143,8 +143,11 @@ make py-test-integration # All Python tests. make py-test +# Web app unit + in-memory MCP tests (no external services). +make fe-test-unit + # Web app API integration tests (requires Postgres + builds/starts Next.js). make fe-test-integration ``` -Integration tests use a `data_hub_test` Postgres database and spin up a real Next.js production server. See [CI and deployment](ops/ci-and-deployment.md) for how these run in GitHub Actions. +Integration tests use a `data_hub_test` Postgres database and spin up a real Next.js production server. See [Testing](testing.md) for the full per-package breakdown and [CI and deployment](ci-and-deployment.md) for how these run in GitHub Actions. diff --git a/developer-docs/reference/lambda.md b/developer-docs/lambda.md similarity index 98% rename from developer-docs/reference/lambda.md rename to developer-docs/lambda.md index 11ed7181..9b3e4ca6 100644 --- a/developer-docs/reference/lambda.md +++ b/developer-docs/lambda.md @@ -34,7 +34,7 @@ The web app's `GET /api/v1/instruments/:instrumentId/runs/:runId/download-archiv 3. The builder streams each S3 object through `zipfile.ZipFile` into an `_MultipartUploadStream` that buffers writes into ~16 MB parts and flushes each via `UploadPart`. Memory stays bounded regardless of total archive size, so a 200+ GB run zips inside the Lambda's standard memory budget. Entries are written `ZIP_STORED` (no compression — instrument output rarely compresses) with `force_zip64=True` (so individual entries ≥ 4 GB don't blow up the writer). 4. On success, the Lambda returns `{ archive_bucket, archive_key, size_bytes }`. If `job_id` was supplied, it also PATCHes `/api/v1/archive-jobs/:job_id` with the same fields and `status: "ready"`; on failure it PATCHes `status: "failed"` with `error_message`. The PATCH callback authenticates with `Authorization: Bearer ` — the same PAT the Lambda uses for every other Lambda → API call. The PATCH primarily serves to record terminal state for diagnostics and to surface `failed` quickly — the UI's polling target is the `/download-archive` route itself (whose first action is an S3 HEAD against the canonical archive key), so a finished build is downloadable the moment the multipart upload completes regardless of whether this PATCH lands. -See [Run archives](../ops/run-archives.md) for the full flow, S3 bucket layout, cache semantics, and operator runbook. +See [Run archives](run-archives.md) for the full flow, S3 bucket layout, cache semantics, and operator runbook. ## Supported instruments @@ -92,7 +92,7 @@ Available commands: | `qpcr` | Parse dye channels from an Azure Cielo qPCR Cq Values CSV | | `spectramax` | Parse metadata and raw well data from a SpectraMax `.xls` export | | `tapestation` | Extract the tape type from a TapeStation CSV filename | -| `handler` | Stage a file into a local S3 mirror and invoke `lambda_handler` against the local dev API. See [Testing the Lambda end-to-end](../local-development.md#testing-the-lambda-end-to-end) for the workflow. | +| `handler` | Stage a file into a local S3 mirror and invoke `lambda_handler` against the local dev API. See [Testing the Lambda end-to-end](local-development.md#testing-the-lambda-end-to-end) for the workflow. | The first six subcommands need no S3 or API access — they call into the same parsing/processing utilities the lambda uses, but stop short of the network. `handler` is different: it expects a running dev API and a `LOCAL_S3_MIRROR` directory, and uses the same dispatch path production uses. diff --git a/developer-docs/local-development.md b/developer-docs/local-development.md index 289dbd60..e8285354 100644 --- a/developer-docs/local-development.md +++ b/developer-docs/local-development.md @@ -133,12 +133,12 @@ Some features depend on services that aren't running in this workflow. Each one | File reprocessing | The reprocess endpoint returns null and no Lambda is invoked | Same | | Slack channel notifications on new runs | `console.warn` only, no HTTP call | Configure an incoming webhook URL in Settings > Notifications > Slack channel (admins only) | | Slack DM notifications / Connect to Slack | `console.warn` only; the "Connect to Slack" button redirects to Slack but the callback will error without credentials | Set `SLACK_BOT_TOKEN`, `SLACK_CLIENT_ID`, `SLACK_CLIENT_SECRET` | -| Watcher uploads → Lambda → API loop | Not exercised end-to-end; the seed inserts the resulting rows directly. For Lambda-only smoke testing, see [Testing the Lambda end-to-end](#testing-the-lambda-end-to-end) below | Run the watcher (`reference/watcher.md`) and the Lambda (`reference/lambda.md`) end-to-end | +| Watcher uploads → Lambda → API loop | Not exercised end-to-end; the seed inserts the resulting rows directly. For Lambda-only smoke testing, see [Testing the Lambda end-to-end](#testing-the-lambda-end-to-end) below | Run the watcher (`watcher.md`) and the Lambda (`lambda.md`) end-to-end | | Sign in with Google | The button still renders but OAuth callback will 4xx without `AUTH_GOOGLE_ID` / `AUTH_GOOGLE_SECRET` | `vercel env pull` per `getting-started.md` | ## Testing the Lambda end-to-end -Working on a `process_file()` module (or wiring up a brand new one — see [Lambda → Adding a new instrument](reference/lambda.md#adding-a-new-instrument)) and want to run it against the local web app without standing up real S3? The lambda CLI ships a `handler` subcommand that drives `lambda_handler` end-to-end against a gitignored directory mirroring the S3 layout. +Working on a `process_file()` module (or wiring up a brand new one — see [Lambda → Adding a new instrument](lambda.md#adding-a-new-instrument)) and want to run it against the local web app without standing up real S3? The lambda CLI ships a `handler` subcommand that drives `lambda_handler` end-to-end against a gitignored directory mirroring the S3 layout. ```sh cd lambda diff --git a/developer-docs/ops/run-archives.md b/developer-docs/run-archives.md similarity index 97% rename from developer-docs/ops/run-archives.md rename to developer-docs/run-archives.md index b2d58d67..0714f731 100644 --- a/developer-docs/ops/run-archives.md +++ b/developer-docs/run-archives.md @@ -4,7 +4,7 @@ The "Download all" actions on the run detail page and the runs table deliver eve Each archive can mix files from the raw bucket and the processed bucket in a single zip. This matters for instruments that produce processed artifacts via Lambda preprocessing (SpectraMax raw `.xls` → processed CSV; Hina `.nd2` → processed JPG; Azure 600 Gel Doc `.tif` → processed PNG): the run's file rows reference both buckets, and "Download all" zips them together. -This page covers the end-to-end flow, the cache + dedup model, and the on-call runbook. For the Lambda invocation contract, see [Lambda → Function URL (archive build)](../reference/lambda.md#function-url-archive-build). For the HTTP endpoints, see [REST API → Archive jobs](https://arcadia-data-hub-docs.vercel.app/docs/api-reference#archive-jobs). +This page covers the end-to-end flow, the cache + dedup model, and the on-call runbook. For the Lambda invocation contract, see [Lambda → Function URL (archive build)](lambda.md#function-url-archive-build). For the HTTP endpoints, see [REST API → Archive jobs](https://arcadia-data-hub-docs.vercel.app/docs/api-reference#archive-jobs). ## Flow @@ -29,7 +29,7 @@ Every cache miss goes async, regardless of archive size. The route does the chea ## S3 layout -Archives live in a separate bucket per environment, provisioned by [`infra/template.yaml`](../../infra/template.yaml): +Archives live in a separate bucket per environment, provisioned by [`infra/template.yaml`](../infra/template.yaml): - **Bucket:** `arcadia-data-hub-archives-{staging,production}`. - **Public access:** fully blocked. Reads happen exclusively via short-lived presigned GET URLs; writes happen exclusively from the Lambda execution role. diff --git a/developer-docs/reference/shared-library.md b/developer-docs/shared-library.md similarity index 100% rename from developer-docs/reference/shared-library.md rename to developer-docs/shared-library.md diff --git a/developer-docs/testing.md b/developer-docs/testing.md new file mode 100644 index 00000000..e02c2e0f --- /dev/null +++ b/developer-docs/testing.md @@ -0,0 +1,126 @@ +# Testing + +Four independent test suites across two frameworks — one per package, plus a +split between fast/isolated tests and tests that exercise a real Postgres + +Next.js server. This page is the map; see [Conventions](conventions.md) for the +one-line summary and [CI and deployment](ci-and-deployment.md) for how these +run in GitHub Actions. + +## Running tests + +| Command | Package(s) | What it runs | +| --- | --- | --- | +| `make py-test-unit` | `lambda/`, `watcher/` | pytest, excluding `@pytest.mark.integration` — no external services | +| `make py-test-integration` | `lambda/`, `watcher/` | pytest, only `@pytest.mark.integration` — builds/starts Next.js against Postgres | +| `make py-test` | `lambda/`, `watcher/` | All pytest tests (unit + integration) | +| `make fe-test-unit` | `web/` | Vitest `tests/unit/` + `tests/mcp/` — no Postgres, no server | +| `make fe-test-integration` | `web/` | Vitest `tests/integration/` — builds/starts Next.js against Postgres | +| `make fe-test` | `web/` | `fe-test-unit` then `fe-test-integration` | + +`packages/shared/` has no tests of its own — it ships `testing.py`, the shared +integration-test infrastructure that `lambda/` and `watcher/` both depend on +(see [below](#the-shared-test-server-start_test_server)). + +Integration tests (Python and TypeScript alike) need Postgres reachable at +`127.0.0.1:5432` and create/reuse a `data_hub_test` database. They also build +and start a real Next.js **production** server (`next build && next start`) +rather than `next dev` — dev mode recompiles on every request, which makes a +large suite 5–10x slower, and a production build matches actual deployment +behavior. That build writes to `web/.next`, which **contends with a running +`make dev`** (same directory) — stop the dev server first. See `AGENTS.md` for +this and other local-environment caveats. + +## Python (pytest): `lambda/`, `watcher/` + +Tests are co-located per package (`lambda/tests/`, `watcher/tests/`), with an +`integration/` subdirectory holding everything marked +`@pytest.mark.integration`. Unit tests should never require Postgres, a running +server, or real AWS credentials. + +### The shared test server: `start_test_server()` + +Both packages' integration suites depend on +[`data_hub_shared.testing`](shared-library.md) for an identical +environment with minimal boilerplate. The session-scoped pattern, from +`watcher/tests/integration/conftest.py` and `lambda/tests/integration/conftest.py`: + +```python +@pytest.fixture(scope="session") +def integration_env() -> Generator[IntegrationEnv, None, None]: + with start_test_server() as env: + seed_instruments(env.db_dsn, {...}) + yield env +``` + +`start_test_server()` creates the `data_hub_test` database if missing, pushes +the Drizzle schema via `npx drizzle-kit push --force`, seeds a deterministic +`watcher_release_config` row, builds and starts `next start` on a free port, +waits for it to respond, seeds a test user + personal access token, and yields +an `IntegrationEnv(base_url, api_token, db_dsn)`. The server is torn down +(`terminate()`, then `kill()` after a 10s grace period) when the `with` block +exits. + +Because the server is session-scoped, tests need to reset state between runs +without tearing it down. Both suites do this with an `autouse` fixture that +`truncate_tables()`s only the data tables that change per test — seeded rows +like `instruments` and the PAT stay put so tests don't need to re-seed them: + +```python +_DATA_TABLES = ["files", "instrument_runs"] + +@pytest.fixture(autouse=True) +def reset_db(integration_env: IntegrationEnv) -> None: + truncate_tables(integration_env.db_dsn, _DATA_TABLES) +``` + +### Mocking S3 in Lambda integration tests + +`lambda/tests/integration/conftest.py` patches `data_hub_shared.s3_utils` +directly rather than mocking at the boto3 client level, so `process_file()` +implementations run unmodified end-to-end except for the actual network calls: + +- `mock_s3_download` (autouse) redirects `download_file()` to copy from a + `s3_fixture_files: dict[str, Path]` registry that each test populates before + invoking the handler. +- `mock_s3_upload` (autouse) no-ops `upload_file()` for processors that write + processed artifacts (e.g. Azure 600 Gel Doc). +- `make_s3_event` / `make_function_url_event` build synthetic S3 event / + Function URL payloads matching the real key layout + (`{instrument_id}/{run_id}/{filename}`), including the `quote_plus` + form-encoding real S3 notifications use. + +## TypeScript (Vitest): `web/` + +Three test directories, two Vitest configs: + +- **`tests/unit/` + `tests/mcp/`** run together under `vitest.unit.config.ts` + (`npm run test:unit` / `make fe-test-unit`) — pure functions and in-memory MCP + transport tests. No Postgres, no server, no global setup, 10s timeout. + `tests/mcp/` is grouped in here (not with `tests/integration/`) because it + exercises the MCP server's in-memory transport directly against a mocked + data layer — it's a fast unit-style test, not an HTTP integration test. +- **`tests/integration/`** runs under `vitest.integration.config.ts` + (`npm run test:integration` / `make fe-test-integration`) — real HTTP + requests against a built-and-started Next.js server. `globalSetup` points at + `tests/integration/global-setup.ts`, and `fileParallelism: false` because + every test file shares the one Postgres database and server instance. + +`global-setup.ts` mirrors what `start_test_server()` does for Python, plus one +thing the Python side doesn't need: an in-process HTTP server on a free port +that captures outgoing Slack webhook calls (`slack_channel_config` is seeded to +point at it) and Slack Web API `chat.postMessage` calls (DMs), so +notification-related tests can assert on captured payloads without hitting +Slack. It also strips `LAMBDA_FUNCTION_URL` / `AWS_ROLE_ARN` from the spawned +server's environment so "Lambda not configured" code paths are exercised +regardless of the developer's local `.env` — tests that need a stubbed Lambda +call should mock `fetch` instead of relying on a real Function URL. Values that +test files need (`base_url`, `databaseUrl`, capture-server URLs) are passed +back via `process.env`, since Vitest's global setup runs in a separate worker +from the tests themselves. + +## In CI + +`python-test.yml` runs `make py-test` (unit + integration together, one +Postgres 17 service container). `typescript-test.yml` runs `make fe-test-unit` +then `make fe-test-integration` as separate steps. See +[CI and deployment](ci-and-deployment.md) for the full workflow list. diff --git a/developer-docs/reference/watcher.md b/developer-docs/watcher.md similarity index 100% rename from developer-docs/reference/watcher.md rename to developer-docs/watcher.md diff --git a/infra/template.yaml b/infra/template.yaml index 4a456352..fc644ead 100644 --- a/infra/template.yaml +++ b/infra/template.yaml @@ -505,7 +505,7 @@ Resources: - iam:DetachRolePolicy - iam:TagRole # iam:DeleteRole is intentionally omitted — role replacement - # requires an admin deploy (see ops/ci-and-deployment.md). + # requires an admin deploy (see developer-docs/ci-and-deployment.md). Resource: - !Sub "arn:aws:iam::${AWS::AccountId}:role/data-hub-*-${Environment}" diff --git a/watcher/pyproject.toml b/watcher/pyproject.toml index 2710fdaa..a93e3750 100644 --- a/watcher/pyproject.toml +++ b/watcher/pyproject.toml @@ -39,7 +39,7 @@ windows-service = ["pywin32"] [project.urls] Homepage = "https://github.com/Arcadia-Science/data-hub" # Operator-facing install + auto-update guide — what someone landing on -# the PyPI page actually needs. The developer-facing `developer-docs/reference/watcher.md` +# the PyPI page actually needs. The developer-facing `developer-docs/watcher.md` # describes the editable-checkout workflow and is reachable from the # repository link below. Documentation = "https://arcadia-data-hub-docs.vercel.app/docs/installing-a-watcher" diff --git a/watcher/src/data_hub_watcher/constants.py b/watcher/src/data_hub_watcher/constants.py index 4c5cef79..5e5224b7 100644 --- a/watcher/src/data_hub_watcher/constants.py +++ b/watcher/src/data_hub_watcher/constants.py @@ -213,7 +213,7 @@ def state_db_path(config_dir: Path, environment: str) -> Path: Keyed by `environment` (not `watcher_id`) so a deregister/re-register on the same host reuses the existing dedup history instead of re-uploading - the whole backlog. See `developer-docs/reference/watcher.md` for the rationale. + the whole backlog. See `developer-docs/watcher.md` for the rationale. """ return config_dir / f"watcher-{environment}.db" diff --git a/web/package.json b/web/package.json index 388a6974..6477f162 100644 --- a/web/package.json +++ b/web/package.json @@ -21,8 +21,8 @@ "db:reseed": "npm run db:reset && npm run db:push && npm run db:seed", "db:process-fixtures": "tsx scripts/process-seeded-fixtures.ts", "db:clear-runs": "tsx scripts/clear-run-file-records.ts", - "test:mcp": "vitest run --config vitest.config.ts", - "test:mcp:watch": "vitest --config vitest.config.ts", + "test:unit": "vitest run --config vitest.unit.config.ts", + "test:unit:watch": "vitest --config vitest.unit.config.ts", "test:integration": "vitest run --config vitest.integration.config.ts", "test:integration:watch": "vitest --config vitest.integration.config.ts", "check": "npm run lint:check && npm run typecheck", diff --git a/web/vitest.config.ts b/web/vitest.unit.config.ts similarity index 100% rename from web/vitest.config.ts rename to web/vitest.unit.config.ts From 0302353ee42184a0f2447ef973dbe9c74094cacd Mon Sep 17 00:00:00 2001 From: Wasim Sandhu Date: Tue, 7 Jul 2026 13:32:03 -0700 Subject: [PATCH 4/4] docs: Update README --- README.md | 30 +++++++++++++----------------- developer-docs/README.md | 2 +- 2 files changed, 14 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 54925489..4e276393 100644 --- a/README.md +++ b/README.md @@ -14,15 +14,15 @@ flowchart LR ## Repository structure -| Directory | Description | Docs | -| --- | --- | --- | -| `web/` | Next.js web application and REST API (Vercel) | [API reference](https://arcadia-data-hub-docs.vercel.app/docs/api-reference) | -| `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](developer-docs/lambda.md) | -| `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](developer-docs/watcher.md) | -| `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](developer-docs/shared-library.md) | -| `developer-docs/` | Project documentation | [Index](developer-docs/README.md) | +| Directory | Description | +| --- | --- | +| `web/` | Next.js web application and REST API (Vercel) | +| `lambda/` | AWS Lambda function for instrument data processing | +| `watcher/` | CLI agent for lab instrument PCs | +| `packages/shared/` | Shared Python library (S3, enums, test infra) | +| `developer-docs/` | Project documentation | -## Quick start +## Getting started ```sh # Install Python packages (all workspace members). @@ -41,18 +41,14 @@ cd web && createdb data-hub-local && npm run db:push && cd .. make dev ``` -See the full [Getting Started guide](developer-docs/getting-started.md) for prerequisites and details. +See the full [Getting started guide](developer-docs/getting-started.md) for prerequisites and details. Don't have AWS/Google credentials? [Local development](developer-docs/local-development.md) covers a zero-credential setup for the web app + API + database alone (no watcher or Lambda needed). -## Documentation +Developer docs live in [developer-docs/](developer-docs/README.md). You can find user documentation (self-hosted deployment, watcher installation, adding an instrument, managing tokens) on the [docs site](https://arcadia-data-hub-docs.vercel.app/). -User, operator, and admin documentation (installing a watcher, adding an instrument, managing tokens, deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/), not in this repository. - -Documentation for developing Data Hub itself lives in [developer-docs/](developer-docs/README.md), starting with [Getting started](developer-docs/getting-started.md). - -## Development +## Checks and tests ```sh -# Run formatting, linting, and type checking. +# Format, lint, and type-check everything. make check-all # Run all Python tests. @@ -64,7 +60,7 @@ make py-test-unit # Run Python integration tests (requires Postgres). make py-test-integration -# Run web app unit + in-memory MCP tests. +# Run web app unit tests. make fe-test-unit # Run API integration tests (requires Postgres). diff --git a/developer-docs/README.md b/developer-docs/README.md index cf708768..b9aebcb3 100644 --- a/developer-docs/README.md +++ b/developer-docs/README.md @@ -3,7 +3,7 @@ Documentation for developing Data Hub itself. User, operator, and admin documentation (installing a watcher, adding an instrument, managing tokens, deployment) lives on the [docs site](https://arcadia-data-hub-docs.vercel.app/) -instead — see the root [README](../README.md#documentation) for that split. +instead — see the root [README](../README.md#getting-started) for that split. - [Getting started](getting-started.md) — development setup, environment variables, running locally - [Local development](local-development.md) — zero-credential dev workflow for the web app + API + database (no watcher / Lambda needed)