[Production] Release 2026-07-23

AGENTS.md

@@ -24,7 +24,7 @@ Standard commands live in the `Makefile`, `web/package.json`, `developer-docs/ge
 
 - **PostgreSQL must be started on every fresh VM** — the cluster is installed and the data (roles + databases) persist in the snapshot, but the server process is not running at boot: `sudo pg_ctlcluster 16 main start` (or `sudo service postgresql start`).
 - Postgres is reachable at `postgres://postgres:postgres@127.0.0.1:5432`. Databases `data-hub-local` (dev) and `data_hub_test` (integration tests) already exist. The integration harness (`web/tests/integration/global-setup.ts`) hardcodes these same credentials and creates `data_hub_test` itself if missing.
-- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at `/login` with the "Sign in (dev)" button using email `dev@local` (workspace admin; no password).
+- **Web dev server:** `make dev` (Next.js + Turbopack on http://localhost:3000). Sign in at `/login` with the "Sign in (dev)" button using email `alice@example.com` (workspace admin; no password).
 
 ### Environment file
 

Makefile

@@ -83,7 +83,7 @@ dev:
 	cd web && npm run dev
 
 # Reset the local Postgres database, re-push the Drizzle schema, and load
-# a deterministic seed (dev user + PAT, one instrument per type, watchers,
+# a deterministic seed (Alice + teammates, prod instrument catalog, watchers,
 # runs, files, comments, attributions, archive jobs). See
 # developer-docs/local-development.md for the full local-only dev workflow.
 .PHONY: db-reseed

developer-docs/local-development.md

@@ -20,7 +20,7 @@ make db-reseed
 make dev
 ```
 
-Sign in at `/login` using the "Sign in (dev)" button with the seeded `dev@local` email — no password, no Google Workspace.
+Sign in at `/login` using the "Sign in (dev)" button with the seeded `alice@example.com` email — no password, no Google Workspace.
 
 ## Prerequisites
 
@@ -66,7 +66,7 @@ Explicitly **do not** set the following — leaving them unset is what makes the
 
 ## Sign in (dev only)
 
-`web/lib/auth.ts` registers a `Credentials` provider with id `"dev"` when `process.env.NODE_ENV !== "production"`. The `/login` page renders a matching "Sign in (dev)" form under the Google button. The form accepts any email present in the `user` table and mints a session via the existing JWT strategy — `users.is_admin` is read on sign-in just like for Google sign-ins, so the seeded `dev@local` lands as a workspace admin.
+`web/lib/auth.ts` registers a `Credentials` provider with id `"dev"` when `process.env.NODE_ENV !== "production"`. The `/login` page renders a matching "Sign in (dev)" form under the Google button. The form accepts any email present in the `user` table and mints a session via the existing JWT strategy — `users.is_admin` is read on sign-in just like for Google sign-ins, so the seeded `alice@example.com` lands as a workspace admin.
 
 The dev provider is **not** instantiated in production builds. The form is also conditionally rendered server-side, so a production `npm run build` never ships the affordance.
 
@@ -104,20 +104,20 @@ You can also run `npm run db:seed` on its own — it calls the schema-driven `cl
 
 | Entity | Count | Notes |
 | --- | --- | --- |
-| `user` | 1 | `dev@local`, `is_admin = true` |
-| `personal_access_tokens` | 1 | Wildcard scope, no expiry |
-| `instruments` | 8 (one per `instrument_type`) | First row is `pending`; rest `active` |
-| `watchers` | 7 (for active instruments) | Rotates through `watching` / `registered` / `stopped` |
+| `user` | 26 | `alice@example.com` (admin) + Bob–Zoe teammates (`@example.com`) |
+| `personal_access_tokens` | 1 | Wildcard scope, no expiry (Alice) |
+| `instruments` | 11 | Production catalog (10 active) + 1 pending for the activate-instrument UI |
+| `watchers` | 11 (one per instrument) | Prod-like hostnames; mix of `watching` / `registered` / `stopped` |
 | `watcher_heartbeats` | ~10 per watching watcher | Spread over the last hour |
 | `watcher_events` | 3 per watching watcher | `watcher_started`, `config_synced`, `file_uploaded` |
 | `instrument_runs` | 8 per active instrument | Calendar-relative `acquired_at` (today, yesterday, this week, ~7d / ~10d / ~22d / earlier this month) so date-filter presets and today/this-week stats have distinct non-empty sets; alternating `lambda` / `watcher` source |
-| `files` | 3 per run, or 1 for fixture-bearing runs | Mix of `uploaded` / `completed` / `failed` (and `raw` / `processed` for the 3-file shape). qPCR / gel doc / plate reader runs render exactly one row — the real fixture, bytes copied into `LOCAL_S3_MIRROR` (see [Working with file bytes locally](#working-with-file-bytes-locally)) |
-| `run_comments` | 1 per run | Authored by the dev user; most stamped this week, every 4th last week |
-| `run_attributions` | 1 per run | Dev user attributed |
+| `files` | 1–4 per run | Mix of `uploaded` / `completed` / `failed`. Fixture instruments (qPCR, gel doc, both SpectraMax readers) render the real fixture bytes in `LOCAL_S3_MIRROR`; other instruments use production-shaped synthetic filenames (see [Working with file bytes locally](#working-with-file-bytes-locally)) |
+| `run_comments` | multi-author threads | Q&A / notes across Alice + teammates; richer threads on ~⅓ of runs |
+| `run_attributions` | 1 per run | Rotated across Alice + teammates |
 | `archive_jobs` | 3 | One each of `ready` / `building` / `failed` |
-| `watcher_release_config` | 1 (singleton) | `9.9.9 / 0.1.0 / stable / false` |
+| `watcher_release_config` | 1 (singleton) | `1.0.0 / 0.1.0 / false` (matches seeded watcher version) |
 
-Externally-visible identifiers used in URLs and API paths are deterministic across reseeds, so screenshots, bug reports, and `curl` examples stay stable. Instrument types backed by a real lambda `process_file` (qPCR, gel doc, plate reader) use the canonical kebab-case ids the lambda expects (`azure-cielo-qpcr`, `azure-600-gel-doc`, `spectramax-id3-plate-reader`) with realistic-looking run ids (`Experiment_20260129`, `26.02.02_10.45.05`, `012926_AR_OD600`, …). Other instrument types use cosmetic `seed-<type>` ids and `seed-run-1`…`seed-run-8` since they don't round-trip through any pipeline.
+Externally-visible identifiers used in URLs and API paths are deterministic across reseeds, so screenshots, bug reports, and `curl` examples stay stable. Instrument ids and display names mirror production (`agilent-4150-tapestation`, `instantraman`, `spectramax-id5-plate-reader`, …). Fixture-backed instruments keep realistic run ids (`Experiment_20260129`, `26.02.02_10.45.05`, `012926_AR_OD600`, …); other instruments use production-shaped synthetic run ids and filenames.
 
 Surrogate UUIDs (watcher IDs, archive job IDs, the per-row primary keys on `instrument_runs` and `files`) and the PAT plaintext are regenerated on every reseed — the seed does not use Faker but it does call `crypto.randomUUID()` and `crypto.randomBytes()` where the schema needs server-side IDs.
 
@@ -183,14 +183,14 @@ The wiring lives in [lambda/src/data_hub_lambda/cli.py](../lambda/src/data_hub_l
 
 What this gets you out of the box after `make db-reseed`:
 
-| Instrument type | Seeded fixture | Where it comes from |
+| Instrument type | Seeded fixtures (cycled across runs) | Where they come from |
 | --- | --- | --- |
 | qPCR | `azure_cielo_qpcr_example.csv` | `lambda/tests/fixtures/` |
-| Gel doc | `azure_600_gel_doc_example.tif` | `lambda/tests/fixtures/` |
-| Plate reader | `spectramax_plate_reader_endpoint.xls` | `lambda/tests/fixtures/` |
+| Gel doc | `azure_600_gel_doc_{example,fluorescence,true_color}.tif` | `lambda/tests/fixtures/` |
+| Plate reader (iD3 + iD5) | `spectramax_plate_reader_{endpoint,endpoint_flat,endpoint_sparse,fluorescence,kinetic,well_scan}.xls` | `lambda/tests/fixtures/` |
 | Other instruments | none — files 404 in the mirror | Stage real bytes via `data-hub-process handler` |
 
-The seed copies the fixture into `<LOCAL_S3_MIRROR>/test-raw-data-bucket/<instrument-id>/<run-id>/<filename>` for every seeded run on those instruments, so navigating to `/instruments/azure-cielo-qpcr/runs/Experiment_20260129` shows a real CSV in the file browser, the colony / plate-reader viewers fetch real bytes via `/api/v1/files/<id>/download`, and PNG / TIFF / PDF previews on `RunReportSection` render without 404s. Fixture-bearing runs only have the real fixture file — the synthetic CSV siblings other instruments still get are dropped so the UI only shows files that actually exist on disk.
+The seed cycles every available fixture for an instrument across its seeded runs (so gel-doc screenshots include Chemiluminescence, Fluorescence, and True Color Imaging, not eight copies of the same chemi TIFF). Each run gets one fixture copied to `<LOCAL_S3_MIRROR>/test-raw-data-bucket/<instrument-id>/<run-id>/<filename>`, so navigating to `/instruments/azure-cielo-qpcr/runs/Experiment_20260129` shows a real CSV in the file browser, the colony / plate-reader viewers fetch real bytes via `/api/v1/files/<id>/download`, and PNG / TIFF / PDF previews on `RunReportSection` render without 404s. Fixture-bearing runs only have the real fixture file — the synthetic CSV siblings other instruments still get are dropped so the UI only shows files that actually exist on disk.
 
 When the dev API is reachable during seeding, the seed also drives `data-hub-process handler` over each fixture-bearing run so the dashboard renders processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata) immediately after a reseed. If the API isn't up yet (`npm run db:reseed` ran before `npm run dev`), the seed prints a hint and skips the step — you can re-run it on its own once the dev server is reachable:
 
@@ -201,15 +201,15 @@ npm run dev
 npm run db:process-fixtures
 ```
 
-`npm run db:process-fixtures` mints a fresh PAT for `dev@local`, re-derives the fixture-bearing `(instrument_id, run_id, filename)` triples from the database, and spawns `data-hub-process handler` for each. The wiring lives in [web/scripts/process-fixtures.ts](../web/scripts/process-fixtures.ts) — it's the same module the seed calls — so anything that works during a reseed also works post-hoc.
+`npm run db:process-fixtures` mints a fresh PAT for `alice@example.com`, re-derives the fixture-bearing `(instrument_id, run_id, filename)` triples from the database, and spawns `data-hub-process handler` for each. The wiring lives in [web/scripts/process-fixtures.ts](../web/scripts/process-fixtures.ts) — it's the same module the seed calls — so anything that works during a reseed also works post-hoc.
 
 For instrument types without a fixture (or new file types you're adding components for), the existing CLI flow stays the same: run `data-hub-process handler <instrument-id> <run-id> <filename> --source <FILE>` and the dashboard picks up the file the moment the API row lands.
 
 Components don't need to change — every existing run viewer already fetches `/api/v1/files/<id>/download`, which 302s to whatever `getPresignedDownloadUrl` returns. New custom components for a specific instrument should follow the same pattern (`fetch("/api/v1/files/<id>/download")` for raw bytes, `<img src="/api/v1/files/<id>/download">` for images) and inherit local-mirror support automatically.
 
 A few details worth knowing:
 
-- Adding fixtures for more instruments is two paired entries in [web/lib/db/seed.ts](../web/lib/db/seed.ts): an `INSTRUMENT_FIXTURES` row (`{ filename, contentType, runIds }`) pointing at any file under `lambda/tests/fixtures/`, and a `CANONICAL_INSTRUMENT_ID` mapping to the kebab-case id from `data_hub_shared.enums.Instrument`. Without the canonical id, the seed-time handler step rejects the instrument because `parse_s3_event` only accepts values from that enum.
+- Adding fixtures for more instruments means an `INSTRUMENT_FIXTURES` entry in [web/lib/db/seed.ts](../web/lib/db/seed.ts) keyed by the kebab-case instrument id from `data_hub_shared.enums.Instrument` (`{ files: [{ filename, contentType }, …], runIds }`) pointing at files under `lambda/tests/fixtures/`. List every fixture you want cycled across seeded runs. The handler rejects unknown instrument ids because `parse_s3_event` only accepts values from that enum.
 - The route is gated on `NODE_ENV !== "production"` AND `LOCAL_S3_MIRROR` set; either condition unmet returns 404 unconditionally, so a production build can never expose the filesystem.
 - The MCP tool at `/api/v1/mcp` returns a relative `/api/local-s3/...` URL when the mirror is active — browsers resolve it against the current origin, but non-browser MCP clients on localhost may need to prefix with `http://localhost:3000`.
 

developer-docs/watcher.md

@@ -65,7 +65,7 @@ Starts the file monitoring loop. Before entering the loop it:
 
 While running:
 
-- **File monitor** watches the directory for new/modified files using `watchdog` and waits for each file to stabilize (size + mtime unchanged for the configured stability period). Files that keep changing for longer than 5 minutes are abandoned and surface as a `stability_timeout` error event.
+- **File monitor** watches the directory for new/modified files using `watchdog` and waits for each file to stabilize (size + mtime unchanged for the configured stability period). Files that keep changing past `max_stability_wait_seconds` (default 5 minutes) are abandoned and surface as a `stability_timeout` error event.
 - **Run detector** groups stable files into runs by applying the configured regex to each file's relative path. The first file for a run triggers `POST /instruments/:id/runs`; subsequent files for the same run incrementally `PATCH` only the new entries onto the manifest. Files inside the watch tree that don't match the pattern emit a `pattern_mismatch` event (throttled to one per parent directory) so misconfigured patterns surface in the dashboard.
 - **Uploader** requests a presigned S3 URL from the API and uploads each file via HTTP PUT (auto mode), or processes the server's upload queue (manual mode). The watcher does not need AWS credentials. Each upload retries up to 3 times with exponential backoff (1, 2, 4 s) and is recorded locally with its SHA-256 so retries and restarts don't re-upload the same bytes. In manual mode, queue-poll failures are throttled (1st failure, then every 10th) to keep a sustained outage visible without flooding the events stream.
 - **Upload worker** (manual mode only) polls the server's upload queue on its own long-lived thread every 60 seconds, decoupled from the heartbeat so a slow or large upload can't delay heartbeats and make a busy watcher look offline. On shutdown it is stopped and joined before the state DB is closed. Auto mode has no worker: uploads run on the monitor's stability-checker thread via the run detector's upload callback.
@@ -162,6 +162,7 @@ instrument:
   enabled: true
   upload_mode: auto              # "auto" or "manual"
   stability_period_seconds: 5    # 1–300
+  max_stability_wait_seconds: 300  # 1–86400; must be >= stability_period_seconds
   run_detection:
     pattern: '^([^/]+)/'         # regex with one capture group (run ID)
     recursive: true
@@ -281,7 +282,7 @@ The watcher's primary observability surface is the per-watcher event log served
 | --- | --- |
 | `run_report_failed` | POST/PATCH against `/instruments/:id/runs[/:run_id]` failed. `details` includes `operation`, `status_code`, `file_count`. |
 | `config_sync_failed` | The startup `PUT /watchers/:id/config` (or its checksum probe) failed. |
-| `stability_timeout` | A file kept changing past 5 minutes and was abandoned. |
+| `stability_timeout` | A file kept changing past the configured max wait (default 5 minutes) and was abandoned. `details.max_wait_seconds` reports the cap used. |
 | `stable_callback_failed` | The on-stable-file callback raised. |
 | `pattern_mismatch` | A file inside the watch tree did not match `run_detection.pattern`. Throttled to one emission per parent directory per process. |
 | `events_dropped` | Synthetic event prepended after one or more prior batches were dropped. `details.dropped_count` reports the gap size. |

lambda/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "data-hub-lambda"
-version = "0.2.0"
+version = "0.0.0"
 requires-python = ">=3.12"
 license = "MIT"
 license-files = ["LICENSE"]

packages/shared/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "data-hub-shared"
-version = "0.1.0"
+version = "0.0.0"
 requires-python = ">=3.12"
 license = "MIT"
 license-files = ["LICENSE"]