Local dev: end-to-end Lambda + filesystem-backed S3 mirror

.gitignore

@@ -218,4 +218,7 @@ __marimo__/
 .streamlit/secrets.toml
 
 # macOS
-.DS_Store
+.DS_Store
+
+# Local S3 mirror used by `data-hub-process handler` (see docs/local-development.md).
+lambda/.local-s3/

docs/guides/adding-an-instrument.md

@@ -111,6 +111,8 @@ Don't forget to add the import at the top of `handler.py`.
 
 Add unit tests in `lambda/tests/` for the new processor. Integration tests will automatically cover the new instrument if it's registered in the shared library.
 
+For a quick end-to-end smoke against your local web app — without S3, AWS credentials, or LocalStack — use `data-hub-process handler` to drive `lambda_handler` against a gitignored local mirror. See [Testing the Lambda end-to-end](../local-development.md#testing-the-lambda-end-to-end).
+
 ### 4.5 Configure the S3 trigger
 
 Add a `LambdaConfiguration` entry to the `RawDataBucket` resource's `NotificationConfiguration` in `infra/template.yaml`. Each entry specifies a prefix (the instrument ID) and a suffix (the file extension):

docs/local-development.md

@@ -39,15 +39,22 @@ DATABASE_URL=postgres://localhost:5432/data-hub-local
 # Any 32+ character string. NextAuth uses it to sign session JWTs.
 AUTH_SECRET=local-dev-secret-at-least-32-characters!!
 
-# Dummy AWS credentials. The AWS SDK signs presigned URLs locally
-# (HMAC-only — no network call), so any non-empty values work. Actual
-# uploads/downloads against `test-raw-data-bucket` will fail at the
-# browser, which is fine for local dev (see "What's deliberately
-# missing" below).
+# Dummy AWS credentials. The local-mirror branch in `web/lib/s3.ts`
+# bypasses the AWS SDK entirely when LOCAL_S3_MIRROR is set, but a
+# few server-side modules instantiate the SDK at import time so any
+# non-empty values keep them happy.
 AWS_ACCESS_KEY_ID=test-key
 AWS_SECRET_ACCESS_KEY=test-secret
 AWS_REGION=us-east-1
 S3_RAW_DATA_BUCKET=test-raw-data-bucket
+
+# Filesystem-backed S3 mirror. When set (and NODE_ENV != production),
+# `web/lib/s3.ts` swaps presigned-URL generation, HEAD checks, and
+# server-side stream reads for `<root>/<bucket>/<key>` lookups, and
+# the seed copies fixture bytes into the mirror so seeded runs
+# render real bytes. Path is resolved relative to web/. See
+# "Working with file bytes locally" below.
+LOCAL_S3_MIRROR=../lambda/.local-s3
 ```
 
 Explicitly **do not** set the following — leaving them unset is what makes the relevant features short-circuit cleanly:
@@ -104,13 +111,13 @@ You can also run `npm run db:seed` on its own — it calls the schema-driven `cl
 | `watcher_heartbeats` | ~10 per watching watcher | Spread over the last hour |
 | `watcher_events` | 3 per watching watcher | `watcher_started`, `config_synced`, `file_uploaded` |
 | `instrument_runs` | 5 per active instrument | Spread across the last ~2 weeks (3, 6, 9, 12, 15 days back), alternating `lambda` / `watcher` source |
-| `files` | 3 per run | Mix of `uploaded` / `completed` / `failed`, `raw` / `processed` |
+| `files` | 3 per run, or 1 for fixture-bearing runs | Mix of `uploaded` / `completed` / `failed` (and `raw` / `processed` for the 3-file shape). qPCR / gel doc / plate reader runs render exactly one row — the real fixture, bytes copied into `LOCAL_S3_MIRROR` (see [Working with file bytes locally](#working-with-file-bytes-locally)) |
 | `run_comments` | 1 per run | Authored by the dev user |
 | `run_attributions` | 1 per run | Dev user attributed |
 | `archive_jobs` | 3 | One each of `ready` / `building` / `failed` |
 | `watcher_release_config` | 1 (singleton) | `9.9.9 / 0.1.0 / stable / false` |
 
-Externally-visible identifiers used in URLs and API paths — instrument IDs (`seed-<type>`) and run IDs (`seed-run-1`, …) — are deterministic across reseeds, so screenshots, bug reports, and `curl` examples against `/api/v1/instruments/seed-plate-reader/runs/seed-run-1` stay stable.
+Externally-visible identifiers used in URLs and API paths are deterministic across reseeds, so screenshots, bug reports, and `curl` examples stay stable. Instrument types backed by a real lambda `process_file` (qPCR, gel doc, plate reader) use the canonical kebab-case ids the lambda expects (`azure-cielo-qpcr`, `azure-600-gel-doc`, `spectramax-id3-plate-reader`) with realistic-looking run ids (`Experiment_20260129`, `26.02.02_10.45.05`, `012926_AR_OD600`, …). Other instrument types use cosmetic `seed-<type>` ids and `seed-run-1`…`seed-run-5` since they don't round-trip through any pipeline.
 
 Surrogate UUIDs (watcher IDs, archive job IDs, the per-row primary keys on `instrument_runs` and `files`) and the PAT plaintext are regenerated on every reseed — the seed does not use Faker but it does call `crypto.randomUUID()` and `crypto.randomBytes()` where the schema needs server-side IDs.
 
@@ -120,18 +127,97 @@ Some features depend on services that aren't running in this workflow. Each one
 
 | Feature | Behavior locally | How to enable |
 | --- | --- | --- |
-| File download | Presigned URL renders but GET fails — no real S3 object | Point S3 env vars at a real bucket or LocalStack/MinIO |
-| File upload (from watcher) | `request-upload-url` returns a usable signed URL, but actually PUTting fails | Same |
+| File download | Served from `LOCAL_S3_MIRROR` if set (real bytes for seeded qPCR / gel doc / plate reader runs out of the box; other instruments staged via `data-hub-process handler`). 404s when unset. See [Working with file bytes locally](#working-with-file-bytes-locally) | Point S3 env vars at a real bucket or LocalStack/MinIO |
+| File upload (from watcher) | `request-upload-url` returns a same-origin URL routed to `/api/local-s3/...`; `PUT` writes bytes into the mirror | Same |
 | Run archive ("Download all") | 503 "Archive builder is not configured" | Set `LAMBDA_FUNCTION_URL` + `S3_ARCHIVES_BUCKET` and grant `lambda:InvokeFunctionUrl` |
 | File reprocessing | The reprocess endpoint returns null and no Lambda is invoked | Same |
 | Slack notifications on new runs | `console.warn` only, no HTTP call | Set `SLACK_WEBHOOK_URL` |
-| Watcher uploads → Lambda → API loop | Not exercised; the seed inserts the resulting rows directly | Run the watcher (`docs/watcher.md`) and the Lambda (`docs/lambda.md`) end-to-end |
+| Watcher uploads → Lambda → API loop | Not exercised end-to-end; the seed inserts the resulting rows directly. For Lambda-only smoke testing, see [Testing the Lambda end-to-end](#testing-the-lambda-end-to-end) below | Run the watcher (`docs/watcher.md`) and the Lambda (`docs/lambda.md`) end-to-end |
 | Sign in with Google | The button still renders but OAuth callback will 4xx without `AUTH_GOOGLE_ID` / `AUTH_GOOGLE_SECRET` | `vercel env pull` per `docs/getting-started.md` |
 
+## Testing the Lambda end-to-end
+
+Working on a `process_file()` module (or wiring up a brand new one — see [Adding an instrument](guides/adding-an-instrument.md)) and want to run it against the local web app without standing up real S3? The lambda CLI ships a `handler` subcommand that drives `lambda_handler` end-to-end against a gitignored directory mirroring the S3 layout.
+
+```sh
+cd lambda
+
+# Point the inner DataHubClient at the local dev API. The PAT is printed
+# by `npm run db:seed` / `make db-reseed`.
+export DATA_HUB_API_URL=http://localhost:3000/api/v1
+export DATA_HUB_API_KEY=dhub_<paste-from-seed-output>
+
+# instrument_id / run_id / filename match the kebab-case S3 key layout the
+# real Lambda expects; --source is the local file you want "uploaded".
+uv run data-hub-process handler \
+  agilent-4150-tapestation \
+  run-1 \
+  sample.csv \
+  --source ~/Downloads/sample.csv
+```
+
+What happens under the hood:
+
+1. The CLI copies `--source` into `lambda/.local-s3/test-raw-data-bucket/<instrument-id>/<run-id>/<filename>`. That mirror directory is gitignored.
+2. `AWS_S3_RAW_DATA_BUCKET` / `AWS_S3_PROCESSED_DATA_BUCKET` are set so `process_file()` modules see consistent bucket names.
+3. `data_hub_shared.s3_utils.download_file` and `upload_file` are monkey-patched for the duration of the call to `shutil.copy2` from/to the mirror — no boto3, no AWS credentials, no LocalStack.
+4. A synthetic S3 event is built and `lambda_handler(event, ctx)` runs the same dispatch path production uses, calling the dev API at `localhost:3000` for the run/file upserts.
+
+After it returns, navigate to `http://localhost:3000/instruments/<instrument-id>/runs/<run-id>` to inspect what landed; processed artifacts show up under `lambda/.local-s3/test-processed-data-bucket/...`.
+
+Useful flags (`uv run data-hub-process handler --help` for the full list):
+
+| Flag | Default | Purpose |
+| --- | --- | --- |
+| `--source FILE` | required | Local file to stage as the "uploaded" raw object. |
+| `--mirror-root DIR` | `<repo>/lambda/.local-s3` (or `$LOCAL_S3_MIRROR`) | Where the mirror lives on disk. |
+| `--raw-bucket NAME` | `test-raw-data-bucket` | First path segment under the mirror for the raw file. |
+| `--processed-bucket NAME` | `test-processed-data-bucket` | First path segment for `upload_file` calls from the processor. |
+
+The wiring lives in [lambda/src/data_hub_lambda/cli.py](../lambda/src/data_hub_lambda/cli.py) (`handler` subcommand) and [lambda/src/data_hub_lambda/local_s3_mirror.py](../lambda/src/data_hub_lambda/local_s3_mirror.py) (`patched_s3` context manager). The same patch surface backs the integration suite at [lambda/tests/integration/conftest.py](../lambda/tests/integration/conftest.py), so anything that works under the CLI is exercised in CI too.
+
+## Working with file bytes locally
+
+`LOCAL_S3_MIRROR` makes the Next.js app share the same on-disk layout the lambda CLI writes to. When it's set (and `NODE_ENV != production`), the four helpers in [web/lib/s3.ts](../web/lib/s3.ts) — `getPresignedDownloadUrl`, `getPresignedUploadUrl`, `headS3Object`, `getS3ObjectStream` — short-circuit AWS and serve from disk. The HTTP face of the mirror is a single dev-only catch-all at [web/app/api/local-s3/[bucket]/[...key]/route.ts](../web/app/api/local-s3/%5Bbucket%5D/%5B...key%5D/route.ts) that handles GET (download with optional `Content-Disposition`) and PUT (writes bytes from the request body). Note the folder is `local-s3` (no leading underscore) — the App Router treats `_`-prefixed folders as private and excludes them from routing.
+
+What this gets you out of the box after `make db-reseed`:
+
+| Instrument type | Seeded fixture | Where it comes from |
+| --- | --- | --- |
+| qPCR | `azure_cielo_qpcr_example.csv` | `lambda/tests/fixtures/` |
+| Gel doc | `azure_600_gel_doc_example.tif` | `lambda/tests/fixtures/` |
+| Plate reader | `spectramax_plate_reader_endpoint.xls` | `lambda/tests/fixtures/` |
+| Other instruments | none — files 404 in the mirror | Stage real bytes via `data-hub-process handler` |
+
+The seed copies the fixture into `<LOCAL_S3_MIRROR>/test-raw-data-bucket/<instrument-id>/<run-id>/<filename>` for every seeded run on those instruments, so navigating to `/instruments/azure-cielo-qpcr/runs/Experiment_20260129` shows a real CSV in the file browser, the colony / plate-reader viewers fetch real bytes via `/api/v1/files/<id>/download`, and PNG / TIFF / PDF previews on `RunReportSection` render without 404s. Fixture-bearing runs only have the real fixture file — the synthetic CSV siblings other instruments still get are dropped so the UI only shows files that actually exist on disk.
+
+When the dev API is reachable during seeding, the seed also drives `data-hub-process handler` over each fixture-bearing run so the dashboard renders processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata) immediately after a reseed. If the API isn't up yet (`npm run db:reseed` ran before `npm run dev`), the seed prints a hint and skips the step — you can re-run it on its own once the dev server is reachable:
+
+```sh
+# in the terminal where the dev server is running
+npm run dev
+# in another terminal, after the dev server is reachable
+npm run db:process-fixtures
+```
+
+`npm run db:process-fixtures` mints a fresh PAT for `dev@local`, re-derives the fixture-bearing `(instrument_id, run_id, filename)` triples from the database, and spawns `data-hub-process handler` for each. The wiring lives in [web/scripts/process-fixtures.ts](../web/scripts/process-fixtures.ts) — it's the same module the seed calls — so anything that works during a reseed also works post-hoc.
+
+For instrument types without a fixture (or new file types you're adding components for), the existing CLI flow stays the same: run `data-hub-process handler <instrument-id> <run-id> <filename> --source <FILE>` and the dashboard picks up the file the moment the API row lands.
+
+Components don't need to change — every existing run viewer already fetches `/api/v1/files/<id>/download`, which 302s to whatever `getPresignedDownloadUrl` returns. New custom components for a specific instrument should follow the same pattern (`fetch("/api/v1/files/<id>/download")` for raw bytes, `<img src="/api/v1/files/<id>/download">` for images) and inherit local-mirror support automatically.
+
+A few details worth knowing:
+
+- Adding fixtures for more instruments is one entry in `INSTRUMENT_FIXTURES` in [web/lib/db/seed.ts](../web/lib/db/seed.ts), pointing at any file under `lambda/tests/fixtures/`.
+- The route is gated on `NODE_ENV !== "production"` AND `LOCAL_S3_MIRROR` set; either condition unmet returns 404 unconditionally, so a production build can never expose the filesystem.
+- The MCP tool at `/api/v1/mcp` returns a relative `/api/local-s3/...` URL when the mirror is active — browsers resolve it against the current origin, but non-browser MCP clients on localhost may need to prefix with `http://localhost:3000`.
+
 ## Where the seed lives
 
 - [web/lib/db/seed.ts](../web/lib/db/seed.ts) — shared builder functions (`seedDevUser`, `seedInstruments`, `seedRuns`, etc.) plus a schema-driven `clearAll()`.
 - [web/scripts/seed-database.ts](../web/scripts/seed-database.ts) — the entry point that `npm run db:seed` runs.
+- [web/scripts/process-fixtures.ts](../web/scripts/process-fixtures.ts) — shared probe/spawn logic for the post-seed handler step.
+- [web/scripts/process-seeded-fixtures.ts](../web/scripts/process-seeded-fixtures.ts) — `npm run db:process-fixtures` entry point for re-running the handler step on demand.
 
 The same builders back the integration test harness in [web/tests/integration/helpers.ts](../web/tests/integration/helpers.ts), so any new table added to `web/lib/db/schema.ts` is automatically included in `clearAll()` and only needs a new builder function if you want it populated by the dev seed.