Local dev: end-to-end Lambda + filesystem-backed S3 mirror

[wasimxyz]

Summary

Local-dev workflow improvements that together let a developer iterate on process_file() modules and custom run/file viewer components against the local web app without S3, AWS credentials, or LocalStack.

1. data-hub-process handler CLI

New subcommand on the lambda CLI that runs lambda_handler end-to-end against a gitignored lambda/.local-s3/ directory mirroring the S3 layout:

• Stages --source FILE into <mirror>/<raw-bucket>/<instrument-id>/<run-id>/<filename>.
• Monkey-patches data_hub_shared.s3_utils.download_file / upload_file to copy via shutil.copy2 (no boto3, no credentials).
• Builds a synthetic S3 event and invokes lambda_handler(event, ctx) so the same dispatch path production uses runs against the local Next.js dev API.

Reusable patch surface in lambda/src/data_hub_lambda/local_s3_mirror.py mirrors what tests/integration/conftest.py already does for CI.

2. Filesystem-backed S3 mirror in web/

When LOCAL_S3_MIRROR is set (and NODE_ENV != production), the four helpers in web/lib/s3.ts short-circuit the AWS SDK and resolve to <root>/<bucket>/<key> via a dev-only catch-all at /api/local-s3/{bucket}/{...key}:

• GET streams the file with optional Content-Disposition (matches presigned-URL behavior).
• PUT writes the request body to disk via pipeline() for clean error/abort handling.
• Both 404 unconditionally when production or the env var is unset.

(Folder is local-s3 not _local-s3 — the App Router treats _-prefixed folders as private and excludes them from routing.)

Components don't change: every existing run viewer fetches /api/v1/files/<id>/download, which 302s to whatever getPresignedDownloadUrl returns, so swapping the backend is invisible.

3. Realistic seed + automated handler run

• Instrument types backed by a real process_file (qPCR, gel doc, plate reader) seed under their canonical kebab ids (azure-cielo-qpcr, azure-600-gel-doc, spectramax-id3-plate-reader) with instrument-shaped run ids (Experiment_20260129, 26.02.02_10.45.05, 012926_AR_OD600, …). Other types keep seed-<type> / seed-run-N.
• Fixture-bearing runs render exactly one file row — the real fixture — instead of mixing in synthetic CSV siblings.
• After seeding, the seed probes the dev API with the just-minted PAT and (when reachable) spawns data-hub-process handler for each fixture-bearing run so processed artifacts (gel-doc PNGs, plate-reader CSVs, qPCR metadata) appear in the dashboard immediately. If the API is down, it prints a hint and skips. npm run db:process-fixtures re-runs the same step on demand once npm run dev is up.

Files

• lambda/src/data_hub_lambda/cli.py — new handler subcommand
• lambda/src/data_hub_lambda/local_s3_mirror.py — patched_s3 context manager
• web/lib/s3-local-mirror.ts — getLocalMirrorRoot, resolveMirrorPath, URL builders, MIME map
• web/app/api/local-s3/[bucket]/[...key]/route.ts — GET + PUT handlers
• web/lib/s3.ts — local-mirror branch on each of the four exports
• web/lib/db/seed.ts — CANONICAL_INSTRUMENT_ID, INSTRUMENT_FIXTURES, fixture copying
• web/scripts/process-fixtures.ts — shared probe + spawn helper
• web/scripts/process-seeded-fixtures.ts — npm run db:process-fixtures entry point
• web/scripts/seed-database.ts — calls processSeededFixtures after seeding
• web/.env.example, docs/local-development.md, docs/guides/adding-an-instrument.md — documentation
• .gitignore — lambda/.local-s3/

Test plan

• make check-all passes (ruff, pyright, prettier, eslint, tsc).
• make db-reseed && make dev then visit http://localhost:3000/instruments/azure-cielo-qpcr/runs/Experiment_20260129 — fixture CSV downloads and processed metadata is set on the run.
• Visit http://localhost:3000/instruments/azure-600-gel-doc/runs/26.02.02_10.45.05 — TIFF renders and processed PNG appears under the run.
• cd lambda && uv run data-hub-process handler agilent-4150-tapestation run-1 sample.csv --source <path> — file row appears with viewable bytes.
• Reseed with npm run dev not running → seed prints the skip hint; npm run db:process-fixtures after starting dev produces the same result as case 1.
• Production-safety: setting LOCAL_S3_MIRROR with NODE_ENV=production does not expose /api/local-s3/... (returns 404).
• Existing integration suite (web/tests/integration/, lambda/tests/integration/) continues to pass.

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
data-hub	Ready	Preview, Comment	May 22, 2026 6:25pm