Add first-class support for the Hina microscope

[wasimxyz]

Summary

Adds first-class support for the Hina microscope as a new instrument in Data Hub, end-to-end:

• Lambda: converts uploaded .nd2 files into composite RGB JPG overlays using arcadia-microscopy-tools, and extracts run-level metadata (channels, dimensions, sizes).
• Web app: a HinaRunsTable with filterable Channels / Dimensions / Sizes columns and a HinaMicroscopeRunDetail variant whose report is an image carousel over the per-file JPGs.
• Infra: raises Lambda limits to Lambda hard maxes so 3–4 GB ND2 files actually process.

Changes

Lambda / processing (201bef4, 0fef702)

• New lambda/src/data_hub_lambda/hina_microscope/ package:
  • image_processing.ND2Processor — loads ND2, reduces to 2D (Z → max projection, T/P → first index), percentile-stretches (1st–99th) each channel, composites via overlay_channels on a brightfield (or zero) background, exports JPG (quality 90).
  • parse_metadata — serializes sizes, channels (name / ex / em / hex), and DimensionFlags to JSON.
  • process_file — download raw → produce JPG → upload processed → create/update file records; run metadata is parsed once per run via an if not run.metadata gate (first file wins).
• hina CLI subcommand added to data-hub-process.
• S3 trigger for hina-microscope/*.nd2 added to infra/template.yaml.
• arcadia-microscopy-tools>=0.3.2 added to lambda/pyproject.toml.
• Lambda globals raised to hard maxes for ND2 headroom: Timeout: 900, MemorySize: 10240, per-function EphemeralStorage: 10240.
• Handler: added a pre-cleanup _cleanup_tmp() call at invocation start so warm containers that were SIGKILL'd (OOM/timeout) don't start a new run with stale /tmp downloads.
• Unit tests for axis reduction, percentile rescaling, metadata serialization, and the single-shot metadata gate.

Shared (201bef4)

• Instrument.HINA_MICROSCOPE = "hina-microscope" + INSTRUMENT_ID_TO_NAME_MAP entry.

Web app — runs table & detail (a7b7754, b3f6f14)

• Drizzle migration 0008_chilly_princess_powerful.sql: ALTER TYPE instrument_type ADD VALUE 'hina_microscope', with the new value wired into the edit-instrument dialog picker and the schema.
• HinaRunsTable with Channels / Dimensions / Sizes columns (channel name + hex swatch, dimension flags, formatted size string).
• HinaMicroscopeRunDetail variant with a HinaReportSection powered by a new shadcn ui/carousel (embla-carousel-react) over processed JPGs.
• Shared helpers getMetadataRecord and getMetadataObjectArray (+ metadata badges) to safely read nested sizes / channels.
• nuqs keys added to instrumentDetailSearchParams: hina_channel, hina_dimension, hina_size (raw JSONB equality on sizes so key orderings collapse to one option).
• API: instrument-runs.ts returns distinct Hina metadata values for column filter dropdowns.
• Light-theme fix: near-white channel swatches blend with --foreground for border/text and ring with --border so the true channel color still reads.

Driveby changes

• Watcher-gated uploads (c76bbc9): when no watcher is heartbeating, per-row and bulk Upload buttons are disabled with a tooltip explaining why, so files don't get stuck in upload_requested with no agent to action them. Adds WatcherStatusProvider, WatcherGatedUploadButton, and threads isWatcherOnline through RunDetailProps → each variant → RunFilesSection / RunFilesTable. Applies to all instruments, not just Hina.
• Drizzle config (865b437): small config update alongside the new migration.
• docs/lambda.md updated with the new instrument row, CLI subcommand, and arcadia-microscopy-tools dependency.
• Cursor review fixes (5dd58ba).

Breaking changes

• DB migration required: 0008_chilly_princess_powerful.sql adds hina_microscope to the instrument_type pg enum. Must be applied before deploying the web app build that references it.
• Lambda redeploy required: new S3 trigger + raised Timeout / Memory / EphemeralStorage. All functions under the shared Globals block now run at 10 GB memory / 900 s timeout — verify this is acceptable cost-wise for the other instrument handlers (or scope the overrides per-function if not).
• No API contract breakage.

Testing

• make check-all is green (format / lint / type checks)
• pytest lambda/tests/hina_microscope/ passes (axis reduction, percentile rescale, metadata serialization, single-shot metadata gate)
• Run the data-hub-process hina <file.nd2> CLI locally against a real ND2 and verify:
  • JPG renders with expected channel colors / contrast
  • Z-stacks max-project correctly; T/P fall back to first index
  • parse_metadata output is JSON-serializable with sizes / channels / dimensions
• Deploy Lambda to a staging stack and drop a real hina-microscope/<run>/<file>.nd2 into the raw bucket:
  • Raw file record transitions uploaded → processing → completed
  • Processed JPG appears in the processed bucket and in the run's files list
  • Run-level metadata is populated only on the first file; subsequent files still produce JPGs but don't overwrite metadata
  • Confirm memory + timeout are sufficient for a representative 3–4 GB ND2
  • Warm-container pre-cleanup: trigger an OOM, then invoke again and confirm /tmp is empty at start
• Web app (staging):
  • Instrument picker in edit-instrument dialog includes Hina Microscope
  • HinaRunsTable renders Channels / Dimensions / Sizes with correct swatches and formatted sizes
  • Column filters (hina_channel, hina_dimension, hina_size) populate from distinct values and filter correctly; sizes with different key orderings collapse to one option
  • Near-white channel swatches remain visible in light theme
  • HinaMicroscopeRunDetail carousel paginates through processed JPGs
• Watcher-gated uploads (cross-instrument regression):
  • With watcher online: per-row + bulk Upload buttons are enabled on every instrument
  • Stop the watcher: both buttons disable and show the explanatory tooltip

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
data-hub	Ready	Preview, Comment	Apr 24, 2026 7:30pm