[Production] Release 2026-08-20

.github/workflows/python-test.yml

@@ -63,6 +63,9 @@ jobs:
         run: npm ci
         working-directory: web
 
+      - name: Install ffmpeg
+        run: sudo apt-get update && sudo apt-get install -y ffmpeg
+
       - name: Install Python packages
         run: uv sync --all-packages
 

developer-docs/getting-started.md

@@ -11,6 +11,7 @@ This guide walks through setting up the full Data Hub development environment.
 | Node.js | >= 22 | Web application |
 | PostgreSQL | >= 15 | Database (local development) |
 | Docker | latest | Lambda container builds |
+| ffmpeg | latest | DishCam TIFF → MP4 encode (`brew install ffmpeg` / `apt install ffmpeg`). Encode tests skip if it is missing. |
 | [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html) | latest | Infrastructure deployment (optional — only needed for deploying to AWS) |
 
 ## Clone and install

developer-docs/lambda.md

@@ -50,6 +50,7 @@ Dispatch is by `instrument_type` (Postgres/TS enum), not instrument ID. The regi
 | `epson_v700_scanner` | `epson_v700_scanner` | `.tif` / `.tiff` |
 | `hina_microscope` | `hina_microscope` | `.nd2` |
 | `plate_reader` | `spectramax_plate_reader` | `.xls` |
+| `dishcam` | `dishcam` | `.tif` / `.tiff` / `run.json` |
 | `generic`, `instant_raman` | — | — |
 
 **One type = one vendor's output format.** Names like `qpcr` and `fplc` sound generic, but the parsers behind them are vendor-specific (Azure Cielo, ÄKTA, …). Adding a second vendor under an existing type requires splitting the type, not reusing it.
@@ -98,9 +99,10 @@ Available commands:
 | `qpcr` | Parse dye channels from an Azure Cielo qPCR Cq Values CSV |
 | `spectramax` | Parse metadata and raw well data from a SpectraMax `.xls` export |
 | `tapestation` | Extract the tape type from a TapeStation CSV filename |
+| `dishcam` | Convert a DishCam TIFF stack plus `run.json` into an MP4 preview and JPEG poster |
 | `handler` | Stage a file into a local S3 mirror and invoke `lambda_handler` against the local dev API. See [Testing the Lambda end-to-end](local-development.md#testing-the-lambda-end-to-end) for the workflow. |
 
-The first six subcommands need no S3 or API access — they call into the same parsing/processing utilities the lambda uses, but stop short of the network. `handler` is different: it expects a running dev API and a `LOCAL_S3_MIRROR` directory, and uses the same dispatch path production uses.
+The instrument-specific subcommands need no S3 or API access — they call into the same parsing/processing utilities the lambda uses, but stop short of the network. `handler` is different: it expects a running dev API and a `LOCAL_S3_MIRROR` directory, and uses the same dispatch path production uses.
 
 Examples:
 
@@ -120,8 +122,9 @@ make docker-build-lambda
 
 The Dockerfile is a multi-stage build:
 
-1. **Builder stage**: Uses `uv` to export and install third-party dependencies into the Lambda task root.
-2. **Final stage**: Copies the installed dependencies plus the `data_hub_shared` and `data_hub_lambda` source packages.
+1. **ffmpeg stage**: Copies a static linux/amd64 `ffmpeg` (libx264) from the version-tagged `mwader/static-ffmpeg` image.
+2. **Builder stage**: Uses `uv` to export and install third-party dependencies into the Lambda task root.
+3. **Final stage**: Copies ffmpeg, the installed dependencies, and the `data_hub_shared` and `data_hub_lambda` source packages.
 
 The entry point is `data_hub_lambda.handler.lambda_handler`.
 
@@ -134,6 +137,7 @@ The Lambda function depends on a scientific Python stack:
 - `matplotlib` — plotting
 - `scikit-image` — image processing
 - `tifffile` — TIFF file reading
+- `ffmpeg` — static linux/amd64 binary from `mwader/static-ffmpeg` for DishCam H.264 encode
 - `arcadia-microscopy-tools` — ND2 reading, channel handling, and multi-channel compositing for the Hina microscope
 - `pydantic` — data validation
 - `requests` — HTTP client for the Data Hub API

lambda/Dockerfile

@@ -1,5 +1,9 @@
 FROM ghcr.io/astral-sh/uv:0.8.12 AS uv
 
+# Version-tagged static build with libx264. BtbN autobuild URLs are
+# deleted; this Hub tag stays pullable. linux/amd64 matches Lambda.
+FROM --platform=linux/amd64 mwader/static-ffmpeg:8.1.2 AS ffmpeg
+
 FROM public.ecr.aws/lambda/python:3.12 AS builder
 
 ENV UV_COMPILE_BYTECODE=1
@@ -21,6 +25,7 @@ RUN --mount=from=uv,source=/uv,target=/bin/uv \
 
 FROM public.ecr.aws/lambda/python:3.12
 
+COPY --from=ffmpeg /ffmpeg /usr/local/bin/ffmpeg
 COPY --from=builder ${LAMBDA_TASK_ROOT} ${LAMBDA_TASK_ROOT}
 # Workspace packages are copied as source (not pip-installed) so they land
 # directly on the Python path alongside the third-party deps.

lambda/src/data_hub_lambda/cli.py

@@ -227,6 +227,46 @@ def tapestation(filename: str) -> None:
         click.echo("No tape type found in filename.")
 
 
+# ---------------------------------------------------------------------------
+# DishCam
+# ---------------------------------------------------------------------------
+
+
+@cli.command("dishcam")
+@click.argument("tiff", type=click.Path(exists=True, dir_okay=False, path_type=Path))
+@click.option(
+    "--run-json",
+    type=click.Path(exists=True, dir_okay=False, path_type=Path),
+    required=True,
+    help="Sidecar run.json (fps / frame count).",
+)
+@click.option(
+    "--output-dir",
+    type=click.Path(file_okay=False, path_type=Path),
+    default=None,
+    help="Directory for the MP4 and JPEG poster (default: same directory as TIFF).",
+)
+def dishcam_cmd(tiff: Path, run_json: Path, output_dir: Path | None) -> None:
+    """Convert a DishCam TIFF stack to an MP4 preview and JPEG poster."""
+    from data_hub_lambda.dishcam.encode_video import encode_tiff_stack
+    from data_hub_lambda.dishcam.parse_metadata import (
+        encode_fps,
+        parse_run_json,
+        playback_fps,
+    )
+
+    metadata = parse_run_json(run_json)
+    fps = playback_fps(encode_fps(metadata))
+    dest_dir = output_dir or tiff.parent
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    mp4_path = dest_dir / f"{tiff.stem}.mp4"
+    poster_path = dest_dir / f"{tiff.stem}.jpg"
+    encode_tiff_stack(tiff, mp4_path, poster_path, fps)
+    click.echo(f"Exported MP4: {mp4_path}")
+    click.echo(f"Exported JPEG: {poster_path}")
+    click.echo(json.dumps(metadata, indent=2))
+
+
 # ---------------------------------------------------------------------------
 # End-to-end handler invocation against a local S3 mirror
 # ---------------------------------------------------------------------------

lambda/src/data_hub_lambda/dishcam/__init__.py

@@ -0,0 +1,3 @@
+from data_hub_lambda.dishcam.process_file import process_file
+
+__all__ = ["process_file"]

lambda/src/data_hub_lambda/dishcam/encode_video.py

@@ -0,0 +1,183 @@
+"""Stream a DishCam TIFF stack through ffmpeg into an HTML5 MP4 + JPEG poster."""
+
+from __future__ import annotations
+import logging
+import shutil
+import subprocess
+import tempfile
+from collections.abc import Iterator
+from pathlib import Path
+
+import numpy as np
+import tifffile
+from numpy.typing import NDArray
+
+logger = logging.getLogger(__name__)
+
+# Width cap keeps the preview inside H.264 Level 5.2 / browser limits
+# (a 3040x4056 stack will not play in Safari). `h=-2` keeps height even
+# and aspect; the trunc expression makes width even for yuv420p.
+_SCALE_FILTER = "scale=w='trunc(min(1920,iw)/2)*2':h=-2"
+
+# Progress lines would fill the stderr pipe (~64 KB) and deadlock a
+# long encode. Failures still land in the temp-file capture below.
+_FFMPEG_QUIET = ("-nostats", "-loglevel", "error")
+
+
+def resolve_ffmpeg() -> str:
+    path = shutil.which("ffmpeg")
+    if path is None:
+        raise RuntimeError("ffmpeg is not on PATH")
+    return path
+
+
+def _ffmpeg_rate(fps: float) -> str:
+    """Fixed-point rate. ffmpeg rejects scientific notation like `1e-05`."""
+    return f"{fps:.6f}".rstrip("0").rstrip(".")
+
+
+def _as_rgb24(frame: NDArray) -> NDArray:  # type: ignore[type-arg]
+    """Return a C-contiguous uint8 RGB frame for rawvideo stdin.
+
+    DishCam writes uint8. A later camera with a wider type needs an
+    explicit converter — do not guess 12-bit from a uint16 peak.
+    """
+    if frame.ndim == 2:
+        frame = np.stack([frame, frame, frame], axis=-1)
+    elif frame.ndim == 3 and frame.shape[-1] == 4:
+        frame = frame[..., :3]
+    elif frame.ndim != 3 or frame.shape[-1] != 3:
+        raise ValueError(f"Unsupported TIFF page shape: {frame.shape}")
+    if frame.dtype != np.uint8:
+        raise ValueError(f"Expected uint8 RGB pages, got {frame.dtype}")
+    return np.ascontiguousarray(frame)
+
+
+def _pipe_ffmpeg(cmd: list[str], chunks: Iterator[bytes]) -> None:
+    """Write raw frames to ffmpeg; kill the child if the parent fails.
+
+    stderr goes to a temp file so a chatty encode cannot fill a pipe
+    and deadlock against stdin.
+    """
+    with tempfile.NamedTemporaryFile() as err_file:
+        proc = subprocess.Popen(
+            cmd,
+            stdin=subprocess.PIPE,
+            stdout=subprocess.DEVNULL,
+            stderr=err_file,
+        )
+        assert proc.stdin is not None
+        try:
+            try:
+                for chunk in chunks:
+                    proc.stdin.write(chunk)
+                proc.stdin.close()
+            except BrokenPipeError:
+                pass
+            proc.wait()
+        except Exception:
+            proc.kill()
+            proc.wait()
+            raise
+        err_file.seek(0)
+        stderr = err_file.read().decode(errors="replace")
+    if proc.returncode != 0:
+        raise RuntimeError(f"ffmpeg encode failed: {stderr}")
+
+
+def _write_jpeg(ffmpeg: str, frame: NDArray, dest: Path) -> None:  # type: ignore[type-arg]
+    height, width = frame.shape[:2]
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    cmd = [
+        ffmpeg,
+        "-y",
+        *_FFMPEG_QUIET,
+        "-f",
+        "rawvideo",
+        "-pix_fmt",
+        "rgb24",
+        "-s:v",
+        f"{width}x{height}",
+        "-i",
+        "pipe:0",
+        "-vf",
+        _SCALE_FILTER,
+        "-frames:v",
+        "1",
+        "-q:v",
+        "3",
+        str(dest),
+    ]
+    _pipe_ffmpeg(cmd, iter((frame.tobytes(),)))
+
+
+def encode_tiff_stack(
+    tiff_path: Path,
+    mp4_path: Path,
+    poster_path: Path,
+    fps: float,
+) -> None:
+    """Decode pages one at a time and pipe RGB24 into ffmpeg.
+
+    Loading the full stack would be ~35 MB per 12 MP frame; real runs can
+    be hundreds of frames.
+    """
+    ffmpeg = resolve_ffmpeg()
+    mp4_path.parent.mkdir(parents=True, exist_ok=True)
+
+    with tifffile.TiffFile(tiff_path) as tif:
+        if not tif.pages:
+            raise ValueError(f"{tiff_path.name} has no TIFF pages")
+
+        first = _as_rgb24(tif.pages[0].asarray())
+        height, width = first.shape[:2]
+        _write_jpeg(ffmpeg, first, poster_path)
+
+        cmd = [
+            ffmpeg,
+            "-y",
+            *_FFMPEG_QUIET,
+            "-f",
+            "rawvideo",
+            "-pix_fmt",
+            "rgb24",
+            "-s:v",
+            f"{width}x{height}",
+            "-r",
+            _ffmpeg_rate(fps),
+            "-i",
+            "pipe:0",
+            "-vf",
+            _SCALE_FILTER,
+            "-c:v",
+            "libx264",
+            "-pix_fmt",
+            "yuv420p",
+            "-crf",
+            "23",
+            "-preset",
+            "veryfast",
+            "-movflags",
+            "+faststart",
+            # Every frame is a keyframe so the in-app player can scrub
+            # a short timelapse without waiting for the next GOP.
+            "-g",
+            "1",
+            "-an",
+            str(mp4_path),
+        ]
+        logger.info(
+            "Encoding %s (%dx%d, %s fps) → %s",
+            tiff_path.name,
+            width,
+            height,
+            fps,
+            mp4_path.name,
+        )
+
+        def _chunks() -> Iterator[bytes]:
+            yield first.tobytes()
+            for page in tif.pages[1:]:
+                yield _as_rgb24(page.asarray()).tobytes()
+
+        _pipe_ffmpeg(cmd, _chunks())

lambda/src/data_hub_lambda/dishcam/filenames.py

@@ -0,0 +1,24 @@
+"""Filename helpers for DishCam S3 gates and sibling lookup."""
+
+from __future__ import annotations
+
+RUN_JSON_NAME = "run.json"
+_TIFF_SUFFIXES = (".tif", ".tiff")
+
+
+def is_run_json(filename: str) -> bool:
+    return filename.lower() == RUN_JSON_NAME
+
+
+def is_tiff(filename: str) -> bool:
+    return filename.lower().endswith(_TIFF_SUFFIXES)
+
+
+def matches_filename(filename: str) -> bool:
+    """S3 events for a stack or the sidecar can start encode.
+
+    `run.json` therefore also passes the handler's union gate for every
+    instrument type. The extra instrument lookup is cheap; the per-type
+    gate still rejects it on non-DishCam instruments.
+    """
+    return is_tiff(filename) or is_run_json(filename)

lambda/src/data_hub_lambda/dishcam/parse_metadata.py

@@ -0,0 +1,62 @@
+"""Parse DishCam sidecar `run.json` into run metadata and encode fps."""
+
+from __future__ import annotations
+import json
+from pathlib import Path
+from typing import Any
+
+_METADATA_KEYS = (
+    "fps",
+    "measured_fps",
+    "frames",
+    "duration_seconds",
+    "quality",
+    "format",
+    "started",
+    "finished",
+)
+
+
+def parse_run_json(path: Path) -> dict[str, Any]:
+    """Return the sidecar fields stored on the run.
+
+    Raises `ValueError` when the file is not an object or has no usable fps.
+    """
+    try:
+        payload = json.loads(path.read_text())
+    except json.JSONDecodeError as exc:
+        raise ValueError(f"Invalid run.json: {exc}") from exc
+    if not isinstance(payload, dict):
+        raise ValueError("run.json must be a JSON object")
+
+    metadata = {
+        key: payload[key] for key in _METADATA_KEYS if key in payload and payload[key] is not None
+    }
+    encode_fps(metadata)
+    return metadata
+
+
+def encode_fps(metadata: dict[str, Any]) -> float:
+    """Prefer measured fps when present; otherwise use the planned fps."""
+    raw = metadata.get("measured_fps")
+    if raw is None:
+        raw = metadata.get("fps")
+    if raw is None:
+        raise ValueError("run.json is missing fps and measured_fps")
+    try:
+        fps = float(raw)
+    except (TypeError, ValueError) as exc:
+        raise ValueError(f"run.json fps is not a number: {raw!r}") from exc
+    if fps <= 0:
+        raise ValueError(f"run.json fps must be positive, got {fps}")
+    return fps
+
+
+# Multi-day captures can land well below 1 fps. The in-app player is a
+# preview, so floor the encode rate rather than playing in real time.
+MIN_PLAYBACK_FPS = 10.0
+
+
+def playback_fps(capture_fps: float) -> float:
+    """Return the fps used for the MP4 preview, not the stored metadata."""
+    return max(capture_fps, MIN_PLAYBACK_FPS)