[Production] Release 2026-07-27

developer-docs/ci-and-deployment.md

@@ -110,7 +110,7 @@ On pushes to `staging` or `production`, the **Deploy Lambda** workflow:
 
 Secrets (`DATA_HUB_API_KEY`, etc.) are stored in GitHub environment secrets scoped to each environment.
 
-> **Note:** The CI deploy role has intentionally narrow permissions — enough to push a new container image, update the existing CloudFormation stack, modify the data buckets' S3 event notifications (so new instrument triggers roll out via CI), and update the data buckets' CORS configuration, but _not_ enough to create the stack from scratch or to add/remove S3 buckets or Lambda functions. Initial stack creation and structural infrastructure changes must be performed by an admin with broader AWS permissions. Once the stack exists, routine image-update deploys and new-trigger rollouts through CI work without issue.
+> **Note:** The CI deploy role has intentionally narrow permissions — enough to push a new container image, update the existing CloudFormation stack, modify the data buckets' S3 event notifications, and update the data buckets' CORS configuration, but _not_ enough to create the stack from scratch or to add/remove S3 buckets or Lambda functions. Initial stack creation and structural infrastructure changes must be performed by an admin with broader AWS permissions. Once the stack exists, routine image-update deploys through CI work without issue.
 >
 > The deploy that first grants `s3:PutBucketCORS` must be run by an admin via `make sam-deploy` (CI can't grant itself a permission and use it in the same changeset). CORS edits after that roll out through CI.
 >
@@ -144,23 +144,9 @@ make docker-push-lambda ENV=staging
 make sam-deploy ENV=staging
 ```
 
-#### Adding an S3 trigger for a new instrument
+#### S3 notifications
 
-Instruments that support automated preprocessing need an S3 event trigger so the Lambda runs as files land. The processor code lives in `data-hub-lambda` (see [Lambda → Adding a new instrument](lambda.md#adding-a-new-instrument)); this is the infrastructure half. Add a `LambdaConfiguration` entry to the `RawDataBucket` resource's `NotificationConfiguration` in `infra/template.yaml`:
-
-```yaml
-- Event: s3:ObjectCreated:*
-  Filter:
-    S3Key:
-      Rules:
-        - Name: prefix
-          Value: <instrument-id>/
-        - Name: suffix
-          Value: .csv
-  Function: !GetAtt DataHubFunction.Arn
-```
-
-The CI deploy role has permission to roll new triggers out, so the trigger goes live on the next deploy — either the [automated workflow](#automated-deployment-deploy-lambdayml) or a manual `make sam-deploy`. No manual AWS step is needed once the code and trigger are merged.
+The raw bucket uses a single catch-all `ObjectCreated:*` notification on the Lambda. New instrument types do **not** need a new `LambdaConfiguration` entry — register a processor by `instrument_type` instead (see [Lambda → Adding a new instrument / processor](lambda.md#adding-a-new-instrument--processor)). Deploy the type-dispatch handler before changing notification filters when rolling this out to an environment that still has per-ID rules.
 
 ### Watcher (PyPI)
 

developer-docs/conventions.md

@@ -10,25 +10,27 @@ All raw data files are stored in S3 with the key pattern:
 {instrument_id}/{run_id}/{filename}
 ```
 
-- **`instrument_id`** — kebab-case identifier matching the `Instrument` enum (e.g., `akta-fplc`).
+- **`instrument_id`** — kebab-case identifier for the instrument row (e.g., `akta-fplc`).
 - **`run_id`** — unique identifier for the run, either extracted from the filename prefix or from a subdirectory name.
 - **`filename`** — the original filename.
 
 The S3 bucket name follows the template `arcadia-data-hub-raw-{environment}`, where `environment` is `staging` or `production`.
 
-## Instrument IDs
+## Instrument IDs and types
 
 Instrument IDs are kebab-case strings (lowercase letters, numbers, hyphens). They serve as:
 
 - S3 key prefixes
 - API resource identifiers
-- Enum values in `data_hub_shared.enums.Instrument`
+- Primary keys on the `instruments` table
 
-When adding a new instrument, the ID must be registered in three places:
+Lambda dispatch and web reprocess eligibility use **`instrument_type`**, not the ID. When adding a processable instrument:
 
-1. `Instrument` enum in `packages/shared/src/data_hub_shared/enums.py`
-2. `INSTRUMENT_ID_TO_NAME_MAP` in `packages/shared/src/data_hub_shared/constants.py`
-3. Dispatch logic in `lambda/src/data_hub_lambda/handler.py`
+1. Set (or add) the appropriate `instrument_type` on the instrument row
+2. Register a processor for that type in `lambda/src/data_hub_lambda/processors.py`
+3. Add the same type to `PROCESSABLE_INSTRUMENT_TYPES` in `web/lib/instruments/processable-types.ts`
+
+The shared `Instrument` enum in `packages/shared` is optional legacy naming for watcher/CLI display — it is not the Lambda support gate.
 
 ## Environment variables
 

developer-docs/first-time-deployment.md

@@ -82,7 +82,11 @@ npm run db:migrate
 
 ### Create an API key for the Lambda
 
-Sign in with an account listed in `ADMIN_EMAILS`, then create a personal access token under Settings. The AWS stack and the Lambda use this token as `DATA_HUB_API_KEY` to call the Data Hub API, so create it now and keep it for [step 4](#4-deploy-the-aws-infrastructure). See [Issue and revoke tokens](https://datahub.arcadiascience.com/docs/manage-tokens) for the token UI.
+Sign in with an account listed in `ADMIN_EMAILS`, then create a personal access token under Settings. Use the **Lambda** scope preset (or an equivalent list that includes `instruments:read`, `runs:create`, `runs:update`, `files:create`, `files:update`, and `archive-jobs:write`). The Lambda looks up each instrument's type before dispatching, so a token without `instruments:read` will 403 on every S3 event.
+
+The AWS stack and the Lambda use this token as `DATA_HUB_API_KEY` to call the Data Hub API, so create it now and keep it for [step 4](#4-deploy-the-aws-infrastructure). See [Issue and revoke tokens](https://datahub.arcadiascience.com/docs/manage-tokens) for the token UI.
+
+If you previously minted a Lambda token from an older preset that omitted `instruments:read`, revoke it and create a new one with the updated Lambda preset, then update the `DATA_HUB_API_KEY` secret / SAM parameter for each environment.
 
 ## 3. Bootstrap AWS (once per account)
 
@@ -96,7 +100,7 @@ make sam-bootstrap
 
 ## 4. Deploy the AWS infrastructure
 
-The storage and processing layer — the S3 buckets and the data-processing Lambda — is defined in `infra/template.yaml` and deployed with [AWS SAM](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/). The stack creates the raw/processed S3 buckets, the Lambda function (container image, function URL), the per-instrument S3 event triggers, and the IAM roles for Lambda execution, CI deploys (OIDC), and Vercel web app S3 access (OIDC).
+The storage and processing layer — the S3 buckets and the data-processing Lambda — is defined in `infra/template.yaml` and deployed with [AWS SAM](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/). The stack creates the raw/processed S3 buckets, the Lambda function (container image, function URL), a catch-all S3 `ObjectCreated:*` notification on the raw bucket, and the IAM roles for Lambda execution, CI deploys (OIDC), and Vercel web app S3 access (OIDC).
 
 **1. Get the bootstrap stack outputs.**
 

developer-docs/lambda.md

@@ -8,22 +8,23 @@ The Lambda has three invocation paths:
 
 ### S3 trigger (automatic)
 
-1. An S3 `PutObject` event triggers the Lambda function.
+1. An S3 `ObjectCreated:*` event on the raw bucket triggers the Lambda (catch-all; no per-instrument prefix/suffix filters).
 2. The handler parses the S3 key to extract the instrument ID, run ID, and filename. The expected key layout is `{instrument_id}/{run_id}/{filename}`.
-3. It dispatches to the appropriate instrument processor based on the instrument ID.
-4. The processor downloads the raw file from S3, preprocesses it (e.g., extracting metadata), and creates/updates the run and files via the Data Hub API.
+3. For S3-triggered events, a cheap union of processor filename gates runs first. Non-matching files no-op without an API call.
+4. The handler fetches the instrument via `GET /instruments/:id` and looks up a processor by `instrument_type` in `data_hub_lambda.processors`. Unmapped types (including `generic`) and per-type gate failures no-op.
+5. The processor downloads the raw file from S3, preprocesses it, and creates/updates the run and files via the Data Hub API using the event's `instrument_id`.
 
 Slack notifications are sent by the **web app**, not the Lambda — see [Slack notifications](#slack-notifications) below.
 
 ### Function URL (manual reprocessing)
 
 When a file fails processing (or needs to be re-run), users can trigger reprocessing from the run detail page in the web app. This invokes the Lambda's Function URL instead of going through S3:
 
-1. The user clicks **Reprocess** on a failed or completed file in the web dashboard.
+1. The user clicks **Reprocess** on an uploaded, failed, or completed file in the web dashboard.
 2. The web app's `POST /api/v1/files/:fileId/reprocess` endpoint transitions the file to `processing` status, clears any previous error, and sends a POST request to the Lambda Function URL.
 3. The Function URL is configured with `AuthType: AWS_IAM`, so the web app SigV4-signs the request using credentials it gets via Vercel OIDC federation (the `WebAppS3Role` IAM role, which has `lambda:InvokeFunctionUrl` on this function's ARN). The body is a JSON payload containing a synthetic S3 event.
 4. The Lambda handler detects the Function URL invocation (via `requestContext` in the event) and parses the S3 event from the request body. Inbound auth is enforced by AWS itself in front of the function — the handler never sees an unauthenticated request.
-5. From here, processing follows the same dispatch logic as the S3 trigger path (steps 2–4 above).
+5. From here, processing follows the same type-based dispatch as the S3 trigger path, except **filename gates are skipped** — a user clicking Reprocess has stated intent, so the handler must not leave the file stranded in `processing`.
 
 ### Function URL (archive build)
 
@@ -36,43 +37,48 @@ The web app's `GET /api/v1/instruments/:instrumentId/runs/:runId/download-archiv
 
 See [Run archives](run-archives.md) for the full flow, S3 bucket layout, cache semantics, and operator runbook.
 
-## Supported instruments
+## Supported instrument types
 
-| Instrument | Module | Instrument ID |
+Dispatch is by `instrument_type` (Postgres/TS enum), not instrument ID. The registry lives in `lambda/src/data_hub_lambda/processors.py`; the web reprocess gate mirrors the same keys in `web/lib/instruments/processable-types.ts`.
+
+| `instrument_type` | Module | Filename gate (S3 events only) |
 | --- | --- | --- |
-| Agilent 4150 TapeStation | `agilent_4150_tapestation` | `agilent-4150-tapestation` |
-| Akta FPLC | `akta_fplc` | `akta-fplc` |
-| Azure 600 Gel Doc | `azure_600_gel_doc` | `azure-600-gel-doc` |
-| Azure Cielo qPCR | `azure_cielo_qpcr` | `azure-cielo-qpcr` |
-| Epson V700 Scanner | `epson_v700_scanner` | `epson-v700-scanner` |
-| Hina Microscope | `hina_microscope` | `hina-microscope` |
-| InstantRaman | _(no Lambda processor)_ | `instant-raman` |
-| SpectraMax iD3 Plate Reader | `spectramax_plate_reader` | `spectramax-id3-plate-reader` |
-| SpectraMax iD5 Plate Reader | `spectramax_plate_reader` | `spectramax-id5-plate-reader` |
-
-Each processor module exposes a `process_file()` function that accepts the run ID and filename (and instrument ID for SpectraMax readers) and reports progress back through the Data Hub API.
+| `tape_station` | `agilent_4150_tapestation` | `.pdf` |
+| `fplc` | `akta_fplc` | `.pdf` |
+| `gel_doc` | `azure_600_gel_doc` | `.tif` / `.tiff` |
+| `qpcr` | `azure_cielo_qpcr` | ends with `_cq values.csv` |
+| `epson_v700_scanner` | `epson_v700_scanner` | `.tif` / `.tiff` |
+| `hina_microscope` | `hina_microscope` | `.nd2` |
+| `plate_reader` | `spectramax_plate_reader` | `.xls` |
+| `generic`, `instant_raman` | — | — |
+
+**One type = one vendor's output format.** Names like `qpcr` and `fplc` sound generic, but the parsers behind them are vendor-specific (Azure Cielo, ÄKTA, …). Adding a second vendor under an existing type requires splitting the type, not reusing it.
+
+Seeded `jolene-fplc` stays `generic` until an operator confirms its PDFs match the ÄKTA processor and edits the type to `fplc`. Typing an unknown FPLC as `fplc` would feed non-ÄKTA files into that parser.
+
+Each processor module exposes `process_file(instrument_id, run_id, filename)` and reports progress through the Data Hub API.
 
 ## Slack notifications
 
 Slack channel notifications are sent by the **web app** (`web/lib/slack.ts`), not the Lambda. When the Lambda's `process_file` calls `POST /api/v1/instruments/:instrumentId/runs` to register a newly-detected run, that endpoint posts a single message per run to the incoming webhook URL configured in Settings > Notifications > Slack channel (workspace admins only). Subsequent files for the same run do not re-notify because the upsert is idempotent on `(instrument_id, run_id)`. File-level failures remain visible in the web app via the file row's `status='failed'` and `error_message` fields.
 
-## Adding a new instrument
+## Adding a new instrument / processor
 
-1. **Register the instrument.** Add a new member to the `Instrument` enum in `packages/shared/src/data_hub_shared/enums.py` and a corresponding entry in the `INSTRUMENT_ID_TO_NAME_MAP` in `packages/shared/src/data_hub_shared/constants.py`.
+1. **Add or reuse an `instrument_type`.** If this is a new vendor format, extend `instrumentTypeEnum` in `web/lib/db/schema.ts` and generate an `ALTER TYPE ... ADD VALUE` migration. Create the instrument row in the web app with that type (or edit an existing row). The shared `Instrument` enum in `packages/shared` is optional — only needed for watcher/CLI display naming, not for Lambda dispatch.
 
-2. **Create a processor module.** Add a new module under `lambda/src/data_hub_lambda/` (e.g., `new_instrument.py`). It must expose:
+2. **Create a processor module** under `lambda/src/data_hub_lambda/` that exposes:
 
    ```python
-   def process_file(run_id: str, filename: str) -> None:
+   def process_file(instrument_id: str, run_id: str, filename: str) -> None:
        """Process a file, reporting progress via the Data Hub API."""
        ...
    ```
 
-3. **Register the dispatch.** Add an `elif` branch in the `lambda_handler` function in `lambda/src/data_hub_lambda/handler.py` that maps the new instrument ID to your `process_file` function.
+3. **Register it** in `lambda/src/data_hub_lambda/processors.py` (type → `process_file` + `matches_filename`) and add the same type string to `PROCESSABLE_INSTRUMENT_TYPES` in `web/lib/instruments/processable-types.ts`.
 
-4. **Add tests.** Add unit tests in `lambda/tests/` for the new processor.
+4. **Add tests** for the processor and for the new registry gate.
 
-5. **Configure the S3 trigger and deploy.** See [CI and deployment → Adding an S3 trigger for a new instrument](ci-and-deployment.md#adding-an-s3-trigger-for-a-new-instrument) for the `infra/template.yaml` trigger entry and the deploy steps.
+5. **Deploy the Lambda image.** The raw bucket already notifies on all `ObjectCreated:*` events — no new S3 trigger entry is required.
 
 ## Local processing CLI
 

developer-docs/shared-library.md

@@ -6,15 +6,15 @@
 
 ### `enums`
 
-Defines the `Instrument` enum, whose kebab-case values are used as S3 key prefixes and as the canonical instrument identifiers throughout the system.
+Defines the optional `Instrument` enum for watcher/CLI display naming. Lambda dispatch and reprocess eligibility use `instrument_type` on the web app's `instruments` table, not this enum.
 
 ```python
 from data_hub_shared.enums import Instrument
 
 Instrument.AKTA_FPLC.value  # "akta-fplc"
 ```
 
-Currently supported instruments:
+Enum members available for display naming:
 
 | Enum member | Value |
 | --- | --- |
@@ -27,11 +27,7 @@ Currently supported instruments:
 
 ### `constants`
 
-Maps between instrument IDs and human-readable display names:
-
-```python
-from data_hub_shared.constants import INSTRUMENT_ID_TO_NAME_MAP, INSTRUMENT_NAME_TO_ID_MAP
-```
+Reserved for cross-package constants. Instrument display names live in the web app's `instruments` table; Lambda dispatch uses `instrument_type` rather than a shared ID→name map.
 
 ### `config`
 

infra/template.yaml

@@ -88,89 +88,12 @@ Resources:
               - Content-Length
             MaxAge: 3600
       NotificationConfiguration:
+        # Catch-all: the Lambda handler resolves instrument_type via the API
+        # and applies per-processor filename gates. Deploy the type-dispatch
+        # handler before (or with) this notification change — an older
+        # ID-dispatched handler would process every ObjectCreated event.
         LambdaConfigurations:
           - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: agilent-4150-tapestation/
-                  - Name: suffix
-                    Value: .pdf
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: akta-fplc/
-                  - Name: suffix
-                    Value: .pdf
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: azure-600-gel-doc/
-                  - Name: suffix
-                    Value: .tif
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: azure-cielo-qpcr/
-                  - Name: suffix
-                    # S3 requires spaces in filter values to be replaced
-                    # with `+` — this matches keys ending in `_Cq Values.csv`.
-                    Value: _Cq+Values.csv
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: epson-v700-scanner/
-                  - Name: suffix
-                    Value: .tif
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: epson-v700-scanner/
-                  - Name: suffix
-                    Value: .tiff
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: hina-microscope/
-                  - Name: suffix
-                    Value: .nd2
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: spectramax-id3-plate-reader/
-                  - Name: suffix
-                    Value: .xls
-            Function: !GetAtt DataHubFunction.Arn
-          - Event: s3:ObjectCreated:*
-            Filter:
-              S3Key:
-                Rules:
-                  - Name: prefix
-                    Value: spectramax-id5-plate-reader/
-                  - Name: suffix
-                    Value: .xls
             Function: !GetAtt DataHubFunction.Arn
       Tags:
         - Key: project