Slack: Move run-creation notifications from Lambda to web app

.github/workflows/deploy-lambda.yml

@@ -65,7 +65,6 @@ jobs:
               "EcrImageUri=${IMAGE_URI}" \
               "DataHubApiUrl=${{ secrets.DATA_HUB_API_URL }}" \
               "DataHubApiKey=${{ secrets.DATA_HUB_API_KEY }}" \
-              "SlackWebhookUrl=${{ secrets.SLACK_WEBHOOK_URL }}" \
               "Environment=${{ github.ref_name }}" \
               "GitHubOidcProviderArn=${{ secrets.GH_OIDC_PROVIDER_ARN }}" \
               "VercelOidcProviderArn=${{ secrets.VERCEL_OIDC_PROVIDER_ARN }}" \

Makefile

@@ -146,7 +146,6 @@ endif
 		"EcrImageUri=$(ECR_IMAGE_URI)" \
 		"DataHubApiUrl=$(DATA_HUB_API_URL)" \
 		"DataHubApiKey=$(DATA_HUB_API_KEY)" \
-		"SlackWebhookUrl=$(SLACK_WEBHOOK_URL)" \
 		"GitHubOidcProviderArn=$(GITHUB_OIDC_PROVIDER_ARN)" \
 		"VercelOidcProviderArn=$(VERCEL_OIDC_PROVIDER_ARN)" \
 		"LambdaInvokeToken=$(LAMBDA_INVOKE_TOKEN)"

README.md

@@ -19,7 +19,7 @@ flowchart LR
 | `web-app/` | Next.js web application and REST API (Vercel) | [API reference](docs/api.md) |
 | `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](docs/lambda.md) |
 | `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](docs/watcher.md) |
-| `packages/shared/` | Shared Python library (S3, enums, Slack, test infra) | [Shared library](docs/shared-library.md) |
+| `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](docs/shared-library.md) |
 | `docs/` | Project documentation | — |
 
 ## Quick start

docs/architecture.md

@@ -27,7 +27,7 @@ flowchart LR
 | `web-app/` | `data-hub-web-app` | Next.js web application, REST API, and MCP server. Deployed on Vercel. |
 | `lambda/` | `data-hub-lambda` | AWS Lambda function triggered by S3 uploads. Runs instrument-specific processing pipelines. |
 | `watcher/` | `data-hub-watcher` | CLI agent installed on lab instrument PCs. Detects new files, uploads them to S3, and reports status to the API. |
-| `packages/shared/` | `data-hub-shared` | Shared Python library providing S3 utilities, instrument enums, Slack integration, and test infrastructure. |
+| `packages/shared/` | `data-hub-shared` | Shared Python library providing S3 utilities, instrument enums, and test infrastructure. |
 
 ## Data flow
 
@@ -39,7 +39,7 @@ flowchart LR
 4. It uploads raw files to **S3** at the key `{instrument_id}/{run_id}/{filename}`.
 5. The S3 upload triggers the **Lambda** function.
 6. Lambda downloads the file and dispatches to the appropriate instrument processor for preprocessing (e.g., extracting metadata).
-7. Lambda creates/updates the run and files via the **API** and sends a **Slack** notification.
+7. Lambda creates/updates the run and files via the **API**. The API sends a **Slack** notification once per newly-created run.
 8. Users view the run in the **web dashboard**.
 
 ### Manual upload (manual mode)

docs/ci-and-deployment.md

@@ -133,7 +133,6 @@ cp infra/.env.example infra/.env.staging
 ECR_IMAGE_URI=<image-uri-from-step-2>
 DATA_HUB_API_URL=https://data-hub-env-staging-arcadia-science.vercel.app/api/v1
 DATA_HUB_API_KEY=<your-api-key>
-SLACK_WEBHOOK_URL=<your-slack-webhook>
 GITHUB_OIDC_PROVIDER_ARN=<github-oidc-arn-from-step-1>
 VERCEL_OIDC_PROVIDER_ARN=<vercel-oidc-arn-from-step-1>
 LAMBDA_INVOKE_TOKEN=<shared-secret-for-function-url-auth>
@@ -169,9 +168,10 @@ In your GitHub repo, go to **Settings → Environments**, create a `staging` env
 | `SAM_S3_BUCKET` | SAM CLI managed S3 bucket name (see `sam deploy` output, e.g. `aws-sam-cli-managed-default-samclisourcebucket-*`) |
 | `DATA_HUB_API_URL` | Base API URL for the environment |
 | `DATA_HUB_API_KEY` | API key for Lambda → Data Hub authentication |
-| `SLACK_WEBHOOK_URL` | Slack incoming webhook URL |
 | `LAMBDA_INVOKE_TOKEN` | Shared secret for web app → Lambda Function URL authentication |
 
+Slack notifications are sent by the **web app** (not the Lambda) when a new run is created. Configure `SLACK_WEBHOOK_URL` per environment in the Vercel dashboard alongside the other web-app env vars listed below.
+
 You'll also need the `WebAppRoleArn` and `DataHubFunctionUrl` stack outputs to configure the Vercel web app. In the Vercel dashboard (under the appropriate environment), set:
 
 | Vercel env var | Value |
@@ -190,7 +190,7 @@ On pushes to `staging` or `production`, the **Deploy Lambda** workflow:
 2. Builds and pushes the Docker image to ECR.
 3. Runs `sam deploy` to update the CloudFormation stack.
 
-Secrets (`DATA_HUB_API_KEY`, `SLACK_WEBHOOK_URL`, etc.) are stored in GitHub environment secrets scoped to each environment.
+Secrets (`DATA_HUB_API_KEY`, `LAMBDA_INVOKE_TOKEN`, etc.) are stored in GitHub environment secrets scoped to each environment.
 
 > **Note:** The CI deploy role has intentionally narrow permissions — enough to push a new container image, update the existing CloudFormation stack, and modify the data buckets' S3 event notifications (so new instrument triggers roll out via CI), but _not_ enough to create the stack from scratch or to add/remove S3 buckets or Lambda functions. Initial stack creation and structural infrastructure changes must be performed by an admin with broader AWS permissions. Once the stack exists, routine image-update deploys and new-trigger rollouts through CI work without issue.
 

docs/getting-started.md

@@ -48,6 +48,7 @@ vercel env pull
 | `AWS_REGION` | No | AWS region for S3 presigned URLs (defaults to `us-west-1`) |
 | `AWS_ROLE_ARN` | No | IAM role ARN for Vercel OIDC federation to S3 (only needed on Vercel) |
 | `S3_RAW_DATA_BUCKET` | No | S3 bucket for raw data uploads (defaults to `arcadia-data-hub-raw-staging`) |
+| `SLACK_WEBHOOK_URL` | No | Slack incoming webhook URL — when set, the web app posts a notification each time a new run is created |
 
 ### Lambda / shared library
 
@@ -60,7 +61,6 @@ These are set in the Lambda runtime environment:
 | `AWS_SECRET_ACCESS_KEY` | No | AWS credentials |
 | `AWS_S3_RAW_DATA_BUCKET` | No | S3 bucket for raw data |
 | `AWS_S3_PROCESSED_DATA_BUCKET` | No | S3 bucket for processed data |
-| `SLACK_WEBHOOK_URL` | No | Slack webhook for notifications |
 
 ### Watcher
 

docs/guides/adding-an-instrument.md

@@ -76,8 +76,8 @@ lambda/src/data_hub_lambda/bio_rad_cfx96/
 The processor must expose a `process_file` function:
 
 ```python
-def process_file(run_id: str, filename: str) -> str:
-    """Preprocess a file and return the URL to the run in the web dashboard."""
+def process_file(run_id: str, filename: str) -> None:
+    """Preprocess a file, reporting progress through the Data Hub API."""
     ...
 ```
 
@@ -90,7 +90,6 @@ A typical processor:
 5. Performs instrument-specific preprocessing (parsing, metadata extraction, etc.).
 6. Optionally uploads processed artifacts (CSV, images) to the S3 processed bucket and registers them via `client.create_file(..., category="processed")`. This is the pattern used by the SpectraMax plate reader (processed CSV) and Azure 600 Gel Doc (contrast-enhanced PNG).
 7. Updates the raw file status to `completed`.
-8. Returns the web app URL for the run.
 
 See any existing processor (e.g., `lambda/src/data_hub_lambda/azure_cielo_qpcr/process_file.py` for simple metadata extraction, or `lambda/src/data_hub_lambda/spectramax_plate_reader/process_file.py` for the processed-artifact pattern) for complete examples.
 
@@ -100,7 +99,7 @@ Add an `elif` branch in the `lambda_handler` function in `lambda/src/data_hub_la
 
 ```python
 elif instrument_id == Instrument.BIO_RAD_CFX96.value:
-    result_url = bio_rad_cfx96.process_file(
+    bio_rad_cfx96.process_file(
         run_id=event_info.run_id,
         filename=event_info.filename,
     )
@@ -139,4 +138,4 @@ Even without Step 4, you get a fully functional instrument in Data Hub:
 - Files are downloadable via pre-signed S3 URLs.
 - Watcher health monitoring (heartbeats, events) works in the dashboard.
 
-Lambda preprocessing adds automated metadata extraction and Slack notifications on top of that.
+Lambda preprocessing adds automated metadata extraction on top of that. (Slack notifications fire from the web app whenever a new run is created, regardless of whether a Lambda processor exists for the instrument.)

docs/lambda.md

@@ -1,6 +1,6 @@
 # Lambda
 
-The Data Hub Lambda function preprocesses raw instrument data uploaded to S3 and builds run-archive zips on demand for the web app's "Download all" actions. It can be triggered automatically by S3 events or manually via Function URL invocations from the web app. It runs an instrument-specific preprocessing pipeline (or the archive builder) and reports results back through the API and Slack.
+The Data Hub Lambda function preprocesses raw instrument data uploaded to S3 and builds run-archive zips on demand for the web app's "Download all" actions. It can be triggered automatically by S3 events or manually via Function URL invocations from the web app. It runs an instrument-specific preprocessing pipeline (or the archive builder) and reports results back through the API.
 
 ## How it works
 
@@ -12,8 +12,8 @@ The Lambda has three invocation paths:
 2. The handler parses the S3 key to extract the instrument ID, run ID, and filename. The expected key layout is `{instrument_id}/{run_id}/{filename}`.
 3. It dispatches to the appropriate instrument processor based on the instrument ID.
 4. The processor downloads the raw file from S3, preprocesses it (e.g., extracting metadata), and creates/updates the run and files via the Data Hub API.
-5. On success, a Slack message is sent with a link to view the run in the web dashboard.
-6. On failure, a Slack message is sent with a link to the CloudWatch logs.
+
+Slack notifications are sent by the **web app**, not the Lambda — see [Slack notifications](#slack-notifications) below.
 
 ### Function URL (manual reprocessing)
 
@@ -23,7 +23,7 @@ When a file fails processing (or needs to be re-run), users can trigger reproces
 2. The web app's `POST /api/v1/files/:fileId/reprocess` endpoint transitions the file to `processing` status, clears any previous error, and sends a POST request to the Lambda Function URL.
 3. The request includes an `Authorization: Bearer <LAMBDA_INVOKE_TOKEN>` header and a JSON body containing a synthetic S3 event payload.
 4. The Lambda handler detects the Function URL invocation (via `requestContext` in the event), verifies the Bearer token against the `LAMBDA_INVOKE_TOKEN` environment variable using constant-time comparison, and parses the S3 event from the request body.
-5. From here, processing follows the same dispatch logic as the S3 trigger path (steps 2–6 above).
+5. From here, processing follows the same dispatch logic as the S3 trigger path (steps 2–4 above).
 
 ### Function URL (archive build)
 
@@ -49,7 +49,11 @@ See [Run archives](run-archives.md) for the full flow, S3 bucket layout, cache s
 | SpectraMax iD3 Plate Reader | `spectramax_plate_reader` | `spectramax-id3-plate-reader` |
 | SpectraMax iD5 Plate Reader | `spectramax_plate_reader` | `spectramax-id5-plate-reader` |
 
-Each processor module exposes a `process_file()` function that accepts the run ID and filename (and instrument ID for SpectraMax readers) and returns a URL to the run in the web dashboard.
+Each processor module exposes a `process_file()` function that accepts the run ID and filename (and instrument ID for SpectraMax readers) and reports progress back through the Data Hub API.
+
+## Slack notifications
+
+Slack notifications are sent by the **web app** (`web-app/lib/slack.ts`), not the Lambda. When the Lambda's `process_file` calls `POST /api/v1/instruments/:instrumentId/runs` to register a newly-detected run, that endpoint posts a single message per run to `SLACK_WEBHOOK_URL` (configured per environment in Vercel). Subsequent files for the same run do not re-notify because the upsert is idempotent on `(instrument_id, run_id)`. File-level failures remain visible in the web app via the file row's `status='failed'` and `error_message` fields.
 
 ## Adding a new instrument
 
@@ -58,8 +62,8 @@ Each processor module exposes a `process_file()` function that accepts the run I
 2. **Create a processor module.** Add a new module under `lambda/src/data_hub_lambda/` (e.g., `new_instrument.py`). It must expose:
 
    ```python
-   def process_file(run_id: str, filename: str) -> str:
-       """Process a file and return the URL to the run in the web dashboard."""
+   def process_file(run_id: str, filename: str) -> None:
+       """Process a file, reporting progress via the Data Hub API."""
        ...
    ```
 
@@ -121,4 +125,4 @@ The Lambda function depends on a scientific Python stack:
 - `pydantic` — data validation
 - `requests` — HTTP client for the Data Hub API
 - `aws-lambda-typing` — type stubs for Lambda events/context
-- `data-hub-shared` — shared utilities (S3, Slack, enums)
+- `data-hub-shared` — shared utilities (S3, enums)

docs/shared-library.md

@@ -41,7 +41,6 @@ Shared environment-based configuration. Instantiated as a module-level singleton
 from data_hub_shared.config import config
 
 config.AWS_S3_RAW_DATA_BUCKET
-config.SLACK_WEBHOOK_URL
 ```
 
 | Attribute | Source env var | Default |
@@ -53,7 +52,6 @@ config.SLACK_WEBHOOK_URL
 | `AWS_SESSION_TOKEN` | `AWS_SESSION_TOKEN` | `None` |
 | `AWS_S3_RAW_DATA_BUCKET` | `AWS_S3_RAW_DATA_BUCKET` | `None` |
 | `AWS_S3_PROCESSED_DATA_BUCKET` | `AWS_S3_PROCESSED_DATA_BUCKET` | `None` |
-| `SLACK_WEBHOOK_URL` | `SLACK_WEBHOOK_URL` | `None` |
 
 ### `s3_utils`
 
@@ -69,18 +67,6 @@ Boto3-based S3 utilities. Callers can pass an explicit `s3_client` or let the mo
 | `upload_folder(local_path, s3_uri_prefix)` | Upload all files in a directory |
 | `get_content_type(file_path)` | Guess MIME type for a file |
 
-### `slack`
-
-Posts messages to a Slack channel via a webhook URL:
-
-```python
-from data_hub_shared import slack
-
-slack.send_message("Hello from Data Hub!")
-```
-
-If `SLACK_WEBHOOK_URL` is not set, messages are silently skipped with a warning log.
-
 ### `logger`
 
 Provides a `get_named_logger(name)` helper for consistent logging setup.
@@ -98,5 +84,4 @@ Integration test infrastructure used by both Lambda and watcher test suites. Pro
 ## Dependencies
 
 - `boto3` — AWS SDK for S3 operations
-- `requests` — HTTP client for Slack webhooks
 - `psycopg2-binary` — PostgreSQL driver (dev dependency, used by `testing.py`)

infra/.env.example

@@ -4,7 +4,6 @@
 ECR_IMAGE_URI=
 DATA_HUB_API_URL=
 DATA_HUB_API_KEY=
-SLACK_WEBHOOK_URL=
 GITHUB_OIDC_PROVIDER_ARN=
 VERCEL_OIDC_PROVIDER_ARN=
 LAMBDA_INVOKE_TOKEN=

infra/template.yaml

@@ -32,11 +32,6 @@ Parameters:
     NoEcho: true
     Description: API key for Lambda to authenticate with the Data Hub API
 
-  SlackWebhookUrl:
-    Type: String
-    NoEcho: true
-    Description: Slack incoming webhook URL for Lambda notifications
-
   LambdaInvokeToken:
     Type: String
     NoEcho: true
@@ -223,7 +218,6 @@ Resources:
           AWS_S3_ARCHIVES_BUCKET: !Sub "arcadia-data-hub-archives-${Environment}"
           DATA_HUB_API_URL: !Ref DataHubApiUrl
           DATA_HUB_API_KEY: !Ref DataHubApiKey
-          SLACK_WEBHOOK_URL: !Ref SlackWebhookUrl
           LAMBDA_INVOKE_TOKEN: !Ref LambdaInvokeToken
 
   # ---- S3 → Lambda event wiring ----

lambda/.env.example

@@ -13,8 +13,4 @@ AWS_S3_PROCESSED_DATA_BUCKET=
 # - access to push images to the ECR repository
 # - access to update the Lambda function code
 AWS_ACCESS_KEY_ID=
-AWS_SECRET_ACCESS_KEY=
-
-# This is the Slack webhook URL for the channel to send messages to on workflow completion.
-# See: https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/.
-SLACK_WEBHOOK_URL=
+AWS_SECRET_ACCESS_KEY=

lambda/src/data_hub_lambda/agilent_4150_tapestation/process_file.py

@@ -3,7 +3,6 @@
 
 from data_hub_lambda.agilent_4150_tapestation.utils import parse_tape_type
 from data_hub_lambda.api_client import get_client
-from data_hub_lambda.constants import DATA_HUB_WEB_URL
 from data_hub_shared import s3_utils
 from data_hub_shared.config import config
 from data_hub_shared.enums import Instrument
@@ -13,15 +12,12 @@
 INSTRUMENT_ID = Instrument.AGILENT_4150_TAPESTATION.value
 
 
-def process_file(run_id: str, filename: str) -> str:
+def process_file(run_id: str, filename: str) -> None:
     """Process a single Agilent 4150 TapeStation file through the Data Hub API.
 
     Args:
         run_id: The run ID (`YYYY-MM-DD - HH-MM-SS` prefix).
         filename: The original filename (e.g. `2026-02-18 - 18-00-04-gDNA_peakTable.csv`).
-
-    Returns:
-        The web app URL for the instrument run.
     """
     logger.info("Processing Agilent 4150 TapeStation file: %s (run: %s)", filename, run_id)
 
@@ -62,5 +58,3 @@ def process_file(run_id: str, filename: str) -> str:
         logger.error("Error processing file: %s", e)
         client.update_file(file_id, status="failed", error_message=str(e))
         raise
-
-    return f"{DATA_HUB_WEB_URL}/instruments/{INSTRUMENT_ID}/runs/{run_id}"

lambda/src/data_hub_lambda/akta_fplc/process_file.py

@@ -2,7 +2,6 @@
 import logging
 
 from data_hub_lambda.api_client import get_client
-from data_hub_lambda.constants import DATA_HUB_WEB_URL
 from data_hub_shared import s3_utils
 from data_hub_shared.config import config
 from data_hub_shared.enums import Instrument
@@ -12,15 +11,12 @@
 INSTRUMENT_ID = Instrument.AKTA_FPLC.value
 
 
-def process_file(run_id: str, filename: str) -> str:
+def process_file(run_id: str, filename: str) -> None:
     """Process a single Akta FPLC file through the Data Hub API.
 
     Args:
         run_id: The run ID (filename stem).
         filename: The original filename (e.g. `2025-09-23_test.pdf`).
-
-    Returns:
-        The web app URL for the instrument run.
     """
     logger.info("Processing Akta FPLC file: %s (run: %s)", filename, run_id)
 
@@ -54,5 +50,3 @@ def process_file(run_id: str, filename: str) -> str:
         logger.error("Error processing file: %s", e)
         client.update_file(file_id, status="failed", error_message=str(e))
         raise
-
-    return f"{DATA_HUB_WEB_URL}/instruments/{INSTRUMENT_ID}/runs/{run_id}"

lambda/src/data_hub_lambda/azure_600_gel_doc/process_file.py

@@ -4,7 +4,6 @@
 from data_hub_lambda.api_client import get_client
 from data_hub_lambda.azure_600_gel_doc.image_processing import TIFFProcessor
 from data_hub_lambda.azure_600_gel_doc.parse_metadata import parse_metadata
-from data_hub_lambda.constants import DATA_HUB_WEB_URL
 from data_hub_shared import s3_utils
 from data_hub_shared.config import config
 from data_hub_shared.enums import Instrument
@@ -14,7 +13,7 @@
 INSTRUMENT_ID = Instrument.AZURE_600_GEL_DOC.value
 
 
-def process_file(run_id: str, filename: str) -> str:
+def process_file(run_id: str, filename: str) -> None:
     """Process a single Azure 600 Gel Doc file through the Data Hub API.
 
     Downloads the raw TIFF, runs it through the image processing pipeline
@@ -24,9 +23,6 @@ def process_file(run_id: str, filename: str) -> str:
     Args:
         run_id: The run ID (filename stem).
         filename: The original filename (e.g. `26.04.01_16.51.59.tif`).
-
-    Returns:
-        The web app URL for the instrument run.
     """
     logger.info("Processing Azure 600 Gel Doc file: %s (run: %s)", filename, run_id)
 
@@ -87,5 +83,3 @@ def process_file(run_id: str, filename: str) -> str:
         logger.error("Error processing file: %s", e)
         client.update_file(file_id, status="failed", error_message=str(e))
         raise
-
-    return f"{DATA_HUB_WEB_URL}/instruments/{INSTRUMENT_ID}/runs/{run_id}"