Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/python-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ jobs:

- name: Install Node.js packages
run: npm ci
working-directory: web-app
working-directory: web

- name: Install Python packages
run: uv sync --all-packages
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/typescript-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,23 @@ on:
- production
paths:
- .github/workflows/typescript-lint.yml
- web-app/**
- web/**
- Makefile
pull_request:
branches:
- staging
- production
paths:
- .github/workflows/typescript-lint.yml
- web-app/**
- web/**
- Makefile

jobs:
typescript-lint:
runs-on: ubuntu-latest
defaults:
run:
working-directory: web-app
working-directory: web
steps:
- uses: actions/checkout@v6

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/typescript-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ on:
- production
paths:
- .github/workflows/typescript-test.yml
- web-app/**
- web/**
- Makefile
pull_request:
branches:
- staging
- production
paths:
- .github/workflows/typescript-test.yml
- web-app/**
- web/**
- Makefile

jobs:
Expand Down Expand Up @@ -51,7 +51,7 @@ jobs:

- name: Install packages
run: npm ci
working-directory: web-app
working-directory: web

- name: Run MCP tests
run: make fe-test-mcp
Expand Down
16 changes: 8 additions & 8 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -49,35 +49,35 @@ py-check-watcher-version:
# Web app.
.PHONY: fe-format
fe-format:
cd web-app && npm run format
cd web && npm run format

.PHONY: fe-lint
fe-lint:
cd web-app && npm run lint
cd web && npm run lint

.PHONY: fe-typecheck
fe-typecheck:
cd web-app && npm run typecheck
cd web && npm run typecheck

.PHONY: fe-test-mcp
fe-test-mcp:
cd web-app && npm run test:mcp
cd web && npm run test:mcp

.PHONY: fe-test-integration
fe-test-integration:
cd web-app && npm run test:integration
cd web && npm run test:integration

.PHONY: fe-test
fe-test:
cd web-app && npm run test:mcp && npm run test:integration
cd web && npm run test:mcp && npm run test:integration

.PHONY: dev
dev:
cd web-app && npm run dev
cd web && npm run dev

.PHONY: fe-build
fe-build:
cd web-app && npm run build
cd web && npm run build

# Formatting, linting, and type checking.
.PHONY: py-check
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ flowchart LR

| Directory | Description | Docs |
| --- | --- | --- |
| `web-app/` | Next.js web application and REST API (Vercel) | [API reference](docs/api.md) |
| `web/` | Next.js web application and REST API (Vercel) | [API reference](docs/api.md) |
| `lambda/` | AWS Lambda function for instrument data processing | [Lambda docs](docs/lambda.md) |
| `watcher/` | CLI agent for lab instrument PCs | [Watcher docs](docs/watcher.md) |
| `packages/shared/` | Shared Python library (S3, enums, test infra) | [Shared library](docs/shared-library.md) |
Expand All @@ -29,13 +29,13 @@ flowchart LR
uv sync --all-packages

# Install web app dependencies.
cd web-app && npm install && cd ..
cd web && npm install && cd ..

# Set up environment variables for the web app.
cd web-app && vercel env pull && cd ..
cd web && vercel env pull && cd ..

# Create and initialize the local database.
cd web-app && createdb data-hub-local && npm run db:push && cd ..
cd web && createdb data-hub-local && npm run db:push && cd ..

# Start the dev server.
make dev
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ flowchart LR

| Directory | Package | Description |
| --- | --- | --- |
| `web-app/` | `data-hub-web-app` | Next.js web application, REST API, and MCP server. Deployed on Vercel. |
| `web/` | `data-hub-web` | Next.js web application, REST API, and MCP server. Deployed on Vercel. |
| `lambda/` | `data-hub-lambda` | AWS Lambda function triggered by S3 uploads. Runs instrument-specific processing pipelines. |
| `watcher/` | `data-hub-watcher` | CLI agent installed on lab instrument PCs. Detects new files, uploads them to S3, and reports status to the API. |
| `packages/shared/` | `data-hub-shared` | Shared Python library providing S3 utilities, instrument enums, and test infrastructure. |
Expand Down Expand Up @@ -56,7 +56,7 @@ Steps 1–3 are the same, but the watcher does not upload immediately. Instead:
- **API-driven coordination.** The watcher registers with the API, syncs its YAML config, and sends periodic heartbeats. This lets the web dashboard show watcher health and manage upload queues.
- **Presigned URLs from the API.** The web app generates presigned S3 upload and download URLs so watchers and browsers can transfer files directly to/from S3 without routing data through the API. On Vercel, the app assumes an IAM role via OIDC federation (no long-lived AWS credentials).
- **Lambda-built run archives.** The "Download all" actions on a run delegate to the Lambda, which streams files from the raw + processed buckets directly into a zip in a separate `arcadia-data-hub-archives-{env}` bucket via S3 multipart upload. The web app then 302s the browser to a short-lived presigned URL on that archive — bytes never traverse Vercel, eliminating Fast Origin Transfer for run downloads. Builds are cached at `runs/{instrument_id}/{run_id}/{fingerprint}.zip` and expire after 7 days. Every cache miss is dispatched asynchronously: the route inserts an `archive_jobs` row, schedules the Lambda invocation via `after()`, returns `202 { job_id }`, and the UI polls the same `/download-archive` URL (which HEADs S3 on each call) until the artifact appears — making the S3 object, not the row's `status`, the source of truth for "ready". See [Run archives](run-archives.md) for the full flow and runbook.
- **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `<head>` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web-app/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows all generic crawlers (with an allow-list for unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on every response.
- **Public page metadata, gated bodies.** Routes commonly shared into Slack/Notion (dashboard, instruments, instrument and run detail, settings) are reachable without a session so link unfurlers can read `<head>` metadata; the page or layout renders a `SignInRequired` CTA in place of the real body when there's no session. `/watchers/*` stays redirected to `/login` by `web/proxy.ts`. Three independent layers prevent search indexing: a `robots` field on the root layout, an `app/robots.ts` that disallows all generic crawlers (with an allow-list for unfurl bots), and an `X-Robots-Tag: noindex, nofollow` header on every response.
- **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so they stay consistent across Lambda and the watcher without duplicating code.
- **MCP for AI access.** The web app includes a [Model Context Protocol](https://modelcontextprotocol.io/) server at `/api/v1/mcp` that exposes read-only tools, resources, and prompts. AI clients (e.g. Claude Desktop, Cursor) can query instruments, runs, and system status using a personal access token.
- **Integration tests against a real server.** The shared `testing.py` module spins up a real Next.js server backed by a Postgres database, so Lambda and watcher integration tests exercise the actual API surface.
6 changes: 3 additions & 3 deletions docs/ci-and-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The Next.js app is deployed on [Vercel](https://vercel.com/arcadia-science/data-
Environment variables are managed in the Vercel dashboard and can be pulled locally with:

```sh
cd web-app
cd web
vercel env pull
```

Expand All @@ -68,7 +68,7 @@ Staging and production each have a dedicated PostgreSQL instance hosted on [Rend
Schema changes are applied with Drizzle:

```sh
cd web-app
cd web

# Generate migration files from schema changes.
npm run db:generate
Expand Down Expand Up @@ -168,7 +168,7 @@ In your GitHub repo, go to **Settings → Environments**, create a `staging` env
| `DATA_HUB_API_URL` | Base API URL for the environment |
| `DATA_HUB_API_KEY` | API key for Lambda → Data Hub authentication (also used by the Lambda's archive-job PATCH callback) |

Slack notifications are sent by the **web app** (not the Lambda) when a new run is created. Configure `SLACK_WEBHOOK_URL` per environment in the Vercel dashboard alongside the other web-app env vars listed below.
Slack notifications are sent by the **web app** (not the Lambda) when a new run is created. Configure `SLACK_WEBHOOK_URL` per environment in the Vercel dashboard alongside the other web app env vars listed below.

You'll also need the `WebAppRoleArn` and `DataHubFunctionUrl` stack outputs to configure the Vercel web app. In the Vercel dashboard (under the appropriate environment), set:

Expand Down
8 changes: 4 additions & 4 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ cd data-hub
uv sync --all-packages

# Install web app dependencies.
cd web-app && npm install && cd ..
cd web && npm install && cd ..
```

The Python workspace is managed by uv. The root `pyproject.toml` defines three workspace members — `lambda`, `watcher`, and `packages/shared` — and all are installed together by `uv sync --all-packages`.
Expand All @@ -35,7 +35,7 @@ The Python workspace is managed by uv. The root `pyproject.toml` defines three w
The web app requires the following variables. The easiest way to get them is via the Vercel CLI:

```sh
cd web-app
cd web
vercel env pull
```

Expand Down Expand Up @@ -72,7 +72,7 @@ The watcher reads its configuration from a YAML file at `~/.data-hub/config.yaml
## Database setup

```sh
cd web-app
cd web

# Create a local PostgreSQL database.
createdb data-hub-local
Expand All @@ -98,7 +98,7 @@ Other database commands:
make dev

# Or equivalently:
cd web-app && npm run dev
cd web && npm run dev
```

The app runs at [http://localhost:3000](http://localhost:3000).
Expand Down
2 changes: 1 addition & 1 deletion docs/lambda.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ Each processor module exposes a `process_file()` function that accepts the run I

## Slack notifications

Slack notifications are sent by the **web app** (`web-app/lib/slack.ts`), not the Lambda. When the Lambda's `process_file` calls `POST /api/v1/instruments/:instrumentId/runs` to register a newly-detected run, that endpoint posts a single message per run to `SLACK_WEBHOOK_URL` (configured per environment in Vercel). Subsequent files for the same run do not re-notify because the upsert is idempotent on `(instrument_id, run_id)`. File-level failures remain visible in the web app via the file row's `status='failed'` and `error_message` fields.
Slack notifications are sent by the **web app** (`web/lib/slack.ts`), not the Lambda. When the Lambda's `process_file` calls `POST /api/v1/instruments/:instrumentId/runs` to register a newly-detected run, that endpoint posts a single message per run to `SLACK_WEBHOOK_URL` (configured per environment in Vercel). Subsequent files for the same run do not re-notify because the upsert is idempotent on `(instrument_id, run_id)`. File-level failures remain visible in the web app via the file row's `status='failed'` and `error_message` fields.

## Adding a new instrument

Expand Down
4 changes: 2 additions & 2 deletions docs/run-archives.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Archives live in a separate bucket per environment, provisioned by [`infra/templ
The fingerprint is a sorted SHA-256 of `(file_id, s3_key)` pairs:

```ts
// web-app/lib/api/archive-builder.ts
// web/lib/api/archive-builder.ts
[...files].map((f) => `${f.id}:${f.s3Key}`).sort().join("|")
```

Expand All @@ -61,7 +61,7 @@ Properties this gives us:
Two simultaneous "Download all" clicks must not double-invoke the Lambda. The schema enforces this with a *partial* unique index:

```sql
-- web-app/drizzle/0016_add_archive_jobs.sql
-- web/drizzle/0016_add_archive_jobs.sql
CREATE UNIQUE INDEX archive_jobs_inflight_unique_idx
ON archive_jobs (instrument_run_id, fingerprint)
WHERE status IN ('pending', 'building');
Expand Down
4 changes: 2 additions & 2 deletions lambda/src/data_hub_lambda/handler.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ def _handle_build_archive(payload: dict[str, Any]) -> dict[str, Any]:
"""Run the archive builder and (optionally) PATCH the originating job.

Sync callers (``payload["job_id"]`` absent) get the build result inline.
Async callers send ``"job_id"`` so the web-app job row gets PATCHed when
Async callers send ``"job_id"`` so the web app job row gets PATCHed when
the build finishes — the HTTP response is the ack of acceptance and the
actual outcome is delivered out-of-band.
"""
Expand Down Expand Up @@ -246,7 +246,7 @@ def _post_archive_job_status(

Failures here are logged but never re-raised — the build itself succeeded
or failed for its own reasons, and we don't want a callback failure to
mask that. The web-app's UI does not depend on this PATCH landing
mask that. The web app's UI does not depend on this PATCH landing
either: it polls ``/download-archive`` which short-circuits on an S3
HEAD, so a finished build is downloadable the moment the multipart
upload completes.
Expand Down
10 changes: 5 additions & 5 deletions packages/shared/src/data_hub_shared/testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
_TOKEN_PREFIX = "dhub_"

# testing.py lives at packages/shared/src/data_hub_shared/ — walk up 4 levels to the repo root.
_WEB_APP_DIR = Path(__file__).resolve().parents[4] / "web-app"
_WEB_DIR = Path(__file__).resolve().parents[4] / "web"


# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -115,7 +115,7 @@ def wait_for_server(


# ---------------------------------------------------------------------------
# Token helpers — must match web-app/lib/tokens.ts
# Token helpers — must match web/lib/tokens.ts
# ---------------------------------------------------------------------------


Expand Down Expand Up @@ -230,7 +230,7 @@ def start_test_server() -> Generator[IntegrationEnv, None, None]:
# 2. Push the Drizzle schema.
subprocess.run(
["npx", "drizzle-kit", "push", "--force"],
cwd=str(_WEB_APP_DIR),
cwd=str(_WEB_DIR),
env={**os.environ, "DATABASE_URL": _DATABASE_URL},
check=True,
capture_output=True,
Expand Down Expand Up @@ -260,7 +260,7 @@ def start_test_server() -> Generator[IntegrationEnv, None, None]:

build_result = subprocess.run(
["npx", "next", "build"],
cwd=str(_WEB_APP_DIR),
cwd=str(_WEB_DIR),
env=server_env,
capture_output=True,
)
Expand All @@ -273,7 +273,7 @@ def start_test_server() -> Generator[IntegrationEnv, None, None]:

server_proc = subprocess.Popen(
["npx", "next", "start", "-p", str(port)],
cwd=str(_WEB_APP_DIR),
cwd=str(_WEB_DIR),
env=server_env,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
Expand Down
2 changes: 1 addition & 1 deletion watcher/tests/integration/test_update_check.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Integration tests: watcher update-check endpoint via DataHubClient.

The TS-side tests in `web-app/tests/integration/watchers.test.ts` cover
The TS-side tests in `web/tests/integration/watchers.test.ts` cover
the HTTP shape; this file ensures the Python `DataHubClient` plumbing
parses the response into the typed `WatcherUpdateInfoResponse` model
correctly so the CLI / in-process updater can consume it.
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
68 changes: 66 additions & 2 deletions web-app/package-lock.json → web/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading