Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions docs/guides/upgrading-the-watcher.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ If you want a specific PC to stay on a particular release rather than tracking t
uv tool install data-hub-watcher==<pinned>
```

Run `data-hub-watcher self-update --check` afterwards to confirm what the server's target is. As long as the pinned version matches `latest_version`, the auto-update tick is a no-op. The moment the server's target moves past your pin, the next tick will try to upgrade past it again — pinning is per-machine state, not server-side state. To park a fleet on a given version intentionally, the right knob is the server-side `WATCHER_LATEST_VERSION` env var (see [Cutting a new release](#cutting-a-new-release)).
Run `data-hub-watcher self-update --check` afterwards to confirm what the server's target is. As long as the pinned version matches `latest_version`, the auto-update tick is a no-op. The moment the server's target moves past your pin, the next tick will try to upgrade past it again — pinning is per-machine state, not server-side state. To park a fleet on a given version intentionally, the right knob is the server-side **Latest version** field on the **Settings → Watchers** page in Data Hub (see [Cutting a new release](#cutting-a-new-release)).

## Cutting a new release

Expand Down Expand Up @@ -99,26 +99,26 @@ Manual dispatch from any branch other than `production` is refused by the workfl

### 4. Roll the release out

Once the new version is live on PyPI, bump the server-side `WATCHER_LATEST_VERSION` env var in Vercel (per environment) so the `/update-check` endpoint advertises the new target. Lab PCs running an auto-update-capable build will pick it up on their next hourly tick.
Once the new version is live on PyPI, open **Settings → Watchers** in Data Hub (admins only) and set **Latest version** to the new release. Save; the `/update-check` endpoint will start advertising the new target immediately. Lab PCs running an auto-update-capable build will pick it up on their next hourly tick.

The supported watcher release env vars are:
The supported watcher release fields are:


| Env var | Purpose |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `WATCHER_LATEST_VERSION` | Required to advertise a release. `null`/unset means "no update info available" and the watcher skips its update attempt. |
| `WATCHER_MIN_SUPPORTED_VERSION` | Optional floor; surfaced in the response for future use. Not yet enforced server-side. |
| `WATCHER_RELEASE_CHANNEL` | Defaults to `stable`. Surfaced in the response and shown in `self-update` output. |
| `WATCHER_MANDATORY_UPDATE` | Set to `true` / `1` to flag the release as mandatory (see below). |
| Field | Purpose |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| **Latest version** | Required to advertise a release. Leaving it blank means "no update info available" and the watcher skips its update attempt. |
| **Minimum supported version** | Optional floor; surfaced in the response for future use. Not yet enforced server-side. |
| **Release channel** | Defaults to `stable`. Surfaced in the response and shown in `self-update` output. |
| **Mandatory update** | Toggle on to flag the release as mandatory (see below). |


Do **not** bump `WATCHER_LATEST_VERSION` ahead of the PyPI publish — the watcher's upgrade subprocess will fail to resolve a version that doesn't yet exist on the index, and you'll see a wave of `update_failed` events from the fleet. Always: tag → publish → verify → bump env var.
Do **not** save a **Latest version** ahead of the PyPI publish — the watcher's upgrade subprocess will fail to resolve a version that doesn't yet exist on the index, and you'll see a wave of `update_failed` events from the fleet. Always: tag → publish → verify → save.

If the rollout needs to be paged through (e.g. you want only `staging` lab PCs to see the new version while you babysit it for a day), bump only the staging environment's `WATCHER_LATEST_VERSION`. The `production` environment keeps advertising the previous version until you bump it explicitly.
If the rollout needs to be paged through (e.g. you want only `staging` lab PCs to see the new version while you babysit it for a day), save the new version on the staging deployment only. Each Vercel environment has its own database, so the `production` deployment keeps advertising the previous version until you also open production's settings page and save the bump there.

## Mandatory updates

By default the activity-window guard means a watcher mid-acquisition won't auto-update — it'll wait for the next idle window. For releases that fix a security issue, a wire-protocol break, or any other case where running the known-bad version is worse than a brief outage, set `WATCHER_MANDATORY_UPDATE=true` alongside the version bump. Mandatory rollouts skip the activity-window guard and fire on the very next hourly check on every lab PC.
By default the activity-window guard means a watcher mid-acquisition won't auto-update — it'll wait for the next idle window. For releases that fix a security issue, a wire-protocol break, or any other case where running the known-bad version is worse than a brief outage, toggle **Mandatory update** on alongside the version bump on the **Settings → Watchers** page. Mandatory rollouts skip the activity-window guard and fire on the very next hourly check on every lab PC.

Use this sparingly. The activity-window guard exists for a reason — a forced upgrade in the middle of a multi-hour microscopy run will lose data. Reserve it for cases where leaving the bad version in place is strictly worse than restarting the watcher in flight.

Expand All @@ -128,10 +128,10 @@ Note also that mandatory rollouts are versioned, not absolute: the server compar

Rollback is just another release. To revert the fleet from `0.3.0` to `0.2.5`:

1. Set `WATCHER_LATEST_VERSION=0.2.5` in Vercel for the affected environment(s).
2. Set `WATCHER_MANDATORY_UPDATE=true` if you need the rollback to bypass the activity-window guard (most rollback scenarios qualify — you're rolling back precisely because the running version is misbehaving).
1. Open **Settings → Watchers** in Data Hub and set **Latest version** to `0.2.5` for the affected environment(s).
2. Toggle **Mandatory update** on if you need the rollback to bypass the activity-window guard (most rollback scenarios qualify — you're rolling back precisely because the running version is misbehaving).
3. Wait for the next hourly tick on each PC. Lab PCs running an auto-update-capable build will downgrade themselves; PCs being upgraded manually need a `data-hub-watcher self-update` (or an `uv tool install data-hub-watcher==0.2.5` if `self-update` itself is what's broken).
4. Once the fleet has converged, set `WATCHER_MANDATORY_UPDATE` back to `false`.
4. Once the fleet has converged, toggle **Mandatory update** back off.

There is no separate "yank" step — a rolled-back release is still on PyPI and still reinstallable, just not advertised by `/update-check`.

Expand All @@ -149,20 +149,20 @@ The upgrade subprocess started but didn't end up running the new version on the

When `details.attempted_subprocess` is `false`, the auto-updater never ran the upgrade command — it refused before starting one. The `details.reason` field tells you why:

- `**install method '<editable|unknown>' not eligible for auto-update**` — the watcher detected a development-style install (editable `uv sync`, or a distribution whose metadata couldn't be located) and refused so it wouldn't silently shadow the source tree with an index build. Resolve by switching the host to a PyPI install (`uv tool install data-hub-watcher`) or, on a developer machine, ignoring the event. To avoid spamming the events stream, the watcher emits this at most once per server target — a rebump of `WATCHER_LATEST_VERSION` will trigger one fresh event per stuck PC.
- `**install method '<editable|unknown>' not eligible for auto-update**` — the watcher detected a development-style install (editable `uv sync`, or a distribution whose metadata couldn't be located) and refused so it wouldn't silently shadow the source tree with an index build. Resolve by switching the host to a PyPI install (`uv tool install data-hub-watcher`) or, on a developer machine, ignoring the event. To avoid spamming the events stream, the watcher emits this at most once per server target — saving a new **Latest version** on the **Settings → Watchers** page will trigger one fresh event per stuck PC.

### Auto-update never fires

Check, in order:

- The watcher is running as a service, not in a console window. Foreground `data-hub-watcher watch` does run the in-process updater, but on a developer-style install it'll typically be refused as editable.
- The environment isn't `preview`. Auto-update is hard-disabled there.
- The activity-window guard isn't holding things up. The instrument has to have been quiet for several heartbeats; on a busy plate reader you may simply never hit the idle window. Use the CLI path (`data-hub-watcher self-update`) for an immediate upgrade, or set `WATCHER_MANDATORY_UPDATE=true` if the release warrants it.
- The activity-window guard isn't holding things up. The instrument has to have been quiet for several heartbeats; on a busy plate reader you may simply never hit the idle window. Use the CLI path (`data-hub-watcher self-update`) for an immediate upgrade, or toggle **Mandatory update** on the **Settings → Watchers** page if the release warrants it.
- The dashboard's **Last Heartbeat** is recent. If the watcher has gone stale, it's not ticking and won't auto-update.

### `data-hub-watcher self-update --check` says "(none configured)"

`WATCHER_LATEST_VERSION` is unset for that environment. Either you're running against a `preview` build that doesn't have the env var set, or someone unset it in Vercel. The CLI returns successfully and treats this as "no update available" — the same response code path used by an up-to-date watcher — so this is benign, just informational.
The **Settings → Watchers** page has a blank **Latest version** for that environment. Either you're running against a `preview` build that has never had a release saved, or an admin cleared the field in Data Hub. The CLI returns successfully and treats this as "no update available" — the same response code path used by an up-to-date watcher — so this is benign, just informational.

### "Refusing to self-update an editable / unknown install"

Expand Down
48 changes: 42 additions & 6 deletions packages/shared/src/data_hub_shared/testing.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,40 @@ def seed_auth(dsn: str) -> str:
return token_plaintext


def seed_watcher_release(
dsn: str,
*,
latest_version: str = "9.9.9",
min_supported_version: str = "0.1.0",
channel: str = "stable",
mandatory: bool = False,
) -> None:
"""Upsert the singleton ``watcher_release_config`` row.

The ``update-check`` endpoint reads from this table; tests assert on
the seeded values. Previously these were plumbed via ``WATCHER_*``
env vars; the source of truth is now the DB, edited via the
admin-only ``/settings/watchers`` page in the web app.
"""
conn = psycopg2.connect(dsn)
conn.autocommit = True
with conn.cursor() as cur:
cur.execute(
"""INSERT INTO watcher_release_config
(id, latest_version, min_supported_version, channel, mandatory)
VALUES
(true, %s, %s, %s, %s)
ON CONFLICT (id) DO UPDATE SET
latest_version = EXCLUDED.latest_version,
min_supported_version = EXCLUDED.min_supported_version,
channel = EXCLUDED.channel,
mandatory = EXCLUDED.mandatory,
updated_at = now()""",
(latest_version, min_supported_version, channel, mandatory),
)
conn.close()


def seed_instruments(dsn: str, instruments: dict[str, str]) -> None:
"""Insert instrument rows (ON CONFLICT DO NOTHING)."""
conn = psycopg2.connect(dsn)
Expand Down Expand Up @@ -236,6 +270,12 @@ def start_test_server() -> Generator[IntegrationEnv, None, None]:
capture_output=True,
)

# 2a. Seed the singleton watcher_release_config row so the
# update-check endpoint returns deterministic values during
# integration tests. Previously this came from WATCHER_* env
# vars; the source of truth is now the DB.
seed_watcher_release(_PG_TEST_DSN)

# 3. Build and start the Next.js production server.
port = get_free_port()
base_url = f"http://127.0.0.1:{port}"
Expand All @@ -250,12 +290,8 @@ def start_test_server() -> Generator[IntegrationEnv, None, None]:
"AWS_SECRET_ACCESS_KEY": os.environ.get("AWS_SECRET_ACCESS_KEY", "test-secret"),
"AWS_REGION": os.environ.get("AWS_REGION", "us-east-1"),
"S3_RAW_DATA_BUCKET": os.environ.get("S3_RAW_DATA_BUCKET", "data-hub-test-raw"),
# Stable defaults for the watcher update-check endpoint so Python
# integration tests can assert on a known target version.
"WATCHER_LATEST_VERSION": os.environ.get("WATCHER_LATEST_VERSION", "9.9.9"),
"WATCHER_MIN_SUPPORTED_VERSION": os.environ.get("WATCHER_MIN_SUPPORTED_VERSION", "0.1.0"),
"WATCHER_RELEASE_CHANNEL": os.environ.get("WATCHER_RELEASE_CHANNEL", "stable"),
"WATCHER_MANDATORY_UPDATE": os.environ.get("WATCHER_MANDATORY_UPDATE", "false"),
# Watcher release defaults are seeded into watcher_release_config
# above (see seed_watcher_release); the env-var fallback is gone.
}

build_result = subprocess.run(
Expand Down
5 changes: 3 additions & 2 deletions watcher/tests/test_updater.py
Original file line number Diff line number Diff line change
Expand Up @@ -767,8 +767,9 @@ def test_editable_refusal_throttled_to_one_event_per_target(
def test_editable_refusal_re_emits_when_target_advances(
self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# When the server bumps `WATCHER_LATEST_VERSION` to a new
# release, we *do* want to re-notify — the dashboard event for
# When the server advertises a new release (via the admin-only
# /settings/watchers page), we *do* want to re-notify —
# the dashboard event for
# the old target doesn't tell the admin this PC is missing the
# new one. Throttling is per-target, not "fire at most once
# ever".
Expand Down
13 changes: 5 additions & 8 deletions web/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,8 @@ LAMBDA_FUNCTION_URL=
# Leave empty to disable Slack notifications.
SLACK_WEBHOOK_URL=

# Watcher release metadata served by GET /api/v1/watchers/:id/update-check.
# Leave WATCHER_LATEST_VERSION unset to disable self-updates (the endpoint
# returns latest_version: null and clients skip the upgrade). Set
# WATCHER_MANDATORY_UPDATE to "true" or "1" to force the rollout.
WATCHER_LATEST_VERSION=
WATCHER_MIN_SUPPORTED_VERSION=
WATCHER_RELEASE_CHANNEL=stable
WATCHER_MANDATORY_UPDATE=false
# Watcher release metadata is now stored in the `watcher_release_config`
# table and edited at /settings/watchers (admin-only). Until an admin
# saves a row, GET /api/v1/watchers/:id/update-check returns
# latest_version: null and watchers skip the self-update — the same
# behavior as leaving WATCHER_LATEST_VERSION unset used to be.
Loading