Watcher: Per-environment .env files, file creation time, API key sanitization, and Windows service auto-restart

.github/workflows/python-test.yml

@@ -9,6 +9,7 @@ on:
       - .github/workflows/python-test.yml
       - lambda/**
       - packages/**
+      - watcher/**
       - pyproject.toml
       - uv.lock
       - Makefile
@@ -20,6 +21,7 @@ on:
       - .github/workflows/python-test.yml
       - lambda/**
       - packages/**
+      - watcher/**
       - pyproject.toml
       - uv.lock
       - Makefile

docs/guides/installing-a-watcher.md

@@ -41,7 +41,7 @@ The wizard will walk you through:
 
 1. **Environment** — choose `staging` (for testing), `production`, or `preview` (for testing against a Vercel preview deployment). If you choose `preview`, you'll be prompted for the deployment's API base URL (e.g. `https://data-hub-git-my-branch.vercel.app/api/v1`).
 
-2. **API key** — paste the personal access token. The key is saved to `~/.data-hub/.env`. You can also set the `DATA_HUB_API_KEY` environment variable before running `init` to skip this prompt.
+2. **API key** — paste the personal access token. The key is saved to `~/.data-hub/.env.<environment>` (e.g. `~/.data-hub/.env.staging`), so each environment keeps its own key and you can switch between them by re-running `init` without re-entering credentials. You can also set the `DATA_HUB_API_KEY` environment variable before running `init` to skip this prompt.
 
 3. **Instrument** — select an existing instrument from the list, or register a new one by choosing the last option. New instruments start as `pending` and must be activated by an admin in the web app before the watcher can start.
 
@@ -59,7 +59,7 @@ The wizard will walk you through:
    - **`auto`** — files are uploaded to S3 immediately after detection.
    - **`manual`** — files are reported to the server but not uploaded until an admin approves them via the upload queue.
 
-The wizard saves configuration to `~/.data-hub/config.yaml`, the API key to `~/.data-hub/.env`, and syncs the config to the server.
+The wizard saves configuration to `~/.data-hub/config.yaml`, the API key to `~/.data-hub/.env.<environment>`, and syncs the config to the server.
 
 ## Starting the watcher
 

docs/watcher.md

@@ -37,11 +37,11 @@ uv run data-hub-watcher watch
 Interactive setup wizard that:
 
 1. Prompts for the environment (`staging`, `production`, or `preview`). Choosing `preview` also prompts for a custom API base URL.
-2. Prompts for an API key (or reads `DATA_HUB_API_KEY` from the environment). The key is saved to `~/.data-hub/.env`.
+2. Prompts for an API key (or reads `DATA_HUB_API_KEY` from the environment). The key is saved to a per-environment file at `~/.data-hub/.env.<environment>` (e.g. `.env.staging`), so switching between environments later doesn't require re-entering it.
 3. Fetches existing instruments from the API or registers a new one.
 4. Prompts for the watch directory, file patterns, run detection pattern, stability period, and upload mode.
 5. Registers the watcher with the API.
-6. Saves the config to `~/.data-hub/config.yaml`, the API key to `~/.data-hub/.env`, and syncs the config to the API.
+6. Saves the config to `~/.data-hub/config.yaml`, the API key to `~/.data-hub/.env.<environment>`, and syncs the config to the API.
 
 ### `watch`
 
@@ -105,7 +105,7 @@ Manage the watcher as a Windows service:
 
 ## Configuration
 
-The config file lives at `~/.data-hub/config.yaml` by default. Override with `--config` or the `DATA_HUB_CONFIG_PATH` environment variable. The API key is stored separately in `~/.data-hub/.env`.
+The config file lives at `~/.data-hub/config.yaml` by default. Override with `--config` or the `DATA_HUB_CONFIG_PATH` environment variable. The API key is stored separately in `~/.data-hub/.env.<environment>` (e.g. `.env.staging`, `.env.production`, or `.env.preview`); the legacy `~/.data-hub/.env` is also loaded for backwards compatibility, with the per-environment file taking precedence.
 
 ### Config file format
 

pyproject.toml

@@ -6,7 +6,7 @@ dev = [
     "pre-commit>=4.2.0",
     "psycopg2-binary>=2.9.10",
     "pyright>=1.1.408",
-    "pytest>=8.3.5",
+    "pytest>=9.0.3",
     "ruff>=0.11.11",
 ]
 

watcher/src/data_hub_watcher/api_client.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 import logging
 import os
+from datetime import datetime, timezone
 from typing import Any
 
 import requests
@@ -203,12 +204,17 @@ def request_upload_url(
         filename: str,
         content_type: str | None = None,
         size_bytes: int | None = None,
+        file_created_at_ts: float | None = None,
     ) -> PresignedUploadResponse:
         payload: dict[str, Any] = {"filename": filename}
         if content_type:
             payload["content_type"] = content_type
         if size_bytes is not None:
             payload["size_bytes"] = size_bytes
+        if file_created_at_ts:
+            payload["file_created_at"] = datetime.fromtimestamp(
+                file_created_at_ts, tz=timezone.utc
+            ).isoformat()
         resp = self._request(
             "POST",
             f"/instruments/{instrument_id}/runs/{run_id}/request-upload-url",

watcher/src/data_hub_watcher/cli.py

@@ -21,6 +21,8 @@
     DEFAULT_STABILITY_PERIOD_SECONDS,
     RUN_DETECTION_PRESETS,
     STATE_DB_FILENAME,
+    SUPPORTED_ENVIRONMENTS,
+    env_file_path,
     load_env,
     resolve_config_path,
     save_api_key,
@@ -76,6 +78,49 @@ def _resolve_path(ctx: click.Context) -> Path:
     return resolve_config_path(ctx.obj.get("config_path"))
 
 
+API_KEY_PREFIX = "dhub_"
+
+# Invisible characters that some Windows clipboards (Outlook, Teams, Word, etc.)
+# silently inject when an operator copies an API key. Stripping them here
+# avoids 401s caused by a hash mismatch on the server.
+_INVISIBLE_CHARS = (
+    "\u00a0",  # non-breaking space
+    "\u200b",  # zero-width space
+    "\u200c",  # zero-width non-joiner
+    "\u200d",  # zero-width joiner
+    "\ufeff",  # BOM / zero-width no-break space
+)
+
+
+def _clean_api_key(value: str) -> str:
+    """Normalize and validate an API key entered by the operator.
+
+    Pasting into a hidden ``click.prompt`` on Windows frequently introduces
+    stray whitespace (CR, LF, NBSP) or zero-width characters from rich-text
+    clipboards. We strip those defensively and then verify the value still
+    looks like a Data Hub PAT before any network call so the operator sees a
+    clear error instead of a confusing 401.
+    """
+    cleaned = value
+    for ch in _INVISIBLE_CHARS:
+        cleaned = cleaned.replace(ch, "")
+    cleaned = cleaned.strip()
+
+    if not cleaned:
+        raise click.ClickException("API key is empty.")
+    if any(c.isspace() for c in cleaned):
+        raise click.ClickException(
+            "API key contains whitespace. Re-copy the key — your clipboard "
+            "may have included a line break or non-breaking space."
+        )
+    if not cleaned.startswith(API_KEY_PREFIX):
+        raise click.ClickException(
+            f"API key must start with '{API_KEY_PREFIX}'. Re-copy the key from "
+            "the Data Hub UI; the value may have been truncated on paste."
+        )
+    return cleaned
+
+
 def _make_client(
     environment: str, api_key: str | None = None, api_base_url: str | None = None
 ) -> DataHubClient:
@@ -92,6 +137,9 @@ def _load_and_client(ctx: click.Context) -> tuple[WatcherConfig, DataHubClient,
     """Load config and build a matching API client. Returns (config, client, path)."""
     path = _resolve_path(ctx)
     cfg = load_config(path)
+    # Overlay the env-specific file (e.g. ``.env.staging``) so the API key
+    # picked up by ``DataHubClient`` always matches the configured environment.
+    load_env(cfg.environment)
     client = _make_client(cfg.environment, api_base_url=cfg.api_base_url)
     return cfg, client, path
 
@@ -115,8 +163,16 @@ def _setup_file_logging() -> None:
 
 
 @cli.command()
+@click.option(
+    "--show-key",
+    is_flag=True,
+    help=(
+        "Echo the API key as it is typed/pasted. Useful on Windows terminals "
+        "where hidden input is unreliable for paste."
+    ),
+)
 @click.pass_context
-def init(ctx: click.Context) -> None:
+def init(ctx: click.Context, show_key: bool) -> None:
     """Interactive setup wizard + API registration."""
     path = _resolve_path(ctx)
     if path.exists():
@@ -126,7 +182,7 @@ def init(ctx: click.Context) -> None:
     # 1. Environment
     environment = click.prompt(
         "Environment",
-        type=click.Choice(["staging", "production", "preview"], case_sensitive=False),
+        type=click.Choice(list(SUPPORTED_ENVIRONMENTS), case_sensitive=False),
     )
 
     api_base_url: str | None = None
@@ -136,10 +192,24 @@ def init(ctx: click.Context) -> None:
         )
         api_base_url = raw_url.rstrip("/")
 
-    # 2. API key
-    api_key = os.environ.get("DATA_HUB_API_KEY", "")
-    if not api_key:
-        api_key = click.prompt("DATA_HUB_API_KEY", hide_input=True)
+    # 2. API key — overlay any existing per-environment env file so the user
+    # doesn't have to re-enter a key they've already saved for this target.
+    load_env(environment)
+    existing_key = os.environ.get("DATA_HUB_API_KEY", "")
+    env_specific_path = env_file_path(environment)
+    hide_input = not show_key
+    if existing_key and env_specific_path.exists():
+        click.echo(f"Found saved API key for {environment} at {env_specific_path}.")
+        if click.confirm("Use the saved key?", default=True):
+            api_key = existing_key
+        else:
+            api_key = click.prompt("DATA_HUB_API_KEY", hide_input=hide_input)
+    elif existing_key:
+        api_key = existing_key
+    else:
+        api_key = click.prompt("DATA_HUB_API_KEY", hide_input=hide_input)
+
+    api_key = _clean_api_key(api_key)
 
     client = _make_client(environment, api_key=api_key, api_base_url=api_base_url)
 
@@ -153,7 +223,7 @@ def init(ctx: click.Context) -> None:
             "The API key was not saved. Please re-run init with a valid key."
         ) from exc
 
-    env_path = save_api_key(api_key)
+    env_path = save_api_key(api_key, environment)
     click.echo(f"API key saved to {env_path}")
 
     if instruments:
@@ -779,29 +849,27 @@ def _windows_only() -> None:
     "env_path_override",
     type=click.Path(dir_okay=False),
     default=None,
-    help="Path to the .env file. Defaults to ~/.data-hub/.env.",
+    help="Path to the .env file. Defaults to ~/.data-hub/.env.<environment>.",
 )
 @click.pass_context
 def service_install(ctx: click.Context, env_path_override: str | None) -> None:
     """Install the watcher as a Windows service."""
     _windows_only()
     path = _resolve_path(ctx)
-    load_config(path)
+    cfg = load_config(path)
 
     from data_hub_watcher.service import install_service
 
     if env_path_override is not None:
         env_path = Path(env_path_override).resolve()
     else:
-        from data_hub_watcher.constants import DEFAULT_CONFIG_DIR, ENV_FILENAME
-
-        env_path = (DEFAULT_CONFIG_DIR / ENV_FILENAME).resolve()
+        env_path = env_file_path(cfg.environment).resolve()
 
     if not env_path.exists():
         click.echo(
             click.style(
                 f"⚠ Warning: {env_path} does not exist. "
-                "Run 'data-hub-watcher login' first or pass --env-path.",
+                "Run 'data-hub-watcher init' first or pass --env-path.",
                 fg="yellow",
             ),
             err=True,

watcher/src/data_hub_watcher/constants.py

@@ -13,6 +13,7 @@
 DEFAULT_CONFIG_DIR = Path("~/.data-hub").expanduser()
 DEFAULT_CONFIG_FILENAME = "config.yaml"
 ENV_FILENAME = ".env"
+SUPPORTED_ENVIRONMENTS: tuple[str, ...] = ("staging", "production", "preview")
 
 HEARTBEAT_INTERVAL_SECONDS = 60
 DEFAULT_STABILITY_PERIOD_SECONDS = 5
@@ -68,24 +69,47 @@
 CONFIG_PATH_ENV_VAR = "DATA_HUB_CONFIG_PATH"
 
 
-def load_env() -> None:
-    """Load ``~/.data-hub/.env`` into the process environment.
+def env_file_path(environment: str | None = None) -> Path:
+    """Return the env file path for *environment* (or the base file if ``None``).
 
-    Existing environment variables take precedence (``override=False``),
-    so an explicit ``DATA_HUB_API_KEY`` export still wins.
+    Examples:
+        ``env_file_path()``           -> ``~/.data-hub/.env``
+        ``env_file_path("staging")``  -> ``~/.data-hub/.env.staging``
     """
-    env_path = DEFAULT_CONFIG_DIR / ENV_FILENAME
-    load_dotenv(env_path)
+    if environment:
+        return DEFAULT_CONFIG_DIR / f"{ENV_FILENAME}.{environment}"
+    return DEFAULT_CONFIG_DIR / ENV_FILENAME
 
 
-def save_api_key(api_key: str) -> Path:
-    """Persist *api_key* to ``~/.data-hub/.env`` and return the file path.
+def load_env(environment: str | None = None) -> None:
+    """Load env files from ``~/.data-hub/`` into the process environment.
+
+    Always loads the base ``.env`` first. If *environment* is provided,
+    overlays ``.env.<environment>`` on top so its values take precedence.
+    Existing process-level environment variables (e.g. an explicit
+    ``DATA_HUB_API_KEY`` export) still win over the base file but are
+    overridden by the env-specific file when one is supplied — this lets
+    operators switch environments simply by changing the config without
+    re-exporting their key.
+    """
+    load_dotenv(env_file_path())
+    if environment:
+        load_dotenv(env_file_path(environment), override=True)
+
+
+def save_api_key(api_key: str, environment: str | None = None) -> Path:
+    """Persist *api_key* to the env file for *environment* and return its path.
+
+    When *environment* is provided the key is written to
+    ``~/.data-hub/.env.<environment>`` so each deployment target keeps its
+    own credentials. Without *environment* the legacy ``~/.data-hub/.env``
+    file is used.
 
     Preserves any other variables already present in the file and
     single-quotes the value to guard against special characters
     (``#``, ``=``, whitespace) that would confuse dotenv parsers.
     """
-    env_path = DEFAULT_CONFIG_DIR / ENV_FILENAME
+    env_path = env_file_path(environment)
     env_path.parent.mkdir(parents=True, exist_ok=True)
 
     key_line = f"DATA_HUB_API_KEY='{api_key}'\n"