Watcher: Move manual-mode uploads to a dedicated worker thread

[wasimxyz]

Summary

Manual-mode upload-queue polling ran on the heartbeat tick, so a slow or large upload delayed heartbeats and the dashboard flagged busy watchers as offline (prompting operator restarts). On stop, a long-running upload could also outlive the heartbeat join and write to the state DB after close(), raising sqlite3.ProgrammingError: Cannot operate on a closed database.

This moves manual-mode uploads off the heartbeat thread onto a dedicated long-lived worker and fixes the shutdown ordering.

• Add UploadQueueWorker (in uploader.py): owns the poll loop on its own thread and polls every UPLOAD_POLL_INTERVAL_SECONDS (60s), decoupled from the heartbeat.
• Give Uploader an optional shared stop_event: the retry backoff waits on it (interruptible), poll_upload_queue bails out between files on shutdown, and a stop mid-upload returns False (defer, not a spurious failure event).
• Wire into runtime: build_runtime builds a worker in manual mode only (None in auto mode); start_runtime starts it before the initial scan; _on_tick no longer polls.
• stop_runtime stops the worker before state_db.close(), and skips the close if the worker can't stop within UPLOAD_WORKER_STOP_TIMEOUT_SECONDS (30s), restoring the writer-threads-joined-before-close invariant StateDB.close documents.
• Bump watcher version to 0.4.0; update developer-docs/reference/watcher.md.

Auto mode is unchanged: uploads still run on the monitor's stability-checker thread via the run detector's callback.

Test plan

• make py-check (ruff format + lint + pyright) clean
• Watcher unit suite passes (434 tests), including new coverage for interruptible backoff, between-files bailout, worker start/stop, and stop_runtime conditional-close ordering
• Manual smoke: manual-mode watcher keeps heartbeating during a large upload batch; stop/restart mid-upload no longer logs a closed-DB error

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
data-hub	Ready	Preview, Comment	Jul 2, 2026 12:01am