Watcher: Speed up initial scan, parallelize uploads, eliminate StateDB lock contention

[wasimxyz]

Summary

Four hot-path performance optimizations on the lab-PC watcher loop, scoped to watcher/src/data_hub_watcher. Targets the symptoms operators actually feel: minutes-long startup scans, sequential uploads bottlenecking large runs, and a process-wide SQLite lock that defeated WAL mode.

Initial scan rewrite (monitor.py)

• Replaced Path.rglob + is_file + double stat() with an os.scandir-based walk against cached DirEntry.stat(follow_symlinks=False).
• Pre-compiled fnmatch patterns into a single anchored regex shared between the scan loop and the watchdog _EventHandler.
• New _load_dedup_index builds an in-memory dict[str, list[(size, mtime)]] from one bulk SELECT per table, replacing two SQL round trips per scanned file with one Python dict lookup.

StateDB thread-local connections (state.py)

• Per-thread sqlite3.Connection via threading.local() opened lazily through a _conn property (existing state_db._conn test access keeps working).
• Process-wide _lock replaced by a smaller _write_lock only around mutations; reads run unlocked under WAL.
• New iter_uploaded_stat_keys / iter_detected_stat_keys bulk loaders for the rewritten scan.
• Per-connection PRAGMAs added: cache_size=-65536, temp_store=MEMORY, mmap_size=268435456, busy_timeout=5000.
• close() now reaches every thread's connection.

SHA-256 chunking + overlap (util.py, uploader.py)

• New HASH_CHUNK_SIZE = 1 << 20 (1 MiB) constant replaces the 8 KiB streaming buffer (~128x fewer read() syscalls on multi-GiB instrument files).
• Uploader._upload_single runs file_sha256 and client.request_upload_url on a 2-thread ThreadPoolExecutor so the hash and the presign network round trip overlap.

Parallel uploads (models.py, runtime.py, uploader.py)

• New InstrumentConfig.upload_parallelism: int = Field(default=4, ge=1, le=32). Defaulted, so existing config YAMLs need no migration.
• Plumbed through runtime.build_runtime into the Uploader constructor.
• Uploader.upload_files now uses a per-batch ThreadPoolExecutor sized by min(parallelism, len(files)), with a fast path when parallelism == 1.
• Single requests.Session shared across S3 PUTs for TLS keep-alive across parallel uploads.
• Counter mutations protected by a new _counters_lock.

Test plan

• make check-all clean (ruff format/lint, pyright, frontend formatter/lint/typecheck)
• All 360 watcher unit tests pass
• New TestUploadFilesParallelism (4 tests) — serial fast path, true concurrency proven via threading.Barrier(3), partial-failure handling, empty-list behaviour, parallelism=0 rejection
• New TestBulkLoadIterators, TestStatInDedupIndex, TestInitialScanBulkLoad — last asserts exactly one SELECT per table and zero per-row has_stat_match calls in the scan
• New tests/test_state_db_concurrency.py — 40-thread × 25-write contention, concurrent reads during writes, per-thread connection isolation, close() shutdown coverage
• New TestUploadParallelism covers default/range/rejection cases for the new field
• Updated TestPutToPresignedUrl.test_successful_put to use an Uploader instance and patch the shared session

Out of scope (deferred)

The remaining items from the wider perf review (heartbeat thread split, _persist_detected_files delta-only writes, parallel _pending snapshot, RunState.files set membership, mimetype caching) are intentionally not in this PR. Easier to land + measure these four hot paths first.

Made with Cursor

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
data-hub	Ready	Preview, Comment	May 14, 2026 6:31pm