Cancel pending upload requests on watch-directory change - #93
Merged
wasimxyz merged 2 commits intoJun 12, 2026
Conversation
Fix ENG-1397, where changing a watcher's watch_directory left pending upload requests pointing at relative paths under the old root, so the watcher re-emitted a "Queued file missing" error on every heartbeat poll. - Server: revert an instrument's pending upload requests to `detected` (clearing upload_requested_at) when watch_directory changes, plus a config_synced cancellation event. Allow upload_requested -> detected in PATCH /files/:id so the watcher can cancel a single stuck request. - Watcher: cap manual-mode queue retries at MAX_QUEUE_FILE_ATTEMPTS=3 (covering both missing-on-disk and upload failures), throttle the missing-file error to once per file, then cancel the request server-side. - Tests and docs for both paths. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
wasimxyz
changed the title
Co-authored-by: Cursor <cursoragent@cursor.com>
wasimxyz
deleted the
wasimamiri/eng-1397-queued-file-missing-errors-when-switching-watch-directory
branch
June 12, 2026 00:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In manual mode the uploader resolves each queued file as
watch_dir / relative_path, butrelative_pathis anchored to the watch directory active at detection time. After an operator changes a watcher'swatch_directory, every pending request becomes unresolvable and the queue re-poll re-emits aQueued file missingerror on every heartbeat tick forever (the reported flood was 8611+ events).Two coordinated fixes plus a per-file give-up path:
PUT /watchers/:id/configchangeswatch_directory, all of that instrument'supload_requestedfiles are reverted todetected(clearingupload_requested_at) so the queue drains immediately. Gated on a known previous directory so the first config push and unrelated config edits never revert spuriously. Aconfig_syncedevent withdetails.kind="upload_requests_cancelled"records the cancellation.PATCH /files/:idnow allowsupload_requested -> detected(clearingupload_requested_at) so the watcher can remove one stuck request from the queue.MAX_QUEUE_FILE_ATTEMPTS=3heartbeat polls, covering both missing-on-disk and upload/network failures. TheQueued file missingerror is emitted once per file (throttled), and after the cap the watcher cancels the request server-side (cancel_upload_request) and emits anupload_request_cancelledgive-up event. Attempt counters reset on restart, so a transient outage longer than ~3 ticks is recovered on the next start.Trade-off
A transient failure that lasts longer than ~3 heartbeat ticks (~3 min) causes the file to be reverted to
detecteduntil a human re-requests it or the watcher re-detects it. This is the chosen behavior over retrying forever.Changes
web/lib/api/watchers.ts:extractWatchDirectory+revertPendingUploadRequestshelpers.web/app/api/v1/watchers/[watcherId]/config/route.ts: revert + cancellation event onwatch_directorychange.web/app/api/v1/files/[fileId]/route.ts: allowupload_requested -> detected, clearupload_requested_at.watcher/:cancel_upload_requestclient method,MAX_QUEUE_FILE_ATTEMPTSconstant, per-file attempt cap inUploader, new event kinds in the taxonomy.watcher/tests/test_uploader.py,web/tests/integration/{files,upload-request-cancellation}.test.ts) anddocs/watcher.md.Test plan
make check-all(ruff lint/format, pyright, prettier, eslint, tsc)upload_requested -> detectedcases)Made with Cursor