Skip to content

feat: add POST /tabs/:tabId/set_input_files endpoint - #4164

Closed
MrReasonable wants to merge 96 commits into
jo-inc:masterfrom
MrReasonable:master
Closed

feat: add POST /tabs/:tabId/set_input_files endpoint#4164
MrReasonable wants to merge 96 commits into
jo-inc:masterfrom
MrReasonable:master

Conversation

@MrReasonable

@MrReasonable MrReasonable commented May 24, 2026

Copy link
Copy Markdown

Why

AI agents using camofox-browser often need to apply for jobs or submit documents on behalf of users. Many job sites and portals use <input type="file"> elements for CV/resume uploads — but browsers intentionally block JavaScript from setting file inputs, so a standard type or click action cannot attach a file. This endpoint fills that gap by exposing Playwright's privileged setInputFiles channel via the REST API.

What

Adds POST /tabs/:tabId/set_input_files to programmatically attach files to <input type="file"> elements.

  • Accepts either a snapshot ref (preferred, with automatic stale-ref refresh) or a CSS selector to target the file input
  • File paths are sandboxed to a configurable uploads directory (CAMOFOX_UPLOADS_DIR, default /home/node/cv-uploads) to prevent the endpoint being used to read arbitrary files from the container
  • Emits a tab:set_input_files plugin event consistent with the rest of the interaction surface
  • Full OpenAPI/Swagger documentation included

Test plan

  • Attaches file via CSS selector — filename appears in snapshot
  • Attaches file via element ref — filename appears in snapshot
  • Rejects paths outside the uploads directory with 400
  • Rejects request with neither ref nor selector with 400
  • Returns 404 for unknown tab ID
  • Full e2e test suite passes (npm run test:e2e)

🤖 Generated with Claude Code

- createTabState gets navigateAbort AbortController field
- navigateCurrentPage races page.goto() with abort signal
- DELETE /tabs/:tabId calls abort() before safePageClose, cancelling
  in-flight navigation immediately instead of waiting 30s timeout
- Add camofox_snapshot_bytes histogram for snapshot size tracking
- 6 new abort tests passing
…tion, fixes

Backported from camofox-browser v1.5.2:

- Lazy Prometheus metrics: off by default, enable with PROMETHEUS_ENABLED=1.
  Noop stubs when disabled — no prom-client import overhead. Fixes OpenClaw
  install blocking on optional dep.
- Extract extractPageImages to lib/images.js (OpenClaw scanner isolation)
- Extract actionFromReq + classifyError to lib/request-utils.js
- buildRefs timeout leak: clearTimeout in both success and error paths
- YT transcript session cleanup: close phantom __yt_transcript__ context
  when all tabs are gone
- Session cleanup on empty tab groups: tab reaper now closes sessions with
  zero remaining tabs + triggers browser idle shutdown
- Horizontal scroll: mouse.wheel now supports left/right directions
- Delete tab/group: accept userId from query string or body
- plugin.ts: proc.kill() on startup timeout, screenshot Content-Type guard
  (returns text error instead of base64-encoded JSON on non-image response)
- 2 new test files: sessionCleanup.test.js, updated screenshotToolResult.test.js
Race 1 (YT transcript): browserTranscript cleanup counted tabGroups which
was always 0 (YT pages aren't registered in tabGroups). First concurrent
request to finish would close the context, killing other requests' pages.
Fix: use context.pages() to check actual live pages before closing.

Race 2 (tab reaper / session expiry): context.close() fired without await,
then session deleted from map. A request that already got the session ref
(between getSession return and newPage) would use a closing context.
Fix: set session._closing = true before teardown; getSession() treats
_closing sessions as dead and creates a new one.

Both fixes also applied to the session expiry timer (same pattern).
16 unit tests added covering _closing flag, getSession skip, YT concurrent
cleanup, and session expiry sentinel.
When a click/navigate/open_url times out (e.g., Cloudflare holding the
connection for 30s), the Decodo proxy session becomes poisoned — all
subsequent requests through the same BrowserContext hang. Previously,
timeouts only tracked per-tab consecutive counts without destroying the
session, so recovery tabs inherited the poisoned proxy.

Now navigation-related timeouts (click, navigate, open_url) destroy the
entire user session, so the next request gets a fresh BrowserContext
with a fresh proxy session. Non-navigation timeouts (type, scroll) still
use the per-tab consecutive timeout tracking.

Root cause: WWDC cron clicked a MacRumors Cloudflare-protected link,
the proxy session got poisoned, recovery tab's prewarm to google.com
also timed out on the same session, browser agent gave up entirely.
…act, crash reporter, CI improvements

New features from upstream:
- OpenAPI spec auto-generated from JSDoc + swagger-stripey docs at /docs (jo-inc#78)
- Opt-in session tracing via Playwright (jo-inc#68)
- Structured extract endpoint with JSON Schema + x-ref hints (jo-inc#70)
- session:destroying event for persistence checkpoint (jo-inc#75)
- Anonymized crash/frustration reporter with GitHub App auth
- GitHub Actions CI workflow for unit tests

Also: compressed fox.png, added README badges, GitHub Sponsors funding link
Upstream refactor moves crash reporter credentials to camofox.config.json.
Watchdog now includes active sessions/tabs/URLs in event-loop stall reports.

jo-browser config uses its own GitHub App (Jo Browser Crash Reporter,
app ID 3505356) targeting jo-inc/jo-browser.
Drifts >120s are OS sleep/hibernate, not real event-loop stalls.
Skip them and suppress the next 5 ticks to avoid post-wake jitter
false positives (like the 6s stall in #83).

Fixes #82, #83
# Conflicts:
#	lib/reporter.js
#	openapi.json
Merge artifact from v1.7.4 sync introduced duplicate 'let lastHeapUsed'
at lines 976 and 1004 in reporter.js. Node.js SyntaxError on startup
caused all 3 Fly machines to crash-loop and exhaust retries.

All machines have been stopped since ~05:11 UTC, breaking browser-dependent
cron jobs and use_browser tool calls.
deploy.yml had zero dependency on CI — deployed immediately on push
regardless of whether tests passed. This is how the duplicate let
declaration shipped to prod and took down all 3 machines for 14+ hours.

Now deploy requires: syntax check of all JS files + reporter unit tests.
- Install @sentry/node, init from SENTRY_DSN env var
- lib/sentry.js: isolated module (no process.env, scanner-safe)
- lib/config.js: add sentryDsn config key
- server.js: capture 500s in sendError, uncaughtException, unhandledRejection
- setupExpressErrorHandler after all routes, flush on graceful shutdown
- fly.toml: add SENTRY_DSN for jo-browser project
- StaleRefsError (422) filtered out via beforeSend
Root cause: .internal DNS bypassed Fly Proxy — all traffic pinned to one
machine. 11 concurrent BrowserContexts × ~300MB = 3.3GB on 4GB → OOM.

Changes:
- MAX_SESSIONS=10 (was 50 default — way too high for 4GB)
- concurrency type connections soft=25 (was requests soft=8 — wrong metric)
- Memory admission control: 503 + fly-replay at >90% RAM
- Memory pressure eviction: 30s loop, multi-evict until <80% RAM
- Session overflow redirect: fly-replay when >ceil(MAX/3) sessions
- tabNotFoundResponse: 410 Gone for local tabs after browser restart
- SSL error handling: 502 recoverable:false for SEC_ERROR
- Sentry: skip capture for intentional 503s
Dead context, timeouts, proxy errors, nav aborts, and tab lifecycle
errors are expected operational behavior — not bugs. They were flooding
Sentry with noise issues, hiding real problems.

Added classifyError patterns for NS_ERROR_ABORT, tab deleted navigation
abort, and page crash. Expanded beforeSend to drop operational errors.
…, timeout tuning

Fix 1: Drain tab locks BEFORE closing browser context in closeSession().
Queued operations now get clean 'Tab destroyed' (410) instead of cascading
'Target page closed' (500) errors. Dead-context errors in handleRouteError
now return 503 with session_expired code — the root cause (proxy/timeout)
is still reported, but cascade errors no longer flood Sentry.

Fix 2: Transparent retry on proxy/timeout errors during navigation.
When page.goto fails with NS_ERROR_PROXY_FORBIDDEN or timeout, destroy
the session, get a fresh proxy, and retry once before failing to caller.
Applied to POST /tabs (create with URL), POST /tabs/:tabId/navigate, and
POST /tabs/open. Uses existing context rotation pattern.

Fix 3: Increase goBack timeout from 10s to 20s.
Back navigation uses browser cache and shouldn't need aggressive timeouts.
10s was too short for complex SPA re-renders.
# Conflicts:
#	camofox.config.json
#	package-lock.json
#	server.js
@SSOURABH58

SSOURABH58 commented Jul 8, 2026

Copy link
Copy Markdown

Thanks for putting this together! I've been testing file uploads and found some edge cases that we should consider handling here to make this endpoint more robust.

The Issue:
Currently, the PR relies solely on locator.setInputFiles() / page.setInputFiles(). This approach works for simple file inputs but has limitations:

  1. It can fail/500 if the <input type="file"> element is hidden.
  2. It silently fails on custom dropzones (e.g., react-dropzone). On these custom widgets, the site typically reads the file from the trusted file-chooser event payload rather than directly from the hidden input element. setInputFiles directly on the input bypasses this payload.

Proposed Improvement:
To support both plain inputs and custom dropzone widgets, we can update the set_input_files implementation in server.js (around the withTabLock block) to use a two-path approach.

We can enable file chooser interception and attempt to click the element to trigger the chooser event, falling back to direct setInputFiles if that fails.

Here is a conceptual example of how we can update the logic:

// Inside withTabLock, before interacting:
await tabState.page.setFileChooserInterceptedBy(true);

const fileChooserPromise = tabState.page.waitForEvent('filechooser', { timeout: 3000 }).catch(() => null);

// Try clicking the visible dropzone wrapper or the input itself (not the hidden input if possible)
// (We could potentially auto-detect the nearest clickable ancestor or accept a dropzone selector)
await locator.click({ force: true }).catch(() => {});

const fileChooser = await fileChooserPromise;

if (fileChooser) {
  // Intercept the filechooser and set the files via the chooser element.
  // Since the server FileChooser might lack .accept(), use element().setInputFiles
  await fileChooser.element().setInputFiles(resolved, { timeout: 10000 });
} else {
  // Fallback: Direct target.setInputFiles for plain inputs if no chooser was triggered
  await locator.setInputFiles(resolved, { timeout: 10000 });
}

This ensures we intercept the trusted chooser event for modern upload widgets while preserving the direct fallback (Plan A) for simpler inputs.

(Also referencing #3115 as this relates to similar input interaction logic.) Let me know what you think!

@MrReasonable

Copy link
Copy Markdown
Author

Thanks @SSOURABH58 — this is a genuinely good catch. You're right that direct setInputFiles on the hidden input silently no-ops for widgets (react-dropzone et al.) that read from the trusted file-chooser event, and that's worth handling.

I've pushed a change that adds this as an opt-in second path rather than replacing the default (68ba18a):

  • Direct input stays the default. ref / selector still set files straight on the <input type="file"> via setInputFiles — deterministic, works on hidden inputs (Playwright bypasses the visibility actionability check for file inputs), and adds no latency for the common case.
  • New dropzoneRef / dropzoneSelector target the visible dropzone element. When supplied, the server registers a filechooser waiter, clicks that element, and feeds the chooser via fileChooser.setFiles(...).

A couple of implementation notes vs. the sketch:

  • I went with Playwright's implicit chooser interception — attach a page.waitForEvent('filechooser') listener before the click and it intercepts automatically; there isn't a page.setFileChooserInterceptedBy(...) on the API surface, and fileChooser.setFiles() takes the paths directly.
  • Rather than auto-detecting the clickable ancestor (which is guesswork and can force-click the wrong element / trigger navigation), the caller names the dropzone explicitly. Force-click is used only as a fallback for a blocked/timed-out click, so strict-mode violations and detached nodes still surface honestly.
  • Security: the file-chooser path feeds the same CAMOFOX_UPLOADS_DIR-sandboxed paths as the direct path (now checked on the realpath, so an in-dir symlink can't escape the allowlist), and it returns a clear 400 if the click opens no chooser, pointing plain-<input> callers back to ref/selector.

Tests cover both dropzone paths (selector + ref) against a react-dropzone-style page, sandbox enforcement on the new path, and the no-chooser case.

One thing on the #3115 reference: it already prototypes this file-chooser approach with the correct APIs (waitForEvent('filechooser') + chooser.setFiles()) — worth a look — but its /upload endpoint overlaps this one and doesn't sandbox the file paths, so I'd be wary of landing them in a way that reintroduces arbitrary-file access. Happy to reconcile if a maintainer wants a single endpoint.

Leo Mercer and others added 20 commits July 11, 2026 16:19
Adds a new route for programmatically attaching files to <input type="file">
elements via Playwright's setInputFiles. File paths are restricted to a
configurable uploads directory (CAMOFOX_UPLOADS_DIR, default /home/node/cv-uploads)
to prevent arbitrary file exfiltration. Supports both ref-based and CSS
selector targeting, with auto-refresh of stale refs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers selector and ref-based file attachment, path-traversal rejection,
missing ref/selector validation, and 404 for unknown tabs. Global setup
now creates a temp uploads directory and passes CAMOFOX_UPLOADS_DIR to
the test server so the sandbox restriction is exercisable in CI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds an opt-in file-chooser path to POST /tabs/:tabId/set_input_files for
custom dropzone widgets (e.g. react-dropzone) that read the file from the
trusted filechooser event rather than the hidden <input>. Direct
setInputFiles on the input silently no-ops for these widgets.

- New dropzoneRef / dropzoneSelector params target the visible dropzone
  element. The server clicks it, intercepts the resulting filechooser, and
  feeds it the same uploads-dir-sandboxed paths, so the security boundary
  is unchanged.
- Direct ref/selector path is untouched and remains the default.
- Returns `via: "input" | "filechooser"` so callers can see which
  mechanism ran; emits the same via/dropzone fields on the plugin event.
- Returns 400 (not 500) when the dropzone click opens no chooser, with a
  message pointing plain-input callers back to ref/selector.
- Regenerates openapi.json, which was missing the endpoint entirely (the
  original commit never re-ran generate-openapi).

Tests: adds a react-dropzone-style test page and e2e coverage for the
dropzone path via selector and ref, sandbox enforcement on that path, the
no-chooser 400, and the via:input result. Full suite (10 tests) passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses an adversarial review of the endpoint:

Security
- Sandbox containment is now checked on the realpath of each path, not the
  pre-resolved string. A symlink planted inside the uploads dir can no longer
  smuggle an out-of-tree target (e.g. /etc/passwd) past the allowlist, on
  either the direct or the file-chooser sink. The allowlist root is itself
  realpath'd once (the mount may be a symlink).

Functionality
- Reject directories and other non-regular files (fs.access(R_OK) succeeds on
  a dir, which previously slipped through to a Playwright 500).
- Dropzone chooser wait (15s) now exceeds the click timeout (10s) so a slow
  click that opens the chooser can't lose a race to an equal-length timer and
  report a false "no chooser".
- Narrow the dropzone force-click fallback to blocked/timed-out clicks only;
  strict-mode violations, detached nodes and closed targets now surface with
  their real diagnostic instead of being retried with force.
- Require userId (was silently returning 404 tab-not-found).

Code quality
- Extract timeout constants; move CAMOFOX_UPLOADS_DIR into lib/config.js;
  log client (4xx) failures at info level instead of error.

Tests
- Add coverage: symlink escape, directory rejection, missing userId, dropzone
  precedence over a direct target, and that a direct setInputFiles on the
  visible div fails (justifying the dropzone path). Harden the dropzoneRef
  test so it fails loudly instead of silently falling back to the selector
  (the dropzone now carries role=button so it appears in the snapshot).
  Full suite: 15/15 pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update pinned Camoufox version (Firefox 152) across Makefile and both
Dockerfiles. Upstream marked the FF135 build unsuitable for modern
anti-bot systems.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skyfallsin

Copy link
Copy Markdown
Contributor

Thanks for the upload work.

This PR’s user-facing need is now covered by the production upload endpoint:

POST /tabs/:tabId/upload

It supports upload-root containment, symlink-safe path checks, direct file inputs, ref/selector triggers, native filechooser fallback, and post-upload ref refresh.

The branch is now conflicting and contains many unrelated server, Docker, auth, and infrastructure changes. Maintaining a second upload route would duplicate the current API and increase drift risk.

I am closing this PR as superseded. If there is a specific upload case that /upload does not handle, please open a small focused issue or PR against the current implementation.

@skyfallsin skyfallsin closed this Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants