feat: opt-in persistent browser profiles (CAMOFOX_PERSISTENT_PROFILES) - #6525
feat: opt-in persistent browser profiles (CAMOFOX_PERSISTENT_PROFILES)#6525juniperbevensee wants to merge 4 commits into
Conversation
Sessions are created with browser.newContext({ storageState }), which persists
cookies + localStorage but NOT IndexedDB / Service Workers / Cache Storage. Sites
that keep auth in IndexedDB (e.g. Telegram Web) cannot be restored from
storageState, so their logins do not survive session recreation.
Add an opt-in mode: when CAMOFOX_PERSISTENT_PROFILES=1, a session launches a
dedicated context via firefox.launchPersistentContext(userDataDir, ...), keyed by
sha256(userId) under CAMOFOX_USERDATA_DIR. The full on-disk Firefox profile
(IndexedDB included) then survives session reap/recreate. Reuses the shared Xvfb
display and the same camoufox launchOptions as the pooled browser, so persistent
contexts get identical stealth. storageState injection is skipped for persistent
sessions (the on-disk profile supersedes it).
Opt-in because each persistent identity is its own browser process; the default
shared-browser / ephemeral-context model is unchanged when the flag is off.
Verified: with the flag on, a value written to IndexedDB survives closing the
session and recreating it (reloaded from userDataDir); with the flag off, the
same value is MISSING after recreate.
…able Docker port-forwarding (-p 127.0.0.1:6081:6080) proxies to the container IP, not container loopback — so binding websockify to 127.0.0.1 INSIDE the container made the published port dead. Bind 0.0.0.0 inside; the host -p mapping remains the security boundary (still loopback-only + SSH-tunnel access). Fixes the dead noVNC that blocked the browser-login VNC handoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A closed tab left agents unable to recover: the hermes browser tool reopens a
fresh tab with 'open about:blank', but validateUrl rejected any non-http(s)
scheme, so tab-create 400'd ('Blocked URL scheme: about:') and the agent could
never get a working surface back. Observed live — an agent told to stop had its
tab closed, then spun retrying a dead tab / failing to reopen.
about:blank is the canonical empty page (a freshly created page already sits on
it, and the server itself does page.goto('about:blank') elsewhere); the scheme
guard is meant to block file:/chrome:/data:, not the inert blank page.
Moved validateUrl into lib/request-utils.js (where the sibling error classifier
already lives) so it is unit-testable, added the about:blank allowance, removed
the now-dead ALLOWED_URL_SCHEMES const from server.js. New unit test covers
about:blank/http/https allowed, file/chrome/data/about:config blocked, garbage
rejected. Verified all cases green.
Co-authored-by: Juniper Bevensee <juniperbevensee@users.noreply.github.com>
… + Dockerfile.simple Two production fixes lived only as uncommitted changes on the Mini's build checkout, at risk of being lost on any rebuild-from-git: - persistentWindowSize(): pins the persistent browser window to the display size (CAMOFOX_WINDOW / VNC_RESOLUTION, default 1280x720) so pages don't render at the camoufox fingerprint's screen size (e.g. 2560x1440) and overflow the VNC view. Applied to both launch paths. - Dockerfile.simple: the ARM (aarch64) build recipe that actually builds the Mini's camofox-browser:persistent image — copies a prebuilt dist/camoufox zip instead of downloading, for reproducible local builds. dist/ stays gitignored (build input, not source). The fork's feat/persistent-profiles is now the single source of truth for the deployed image: persistent-profiles + VNC_BIND + about:blank + window-size + the build recipe. Reconciles the divergence that made rebuilds drop fork work.
|
thanks for putting this together. I’m thinking a little more about the persistent-profile approach and whether it’s the right boundary for this particular use case before committing either way. stay tuned. |
|
Thanks for the persistent-profile implementation and the clear verification work. Since this PR was opened, the built-in persistence plugin gained opt-in IndexedDB storage-state support. This covers the stated login-persistence use case without adding one Firefox process and profile directory per user: {
"plugins": {
"persistence": {
"indexedDB": true
}
}
}The branch is now conflicting and also includes unrelated VNC, URL-validation, and Docker changes. A persistent-profile mode would also need a redesigned reset/delete contract so I am closing this PR as superseded by the current persistence path. Please open a new focused proposal if full Firefox-profile persistence is still needed for a case that IndexedDB storage state cannot support. |
Problem
Sessions are created with
browser.newContext({ storageState }).storageStatepersists cookies + localStorage but not IndexedDB / Service Workers / Cache Storage. Sites that keep their auth in IndexedDB — Telegram Web (WebK) is a clear example — therefore can't be restored from a persisted session: after a session reaps and a new context is created, the user is logged out even thoughstorage-state.jsonwas reloaded.Change
Opt-in persistent profiles. When
CAMOFOX_PERSISTENT_PROFILES=1, a session launches a dedicated context viafirefox.launchPersistentContext(userDataDir, …)instead ofbrowser.newContext(…), keyed bysha256(userId)underCAMOFOX_USERDATA_DIR(default~/.camofox/userdata). The full on-disk Firefox profile — IndexedDB included — then survives session reap/recreate (and, if the dir is on a volume, container recreate).launchOptionsas the pooled browser, so persistent contexts get identical stealth.storageStateinjection is skipped for persistent sessions (the on-disk profile supersedes it).Two new config keys (
lib/config.js):persistentProfiles,userDataDir.Verification
Built the image, ran two containers (flag on / flag off), and for each: opened a tab, wrote a key to IndexedDB, closed the session, recreated it for the same
userId, and read the key back.CAMOFOX_PERSISTENT_PROFILES=1MISSING(lost)Notes / trade-offs