hackbrowser: dedicated worker diagnostic log + full browser lifecycle logging

[badchars]

Summary

The hackbrowser crawl worker runs as a separate subprocess, and its internal state — browser/page lifecycle, crawl progress, and errors — is not reliably surfaced anywhere the operator can read. When a crawl fails or stops early (see #104), there is no persisted record of what actually happened (e.g. whether a page closed, a popup was dismissed, the renderer crashed, or the browser disconnected). This has repeatedly blocked troubleshooting.

Goal

Give the worker its own persistent diagnostic log and record every browser lifecycle event, so a failed/partial crawl can be diagnosed after the fact without guessing from a single stack trace.

Proposed changes

1. Dedicated worker log file

Write worker diagnostics to a per-session file in the runtime data dir:

<data>/log/hackbrowser-<sessionID>.log

• Lives in the data dir (outside the repo), so it is never committed.
• Independent of the main log / IPC relay, which does not reliably carry worker logs today.

2. Log all browser lifecycle events

Attach listeners and log each event with context:

• page.on("close") — page closed (navigation / window.close / popup dismissed)
• page.on("crash") — renderer crash (distinct from a close)
• context.on("page") — a new page/popup/tab was opened
• context.on("close") — context closed
• browser.on("disconnected") — browser went away

Listening to close vs crash separately means we can tell a deliberate close from a crash without inferring it from an error string.

3. Context attached to each event

• timestamp
• URL of the page/target
• open page count (context.pages().length)
• browser.isConnected()
• crawl progress: pages explored so far, queue length

4. Capture-drop diagnostics (ties into #104)

When a request capture is dropped because the target closed mid-flight, log the target state at that moment: pageClosed, browserConnected, and the URL. Combined with the lifecycle log, this pinpoints whether a partial crawl was caused by a page close, a popup, or a browser death.

5. Verbosity

Consider gating the verbose lifecycle logging behind a debug flag / env var so normal runs stay quiet, while troubleshooting can opt in.

Notes

• The per-session log file may contain full request URLs (kept intentionally, for troubleshooting). It stays in the local data dir and is never committed or shared.
• Related: hackbrowser worker crashes (exit 1) mid-crawl when a target closes — unguarded request.allHeaders() #104 (worker crash on closed-target allHeaders()), which this observability work helps root-cause going forward.

[Serhatcck]

Addressed on `dev` — commit `301d5132b` — at a deliberately proportionate scope.

Done:

• Rich context on the lifecycle logs. Added a guarded `lifecycleContext(browser, page?)` helper (url, open-page count, isConnected — every read wrapped so a crash/disconnect handler can never throw while gathering diagnostics) and attached its output to the existing `browser.disconnected` / `page.close` / `page.crash` logs on BOTH the single- and multi-credential paths.
• Popup / new-tab logging. Added a context `page` listener (debug level) that logs when a popup/tab opens, with the same context — no such listener existed.

Verified end-to-end: single-credential crawl logs the popup open and the terminal events with {url, openPages, connected}; a real multi-credential crawl (two creds, manual login) logged `page closed … {credential: …, url, openPages, connected}` for each credential and `browser disconnected … {connected:false}` on close; the guarded reads don't throw even on a not-yet-committed popup URL or a disconnected browser; no regression, 99/99 tests pass.

Intentionally NOT done (proportionality):

• Dedicated per-session log file. The worker's logs are already persisted — the parent relays every worker log record through its own Log pipeline — and hackbrowser worker crashes (exit 1) mid-crawl when a target closes — unguarded request.allHeaders() #104 removed the main mid-crawl crash, so the narrow crash-window loss a separate file would guard is now rare. A separate file duplicates the parent's persistence for little marginal gain, so it was skipped rather than add file-I/O surface. If crash-window log loss proves to be a real problem in practice, reopen and we'll add it as a focused change.
• Verbosity flag (the new lines are already level-gated: lifecycle at error, popup at debug) and crawl-progress plumbing (out of the lifecycle handlers' scope).