[Feature]: unify network interception across explore, diagnostic, and operate

[Astro-Han]

End-to-end tested the full adapter lifecycle (explore → generate → plugin → operate → diagnostic → repair) on v1.6.7. The individual pieces mostly work, but they don't share browser instrumentation, which limits the AI agent story.

All claims below have been verified by both static code analysis and runtime reproduction on v1.6.7.

What was tested

Step	Result	Notes
explore	Runs, but 0 API endpoints on HN and Lobsters	Compensation strategy (iframe re-fetch) exists but rarely triggers
generate	Fails (0 candidates)	Depends on explore finding APIs; confirmed: synthesize hackernews → 0
plugin create	Scaffold OK	YAML template uses nonexistent extract pipeline step (separate bug)
plugin install (TS)	Compile OK, runtime OK	linkHostOpencli() in plugin.ts already creates node_modules/@jackwener/opencli symlink
operate open/state/eval	All work
operate network	0 requests after open	Interceptor injected after goto + wait, misses initial load. Confirmed at runtime.
operate network (post-eval)	Works	Manual fetch after open is captured correctly (visible with --all)
operate init + verify	Works	Template uses correct package exports
diagnostic (TS)	Works	error.code and source both correct
diagnostic (YAML)	source missing	Covered by #808
diagnostic (browser page state)	Minimal info	networkRequests 0, consoleErrors [] (see notes below)
repair loop (manual)	Each step works individually	diagnostic → operate explore → patch → verify all functional

Note on consoleErrors: []: base-page.ts consoleMessages() returns an empty array unconditionally with no subclass override. So diagnostic's consoleErrors: [] does not mean "no errors on page"; it means console errors are not collected at all. This further weakens diagnostic info quality.

Core finding: fragmented network interception paths

The codebase has at least four independent network capture mechanisms (three relevant to the explore/diagnostic/operate story, plus one in record):

System	Global var	Used by	Captures
NETWORK_INTERCEPTOR_JS (cli.ts:310)	__opencli_net	operate	fetch/XHR with method, status, body. Injected after page load, so misses initial requests.
interceptor.ts → installInterceptor()	__opencli_xhr	pipeline intercept step	fetch/XHR with body. Injected before trigger, so no timing gap.
generateTapInterceptorJs	temporary	pipeline tap step	One-shot capture, restores original functions after.
record.ts → generateRecordInterceptorJs	__opencli_record	record	fetch/XHR with method, status, headers, request/response body. Idempotent re-injection with restore.

diagnostic and explore use none of these. They both go through page.networkRequests() → performance.getEntriesByType('resource'), which returns URL and timing metadata but no method, status, or response body.

The key gap: when a pipeline adapter fails, the intercept step may have already captured rich network data into window.__opencli_xhr, but diagnostic.ts only reads page.networkRequests() (the performance API path), not page.getInterceptedRequests().

Suggested improvements (by impact/cost ratio)

High impact, low cost:

• diagnostic: also read page.getInterceptedRequests() when available. Pipeline's intercept step may have already captured the data that diagnostic needs. One additional call in collectPageState.
• diagnostic: implement real consoleMessages() collection (currently base class returns [] unconditionally), or at minimum document that console errors are not captured yet.

Medium impact, medium cost:

• operate open: inject NETWORK_INTERCEPTOR_JS before or during goto() instead of after, so initial page-load requests are captured. Or use CDP-level Fetch.requestPaused / Network.requestWillBeSent which don't have timing issues.
• explore: use page.installInterceptor() (already available on IPage) instead of relying solely on performance.getEntriesByType + iframe re-fetch. This would give explore the same quality network data that pipeline intercept steps get.

Separate bug (not part of network unification):

• plugin create: fix YAML template to use a valid pipeline step (replace extract with evaluate or map). See plugin-scaffold.ts:91-98.

Context

These all feed into the vision where each user's AI agent can autonomously explore a website, generate an adapter, and self-repair when it breaks. The operate path (open → state → eval → init → verify) already works end-to-end today. The automated path (explore → generate) and the repair path (diagnostic → fix) are blocked primarily by network data quality.

Note: #806 already added safety boundaries to diagnostic output (redaction of secrets, size caps, collection timeout, source path resolution). The remaining gap is purely about data richness — getting method/status/body into explore and diagnostic, not safety.

[Astro-Han]

Architectural note: CDP capture already exists

The extension already has a complete CDP-level network capture implementation in extension/src/cdp.ts:

• startNetworkCapture(tabId, pattern) — enables Network.enable and starts collecting
• readNetworkCapture(tabId) — returns accumulated entries and clears buffer
• Listens to Network.requestWillBeSent, Network.responseReceived, Network.loadingFinished, Network.getResponseBody

The NetworkCaptureEntry type captures everything the JS monkey-patch interceptors capture and more:

type NetworkCaptureEntry = {
  kind: 'cdp';
  url: string;
  method: string;
  requestHeaders?: Record<string, string>;
  requestBodyKind?: string;
  requestBodyPreview?: string;
  responseStatus?: number;
  responseContentType?: string;
  responseHeaders?: Record<string, string>;
  responsePreview?: string;
  timestamp: number;
};

IPage already declares startNetworkCapture?() and readNetworkCapture?() as optional methods (types.ts:70-71).

This means the unification path is shorter than it appears. The four JS-level interceptors (__opencli_net, __opencli_xhr, tap temporaries, __opencli_record) can all converge on this existing CDP capture as the primary data source. The pattern is the same as the getInterceptedRequests() gap identified above: capability exists, consumers haven't been wired up.

Suggested convergence order:

1. diagnostic — read from CDP capture (or getInterceptedRequests() as interim). Lowest cost, highest impact.
2. operate open/network — replace NETWORK_INTERCEPTOR_JS with startNetworkCapture before goto(). Fixes the initial-request timing gap.
3. explore — call startNetworkCapture before navigation, read after. Replaces performance.getEntriesByType path entirely.
4. record — replace __opencli_record polling with CDP capture. Preserves the workflow, removes the independent interceptor.
5. intercept/tap — evaluate last; these have precise pattern-matching needs that may still warrant JS-level interception as a complement.

[Astro-Han]

Toward auto loop: what exists, what's missing

Two pieces of infrastructure already exist

1. CDP network capture (extension/src/cdp.ts)

The extension already has a complete CDP-level capture: startNetworkCapture / readNetworkCapture, listening to Network.requestWillBeSent / responseReceived / loadingFinished / getResponseBody. The NetworkCaptureEntry type includes method, headers, status, request body, response preview, timestamp — everything the JS monkey-patch interceptors capture and more.

IPage declares startNetworkCapture?() and readNetworkCapture?() as optional methods (types.ts:70-71). The daemon-backed Page already implements them. But explore, diagnostic, and operate don't call them — they each use weaker, independent paths.

2. Autoresearch engine (autoresearch/engine.ts)

The Karpathy-style iteration loop is already running in-repo: preconditions → review → ideate → modify → commit → verify (extract metric) → decide (keep/revert) → log → repeat. Current presets target operate reliability (eval-browse.ts, 59 browse tasks with deterministic judge criteria).

What's missing for adapter auto-generation

The auto loop for adapter generation (explore → generate → test → diagnose → repair → retry) maps directly onto the existing autoresearch engine. It's not a new system — it's a new preset + a new eval script. But two prerequisites are blocking:

A. Explore evidence quality

Explore currently reads performance.getEntriesByType('resource') — URL and timing only, no method/status/body. Even with CDP capture wired in, explore only works for API-driven SPAs. SSR sites (HN, Lobsters) serve data as HTML; explore finds 0 endpoints because its analysis logic only looks for JSON.

Two complementary fixes:

• Wire CDP capture into explore (replaces performance API path, gives rich data for SPA sites)
• Add a fallback for SSR sites: either DOM structure analysis, or auto-fallback to record with interactFuzz replacing manual interaction (record.ts already has full capture → YAML generation)

B. Candidate verification step

synthesize writes adapter YAML to disk but nobody runs it. generate.ts already chains explore → synthesize → select, but stops there — its own comments mention register → verify steps that aren't implemented. validate.ts and verify.ts exist but aren't wired into the generation pipeline.

The missing piece is an eval-adapter-gen.ts that: explores target sites → synthesizes candidates → runs each candidate → judges output quality (non-empty, expected fields, data freshness) → outputs SCORE=N/M. Then a preset like:

export const adapterGeneration: AutoResearchConfig = {
  goal: 'Increase adapter auto-generation success rate',
  scope: ['src/explore.ts', 'src/synthesize.ts', 'src/browser/dom-helpers.ts', 'src/browser/base-page.ts'],
  metric: 'candidate_pass_rate',
  direction: 'higher',
  verify: 'npx tsx autoresearch/eval-adapter-gen.ts 2>&1 | tail -1',
  guard: 'npm run build',
  minDelta: 1,
};

Suggested order

1. diagnostic: add page.getInterceptedRequests?.() call in collectPageState — lowest cost, immediate value for pipeline adapter failures
2. operate open: replace NETWORK_INTERCEPTOR_JS with startNetworkCapture before goto() — fixes initial-request timing gap
3. explore: wire CDP capture + add SSR fallback path
4. eval-adapter-gen.ts: build the eval script, plug into autoresearch engine
5. record: replace __opencli_record with CDP capture (preserves workflow, removes independent interceptor)

Note: CDPPage (Electron direct-connect mode) doesn't implement startNetworkCapture/readNetworkCapture — only daemon-backed Page does. JS-level interception remains necessary as fallback for that path.