Agent robustness + Harbor eval suite for Search & Auto-Apply agents

[Mayank-glitch-cpu]

Tracking issue for hardening the two server-side agents and building a Harbor-based eval suite with code verifiers. Full design doc: docs/agent-architecture-and-evals.md (architecture diagrams, workflow-vs-agent assessment, eval/verifier specs).

Context

• Search Agent (mcp-server/src/lib/agent/search-agent.ts) is a genuine autonomous tool-use loop, but has no verification: the run result is whatever the model claims plus an upsert tally. Nothing enforces recency, link liveness, or filters on what actually lands in the pipeline.
• Auto-Apply Agent (mcp-server/src/lib/agent/auto-apply-agent.ts) is a code-driven workflow with an approval gate (correct design), but has no recovery loop: conditional fields, multi-page wizards, and off-menu select answers are unhandled.
• There are no agent evals. Plan follows Demystifying evals for AI agents: containerized environments (Harbor), record/replay ATS fixtures with a frozen clock, code verifiers + calibrated LLM judge, pass@k for search / pass^k for auto-apply, multi-model matrix via JOBSYNC_AGENT_MODEL.

Phase 1 — Post-run audit in prod (verifier logic, no new infra)

• search-audit.ts: audit every job upserted during a search run
  • Recency: datePosted parses and falls within the lookback window
  • Liveness: re-verify apply links via the existing board-specific checkers (link-check-tools.ts)
  • Filters: title matches target roles (when roles were specified)
  • No-fabrication: every upserted applyLink must appear in a prior tool result in the transcript
• Collect upserted jobs + observed tool output inside the runSearchAgent loop; run the audit after the loop; surface an AuditReport on SearchResult and flag violations in the pipeline (notes)
• Unit tests (vitest, mocked verifier)

Phase 2 — Fixture recorder + mock ATS server

• Recorder that snapshots Greenhouse/Lever/Ashby/Workday API responses + job pages for ~30 companies on a date D
• Fixture server that replays them; frozen-clock support (env override for "now")
• Mock apply-form server (cloned GH/Lever/Ashby forms) with a request recorder — outcome source for auto-apply grading; doubles as the RL environment

Phase 3 — Search-agent eval suite (Harbor)

• ~10 tasks: happy path, recency trap, dead links, dedupe (pre-seeded seen-state), filter traps, dead slugs, sparse world, empty profile, negative case (adds zero)
• Reference solutions (expected job-link sets) written before the tasks
• Code verifiers (reuse Phase 1 logic): recency, liveness, filters, dedupe, no-fabrication (hard gate), yield, summary faithfulness, budget
• 3-model matrix run (haiku / sonnet / opus) with pass@k + cost-per-added-job report

Phase 4 — Auto-apply eval suite

• Tasks: standard form, open questions, select traps, dry-run safety, email-code gate, CAPTCHA handoff, conditional fields
• Hard gates first: zero submit POSTs before approval; no CAPTCHA bypass attempts; applied-mark ⇔ confirmed submit
• Field-accuracy, select-integrity, grounding (LLM judge) graders; pass^k reporting

Phase 5 — Agent robustness (informed by eval failures)

• Search: structured final report contract (JSON schema) validated against transcript
• Search: prompt caching on system+tools prefix; compact JSON from discovery tools
• Search: budgets (wall-clock, tokens, per-tool-call caps) surfaced to the model
• Auto-apply: re-inspect after fill (conditional fields); code-validate select answers against offered options with one retry
• Auto-apply: bounded page-wizard loop for Workday-style multi-page flows
• Cross-run portal memory (slug 404s, yield stats) injected into kickoff

🤖 Generated with Claude Code

[Mayank-glitch-cpu]

Progress ? Phases 1 & 2 landed (branch UI, not yet committed)

Phase 1 ? post-run audit ?

• mcp-server/src/lib/agent/search-audit.ts ? auditSearchRun() (recency / liveness / title-match / no-fabrication), auditFlagNotes(), summarizeAudit(). Injectable link verifier + frozen-clock-aware now.
• mcp-server/src/tools/link-check-tools.ts ? exposed checkJobLinkLive() so audit reuses the exact board-specific checkers (no prod/grader drift).
• mcp-server/src/lib/pipeline.ts ? annotatePipelineEntries() (notes-only, idempotent, never touches status).
• search-agent.ts ? loop collects upserted jobs + observed tool output; runs audit after the loop; attaches audit to SearchResult; flags offending pipeline rows.
• Tests: search-audit (14), pipeline annotate (4), agent wiring (2 new).

Phase 2 ? fixture record/replay + frozen clock ?

• mcp-server/src/lib/eval/fixture-fetch.ts ? FixtureStore + makeFixtureFetch (record / replay / replay-passthrough) + installFixtureFetch[FromEnv]. Wraps global fetch, so the existing fetchers AND the link verifier replay with zero code changes. Scoped to ATS hosts via isAtsUrl.
• mcp-server/src/lib/eval/clock.ts ? now() / isClockFrozen() honoring JOBSYNC_FAKE_NOW; wired as the audit's default clock (no-op in prod).
• mcp-server/scripts/record-fixtures.ts ? snapshots a real day (board lists + per-job liveness) and writes ground-truth.json for reference solutions.
• Tests: fixture-fetch (8), clock (3).

Design note: chose a fetch interceptor over a standalone mock HTTP server for the search path ? hermetic, faster in Harbor containers, no code changes to fetchers/verifier. A real HTTP server is still needed for Phase 4 (browser-driven auto-apply, since patchright loads real URLs) and will wrap the same FixtureStore.

Status: 47 tests pass, tsc --noEmit clean. Next: Phase 3 (Harbor task scaffold + reference solutions + wire installFixtureFetchFromEnv into the agent runtime entrypoint).