Probe the RAG-dedup gate with the distilled title, not the full body

[jpr5]

Root cause

dedupAgainstRagCorpus (src/atlas/rag-dedup.ts) probed the live RAG search endpoint (GET /api/search on mcp.copilotkit.ai) with the candidate's full distilled title + content body (truncated to ~2 KB) as a single query.

That endpoint is a lexical search: textSearchChunks (src/db/queries.ts) runs tsv @@ plainto_tsquery('english', $1), and plainto_tsquery ANDs every query lexeme — a chunk matches only if it contains all of them. A long, memory-sourced candidate body has hundreds of distinct lexemes spread across many sources, so no single indexed chunk contains them all → 0 hits. With zero hits the containment denominator is empty, so the gate classified the candidate NOVEL and never marked the overlap.

Net: the "must not duplicate the existing RAG corpus" bar (spec §6.2 / §10 bar 6) was silently unenforced for long (memory-sourced) facts — exactly the candidates most likely to re-index already-indexed prose. Short topical queries were unaffected (a distilled title matches fine).

The fix

Send a short, high-signal query built from the candidate's already-distilled title instead of the full body. The title is the short topical phrase the tsvector engine returns strong hits for.

• Title alone, deliberately: appending the structural key components (subsystem / claimSlugHint — synthetic slugs like cpk-runtime, rarely present verbatim in a prose chunk) ANDs the lexical match back down to zero, re-opening the same gap. Verified against prod: "AWS AgentCore" → 5 hits, "AWS AgentCore cpk-runtime" → 0 hits.
• The overlap decision is unchanged: containment is still computed over the FULL candidate token set (candidateFullText); only the query used to find corpus passages is shortened.
• The mark-only / never-drop invariant, idempotent re-annotation, fail-fast streak, and the char/byte/surrogate-safety machinery are all retained (the byte-bound guard now applies to a pathological title). No LLM introduced into the dedup path.

Deterministic, no new dependencies, no prod writes.

RED → GREEN proof (live prod search surface)

Exercised the real AtlasHttpClient + real dedupAgainstRagCorpus against https://mcp.copilotkit.ai/api/search (read-only). The candidate is a genuine long corpus duplicate: a real indexed chunk (AWS AgentCore, docs.copilotkit.ai/deploy/agentcore) restated with a short synthesizing reframe — the shape of a distilled memory fact. Same candidate, same endpoint, only the code differs.

RED (before — full-body probe):

=== CANDIDATE (a genuine corpus duplicate) ===
title: AWS AgentCore
content length: 2569

=== PROBE QUERY that the gate actually sends ===
probe length: 2048
probe head: AWS AgentCore - **Human-in-the-loop** — let users review, approve, or redirect agent actions - **AgentCore memory** — conversation history persists across sessi

=== DIRECT probe of the gate's query against prod ===
hits: 0

=== GATE RESULT (real dedupAgainstRagCorpus vs prod) ===
out length: 1
validated_against: (none — classified NOVEL)
fused_from evidence: []

>>> RED: the genuine corpus duplicate was NOT marked (gate unenforced for long facts).

GREEN (after — title probe):

=== CANDIDATE (a genuine corpus duplicate) ===
title: AWS AgentCore
content length: 2569

=== PROBE QUERY that the gate actually sends ===
probe length: 13
probe head: AWS AgentCore

=== DIRECT probe of the gate's query against prod ===
hits: 5

=== GATE RESULT (real dedupAgainstRagCorpus vs prod) ===
out length: 1
validated_against: rag-corpus-overlap:https://docs.copilotkit.ai/deploy/agentcore
fused_from evidence: [{"kind":"fused_from","ref":"rag-corpus-overlap:https://docs.copilotkit.ai/deploy/agentcore"}]

>>> GREEN: the genuine corpus duplicate was MARKED as overlap.

The live probe was run via a temporary harness (deleted before commit; not in the diff).

Files changed

• src/atlas/rag-dedup.ts — new candidateProbeSource (title-only); candidateProbeQueryText builds the probe from it; header/inline comments updated.
• src/__tests__/atlas-rag-dedup.test.ts — new regression tests (title probe used, subsystem/claimSlugHint NOT appended, and a long candidate that would be missed by the full-body probe is marked via the title probe); byte/surrogate-safety tests re-pointed to a pathological title (the new probe source).

Tests / quality gate

• atlas-* suite: 33 files, 825 tests pass (30 in atlas-rag-dedup).
• Full suite: 171 files, 3195 tests pass (npm test).
• npm run build (tsc): clean. npx tsc --noEmit -p tsconfig.scripts.json: clean. Prettier: my two files are clean.
• Note: the prettier CI job is already failing on main (11 pre-existing unformatted files, none touched here) — a broken-main condition unrelated to this change; left out of scope to keep the diff focused.

🤖 Generated with Claude Code

[jpr5]

Closing this — it's superseded by the semantic pgvector dedup that landed in #135. Rather than working around the lexical probe (distilled title vs full body), the harvest gate now does real embedding-based similarity retrieval, so the full-body-vs-tsvector mismatch this PR was patching no longer exists. Thanks for the initial dig here; the direction moved to semantic.