fix(cluster): correct the vector geometry and stop truncating candidates

ARCHITECTURE.md

@@ -46,6 +46,8 @@ src/                    # CLI tool (published to npm as prism-triage)
   write-gate.ts         # one dry-run-by-default gate every GitHub mutation funnels through (read-only ethos)
   benchmark.ts          # embedding provider benchmark tool (--out per-run results + cluster membership)
   bots.ts               # bot-author detection; excluded from clustering by default
+  # store.ts: vectors are normalised on write; similarity is 1 - d^2/2 because
+  # vec0 is L2. VECTOR_GEOMETRY_VERSION guards stores written before that.
   incident.ts           # incident windows: compile once, match closedAt at read time
   metadata.ts           # the stored-metadata shape, shared by the CLI and App paths
   config.ts             # Zod-validated YAML + env config

CHANGELOG.md

@@ -4,12 +4,18 @@ all notable changes to pr-prism are documented here.
 
 ## [unreleased]
 
+### fixed
+- **duplicate detection was finding a fraction of the duplicates on any corpus of 5000+ items.** three compounding bugs. the `vec0` table is declared without a distance metric, so sqlite-vec returns L2, but `store.search` filtered `1 - distance >= threshold`, which is not cosine at any scale. nothing normalised vectors on write, so that conversion was not even valid to attempt. and `cluster.ts` routed 5000+ items through a candidate-limited path that passed the clustering threshold into that same wrong-scale filter, then truncated whatever survived to K=50. measured on a real 5285-item corpus: **26 clusters found where exact pairwise finds 501**, and the "optimisation" was *slower* (20.6s vs 9.3s). vectors are now normalised on write, similarity is derived as `1 - d^2/2`, and the candidate-limited path is gone - `vec0` KNN is itself a full scan with a `LIMIT`, so there was never an index to approximate with. closes #19
+- stores written before this carry raw vectors and are refused by `search()` with an actionable message rather than answered with confidently wrong similarities. `prism re-embed` clears it, or `backfillVectorGeometry()` converts in place without re-embedding
+- **any cluster count published before this is not comparable.** the model benchmark numbers in particular measured near-exact-duplicate detection, not duplicate detection at the configured threshold
+
 ### changed
 - the `cluster` block (bot filtering) now reaches the server/GitHub-App path, declared per repo in that repo's `config.json` alongside `incidents`. previously a repo-specific bot login was honoured by the CLI and silently ignored by the App. note the key names differ by file format: the CLI's yaml uses `cluster.bot_authors` / `cluster.include_bot_authors`, the App's json uses `cluster.botAuthors` / `cluster.includeBotAuthors`, matching each file's existing convention, so the same repo produced different clusters depending on which path you looked at. closes #27
 - the weekly digest reads per-repo settings instead of a global copy. it was passed `DEFAULT_REPO_CONFIG.similarityThreshold` and clustered *every* repo at the default, ignoring whatever that repo had configured, so its cluster counts disagreed with the backlog scan's for the same repo. `WeeklyDigestConfig` loses `similarityThreshold` and `autoClose`: the first now comes from the repo, the second was already dead
 - incident windows now work on the server/GitHub-App path, not just the CLI. declare them per repo under `incidents` in `{dataDir}/{owner}-{repo}/config.json`. a readable config carrying a malformed window throws on load rather than quietly loading as "no incident", which would rank the affected backlog as rejected. a backlog scan fetches closed items only for repos that declare a window: a window matches on `closedAt`, so without one the extra API calls buy nothing. part of #27
 - the webhook path no longer overwrites what a scan learned. `server/triage.ts` wrote `metadata: { author, state }` wholesale, and because `upsert` replaces `metadata_json` outright, a `pull_request.opened` webhook draining after a backlog scan dropped every other field the scan had stored (labels, diff size, ci status, closing refs, `authorIsBot`). it now writes only the fields the event can actually observe and leaves the rest of the row alone. field names come from the shared `itemMetadata()`, and a name that is not part of it throws rather than silently creating a key nothing reads. part of #27
-- bot-authored items are excluded from clustering by default. automation reuses titles for unrelated content - dependabot files "chore(deps): bump the npm group with 2 updates" week after week - so consecutive bot PRs embed as near-identical and surfaced as duplicates nobody could act on. measured on odysseus-dev/odysseus: every false positive among the extra clusters a high-recall model found was a recurring bot PR, and filtering removed exactly those two clusters (39 -> 37) while leaving all 11 genuine duplicates. set `cluster.include_bot_authors: true` to restore the old behaviour, or list repo-specific automation under `cluster.bot_authors` when a self-hosted bot is not in the built-in list. applies to confirmed (identity) clusters too, not just fuzzy ones
+- incident-closed PRs now rank *between* open and closed rather than as fully open. an item closed by a repository-wide event never got a maintainer verdict, so it must outrank a deliberate close, but it is not evidence of live work the way an open PR is. on the corpus this feature was built for the tiers are 993 open / 1347 merged / 2945 closed, so promoting a ~900-item incident to open-equivalent roughly doubled the tier the ranking exists to order
+- bot-authored items are excluded from clustering by default. automation reuses titles for unrelated content - dependabot files "chore(deps): bump the npm group with 2 updates" week after week - so consecutive bot PRs embed as near-identical and surfaced as duplicates nobody could act on. measured on odysseus-dev/odysseus: every false positive among the extra clusters a high-recall model found was a recurring bot PR, and filtering removed exactly those and nothing else. (the cluster counts originally quoted here were measured before the vector-geometry fix below and are not comparable; the bot PRs were identified by author, which that bug does not affect.) set `cluster.include_bot_authors: true` to restore the old behaviour, or list repo-specific automation under `cluster.bot_authors` when a self-hosted bot is not in the built-in list. applies to confirmed (identity) clusters too, not just fuzzy ones
 - confirmed (identity) duplicate clusters now pick canonical by earliest-created instead of quality score: byte-identical duplicates are a which-was-first question, and a copied PR can outscore the original it was lifted from. fuzzy clusters keep the state/CI/score rule. starmap `canonical`/`contested` for confirmed clusters shift accordingly (value change, schema stays v1)
 
 ### added

src/__tests__/benchmark.test.ts

@@ -349,7 +349,9 @@ describe.sequential("benchmark provider selection", () => {
       expect(fetchMock).toHaveBeenCalledTimes(3);
 
       const store = new VectorStore(benchmarkDatabasePath("owner/repo", "ollama", "local-model", 2));
-      expect(Array.from(store.getAllEmbeddings("owner/repo").values())[0]).toEqual(new Float32Array([0.5, 0.5]));
+      expect(Array.from(store.getAllEmbeddings("owner/repo").values())[0]).toEqual(
+        new Float32Array([1 / Math.SQRT2, 1 / Math.SQRT2]),
+      );
       store.close();
     } finally {
       process.chdir(originalCwd);

src/__tests__/canonical.test.ts

@@ -376,3 +376,46 @@ describe("incident-closed lifecycle ranking", () => {
     expect(selectCanonical(items).number).toBe(31);
   });
 });
+
+describe("incident-closed ranks between open and closed", () => {
+  // Promoting an incident-closed PR to open-equivalent overshoots. On the
+  // odysseus corpus that motivated the feature the tiers are 993 open, 1347
+  // merged, 2945 closed; a ~900-item incident promoted wholesale roughly
+  // doubles the open tier, which is the tier the tool exists to order.
+  // It never got a maintainer verdict, so it outranks a deliberate close, but
+  // it is not evidence of live work the way a genuinely open PR is.
+  const pr = (over: Record<string, unknown>) => c({ type: "pr", ...over } as never);
+
+  it("prefers a genuinely open PR over an incident-closed one", () => {
+    const picked = decideCanonical(
+      [
+        pr({ number: 5559, state: "closed", incidentClosed: true, score: 0.9 }),
+        pr({ number: 5560, state: "open", score: 0.1 }),
+      ],
+      { mode: "pr" },
+    ).canonical;
+    expect(picked.number).toBe(5560);
+  });
+
+  it("prefers an incident-closed PR over a deliberately closed one", () => {
+    const picked = decideCanonical(
+      [
+        pr({ number: 5561, state: "closed", score: 0.9 }),
+        pr({ number: 5562, state: "closed", incidentClosed: true, score: 0.1 }),
+      ],
+      { mode: "pr" },
+    ).canonical;
+    expect(picked.number).toBe(5562);
+  });
+
+  it("still prefers merged over everything", () => {
+    const picked = decideCanonical(
+      [
+        pr({ number: 5563, state: "closed", incidentClosed: true, score: 0.9 }),
+        pr({ number: 5564, state: "merged", score: 0.1 }),
+      ],
+      { mode: "pr" },
+    ).canonical;
+    expect(picked.number).toBe(5564);
+  });
+});

src/__tests__/pipeline.test.ts

@@ -4,7 +4,7 @@ import { join } from "node:path";
 import { afterEach, describe, expect, it, vi } from "vitest";
 import { effectiveEmbeddingConfigHash, embeddingConfigHash } from "../embeddings.js";
 import { parseDuration, reEmbedStoredItems, runScan } from "../pipeline.js";
-import { VectorStore } from "../store.js";
+import { VECTOR_GEOMETRY_VERSION, VectorStore } from "../store.js";
 import type { PipelineContext, PRItem, StoreItem } from "../types.js";
 
 afterEach(() => {
@@ -191,7 +191,7 @@ describe("re-embed configuration identity", () => {
       );
 
       expect(embedBatch).toHaveBeenCalledOnce();
-      expect(store.getEmbedding("owner/repo:issue:1")).toEqual(new Float32Array([0.5, 0.5]));
+      expect(store.getEmbedding("owner/repo:issue:1")).toEqual(new Float32Array([1 / Math.SQRT2, 1 / Math.SQRT2]));
       expect(store.getMeta("embedding_config_hash")).toBe(effectiveEmbeddingConfigHash(providerConfig, 2));
       expect(store.getMeta("embedding_config_hash")).toContain(":vprovider-selected-v1");
     } finally {
@@ -355,3 +355,41 @@ describe("runScan metadata refresh dirty check", () => {
     }
   });
 });
+
+describe("re-embed clears the geometry guard", () => {
+  it("stamps the current vector geometry, since it rewrote every vector", async () => {
+    // The guard's error message tells the operator to run `prism re-embed`.
+    // If that did not clear the marker, the advice would be wrong and the
+    // database would keep refusing to search after being fixed.
+    const dir = mkdtempSync(join(tmpdir(), "prism-reembed-geo-"));
+    const store = new VectorStore(join(dir, "t.db"), 2);
+    try {
+      store.upsert({
+        id: "owner/repo:issue:1",
+        type: "issue",
+        number: 1,
+        repo: "owner/repo",
+        title: "t",
+        bodySnippet: "",
+        embedding: new Float32Array([1, 0]),
+        metadata: { author: "a", state: "open" },
+        createdAt: "2026-01-01T00:00:00Z",
+        updatedAt: "2026-01-01T00:00:00Z",
+      });
+      store.setMeta("vector_geometry_version", "0");
+
+      await reEmbedStoredItems(
+        store,
+        store.getAllItems("owner/repo"),
+        { dimensions: 2, embed: vi.fn(), embedBatch: vi.fn(async (t: string[]) => t.map(() => [0.5, 0.5])) },
+        { provider: "ollama", model: "m" },
+        10,
+      );
+      expect(store.getMeta("vector_geometry_version")).toBe(VECTOR_GEOMETRY_VERSION);
+      expect(() => store.search(new Float32Array([1, 0]), 5, 0.5)).not.toThrow();
+    } finally {
+      store.close();
+      rmSync(dir, { recursive: true, force: true });
+    }
+  });
+});

src/__tests__/store.test.ts

@@ -435,3 +435,90 @@ describe("bot authorship round-trip", () => {
     expect(read.authorIsBot).toBe(true);
   });
 });
+
+describe("vector geometry", () => {
+  const dbs: string[] = [];
+  afterEach(() => {
+    for (const db of dbs) {
+      try {
+        rmSync(resolve(db, ".."), { recursive: true, force: true });
+      } catch {}
+    }
+    dbs.length = 0;
+  });
+
+  function open(dims = 2) {
+    const p = tmpDb();
+    dbs.push(p);
+    return new VectorStore(p, dims);
+  }
+
+  function put(store: VectorStore, id: string, emb: number[]) {
+    store.upsert({
+      id,
+      type: "pr",
+      number: Number(id.split(":").pop()),
+      repo: "o/r",
+      title: id,
+      bodySnippet: "",
+      embedding: new Float32Array(emb),
+      metadata: { author: "a", state: "open" },
+      createdAt: "2026-01-01T00:00:00Z",
+      updatedAt: "2026-01-01T00:00:00Z",
+    });
+  }
+
+  it("search similarity matches exact cosine, not 1 - L2", () => {
+    // The table is L2. `1 - distance` is not cosine at any scale: for unit
+    // vectors L2 = sqrt(2 - 2cos), so a 0.85 threshold on `1 - d` silently
+    // meant cosine ~0.989 and pruned every genuine near-duplicate below that.
+    const store = open(2);
+    put(store, "o/r:pr:1", [1, 0]);
+    put(store, "o/r:pr:2", [Math.cos(0.3), Math.sin(0.3)]); // cos ~= 0.9553
+    const hits = store.search(new Float32Array([1, 0]), 10, 0.9);
+    store.close();
+    expect(hits.map((h) => h.id)).toContain("o/r:pr:2");
+  });
+
+  it("normalises on write, so a provider returning unnormalised vectors still works", () => {
+    // pr-prism does not require providers to return unit vectors, and the
+    // cosine conversion is only valid for them.
+    const store = open(2);
+    put(store, "o/r:pr:1", [3, 0]); // magnitude 3
+    put(store, "o/r:pr:2", [7, 0]); // magnitude 7, identical direction
+    const hits = store.search(new Float32Array([5, 0]), 10, 0.99);
+    store.close();
+    expect(hits.map((h) => h.id).sort()).toEqual(["o/r:pr:1", "o/r:pr:2"]);
+  });
+
+  it("refuses to search a store written under an older geometry", () => {
+    // A database written before normalisation holds raw vectors; searching it
+    // with the corrected formula returns confidently wrong similarities.
+    const p = tmpDb();
+    dbs.push(p);
+    const store = new VectorStore(p, 2);
+    put(store, "o/r:pr:1", [1, 0]);
+    store.setMeta("vector_geometry_version", "0");
+    store.close();
+
+    const reopened = new VectorStore(p, 2);
+    expect(() => reopened.search(new Float32Array([1, 0]), 10, 0.5)).toThrow(/geometry/i);
+    reopened.close();
+  });
+
+  it("backfills an older store in place", () => {
+    const p = tmpDb();
+    dbs.push(p);
+    const store = new VectorStore(p, 2);
+    put(store, "o/r:pr:1", [3, 0]);
+    store.setMeta("vector_geometry_version", "0");
+    store.close();
+
+    const reopened = new VectorStore(p, 2);
+    const migrated = reopened.backfillVectorGeometry();
+    const hits = reopened.search(new Float32Array([1, 0]), 10, 0.99);
+    reopened.close();
+    expect(migrated).toBe(1);
+    expect(hits.map((h) => h.id)).toContain("o/r:pr:1");
+  });
+});

src/canonical.ts

@@ -61,17 +61,23 @@ export interface CanonicalDecision<T> {
  * all ties here and falls through to the score/date rules unchanged.
  *
  * Takes the whole candidate rather than the bare state so an incident-closed
- * item can be ranked where it sat before the incident.
+ * item can be ranked on its own tier.
+ *
+ * Incident-closed sits between open and closed rather than being promoted to
+ * open outright. It never got a maintainer verdict, so it must outrank a
+ * deliberate close; but it is not evidence of live work the way an open PR is,
+ * and a repository-wide event closes items in bulk. On the corpus this feature
+ * was built for the tiers are 993 open, 1347 merged, 2945 closed, so promoting
+ * a ~900-item incident to open-equivalent roughly doubles the tier the ranking
+ * exists to order.
  */
 function statePriority(candidate: { state?: string; incidentClosed?: boolean }): number {
-  // An incident-closed item never reached a maintainer verdict, so it is ranked
-  // where it sat before the incident rather than as a rejection.
-  const state = candidate.state === "closed" && candidate.incidentClosed ? "open" : candidate.state;
-  switch (state) {
+  if (candidate.state === "closed" && candidate.incidentClosed) return 2;
+  switch (candidate.state) {
     case "merged":
-      return 3;
+      return 4;
     case "open":
-      return 2;
+      return 3;
     case "closed":
       return 1;
     default:

src/cluster.ts

@@ -94,43 +94,25 @@ export function findDuplicateClusters(store: VectorStore, items: PRItem[], opts:
   const ids = [...embeddings.keys()];
   const adjacency = new Map<string, Set<string>>();
 
-  // For large datasets, use ANN pre-filtering via store.search() to reduce candidate pairs
-  // For smaller datasets (< 5000), brute force is fast enough
-  const useANN = ids.length >= 5000;
-
-  if (useANN) {
-    // ANN candidate generation: for each item, find top-K nearest neighbors
-    // then verify with exact cosine similarity
-    const K = 50; // candidates per item
-    for (const id of ids) {
-      const emb = embeddings.get(id)!;
-      const candidates = store.search(emb, K, opts.threshold);
-      for (const { id: candidateId } of candidates) {
-        if (candidateId === id) continue;
-        if (!embeddings.has(candidateId)) continue;
-        // Verify with exact cosine similarity
-        const sim = cosineSimilarity(emb, embeddings.get(candidateId)!);
-        if (sim >= opts.threshold) {
-          if (!adjacency.has(id)) adjacency.set(id, new Set());
-          if (!adjacency.has(candidateId)) adjacency.set(candidateId, new Set());
-          adjacency.get(id)?.add(candidateId);
-          adjacency.get(candidateId)?.add(id);
-        }
-      }
-    }
-  } else {
-    // Brute force O(n²) — fine for < 5000 items
-    for (let i = 0; i < ids.length; i++) {
-      const embA = embeddings.get(ids[i])!;
-      for (let j = i + 1; j < ids.length; j++) {
-        const embB = embeddings.get(ids[j])!;
-        const sim = cosineSimilarity(embA, embB);
-        if (sim >= opts.threshold) {
-          if (!adjacency.has(ids[i])) adjacency.set(ids[i], new Set());
-          if (!adjacency.has(ids[j])) adjacency.set(ids[j], new Set());
-          adjacency.get(ids[i])?.add(ids[j]);
-          adjacency.get(ids[j])?.add(ids[i]);
-        }
+  // Every pair, exact cosine. There used to be a candidate-limited path above
+  // 5000 items that called store.search() with the clustering threshold, on the
+  // assumption it was an approximate index worth trading recall for. It was
+  // neither: sqlite-vec's vec0 KNN is itself a full scan with a LIMIT, so the
+  // path bought no speed, while the LIMIT truncated candidates and the
+  // threshold was compared against a distance on the wrong scale. On the 5285
+  // item corpus it found 26 clusters where exact pairwise finds 501, and took
+  // twice as long doing it. 5285 x 768 floats is milliseconds; there is nothing
+  // here to approximate.
+  for (let i = 0; i < ids.length; i++) {
+    const embA = embeddings.get(ids[i])!;
+    for (let j = i + 1; j < ids.length; j++) {
+      const embB = embeddings.get(ids[j])!;
+      const sim = cosineSimilarity(embA, embB);
+      if (sim >= opts.threshold) {
+        if (!adjacency.has(ids[i])) adjacency.set(ids[i], new Set());
+        if (!adjacency.has(ids[j])) adjacency.set(ids[j], new Set());
+        adjacency.get(ids[i])?.add(ids[j]);
+        adjacency.get(ids[j])?.add(ids[i]);
       }
     }
   }