[Graph] The two 3,000-node thresholds in wiki-graph.ts leave no good operating range — below it Louvain blocks the UI, above it every edge weight silently collapses to 1

[CuongGIK]

bug, performance, scalability

Summary

src/lib/wiki-graph.ts has two constants set to the same value:

const MAX_WEIGHTED_GRAPH_NODES = 3_000
const COMMUNITY_WORKER_THRESHOLD = 3_000

They gate opposite behaviours, so a wiki is penalised on both sides of 3,000 nodes:

• Below 3,000 — community detection runs on the main thread (analyzeCommunities only offloads to a worker when nodes.length >= COMMUNITY_WORKER_THRESHOLD), and the weighted retrieval graph is built by reading every markdown file sequentially.
• At/above 3,000 — buildRetrievalGraph is skipped entirely and every edge falls back to weight = 1, silently discarding the whole relevance model.

There is no wiki size at which both halves behave well.

Detail

Above the cap: the relevance model disappears without any notice

if (nodeMap.size <= MAX_WEIGHTED_GRAPH_NODES) {
  retrievalGraph = await buildRetrievalGraph(...)
}
const edges = dedupedEdges.map((e) => {
  let weight = 1
  if (retrievalGraph) { weight = calculateRelevance(nodeA, nodeB, retrievalGraph) }
  return { source: e.source, target: e.target, weight }
})

Everything calculateRelevance contributes is lost at once: source overlap (×4.0 — the strongest signal), Adamic-Adar common neighbours (×1.5), and type affinity (×1.0). Two knock-on effects:

• Louvain gets worse exactly when the wiki gets harder to cluster. detectCommunities passes { weight: edge.weight } to graphology; with all weights equal to 1 it loses every hint about which links matter.
• The weak-edge filter stops filtering. src/components/graph/graph-view.tsx raises edgeVisibilityThreshold to 0.16 above 2,500 nodes — but with all weights pinned to 1, 1 > 0.16 for every edge, so nothing is ever hidden at the exact size where hiding weak edges is the point.

Nothing in the UI indicates that the graph switched to an unweighted mode.

Below the cap: the expensive paths are the ones that run

buildRetrievalGraph (src/lib/graph-relevance.ts) reads the entire wiki one file at a time:

for (const file of mdFiles) {
  let content = ""
  try { content = await readFile(file.path) } catch { continue }
  ...
}

That is one sequential IPC round-trip per page — ~2,900 in my project — on every graph build that misses the cache. By contrast buildWikiGraph itself already uses mapWithConcurrency(..., GRAPH_FILE_READ_CONCURRENCY = 16), so the fast pattern exists in the same file.

Meanwhile Louvain runs on the main thread until 3,000 nodes, which is where the UI freeze on opening the graph comes from. (This may be part of what was reported in #647, which is closed but describes the same symptom.)

Environment / measurements

LLM Wiki v0.6.11, Windows 11. Project: 2,933 markdown files → 2,474 graph nodes after type: query pages are removed by HIDDEN_TYPES, 10,036 unique edges.

So this project sits 526 pages below the cliff: today it gets weighted edges but a main-thread Louvain; after a few more ingest batches it will silently lose all edge weights and gain a worker. Neither state is the good one, and the transition is invisible to the user.

Note that type: query pages do not count toward the cap, which makes the effective limit hard to reason about from the UI — my wiki shows 2,933 files but is 2,474 nodes.

Proposed fixes

1. Decouple the two constants. Move community detection to the worker at a much lower threshold (~500 nodes); there is no reason to block the main thread up to 3,000.
2. Make the weighted path scale instead of switching it off.
   • Read files with the existing mapWithConcurrency helper rather than a sequential loop.
   • Cache the retrieval graph on disk keyed by file mtimes and invalidate incrementally — this is the same caching request as [Bug] Graph API on large projects: 1000-node hard cap, offset ignored, path-form links not in graph, ~24s recompute per request #714, which reports ~24 s per graph API call and eventual hangs on a 4,359-file project.
3. If the cap must stay, degrade gracefully rather than flatly. Even above the cap, directLink and typeAffinity can be computed from data already in memory (nodeData.links, extractType) without a second full pass — that keeps edge weights meaningful without the expensive source/neighbour analysis.
4. Surface the mode. Show in the graph panel when the wiki is running unweighted, and expose the node count against the cap; right now a user has no way to know the model changed under them.