feat(eval): LoCoMo benchmark — full local baseline + cloud-judge cross-check

[Dewinator]

Context

LoCoMo (Snap Research, ACL 2024) is the canonical benchmark for very-long-term conversational memory in LLM agents — 10 conversations, ~340 turns each, ~199 QA pairs per conversation across 5 reasoning categories. It's the right yardstick for empirically measuring how mycelium-as-memory-system compares to a vanilla LLM context window.

Reed asked for this on 2026-05-08. Funding constraint: API-key cost should be near $0 → run voll-lokal first (qwen2.5:7b-instruct as both answerer and judge), do a cloud-judge cross-check only once the local pipeline is green. ChatGPT-Plus OAuth is not a viable substitute for an OpenAI API key for this kind of evaluation (the Plus subscription's tokens are not exposed to platform.openai.com endpoints).

Scaffolding already on `main`

The eval pipeline lives in `experiments/locomo/` and is checked in. `dataset/` and `out/` are gitignored.

• `lib.mjs` — shared env loader (reads `.mcp.json`), Ollama wrapper, dataset helpers.
• `ingest-locomo.mjs` — one project per conversation (`locomo-<sample_id>`), every turn becomes a memory row. Bypasses MemoryService dedup so short turns aren't silently merged.
• `run-locomo.mjs` — for each QA, calls `MemoryService.search` scoped to the conversation's project, sorts retrieved turns chronologically by `dia_id`, sends to qwen2.5:7b-instruct.
• `judge-locomo.mjs` — LLM-as-judge (CORRECT / PARTIAL / WRONG), per-category accuracy, plus retrieval-evidence-recall (fraction of QA where at least one gold evidence `dia_id` was in the top-K). Supports `--judge=ollama|openai|anthropic`.
• `README.md` — quickstart + design notes.

Smoke run is green: 20-turn ingest of `conv-26` + 3 QA + judge → 50% accuracy, 100% retrieval-evidence-recall, ~5–9s per QA on local Ollama.

Sub-tasks (each can be its own tick)

• L1 — Full ingest, all 10 conversations. `node experiments/locomo/ingest-locomo.mjs --all`. Verify each `locomo-<sample_id>` project has the expected turn count.
• L2 — Full QA run, all 10 conversations. Time-budget: ~5 QA/min × 1990 QA ≈ 7 hours single-threaded. Consider parallelising across conversations.
• L3 — Local judge run + per-conversation `judgment.json`. Aggregate per-category accuracy across all 10. Drop into `docs/benchmarks/locomo--local.md`.
• L4 — Cloud-judge cross-check (one conversation, ~$0.10). `--judge=openai --judge-model=gpt-4o-mini`. Compare to local-judge verdicts to measure judge drift.
• L5 — Sweep top-K `{10, 20, 40}` on one conversation. Plot accuracy vs retrieval depth — shows whether mycelium's recall is bottleneck or the answerer is.
• L6 — Ablation: spread on/off, project-scope on/off. Quantify the contribution of spreading-activation and L1 isolation. Spread off = pure vector search. Scope off = noise from other conversations.
• L7 — Write-up: `docs/benchmarks/locomo-baseline.md`. Headline numbers, per-category breakdown, retrieval recall, ablation deltas, comparison to LoCoMo paper's published RAG baselines.

Estimated cost

• L1–L3 (full local run): $0, ~8h wall-clock on the M-series Mac.
• L4 (cloud cross-check, gpt-4o-mini judge, 1 conversation): ~$0.10.
• Optionally L7 with full GPT-4o-mini judge across all 10: ~$1–2.

Acceptance

• `docs/benchmarks/locomo-baseline.md` lands with the numbers above.
• Numbers are reproducible: `node experiments/locomo/ingest-locomo.mjs --conv= --reset` followed by run + judge gives the same accuracy ±2pp on a re-run.

[Dewinator]

Status — full run in flight

Started 2026-05-08 20:40 UTC after the smoke confirmed the pipeline. Still running — final numbers will land in a follow-up comment once all 10 conversations are judged. ETA ~02:30–04:30 local on 2026-05-09.

Confirmed so far

Ingest (all 10 conversations, complete in ~6 min wallclock):

sample_id	turns ingested
conv-26	(full ingest done)
conv-30	(full ingest done)
conv-41	663
conv-42	629
conv-43	(full ingest done)
conv-44	(full ingest done)
conv-47	(full ingest done)
conv-48	681
conv-49	509
conv-50	568

Each conversation lives in its own project (locomo-<sample_id>), so recall is perfectly isolated per-eval.

Smoke (conv-26, 20-turn ingest, 3 QA): 50% accuracy, 100% retrieval-evidence-recall, ~5–9 s per QA on local Ollama qwen2.5:7b-instruct.

Live progress (current run, 22 min in): conv-26 at 62/199 QA predicted (~5–7 s per QA, on track for the full per-conversation budget of ~25 min).

Process

Background script at experiments/locomo/run-and-judge-all.sh. It is resumable — skips any conversation that already has a judgment.json, so a session crash doesn't lose progress. Live log: experiments/locomo/out/full-run.log.

What changed vs. the original spec

• Default answerer + judge is qwen2.5:7b-instruct, not qwen3:8b. qwen3's reasoning tokens overflow the num_predict budget and produce empty output. Fixed in run-locomo.mjs and judge-locomo.mjs.
• Ingest writes directly to memories (bypassing MemoryService.create) to avoid the 0.92-cosine dedup silently dropping short turns ("Yeah.", "Sure."). The salt [D<s>:<t> <speaker>] ... makes every turn a unique row.

I'll post the final per-category accuracy table + retrieval-evidence-recall numbers + a comparison to the LoCoMo paper's published RAG baselines once the run completes.