V2 (maintainers): suggest new CREs when no good match (post-pipeline) #1110

Status

Maintainers only · Version 2 · after the current OIE/librarian pipeline work finishes.

Not for GSoC or drive-by contributors. Do not start implementation until the v1 pipeline (canonical gold fixes, retrieval levers, hop-distance eval) is landed and stable.

Label: existing-maintainers-only

Context

During OIE B2 / production-accuracy work (PR #1088 eval harness), the lab stack included a local Agentic graph/hint gold expand that scored Agentic 5/5 even though prodclone had no Agentic hub Links. That was eval-only (doctored answer key), but it points at a real product capability we want for v2:

When Module C / librarian cannot find a good existing CRE (or only weak multi-hop near-misses), suggest creating a new CRE (and optional Links / Related scaffolding) instead of forcing a bad mapping.

Related findings from hop-distance analysis on canonical gold (SUMMARY winner):

• Exact Links bar ≈ 43%
• Distance-1 soft (exact ∪ Related/Contains neighbor) ≈ 68%
• 0% invented CRE ids in predictions today — the model stays on real CREs; we lack a deliberate “propose new CRE” path

Goal (v2)

1. Detect “no good CRE” cases (e.g. below margin / only multi-hop / empty shortlist).
2. Propose a new CRE candidate (name, description, suggested parent Contains and/or Related neighbors) from section text + graph hints.
3. Keep this clearly separated from production Links writes — review queue / draft artifacts only until mentors approve.
4. Optionally reuse patterns from the eval-only Agentic stub + graph/hint expand, without treating that as a ship metric.

Non-goals (v1 / now)

• Implementing this during the current pipeline accuracy push.
• Counting Agentic stub gold expands toward production accuracy.
• Auto-writing Links to opencre.org without review.

Acceptance criteria (draft, when v2 starts)

• Documented trigger conditions (“no good CRE”).
• Output schema for proposed CRE + suggested graph edges.
• Eval harness hook: count how often we would propose vs force-map (no gold rewrite).
• Explicit flag default off in prod.

References

• PR feat(librarian): Module C retrieval upgrades (stack base) #1088 B2 denominators (local Agentic expand → 58/67 lab only).
• Companion: fix canonical LLM Top 10 gold to AI-topic CREs / AI Exchange (in progress on accuracy track).