Atlas harvest production-readiness (spec-driven blitz) - #135
Merged
Conversation
jpr5
marked this pull request as ready for review
July 7, 2026 02:56
jpr5
added a commit
that referenced
this pull request
Jul 7, 2026
Prompt-only fix so the Atlas distillation judge's **rewrite** branch retains concrete verifiable detail instead of abstracting it into generic WHY prose. ## How this was found A live prod-read-only dry-run of the just-merged harvest pipeline (#135) over real memory fragments. One candidate — a precise admin-ops claim (`POST /admin/:op`, `202/400/401/503`, `timingSafeEqual`, `trust_proxy` fail-closed) — was rewritten by the judge into *"unified authentication enhances security and simplifies access control,"* **dropping every specific.** The 5 already-well-formed candidates were correctly passed through untouched; only the rewrite path over-abstracted. ## Root cause `DISTILLATION_SYSTEM_PROMPT` had no preservation constraint — it asked for "why/how prose" and only guarded against *inventing* reasoning, never against *dropping* facts. Its WHY-vs-WHAT framing had no category for a precise HOW claim, so verifiable mechanics got recoded as low-value WHAT and "elevated" to rationale. ## The fix (3 edits to `DISTILLATION_SYSTEM_PROMPT`, no gate/parsing changes) - Rewrite MUST retain every concrete verifiable detail (endpoints, HTTP/error codes, symbol names, paths, config keys, numbers); sharpen the WHY/HOW *around* them. - A claim already stating a concrete mechanism is `distilled` — keep as-is, don't rewrite up into rationale. - If a rewrite can't keep every specific, return `distilled` instead — losing a verifiable specific is worse than slightly-WHAT-flavored prose. ## Proof (real-LLM red-green — the actual failure surface, not fakes) An opt-in eval (`OPENAI_API_KEY`-gated, `skipIf` so it's out of normal CI) runs the real judge on the admin-ops fragment: - **RED (pre-fix):** verdict `rewritten`, `POST /admin/:op` dropped — 3/3. - **GREEN (post-fix):** verdict `rewritten` retaining `POST /admin/:op` + `timingSafeEqual` + `202/400/401/503` + `trust_proxy` — 4/4. Plus a Layer-1 aimock gate-plumbing contract guard, and this closes the `judgeDistillation` zero-coverage gap. ## Review 7 reviewers (Tier 2), 3 rounds. The prompt diff was clean (source-gate + generalist reviewers: no findings). One in-subject test-quality fix — the eval's `distilled`-fallback branch was made guard-independent (`toContain(["rewritten","distilled"])`, fails loud on a `restatement` regression) instead of a vacuous re-check. Converged with zero in-subject bucket-(a). **Blast radius:** the pass-through cases can only become *more* likely to pass through; `restatement` untouched. 3410 tests green; the eval is `skipIf`-gated. ## Follow-ups (non-blocking, tracked in the CR ledger) - Code-enforce PRESERVE-SPECIFICS with a gate-side token-retention guard (belt-and-braces beyond the prompt). - Strengthen the eval's `distilled` path to assert the gate-layer pass-through invariant. - Add a through-the-gate test for `stripGitHubMetadataHeader`. - Pre-existing `llm.ts` hardening surfaced by review (public→internal floor, `CandidateFragmentSchema.parse`, embed finite-element check).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the Atlas harvest pipeline production-ready: it turns raw company-knowledge fragments (memory, GitHub, Notion) into a deduplicated, quality-gated, human-approvable seed corpus for the Pathfinder MCP retrieval layer.
What this does (by theme)
approvable=falseis enforced across all fact/behavior types; prose facts are human-gated.scripts/atlas-harvest/sandbox/leaf-fleet runbooks and README preflight guidance.Every behavioral fix in this branch carries a red-green test.
Code review
11 unbiased reviewers × 7 rounds (Tier 3, verbatim byte-identical prompts). ~27 bucket-(a) fixes, including:
Round 7 drove the core clean — the primary source-gate reviewer and the silent-failure specialist both returned no findings — and converged with zero load-bearing new bucket-(a). History is 8 plain-English commits by concern; one is a repo-wide prettier formatting-baseline (formatting-only, satisfies the CI prettier check).
Follow-ups (non-blocking, tracked in the CR ledger)
validate.tsisPathLike single-segment path-probe fallback; distillation-gate rewritten-branch empty-guard symmetry.Deferred
Production wire-on (S21) is intentionally deferred — it needs a ratified corpus first. This PR delivers the pipeline and quality gates, not the prod cutover.
Spec: https://app.notion.com/p/3953aa3818528102be21e433fc0e5351