Preserve concrete specifics in Atlas distillation rewrites - #136
Merged
Conversation
DISTILLATION_SYSTEM_PROMPT (judgeDistillation) had no preservation constraint, so on a rewritten verdict the model paraphrased a precise HOW/WHAT claim into generic WHY prose and dropped every concrete identifier — e.g. an admin-ops fragment carrying POST /admin/:op, status codes 202/400/401/503, timingSafeEqual, and trust_proxy fail-closed became "unified authentication enhances security and simplifies access control". enforceDistillation swaps the judge's title/content in wholesale, so the degraded rewrite became the seed. Three prompt changes, no gate/parsing/other-prompt changes: - rewritten bullet: the rewrite MUST retain every concrete verifiable detail (endpoints, status/error codes, symbols, paths, config keys, numbers); add WHY/HOW around them, never paraphrase them away, with the concrete anti-example spelled out. - distilled bullet: a claim already stating a concrete mechanism is distilled — keep as-is, do not rewrite it up into rationale. - Rules: PRESERVE-SPECIFICS is mandatory on rewrite; if a rewrite can't keep every specific, return distilled (pass-through) instead.
Layer-2 real-LLM eval (atlas-distillation-rewrite-eval.test.ts): opt-in, gated on OPENAI_API_KEY via describe.skipIf so it auto-skips in normal CI and runs only with a real key. Drives the REAL OpenAIDistiller (real OpenAI, gpt-4o-mini, temp 0) through judgeDistillation on a WHAT-framed admin-ops fragment that reliably lands the judge on the rewrite branch, and asserts a rewritten verdict retains POST /admin/:op, timingSafeEqual, 401, and trust_proxy (a distilled pass-through is an acceptable pass). This is the red-green proof for the prompt itself — an aimock replay cannot prove a prompt change. Verified RED pre-fix (verdict rewritten, POST /admin/:op dropped) and GREEN post-fix. Layer-1 aimock plumbing guard (atlas-distillation-gate.test.ts): a deterministic fixture whose mocked judge returns a rewritten verdict retaining the tokens, asserting enforceDistillation flows the rewrite content/title through intact. Replay-based, so it locks the gate contract regardless of the prompt.
The distilled branch previously called expectSpecificsRetained on the static ADMIN_OPS_INPUT.content constant — a tautology that always passes regardless of model output, since a distilled verdict returns no content of its own (it is a pass-through signal). A future regression routing a concrete-mechanism claim to distilled would have passed vacuously. Encode the actual contract with a single guard-independent assertion: expect(["rewritten", "distilled"]).toContain(verdict.kind) — a concrete- mechanism claim is acceptable EITHER as a rewrite retaining every specific OR as distilled pass-through (original kept intact), but NEVER as a restatement (which drops it). That assertion runs on every verdict and is not implied by any enclosing branch, so a regression to restatement fails loud. Only the rewritten branch then token-checks the model's own content. (An earlier revision asserted kind === "distilled" inside a kind === "distilled" guard — structurally vacuous, the same defect.) Also drop the duplicated JUDGE_MODEL literal entirely: construct the distiller with no model option so it resolves its own unexported DEFAULT_MODEL exactly as production does — removing the drift surface a hard-coded pin would carry. And mark the gate test's resetMatchCounts() as defensive/no-op for these content-matched (non-sequenceIndex) fixtures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompt-only fix so the Atlas distillation judge's rewrite branch retains concrete verifiable detail instead of abstracting it into generic WHY prose.
How this was found
A live prod-read-only dry-run of the just-merged harvest pipeline (#135) over real memory fragments. One candidate — a precise admin-ops claim (
POST /admin/:op,202/400/401/503,timingSafeEqual,trust_proxyfail-closed) — was rewritten by the judge into "unified authentication enhances security and simplifies access control," dropping every specific. The 5 already-well-formed candidates were correctly passed through untouched; only the rewrite path over-abstracted.Root cause
DISTILLATION_SYSTEM_PROMPThad no preservation constraint — it asked for "why/how prose" and only guarded against inventing reasoning, never against dropping facts. Its WHY-vs-WHAT framing had no category for a precise HOW claim, so verifiable mechanics got recoded as low-value WHAT and "elevated" to rationale.The fix (3 edits to
DISTILLATION_SYSTEM_PROMPT, no gate/parsing changes)distilled— keep as-is, don't rewrite up into rationale.distilledinstead — losing a verifiable specific is worse than slightly-WHAT-flavored prose.Proof (real-LLM red-green — the actual failure surface, not fakes)
An opt-in eval (
OPENAI_API_KEY-gated,skipIfso it's out of normal CI) runs the real judge on the admin-ops fragment:rewritten,POST /admin/:opdropped — 3/3.rewrittenretainingPOST /admin/:op+timingSafeEqual+202/400/401/503+trust_proxy— 4/4.Plus a Layer-1 aimock gate-plumbing contract guard, and this closes the
judgeDistillationzero-coverage gap.Review
7 reviewers (Tier 2), 3 rounds. The prompt diff was clean (source-gate + generalist reviewers: no findings). One in-subject test-quality fix — the eval's
distilled-fallback branch was made guard-independent (toContain(["rewritten","distilled"]), fails loud on arestatementregression) instead of a vacuous re-check. Converged with zero in-subject bucket-(a).Blast radius: the pass-through cases can only become more likely to pass through;
restatementuntouched. 3410 tests green; the eval isskipIf-gated.Follow-ups (non-blocking, tracked in the CR ledger)
distilledpath to assert the gate-layer pass-through invariant.stripGitHubMetadataHeader.llm.tshardening surfaced by review (public→internal floor,CandidateFragmentSchema.parse, embed finite-element check).