fix: 消除对抗式复核的反驳过度 (A 保留默认 + B 视角分化 + C 全票否决)#15
Merged
Conversation
…eralFindings) Architecture findings are BanGD's differentiator, but emitting only those cedes ordinary bugs (off-by-one, swallowed errors, nil deref, races) to Copilot. Add a second result class, generalFindings: concrete diff-evidenced correctness/logic defects, lightweight (no four-段式), filed inline in the PR comment (not as tracked issues). - schema: GeneralFindingSchema + generalFindings (.default([]) for graceful degradation), kept in sync with the tool JSON schema by a test - prompt/system-prompt: 4th output part + quality red-lines (diff-evidenced only, no style nits, no dup of architecture findings, <=6, [] when none) - verify: generalize VerifyOutcome<T>/verifyItems<T>; verifyGeneralFindings runs the SAME adversarial majority-refute pass (refuter prompt reframed to "is this a real correctness defect") - review: verify both finding kinds in parallel; ReviewOutcome.droppedGeneralFindings - format: render general findings inline; omit the section when empty - action(.yml): general_finding_count output; dropped count covers both kinds - DESIGN.md §七: why both classes coexist, structural/delivery differences - tests: rendering, verification, schema-sync, default-omit coverage Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR #13 shipped the generalFindings niche described in the system prompt but with no worked example, while every architecture dimension has one. A wrong-or- absent few-shot is the single biggest quality lever per CLAUDE.md, so add the missing exemplar. - prompts/examples/general-findings.md: a worked example (binary-search off-by- one that infinite-loops) showing the 7-field generalFinding shape AND, just as important, the red-line — what NOT to report (style/naming nits, "add a test", unfounded speculation, dup of an architecture finding), plus the boundary vs architecture findings. - prompt.ts: assembleSystemPrompt takes an always-on generalExample, appended in its own block regardless of selected dimensions (generalFindings is requested on every review). Architecture examples relabeled "架构级 Few-shot 范例". - prompts.ts: PromptTexts.generalExample, loaded unconditionally (not dimension- gated); lives in the prompt-cached system block so marginal token cost ~= nil. - DESIGN.md §七: note the niche now ships with its own few-shot (parity with the per-dimension architecture examples). - tests: new prompt.test.ts (assembleSystemPrompt always-on behavior + user prompt parts); prompts.test.ts loads + red-line check; review.test.ts asserts the exemplar is present regardless of dimension selection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…DESIGN §六.2) Builds the measurement scaffolding that turns "feels accurate" into precision/recall, and runs the first real A/B. The corpus is 6 REAL BanDB PRs (eval/cases.json, frozen from GitHub), labels anchored on author intent or a maintainer-confirmed fix (e.g. #58's crash-consistency + Stat→Seek race, which BanGD flagged and the maintainer fixed in #63). PR bodies are neutralized so the model detects from code, not from a body that states the answer. - src/eval/score.ts: pure precision/recall/F1 scorer; fuzzy match on basename + accepted type/category, one-to-one greedy. Fully unit-tested (13 tests). - src/eval/corpus.ts + eval/cases.json + eval/build-corpus.mjs: the frozen real-PR corpus and its (re-runnable) builder. - src/eval/run.ts: single-variable A/B (adversarial verify off vs on, all else held constant, fresh client per config to isolate tokens); writes docs/评测报告.md. Key from ANTHROPIC_API_KEY/DEEPSEEK_API_KEY or the gitignored eval/.apikey; npm run eval. - docs/评测报告.md: the first run (DeepSeek). Honest, counterintuitive result — on the cheap model the adversarial refuters OVER-refute: recall 100%→0% (true findings killed), F1 35.3%→0%, +173% tokens. Empirically backs DESIGN §二 (model choice is the divide) and motivates §六.3 (perspective-diverse refuters instead of homogeneous ones). Real accuracy must be re-run on Opus. Stacks on the generalFindings branch (eval scores result.generalFindings too). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…findings) The first eval run (docs/评测报告.md) exposed a disaster: on a weak refuter model the verification pass deleted EVERY true finding (recall 100%→0%, F1→0, +173% tokens). Root cause: the refuter prompt said "default to refuted when evidence is thin", and N identical refuters echoed the same over-refute. Fix: - (A) keep-by-default: a refuter rejects a finding only when it can cite concrete evidence it is wrong; "unsure" keeps it. Inverts the old uncertain⇒refute bias. - (B) perspective-diverse lenses: each refuter scrutinizes one angle hardest (diff-grounding / misread / can-it-happen) but returns a HOLISTIC verdict, so N refuters decorrelate. (Holistic, not per-axis — per-axis refuting would let a one-axis FP survive unanimity; DESIGN §六.3.) - (C) unanimous-to-drop: a finding is dropped only if every refuter rejects it — one defender keeps it. Makes killing a real finding hard. Re-run on DeepSeek confirms the fix: recall held 33.3%→33.3% (the true finding #43 survived verification, vs being killed before); F1 22.2%→40%; +5% tokens. The report is honest about confounds: part of the apparent precision gain is a DeepSeek parse-error artifact (#35) and the A/B confounds verify-effect with generation non-determinism — the clean claim is "over-refute eliminated". Also surfaced: DeepSeek returned unparseable output 3/12 times (#58 ×2, #35 ×1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
PR #14 的评测暴露了一个灾难：在弱反驳者模型（DeepSeek）上，对抗式复核把所有真 finding 都误杀了（召回 100%→0%，F1→0，token +173%）。根因是反驳者提示词"存疑即否" + N 个同质反驳者齐声否决。
修复（三件事）
重跑验证（DeepSeek，诚实）
已证实：over-refutation 消除——#43 的真并发 finding 在复核后存活，旧版会杀掉它。
诚实披露的混淆（写进报告，不粉饰）：
详见
docs/评测报告.md的"结论与诚实解读 / 修复前后对比 / 局限"。104 测试全绿（新增 A 的 keep-default 检查、B 的视角分化检查；C 的全票语义替换旧多数否决测试）。
🤖 Generated with Claude Code