feat: execution arbiter default-on for debug/review (v3.12.0)

.claude-plugin/marketplace.json

@@ -9,7 +9,7 @@
     {
       "name": "synod",
       "description": "Multi-agent deliberation with 3-round structured debate keyed on mechanical signals: a debate gate on claim agreement, trust derived from verified file:line citation rates, a lossless claim ledger with mandatory dissent, and 5 modes (review, design, debug, idea, general)",
-      "version": "3.11.0",
+      "version": "3.12.0",
       "author": {
         "name": "quantsquirrel"
       },

CHANGELOG.md

@@ -17,6 +17,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Run the live S0-vs-S3 GSM8K ablation** (`SYNOD_BENCH_LIVE=1 … --live --n 50`) and commit results to `benchmark/results/`. Now measures 50 real questions rather than 10 — but read the power caveat first: this arm bounds cost, it does not settle whether debate helps
 - Validate the judgment-task set before trusting any number it produces: measure separability (bootstrap CIs) and judge-vs-human agreement, per [arXiv:2408.08808](https://arxiv.org/abs/2408.08808). Candidates for an externally-validated primary arm: CODAL-Bench / [CodeUltraFeedback](https://arxiv.org/abs/2403.09032), [CodeJudgeBench](https://arxiv.org/pdf/2507.10535), [Arena-Hard-Auto](https://github.com/lmarena/arena-hard-auto). Licence/redistribution terms unchecked
 - Fix the LiveRunner topology mismatch: Synod's Claude is the **in-session** model (`synod-phase1-solver.md` writes `CLAUDE_SOLVER_RESPONSE` from the session), but LiveRunner calls the Anthropic API with a hardcoded `claude-sonnet-5` / 1024-token config, and runs 2 solvers where Phase 1 has 3 (claude Validator + gemini Architect + openai Explorer). So live numbers do not measure Synod as shipped, and `ANTHROPIC_API_KEY` is a harness artifact rather than a Synod requirement
+- Give the execution arbiter a green baseline — it currently runs the target suite with no "was this already red?" reference, so a **pre-existing** failing test is reported as machine-verified evidence. Deliberately not pre-built: fix it if false signals actually show up in use
+
+---
+
+## [3.12.0] - 2026-07-30
+
+### Changed
+
+- **Execution arbiter is default-on for debug/review** (`SYNOD_EXEC_ARBITER`,
+  `0` → `1`). This is a one-line gate change, not new machinery.
+
+  The gate already required `MODE` to be `debug` or `review`, a `TARGET_PATH`,
+  and a Phase 0.5 probe that collected at least one test. Those three
+  conditions select exactly the situations where execution can settle
+  something — so the extra opt-in flag was suppressing a signal the pipeline
+  had already qualified, and the default path went on settling code disputes
+  by argument. Execution-grounded selection is how SWE-bench SOTA picks among
+  candidates ([CWM, arXiv:2510.02387](https://arxiv.org/abs/2510.02387)) and it
+  is the strongest mechanical signal available, which is the direction 3.8-3.10
+  moved everything else.
+
+  The check is `"${SYNOD_EXEC_ARBITER:-1}" == "1"` rather than `!= "0"`:
+  because this step executes code, an unrecognised value (a typo like
+  `SYNOD_EXEC_ARBITER=flase`) fails toward NOT running.
+
+  **What this means in practice:** in debug/review mode against a repo with a
+  collectable pytest suite, Synod now runs that suite. pytest imports
+  `conftest.py` and every test module during collection, so set
+  `SYNOD_EXEC_ARBITER=0` for targets whose suite has real side effects (live
+  services, shared databases, outbound mail) or is already red — there is no
+  baseline yet, so a pre-existing failure is reported as machine-verified
+  evidence. Bounds are unchanged: `pytest -x -q`, 120s hard timeout, timeout
+  treated as UNSETTLED rather than failing, and the arbiter never blocks the
+  pipeline.
+
+  Docs updated in lockstep: `SKILL.md` flag table, the Phase 2 module, both
+  READMEs' research-basis rows, and the marketplace copy that described the
+  arbiter as opt-in.
 
 ---
 

README.ko.md

@@ -344,7 +344,7 @@ v3.8부터 자기보고 confidence는 통제 신호에서 표시/바닥값 전
 | **dynamic rounds 기계 삭제** (v3.8) | `TOTAL_ROUNDS`는 실행을 바꾸지 않는 플라시보 | 프로토콜 노브는 참가자 강도/다양성 대비 2차 요인 ([arXiv:2511.07784](https://arxiv.org/abs/2511.07784)); 폭이 깊이를 이김 ([arXiv:2605.01566](https://arxiv.org/abs/2605.01566)) | 평탄점은 다중 출처 |
 | **인용 검증기: 파일 존재 + 라인 범위, 모델별** (v3.9) | 증거 게이트가 인용 모양 문자열을 세기만 함 — 날조 `utils.py:9999`도 증거로 점수 | 근거 있는 토론이 우세 (+5.5%, [Tool-MAD, arXiv:2601.04742](https://arxiv.org/abs/2601.04742)); 실패의 21%가 약한 검증 ([MAST, arXiv:2503.13657](https://arxiv.org/abs/2503.13657)) | 단일 preprint들, 방향 수렴; 카운팅 결함은 로컬 확인됨 |
 | **무손실 claim 원장 + 필수 소수 의견 섹션** (v3.9) | Phase 2가 솔버당 한 문장으로 압축 — 문헌이 측정한 사실 소실 메커니즘 그대로 | 라운드 진행 중 핵심 사실의 최대 72% 소실 ([arXiv:2606.03032](https://arxiv.org/abs/2606.03032)); 주관/설계 과제에서 76–89% 드리프트 ([arXiv:2502.19559](https://arxiv.org/abs/2502.19559)); 의견 분기의 ~25%는 소수가 옳음 ([arXiv:2606.29270](https://arxiv.org/abs/2606.29270)) | 단일 preprint들, 상호 보강 |
-| **실행 중재자 (debug/review)** (v3.9, `SYNOD_EXEC_ARBITER=1`) | 실행 가능한 테스트가 있어도 코드 분쟁을 수사로 판정 | 실행 기반 후보 선택이 SWE-bench SOTA의 방식 ([CWM, arXiv:2510.02387](https://arxiv.org/abs/2510.02387)) | 제품/벤치마크 기반 패턴 |
+| **실행 중재자 (debug/review)** (v3.9, **v3.12부터 기본 활성화**) | 실행 가능한 테스트가 있어도 코드 분쟁을 수사로 판정. v3.11까지 `SYNOD_EXEC_ARBITER=1` 뒤에 있어 기본 경로는 여전히 논증으로 판정 | 실행 기반 후보 선택이 SWE-bench SOTA의 방식 ([CWM, arXiv:2510.02387](https://arxiv.org/abs/2510.02387)). 게이트가 이미 debug/review 모드 + `TARGET_PATH` + probe가 테스트 ≥1개 수집을 요구했고 — 실행이 판정할 수 있는 상황을 정확히 골라내는 조건 — 따라서 추가 opt-in 플래그는 파이프라인이 이미 검증한 신호를 억제만 하고 있었음 | 제품/벤치마크 기반 패턴, bounded (pytest `-x`, 하드 타임아웃, 타임아웃=UNSETTLED). 정직한 주의: baseline 없이 대상 스위트를 돌리므로 **사전에 실패하던** 테스트가 machine-verified 증거로 보고됨 — red이거나 부작용 있는 스위트는 `SYNOD_EXEC_ARBITER=0` |
 | **CRIS 루브릭을 기계적 trust로 강등** (v3.10) | trust가 Claude의 자기/경쟁자 채점(C/R/I/S 밴드)이었음 | LLM 판사 trust 개입은 net-negative ([arXiv:2606.29270](https://arxiv.org/abs/2606.29270)); 신뢰성은 판단이 아니라 검증에서 옴 ([MAST, arXiv:2503.13657](https://arxiv.org/abs/2503.13657)). TARGET_PATH 있음: `T = 0.25 + 1.75 × 검증된 인용률` (전량 날조→0.25 제외, 전량 검증→2.0 상한, 판정 불가→중립 1.0). 없음: 균등 1.0 — 자기 채점 대체물 없음. trust-scores.json 스키마는 `basis` 필드 추가로 유지 | 반증은 단일 논문이나 대체 신호는 감사 가능한 실측 |
 | **판단 과제 arm 추가, GSM8K arm은 비용 측정으로 강등** (미출시) | Synod의 유일한 자기 측정이 GSM8K S0-vs-S3 ablation이었고, 라이브 경로는 문서상 50문항이 아니라 단일 단계 10문항을 조용히 실행 — 고쳐도 arm을 구분할 검정력이 없음 | 토론에 대한 부정적 결과는 *검증 가능한* 과제에 집중되고 ([arXiv:2508.17536](https://arxiv.org/abs/2508.17536); Smit et al. ICML 2024), 예측된 이득은 정답을 확인할 수 없는 과제에 있음 → 판별 arm은 개방형이어야 함. `benchmark/judgment_eval.py`는 자체 작성 설계/리뷰 50과제 × 루브릭 4기준을 익명화·위치 교환·루브릭 분해된 교차 계열 판사로 채점하고 ([arXiv:2305.17926](https://arxiv.org/abs/2305.17926), [arXiv:2404.13076](https://arxiv.org/abs/2404.13076), [arXiv:2604.23178](https://arxiv.org/abs/2604.23178)), 위치 뒤집힘률 30% 초과 시 **승자 선언을 거부** | 탈편향 조치는 문헌 근거; 과제 세트는 자체 작성이라 외부 검증 없음, 라이브 실행도 아직 미결제 — 하네스가 두 사실을 모두 명시 |
 | **S0 킬러 베이스라인 하네스 + 라이브 러너** (v3.10) | 핵심 가치 주장을 검증할 ablation이 부재; LiveRunner는 은퇴한 CLI 대상 스텁 | MAD 이득의 대부분은 독립 답변 + 집계로 설명됨 ([Smit et al., ICML 2024](https://arxiv.org/abs/2311.17371); martingale [arXiv:2508.17536](https://arxiv.org/abs/2508.17536)) — S0(독립 + 합성 1회, 교차 대화 0)가 S1/S2/S3와 나란한 일급 arm이 되고 LiveRunner가 현행 direct-API 레인을 대상으로 함 | 하네스는 mock 검증 완료; **라이브 수치는 미실행** — mock S3는 여전히 설계상 정답, mock S0 합성은 정직한 다수결 |

README.md

@@ -348,7 +348,7 @@ so future tuning argues with the citations, not with vibes.
 | **Dynamic rounds machinery deleted** (v3.8) | `TOTAL_ROUNDS` was a session label that never changed execution — a placebo knob | Protocol knobs are second-order versus participant strength/diversity ([arXiv:2511.07784](https://arxiv.org/abs/2511.07784)); width beats depth on the compute Pareto frontier ([arXiv:2605.01566](https://arxiv.org/abs/2605.01566)) | multi-source for the plateau; the deletion itself is a repo fact |
 | **Citation verifier: file-exists + line-in-range, per model** (v3.9) | The evidence gate *counted* citation-shaped strings — fabricated `utils.py:9999` scored as evidence | Grounded debate beats ungrounded (+5.5%, [Tool-MAD, arXiv:2601.04742](https://arxiv.org/abs/2601.04742)); 21% of multi-agent failures trace to weak verification ([MAST, arXiv:2503.13657](https://arxiv.org/abs/2503.13657)) | single-paper 2026 preprints, but convergent direction; the counting flaw was locally verified |
 | **Lossless claim ledger replaces ≤30-word summaries; mandatory Dissent section** (v3.9) | Phase 2 compressed each solver to one sentence — the exact factual-attrition mechanism the literature measures; evidenced minority views could vanish silently | Up to 72% of issue-critical facts erased across rounds while stances homogenize ([The Deliberative Illusion, arXiv:2606.03032](https://arxiv.org/abs/2606.03032)); 76–89% problem drift on subjective/design tasks ([Stay Focused, arXiv:2502.19559](https://arxiv.org/abs/2502.19559)); in ~25% of divergent cases the minority is right and judge-driven majority overrides test net-negative ([Minority Sentinel, arXiv:2606.29270](https://arxiv.org/abs/2606.29270)) | single-paper 2026 preprints, mutually corroborating |
-| **Execution arbiter for debug/review** (v3.9, `SYNOD_EXEC_ARBITER=1`) | Code disputes were settled by rhetoric even when the target repo had a runnable test suite | Execution-grounded candidate selection is how SWE-bench SOTA picks answers ([CWM, arXiv:2510.02387](https://arxiv.org/abs/2510.02387)); models should debate only what execution cannot settle | product/benchmark-backed pattern; Synod's implementation is bounded (pytest `-x`, hard timeout, timeout = UNSETTLED) |
+| **Execution arbiter for debug/review** (v3.9; **default-on since v3.12**) | Code disputes were settled by rhetoric even when the target repo had a runnable test suite. Through v3.11 it stayed behind `SYNOD_EXEC_ARBITER=1`, so the default path still settled code questions by argument | Execution-grounded candidate selection is how SWE-bench SOTA picks answers ([CWM, arXiv:2510.02387](https://arxiv.org/abs/2510.02387)); models should debate only what execution cannot settle. The gate already required debug/review mode + a `TARGET_PATH` + a probe that collected ≥1 test — conditions that select exactly the cases where execution *can* settle something — so the extra opt-in flag suppressed a signal the pipeline had already qualified | product/benchmark-backed pattern; bounded (pytest `-x`, hard timeout, timeout = UNSETTLED). Honest caveat: it runs the target's own suite with no baseline, so a **pre-existing** failing test is reported as machine-verified evidence — set `SYNOD_EXEC_ARBITER=0` for red or side-effecting suites |
 | **CRIS rubric demoted to mechanical trust** (v3.10) | Trust was Claude self-grading itself and rivals on unmeasurable qualities (C/R/I/S bands) | LLM-judge trust overrides tested net-negative ([Minority Sentinel, arXiv:2606.29270](https://arxiv.org/abs/2606.29270)); verification, not judgment, is where reliability comes from ([MAST, arXiv:2503.13657](https://arxiv.org/abs/2503.13657)). With TARGET_PATH: `T = 0.25 + 1.75 × verified-citation-rate` (all-fabricated → excluded at 0.25; all-verified → 2.0 cap; nothing decidable → neutral 1.0). Without: uniform 1.0 — no self-graded substitute. trust-scores.json schema unchanged with a `basis` field | the counter-indication is single-paper; the replacement signal is auditable ground truth |
 | **Judgment-task arm added; GSM8K arm demoted to a cost measurement** (unreleased) | Synod's only self-measurement was an S0-vs-S3 ablation on GSM8K — and its live path silently ran 10 single-step problems instead of the documented 50, so it could not have separated the arms even once fixed | The negative results on debate concentrate on *verifiable* tasks ([Debate or Vote, arXiv:2508.17536](https://arxiv.org/abs/2508.17536); Smit et al. ICML 2024), while the predicted gains are on tasks with no checkable answer — so the discriminating arm must be open-ended. `benchmark/judgment_eval.py` runs 50 authored design/review tasks × 4 rubric criteria through an anonymised, position-swapped, rubric-decomposed cross-family judge ([arXiv:2305.17926](https://arxiv.org/abs/2305.17926), [arXiv:2404.13076](https://arxiv.org/abs/2404.13076), [arXiv:2604.23178](https://arxiv.org/abs/2604.23178)), and **refuses to name a winner** above a 30% position-flip rate | the debiasing measures are literature-backed; the task set is authored by this repo and has no external validation, and no live run has been paid for yet — the harness is honest about both |
 | **S0 killer-baseline harness + live runner** (v3.10) | Synod had never run the one ablation that tests its core value claim; LiveRunner was a `NotImplementedError` stub targeting retired CLIs | Most of MAD's measured gains are explained by independent answers + aggregation ([Smit et al., ICML 2024](https://arxiv.org/abs/2311.17371); martingale result [arXiv:2508.17536](https://arxiv.org/abs/2508.17536)) — S0 (independent + one synthesis pass, zero cross-talk) is now a first-class arm alongside S1/S2/S3, and LiveRunner targets the current direct-API lanes | harness shipped and mock-validated; **live numbers still pending** — mock S3 remains scripted-correct by construction, mock S0 synthesis is an honest majority vote |

marketplace.json

@@ -1,9 +1,9 @@
 {
   "name": "synod",
-  "version": "3.11.0",
+  "version": "3.12.0",
   "displayName": "Synod: Multi-Agent Debate System",
   "description": "Research-backed multi-agent deliberation with Claude, Gemini, and OpenAI - structured 3-round debates for superior decision-making",
-  "longDescription": "Synod brings peer-reviewed multi-agent debate methodologies to Claude Code. Instead of relying on a single LLM prone to confirmation bias and hallucination, Synod orchestrates structured debates between Claude, Gemini, and OpenAI across three rounds: Solver, Critic, and Defense/Prosecution. Originally built on ReConcile (ACL 2024), AgentsCourt, ConfMAD and A-HMAD, and since revised wherever later research contradicted them — see the CHANGELOG for the audit trail. Decisions are keyed on MECHANICAL signals rather than self-reported ones: a debate gate on claim agreement, trust derived from the verified file:line citation rate, a lossless claim ledger with a mandatory Dissent section, anonymised cross-model review by default, and an opt-in execution arbiter that settles code disputes by running the target repo's tests. Self-reported confidence is displayed but never gates a decision. Suited to code review, architecture design, debugging, and brainstorming.",
+  "longDescription": "Synod brings peer-reviewed multi-agent debate methodologies to Claude Code. Instead of relying on a single LLM prone to confirmation bias and hallucination, Synod orchestrates structured debates between Claude, Gemini, and OpenAI across three rounds: Solver, Critic, and Defense/Prosecution. Originally built on ReConcile (ACL 2024), AgentsCourt, ConfMAD and A-HMAD, and since revised wherever later research contradicted them — see the CHANGELOG for the audit trail. Decisions are keyed on MECHANICAL signals rather than self-reported ones: a debate gate on claim agreement, trust derived from the verified file:line citation rate, a lossless claim ledger with a mandatory Dissent section, anonymised cross-model review by default, and an execution arbiter that settles code disputes in debug/review mode by running the target repo's own test suite (default-on; set SYNOD_EXEC_ARBITER=0 to disable). Self-reported confidence is displayed but never gates a decision. Suited to code review, architecture design, debugging, and brainstorming.",
   "author": {
     "name": "quantsquirrel",
     "email": "",
@@ -61,7 +61,7 @@
     "Citation verifier: file:line claims checked against the target repo; fabrications excluded",
     "Mechanical trust: T = 0.25 + 1.75 × verified-citation rate — no self-graded rubric",
     "Lossless claim ledger with a mandatory Dissent section",
-    "Execution arbiter (opt-in): code disputes settled by running the target repo's tests",
+    "Execution arbiter (default-on for debug/review): code disputes settled by running the target repo's tests",
     "Anonymised cross-model review by default",
     "5 specialized modes: review, design, debug, idea, general",
     "Session management and resume capability",

plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "synod",
-  "version": "3.11.0",
+  "version": "3.12.0",
   "description": "Multi-agent deliberation system with Claude, Gemini, and OpenAI - structured debate for better decisions",
   "author": {
     "name": "quantsquirrel"

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "synod"
-version = "3.11.0"
+version = "3.12.0"
 description = "Multi-agent deliberation system for Claude Code"
 readme = "README.md"
 license = {text = "MIT"}

skills/synod/SKILL.md

@@ -36,7 +36,7 @@ You are the **Synod Orchestrator** - a judicial coordinator managing a multi-mod
 | `SYNOD_EVIDENCE_FIRST` | `0` | 증거 우선 Phase 0.5/4.5 활성화 (`1`=enabled, 또는 `--evidence-first`) |
 | `SYNOD_DEBATE_GATE` | `1` | Phase 1.5 합의 게이트 (v3.8부터 기본 활성화); 솔버 합의 시 Phase 2-3 우회, deep/ultra 티어는 항상 전체 토론. `0`=항상 전체 토론 |
 | `SYNOD_ANONYMIZE` | `1` | 숙의 익명화 (v3.8부터 기본 활성화) — 외부 CLI의 브랜드 아첨 방지. `0`=비활성화 |
-| `SYNOD_EXEC_ARBITER` | `0` | v3.9 실행 중재자 — debug/review 모드 + TARGET_PATH 존재 시 대상 테스트 스위트를 bounded 실행해 사실 분쟁을 기계적으로 판정 (`1`=enabled) |
+| `SYNOD_EXEC_ARBITER` | `1` | v3.9 실행 중재자 (v3.12부터 기본 활성화) — debug/review 모드 + TARGET_PATH 존재 + probe가 테스트 1개 이상 수집 시 대상 테스트 스위트를 bounded 실행해 사실 분쟁을 기계적으로 판정. **대상 리포의 테스트를 실제로 실행**하므로(pytest 수집이 `conftest.py`와 테스트 모듈을 import) 부작용 있는 스위트에는 `0`=비활성화 |
 
 > **v3.8에서 제거됨:** `SYNOD_V2_DYNAMIC_ROUNDS` (동적 라운드 수) — 라운드 수는 세션
 > 라벨일 뿐 실행을 바꾸지 않는 플라시보였음. 복잡도는 티어 선택에만 사용되며, 적응형