Skip to content

fix(audit): honesty fixes from benchmark run 1#127

Merged
DocRoms merged 1 commit into
mainfrom
fix/audit-honesty-benchmark-findings
Jul 21, 2026
Merged

fix(audit): honesty fixes from benchmark run 1#127
DocRoms merged 1 commit into
mainfrom
fix/audit-honesty-benchmark-findings

Conversation

@DocRoms

@DocRoms DocRoms commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Findings du benchmark A/B 0.8.13

Le run 1 du gate pré-MEP a attrapé deux défauts d'honnêteté réels : la mécanique terminale (Interrupted, zéro baseline, zéro validation) a par ailleurs tenu parfaitement.

  • Placeholder detector : le code markdown est invisible au compteur : l'index TD documentait {{DO_NOT_1}}/{{DO_NOT_2}} en code inline et un fichier de 43 TDs a échoué le step 8 comme « still the template ». Strippers anti_halluc partagés, durcis fail-closed (5 passes de review adversariale) : spans inline appariés par runs de N backticks (run non fermé = texte gardé), fences lexicales strictes (opener ≥3 backticks / ≤3 espaces / info-string sans backtick ; closer même char / run ≥ / suffixe espaces-tabs only ; non fermé = tout restauré, ligne d'ouverture incluse).
  • Compteurs de réconciliation persistés sur Interrupted : resolved/new étaient calculés puis jetés : « 0 new » s'affichait comme valeur connue alors qu'un TD réellement nouveau existait. La partition exacte atterrit désormais en DB.

Tests

  • Backend 3562 (dont 8 pins nouveaux : repros exacts du banc + faux opener / faux closer / markdown tronqué) · clippy 0 · double review Claude+Codex en room

Gate : après merge → rebuild → session MCP neuve (bridge_info stale=false) → rerun benchmark v3.7 complet.

@DocRoms DocRoms added the ci-test Start all the CI Test steps label Jul 21, 2026
@DocRoms
DocRoms force-pushed the fix/audit-honesty-benchmark-findings branch from 1091c14 to f991bbb Compare July 21, 2026 13:32
@DocRoms DocRoms self-assigned this Jul 21, 2026
@DocRoms
DocRoms requested a review from Copilot July 21, 2026 13:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes two “honesty” issues surfaced by benchmark run 1: (1) placeholder detection falsely counted placeholders that were merely documented inside Markdown code (inline/fenced), and (2) Interrupted audit runs failed to persist reconciliation counters (resolved/new), causing the UI to show misleading “0 new” values.

Changes:

  • Persist td_resolved_since_last and td_new_since_last alongside the other TD counters for Interrupted runs via update_td_counts, and update the Interrupted finalize path to pass these values.
  • Harden Markdown fenced-code stripping to be lexical and fail-closed (unclosed fences restore withheld content) and expose the stripping helpers for reuse within the crate.
  • Update placeholder counting to run on text with fenced blocks and inline code spans removed, and add targeted regression tests for the benchmark failure modes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
backend/src/db/audit_runs.rs Extends update_td_counts to persist reconciliation counters and adds/updates tests for Interrupted runs.
backend/src/core/anti_halluc.rs Reworks fenced-code stripping to strict lexical rules with fail-closed semantics; exposes stripping helpers; adds regression tests.
backend/src/api/audit/validation.rs Applies the shared Markdown strippers before counting raw {{IDENT}} placeholders; adds regression tests to ensure code spans/fences don’t count.
backend/src/api/audit/full.rs Updates the Interrupted terminal write path to pass resolved/new/carried into update_td_counts.

Signed-off-by: Romuald Priol <romuald.priol@protonmail.com>
@DocRoms
DocRoms force-pushed the fix/audit-honesty-benchmark-findings branch from f991bbb to 69bf8ed Compare July 21, 2026 13:39
@DocRoms
DocRoms merged commit aa74d8f into main Jul 21, 2026
9 checks passed
@DocRoms
DocRoms deleted the fix/audit-honesty-benchmark-findings branch July 21, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-test Start all the CI Test steps

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants