Skip to content

Commit 37c5f62

Browse files
shaypal5claude
andauthored
fix(tts): #87 partial — effective-prosody cap addresses Whisper backdoor + helium range (#90)
The bisect on PR #86 showed the residual sp_it_a_0001 WER regression (0.322 vs 04-15's 0.056) is caused by M7 SpeakerState drift compounding with #51's M15 style_map values, producing effective pitch +14 % to +17 % and rate 1.27-1.33x at high-intensity turns. That range simultaneously sounds cartoonish to listeners (May-3 listening test "helium / oompa- loompa") and trips Whisper-large-v3's silence-detection heuristic — the classic length-ratio collapse to ~0.7 that hid the bug for weeks. This PR ships a partial fix: a runtime effective-prosody cap that addresses the canonical Whisper-backdoor fingerprint and the helium- range pitch concern, plus the two detection layers Shay asked for to catch this class of regression in the future. It does NOT fully restore high-intensity WER to the pre-#51 baseline — see #89 for the follow-up workstream. ## Tier-3 Whisper validation (`sp_it_a_0001`) | variant | dur | WER | length_ratio | hyp / ref | |---|---:|---:|---:|---:| | 04-15 reference | 155.9 s | 0.056 | 1.009 | 236 / 234 | | post-#86 main (no cap) | 146.6 s | 0.322 | 0.709 | 166 / 234 | | this PR (cap active) | 149.1 s | **0.129** | **0.906** | 212 / 234 | - Length-ratio recovers above the qa-report --asr 0.85 threshold. - WER reduced 2.5x (0.322 -> 0.129) but still above the 04-15 baseline of 0.056. Failure mode shifts from silence-detector trip (~30 % of words missing) to substitution noise — distinct mechanism requiring a paired listening test to fix without breaking M15 naturalness calibration. Tracked in #89 with insights and four proposed approaches. ## The fix — effective-prosody runtime cap `synthbanshee/tts/renderer._apply_effective_prosody_cap` clamps post- state, post-randomization prosody before SSML emission: - pitch in [-3.0, +2.0] st (~ +/- 12 % Azure) - rate in [0.85, 1.20] - volume left to the existing +/-50 % Azure clamp (Whisper internally normalizes loudness, per #82's lever probe — not a Whisper-trip dimension). Caps are anchored to the pre-#51 effective envelope, which produced the 04-15 reference clips with WER 0.04-0.08. Tighter caps would diverge further from M15 listening-test calibration; looser caps would re-trip Whisper. Each cap activation logs a warning and is recorded per turn. ## Detection layer 1 — static prosody-cap activations in metadata - `DialogueTurn.effective_prosody_caps` carries per-turn cap events. - `cli.py` rolls them up into `ClipMetadata.generation_metadata.effective_prosody_caps` (new `EffectiveProsodyCapEvent` model in labels/schema.py). - `qa-report` surfaces a new "Effective-Prosody Cap Activations (#87)" table per clip — runs on every batch, no Azure / Whisper required. Tier-3 render of sp_it_a_0001 recorded 14 cap activations across 7 high-intensity turns; metadata example in PR description. ## Detection layer 2 — `qa-report --asr` Whisper backdoor check New `synthbanshee/package/asr_sanity.py` provides a lazy-loaded `WhisperRunner` and `compute_asr_metrics`. `qa-report --asr` runs Whisper-large-v3 on every clip in a directory, flags clips whose length-ratio falls below `--asr-min-length-ratio` (default 0.85 — the #87 fingerprint sat at ~0.71). Heavy dependencies isolated in the new `eval-asr` optional extra so normal generation/QA stays light. Per the policy decision documented in CLAUDE.md ("ASR sanity check policy"), Tier-3 ASR sanity is local-only (not in CI) for now — see GH issue #88 for the deferred CI re-evaluation triggers. ## Tests - tests/unit/test_effective_prosody_cap.py: 11 tests covering the helper unit, render_utterance integration, and render_scene event propagation to DialogueTurn. - tests/unit/test_qa.py::TestProsodyCapRollup: 3 tests verifying cap-event aggregation in qa-report. - tests/unit/test_asr_sanity.py: 11 tests covering normalize_for_wer, AsrMetrics threshold semantics, and bracket-line stripping in the reference parser. Heavy Whisper inference is exercised by the Tier-3 local run, not these tests. - 1687 unit tests pass (1662 baseline + 25 new); ruff + mypy clean. ## Docs - CLAUDE.md: new "ASR sanity check policy" section + "What NOT to do" bullets pinning the cap thresholds and the Tier-3 local-only policy. - pyproject.toml: new `eval-asr` optional extra. Reduces #87 (does not fully close — see #89 for the residual WER work). Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent a910f2e commit 37c5f62

11 files changed

Lines changed: 937 additions & 17 deletions

File tree

‎AGENTS.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ SFX onset/offset times come **only** from the Stage 3b augmentation log. Speech-
7070
- **Cache key: SHA-256 of the full rendered SSML string** — not `(voice_id, text)`
7171
- **M3a — per-turn RMS gain:** `StyleEntry.rms_target_dbfs` → module-level `_apply_rms_gain()` in `synthbanshee/tts/mixer.py`; 4-tuple segment API `(wav_bytes, pause_s, speaker_id, rms_target_dbfs)`
7272
- **Temp WAV from mixer must be `subtype="FLOAT"`** — never `PCM_16` before `preprocess()` to avoid hard-clipping M3 gain
73+
- **#87 effective-prosody cap:** `synthbanshee/tts/renderer._apply_effective_prosody_cap` clamps post-state, post-randomization prosody to `pitch ∈ [−3.0, +2.0] st`, `rate ∈ [0.85, 1.20]`. Volume is left to `_volume_to_string`'s ±50% Azure clamp (Whisper internally normalizes loudness; #82 lever probe). Cap activations are recorded per-turn in `DialogueTurn.effective_prosody_caps` and rolled up into `ClipMetadata.generation_metadata.effective_prosody_caps`. Caps are anchored to the pre-#51 effective envelope (the 04-15 reference baseline that Whisper handles with WER 0.04–0.08); changing them requires a paired listening test + Whisper sanity check (see "ASR sanity check policy" below).
7374

7475
## Splits
7576

@@ -85,6 +86,23 @@ SFX onset/offset times come **only** from the Stage 3b augmentation log. Speech-
8586
3. JSON parses as `ClipMetadata` with `is_synthetic=True`
8687
4. Filename stem is ASCII-only lowercase
8788

89+
## ASR sanity check policy (#87)
90+
91+
The Whisper-large-v3 sanity check (`synthbanshee qa-report --asr`) detects audio that synthesizes fine to a listener but trips Whisper's silence-detection / segmentation heuristic — the failure mode that surfaced in #87 and went undetected on `sp_it_a_0001` for weeks. The canonical fingerprint is **length-ratio collapse** (`hyp_words / ref_words < 0.85`).
92+
93+
**Cost:** ~5–20 s per clip on M-series MPS, ~3 GB Whisper-large-v3 download on first use. Real Azure render to populate the cache is an additional ~$0.05–$0.20 per scene.
94+
95+
**Policy — for the foreseeable future, this check is _local-only_, not in CI:**
96+
97+
1. **Unit tests** for the cap logic (in `tests/unit/test_effective_prosody_cap.py`) run on every PR. They cover ~90 % of regressions with no Azure / Whisper cost.
98+
2. **`qa-report --asr` (Tier-3)** is run **locally**, **once**, **after all PR reviews and fixes are done**, **and only for PRs that might affect audio rendering** (anything touching `synthbanshee/tts/`, `synthbanshee/script/`, `synthbanshee/augment/`, or the speaker / scene / acoustic / project YAML configs). PRs that only touch labels, packaging, docs, or tests skip it.
99+
3. **Cost discipline:** use this command **very sparingly** — each invocation incurs Azure spend (~$1–$2 per real-render run). Prefer running against a directory of already-cached clips when possible; only re-render when a code change invalidates the SSML cache.
100+
4. **Result documentation:** when run, paste the `qa-report --asr` output (table of any clips with `length_ratio < 0.85`, plus the green-tick "no asr warnings" line if all clean) into the PR body's "Test plan" section under a "Tier-3 ASR sanity (local)" sub-heading.
101+
102+
**Future CI consideration:** see GH issue tracking the addition of a scheduled / labeled Whisper-based CI workflow, including repo-secret management for `AZURE_TTS_KEY` and Azure spend caps. Re-evaluate when scene volume increases enough that local manual runs become impractical.
103+
104+
**Install:** `uv pip install --python .venv/bin/python -e ".[eval-asr]"` (heavyweight optional extra).
105+
88106
## Environment variables
89107

90108
```
@@ -153,3 +171,5 @@ When addressing PR review comments (Copilot, human, or bot):
153171
- Don't generate clips shorter than 3.0 s
154172
- Don't add a new typology to `taxonomy.yaml` without adding it to `_TYPOLOGY_INTENSITY_MAP`
155173
- Don't pin `pr-agent-context` workflow references to a specific patch version — keep `@v4` floating
174+
- Don't relax the #87 effective-prosody cap (`_EFFECTIVE_PITCH_*`, `_EFFECTIVE_RATE_*` in `synthbanshee/tts/renderer.py`) without a paired listening test + `qa-report --asr` Tier-3 run; the cap defends both naturalness (May-3 listening test) and Whisper transcription
175+
- Don't merge a PR that touches audio rendering (`synthbanshee/tts/`, `synthbanshee/script/`, `synthbanshee/augment/`, or speaker / scene / acoustic / project YAMLs) without running `qa-report --asr` locally first and pasting the result into the PR's test plan (see "ASR sanity check policy")

‎pyproject.toml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,17 @@ google-tts = [
5858
"google-cloud-texttospeech>=2.16",
5959
]
6060

61+
# Whisper-based ASR sanity for `qa-report --asr` (#87 length-ratio detector).
62+
# Heavyweight: pulls torch + transformers + ~3 GB of Whisper-large-v3 weights
63+
# on first use. Not required for normal generation/QA — only install when
64+
# running the ASR sanity check locally (see CLAUDE.md "ASR sanity check policy").
65+
eval-asr = [
66+
"torch>=2.0",
67+
"transformers>=4.40",
68+
"accelerate>=0.30",
69+
"jiwer>=3.0",
70+
]
71+
6172
dev = [
6273
"pytest>=8.0",
6374
"pytest-cov>=5.0",

‎synthbanshee/cli.py‎

Lines changed: 112 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -628,7 +628,7 @@ def vlog(msg: str) -> None:
628628
from collections import Counter
629629

630630
from synthbanshee import __version__
631-
from synthbanshee.labels.schema import GenerationMetadata
631+
from synthbanshee.labels.schema import EffectiveProsodyCapEvent, GenerationMetadata
632632

633633
# Per-speaker TTS backend and voice family.
634634
_backends_map: dict[str, str] = {sid: spk.tts_provider for sid, spk in speakers.items()}
@@ -655,6 +655,19 @@ def vlog(msg: str) -> None:
655655
if turn.speaker_id not in _speaker_states and turn.speaker_state_snapshot:
656656
_speaker_states[turn.speaker_id] = turn.speaker_state_snapshot
657657

658+
# #87: roll up per-turn effective-prosody cap activations into clip metadata.
659+
_cap_events: list[EffectiveProsodyCapEvent] = [
660+
EffectiveProsodyCapEvent(
661+
turn_index=i,
662+
intensity=turn.intensity,
663+
dim=ev["dim"],
664+
pre_cap=ev["pre_cap"],
665+
post_cap=ev["post_cap"],
666+
)
667+
for i, turn in enumerate(turns)
668+
for ev in turn.effective_prosody_caps
669+
]
670+
658671
gen_meta = GenerationMetadata(
659672
pipeline_version=__version__,
660673
tts_backend=_backends_map,
@@ -668,6 +681,7 @@ def vlog(msg: str) -> None:
668681
loudness_target_peak_dbfs=preproc_config.target_peak_dbfs,
669682
breathiness_applied=_breathiness_was_applied,
670683
speaker_state_serialized=_speaker_states,
684+
effective_prosody_caps=_cap_events,
671685
)
672686

673687
metadata = label_gen.generate_clip_metadata(
@@ -1444,8 +1458,29 @@ def validate(clip: Path) -> None:
14441458
default=False,
14451459
help="Compute and display M10b run-level aggregation metrics.",
14461460
)
1461+
@click.option(
1462+
"--asr",
1463+
is_flag=True,
1464+
default=False,
1465+
help="Run Whisper-large-v3 on every clip and flag length-ratio collapse "
1466+
"(the #87 'silence-detector trip' fingerprint). Heavyweight: ~5-20s per "
1467+
"clip on M-series MPS, ~3GB model download. Requires the 'eval-asr' extra.",
1468+
)
1469+
@click.option(
1470+
"--asr-min-length-ratio",
1471+
type=float,
1472+
default=0.85,
1473+
show_default=True,
1474+
help="Length-ratio (hyp_words / ref_words) threshold below which a clip is "
1475+
"flagged by --asr. The #87 fingerprint sat at ~0.70; baseline is 1.00.",
1476+
)
14471477
def qa_report(
1448-
data_dir: Path, output: Path | None, max_failure_rate: float, run_summary: bool
1478+
data_dir: Path,
1479+
output: Path | None,
1480+
max_failure_rate: float,
1481+
run_summary: bool,
1482+
asr: bool,
1483+
asr_min_length_ratio: float,
14491484
) -> None:
14501485
"""Run the automated QA suite on a dataset directory.
14511486
@@ -1455,13 +1490,23 @@ def qa_report(
14551490
from synthbanshee.package.qa import (
14561491
_HIGH_EMOTION_DOWNGRADE_RATE,
14571492
_MIN_VOICE_DIVERSITY,
1493+
run_asr_check,
14581494
run_qa,
14591495
)
14601496

14611497
console.print(f"[cyan]Running QA on:[/cyan] {data_dir}")
14621498
report = run_qa(data_dir, max_failure_rate=max_failure_rate, run_summary=run_summary)
14631499
stats = report.stats
14641500

1501+
# #87: opt-in Whisper sanity check. Runs after run_qa so we already
1502+
# know the clips that validated; ASR results attach to the same report.
1503+
if asr:
1504+
console.print(
1505+
f"[cyan]Running ASR sanity (Whisper-large-v3, "
1506+
f"length_ratio < {asr_min_length_ratio}):[/cyan] {data_dir}"
1507+
)
1508+
report.asr_warnings = run_asr_check(data_dir, min_length_ratio=asr_min_length_ratio)
1509+
14651510
table = Table(title="Dataset Statistics")
14661511
table.add_column("Metric")
14671512
table.add_column("Value", justify="right")
@@ -1476,6 +1521,12 @@ def qa_report(
14761521
)
14771522
table.add_row("Unique speakers", str(stats.speaker_count))
14781523
table.add_row("Quality-flagged", str(stats.quality_flagged_clips))
1524+
if stats.clips_with_prosody_cap_activations > 0:
1525+
table.add_row(
1526+
"Clips with prosody-cap activations (#87)",
1527+
f"[yellow]{stats.clips_with_prosody_cap_activations}[/yellow]"
1528+
f" ({stats.prosody_cap_activations_total} total events)",
1529+
)
14791530
missing_jsonl_label = (
14801531
f"[yellow]{stats.clips_missing_strong_labels}[/yellow]"
14811532
if stats.clips_missing_strong_labels > 0
@@ -1517,6 +1568,44 @@ def qa_report(
15171568
if len(report.failed_clip_ids) > 20:
15181569
console.print(f" ... and {len(report.failed_clip_ids) - 20} more")
15191570

1571+
# #87: prosody-cap activations are surfaced as a top-level table because
1572+
# they catch a class of regression that the M10a/M10b checks cannot —
1573+
# specifically, prosody pushed past the listening-test-validated envelope.
1574+
if report.prosody_cap_activations:
1575+
t_caps = Table(title="Effective-Prosody Cap Activations (#87)")
1576+
t_caps.add_column("Clip ID")
1577+
t_caps.add_column("Cap events", justify="right")
1578+
for clip_id, count in sorted(
1579+
report.prosody_cap_activations.items(),
1580+
key=lambda kv: kv[1],
1581+
reverse=True,
1582+
)[:20]:
1583+
t_caps.add_row(clip_id, str(count))
1584+
console.print(t_caps)
1585+
if len(report.prosody_cap_activations) > 20:
1586+
console.print(f" ... and {len(report.prosody_cap_activations) - 20} more")
1587+
1588+
# #87: ASR sanity surfaces clips whose length-ratio collapsed below the
1589+
# configured threshold — the Whisper "silence-detector trip" fingerprint.
1590+
if report.asr_warnings:
1591+
t_asr = Table(title=f"ASR Sanity Failures (#87, length_ratio < {asr_min_length_ratio})")
1592+
t_asr.add_column("Clip ID")
1593+
t_asr.add_column("WER", justify="right")
1594+
t_asr.add_column("len_ratio", justify="right")
1595+
t_asr.add_column("hyp / ref", justify="right")
1596+
for clip_id, m in sorted(report.asr_warnings.items(), key=lambda kv: kv[1]["length_ratio"])[
1597+
:20
1598+
]:
1599+
t_asr.add_row(
1600+
clip_id,
1601+
f"{m['wer']:.3f}",
1602+
f"[red]{m['length_ratio']:.3f}[/red]",
1603+
f"{m['hyp_words']} / {m['ref_words']}",
1604+
)
1605+
console.print(t_asr)
1606+
if len(report.asr_warnings) > 20:
1607+
console.print(f" ... and {len(report.asr_warnings) - 20} more")
1608+
15201609
# --- M10b: run-level summary table ---
15211610
if report.run_summary is not None:
15221611
rs = report.run_summary
@@ -1595,28 +1684,34 @@ def qa_report(
15951684

15961685
if output is not None:
15971686
output.parent.mkdir(parents=True, exist_ok=True)
1598-
report_dict = {
1687+
stats_dict: dict[str, object] = {
1688+
"total_clips": stats.total_clips,
1689+
"failed_clips": stats.failed_clips,
1690+
"total_duration_seconds": stats.total_duration_seconds,
1691+
"speaker_count": stats.speaker_count,
1692+
"quality_flagged_clips": stats.quality_flagged_clips,
1693+
"clips_missing_strong_labels": stats.clips_missing_strong_labels,
1694+
"clips_by_typology": stats.clips_by_typology,
1695+
"clips_by_split": stats.clips_by_split,
1696+
"clips_by_tier": stats.clips_by_tier,
1697+
"intensity_distribution": {str(k): v for k, v in stats.intensity_distribution.items()},
1698+
# #87: surface the prosody-cap counts in the JSON report so CI /
1699+
# post-batch scripts can detect drift without parsing every clip.
1700+
"clips_with_prosody_cap_activations": stats.clips_with_prosody_cap_activations,
1701+
"prosody_cap_activations_total": stats.prosody_cap_activations_total,
1702+
}
1703+
report_dict: dict[str, object] = {
15991704
"data_dir": report.data_dir,
16001705
"passed": report.passed,
16011706
"failure_rate": report.failure_rate,
1602-
"stats": {
1603-
"total_clips": stats.total_clips,
1604-
"failed_clips": stats.failed_clips,
1605-
"total_duration_seconds": stats.total_duration_seconds,
1606-
"speaker_count": stats.speaker_count,
1607-
"quality_flagged_clips": stats.quality_flagged_clips,
1608-
"clips_missing_strong_labels": stats.clips_missing_strong_labels,
1609-
"clips_by_typology": stats.clips_by_typology,
1610-
"clips_by_split": stats.clips_by_split,
1611-
"clips_by_tier": stats.clips_by_tier,
1612-
"intensity_distribution": {
1613-
str(k): v for k, v in stats.intensity_distribution.items()
1614-
},
1615-
},
1707+
"stats": stats_dict,
16161708
"failed_clip_ids": report.failed_clip_ids,
16171709
"quality_flagged": report.quality_flagged,
16181710
"acoustic_warnings": report.acoustic_warnings,
16191711
"structural_warnings": report.structural_warnings,
1712+
# #87: prosody-cap activations + ASR sanity (empty unless --asr).
1713+
"prosody_cap_activations": report.prosody_cap_activations,
1714+
"asr_warnings": report.asr_warnings,
16201715
}
16211716
if report.run_summary is not None:
16221717
rs = report.run_summary

‎synthbanshee/labels/schema.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,23 @@ class PreprocessingApplied(BaseModel):
126126
silence_padded: bool = False
127127

128128

129+
class EffectiveProsodyCapEvent(BaseModel):
130+
"""One activation of the #87 runtime effective-prosody cap.
131+
132+
Recorded per turn and per dimension when the post-state, post-randomization
133+
prosody value would have exceeded the envelope that Whisper-large-v3 can
134+
transcribe and the listener finds natural at high intensities. See
135+
``synthbanshee.tts.renderer._apply_effective_prosody_cap`` for thresholds
136+
and rationale.
137+
"""
138+
139+
turn_index: int = Field(ge=0)
140+
intensity: int = Field(ge=1, le=5)
141+
dim: Literal["rate", "pitch", "volume"]
142+
pre_cap: float
143+
post_cap: float
144+
145+
129146
class GenerationMetadata(BaseModel):
130147
"""Pipeline provenance metadata for ablation studies (spec §4.11).
131148
@@ -161,6 +178,17 @@ class GenerationMetadata(BaseModel):
161178
breathiness_applied: bool = False
162179
speaker_state_serialized: dict[str, dict[str, float]] = Field(default_factory=dict)
163180

181+
effective_prosody_caps: list[EffectiveProsodyCapEvent] = Field(default_factory=list)
182+
"""Per-turn activations of the #87 effective-prosody cap.
183+
184+
One entry per ``(turn_index, dim)`` clamp. An empty list means every turn's
185+
effective prosody fell within the envelope (post-state, post-randomization).
186+
A non-empty list is a static signal that the renderer pulled the turn's
187+
pitch or rate back from the helium / Whisper-trip range — typically a small
188+
number of high-intensity turns near the end of an arc with significant M7
189+
SpeakerState drift. Use ``synthbanshee qa-report`` to flag clips with
190+
``len(effective_prosody_caps) > N``."""
191+
164192

165193
# ---------------------------------------------------------------------------
166194
# Per-clip metadata (one .json per clip)

0 commit comments

Comments
 (0)