You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 37c5f62
Browse filesBrowse the repository at this point in the historyBrowse files
fix(tts): #87 partial — effective-prosody cap addresses Whisper backdoor + helium range (#90)
The bisect on PR #86 showed the residual sp_it_a_0001 WER regression
(0.322 vs 04-15's 0.056) is caused by M7 SpeakerState drift compounding
with #51's M15 style_map values, producing effective pitch +14 % to +17 %
and rate 1.27-1.33x at high-intensity turns. That range simultaneously
sounds cartoonish to listeners (May-3 listening test "helium / oompa-
loompa") and trips Whisper-large-v3's silence-detection heuristic — the
classic length-ratio collapse to ~0.7 that hid the bug for weeks.
This PR ships a partial fix: a runtime effective-prosody cap that
addresses the canonical Whisper-backdoor fingerprint and the helium-
range pitch concern, plus the two detection layers Shay asked for to
catch this class of regression in the future. It does NOT fully
restore high-intensity WER to the pre-#51 baseline — see #89 for the
follow-up workstream.
## Tier-3 Whisper validation (`sp_it_a_0001`)
| variant | dur | WER | length_ratio | hyp / ref |
|---|---:|---:|---:|---:|
| 04-15 reference | 155.9 s | 0.056 | 1.009 | 236 / 234 |
| post-#86 main (no cap) | 146.6 s | 0.322 | 0.709 | 166 / 234 |
| this PR (cap active) | 149.1 s | **0.129** | **0.906** | 212 / 234 |
- Length-ratio recovers above the qa-report --asr 0.85 threshold.
- WER reduced 2.5x (0.322 -> 0.129) but still above the 04-15 baseline
of 0.056. Failure mode shifts from silence-detector trip
(~30 % of words missing) to substitution noise — distinct mechanism
requiring a paired listening test to fix without breaking M15
naturalness calibration. Tracked in #89 with insights and four
proposed approaches.
## The fix — effective-prosody runtime cap
`synthbanshee/tts/renderer._apply_effective_prosody_cap` clamps post-
state, post-randomization prosody before SSML emission:
- pitch in [-3.0, +2.0] st (~ +/- 12 % Azure)
- rate in [0.85, 1.20]
- volume left to the existing +/-50 % Azure clamp (Whisper internally
normalizes loudness, per #82's lever probe — not a Whisper-trip
dimension).
Caps are anchored to the pre-#51 effective envelope, which produced the
04-15 reference clips with WER 0.04-0.08. Tighter caps would diverge
further from M15 listening-test calibration; looser caps would re-trip
Whisper. Each cap activation logs a warning and is recorded per turn.
## Detection layer 1 — static prosody-cap activations in metadata
- `DialogueTurn.effective_prosody_caps` carries per-turn cap events.
- `cli.py` rolls them up into
`ClipMetadata.generation_metadata.effective_prosody_caps`
(new `EffectiveProsodyCapEvent` model in labels/schema.py).
- `qa-report` surfaces a new "Effective-Prosody Cap Activations (#87)"
table per clip — runs on every batch, no Azure / Whisper required.
Tier-3 render of sp_it_a_0001 recorded 14 cap activations across
7 high-intensity turns; metadata example in PR description.
## Detection layer 2 — `qa-report --asr` Whisper backdoor check
New `synthbanshee/package/asr_sanity.py` provides a lazy-loaded
`WhisperRunner` and `compute_asr_metrics`. `qa-report --asr` runs
Whisper-large-v3 on every clip in a directory, flags clips whose
length-ratio falls below `--asr-min-length-ratio` (default 0.85 — the
#87 fingerprint sat at ~0.71). Heavy dependencies isolated in the new
`eval-asr` optional extra so normal generation/QA stays light.
Per the policy decision documented in CLAUDE.md ("ASR sanity check
policy"), Tier-3 ASR sanity is local-only (not in CI) for now — see
GH issue #88 for the deferred CI re-evaluation triggers.
## Tests
- tests/unit/test_effective_prosody_cap.py: 11 tests covering the
helper unit, render_utterance integration, and render_scene event
propagation to DialogueTurn.
- tests/unit/test_qa.py::TestProsodyCapRollup: 3 tests verifying
cap-event aggregation in qa-report.
- tests/unit/test_asr_sanity.py: 11 tests covering normalize_for_wer,
AsrMetrics threshold semantics, and bracket-line stripping in the
reference parser. Heavy Whisper inference is exercised by the
Tier-3 local run, not these tests.
- 1687 unit tests pass (1662 baseline + 25 new); ruff + mypy clean.
## Docs
- CLAUDE.md: new "ASR sanity check policy" section + "What NOT to do"
bullets pinning the cap thresholds and the Tier-3 local-only policy.
- pyproject.toml: new `eval-asr` optional extra.
Reduces #87 (does not fully close — see #89 for the residual WER work).
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: AGENTS.md
+20Lines changed: 20 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,6 +70,7 @@ SFX onset/offset times come **only** from the Stage 3b augmentation log. Speech-
70
70
-**Cache key: SHA-256 of the full rendered SSML string** — not `(voice_id, text)`
71
71
-**M3a — per-turn RMS gain:**`StyleEntry.rms_target_dbfs` → module-level `_apply_rms_gain()` in `synthbanshee/tts/mixer.py`; 4-tuple segment API `(wav_bytes, pause_s, speaker_id, rms_target_dbfs)`
72
72
-**Temp WAV from mixer must be `subtype="FLOAT"`** — never `PCM_16` before `preprocess()` to avoid hard-clipping M3 gain
73
+
-**#87 effective-prosody cap:**`synthbanshee/tts/renderer._apply_effective_prosody_cap` clamps post-state, post-randomization prosody to `pitch ∈ [−3.0, +2.0] st`, `rate ∈ [0.85, 1.20]`. Volume is left to `_volume_to_string`'s ±50% Azure clamp (Whisper internally normalizes loudness; #82 lever probe). Cap activations are recorded per-turn in `DialogueTurn.effective_prosody_caps` and rolled up into `ClipMetadata.generation_metadata.effective_prosody_caps`. Caps are anchored to the pre-#51 effective envelope (the 04-15 reference baseline that Whisper handles with WER 0.04–0.08); changing them requires a paired listening test + Whisper sanity check (see "ASR sanity check policy" below).
73
74
74
75
## Splits
75
76
@@ -85,6 +86,23 @@ SFX onset/offset times come **only** from the Stage 3b augmentation log. Speech-
85
86
3. JSON parses as `ClipMetadata` with `is_synthetic=True`
86
87
4. Filename stem is ASCII-only lowercase
87
88
89
+
## ASR sanity check policy (#87)
90
+
91
+
The Whisper-large-v3 sanity check (`synthbanshee qa-report --asr`) detects audio that synthesizes fine to a listener but trips Whisper's silence-detection / segmentation heuristic — the failure mode that surfaced in #87 and went undetected on `sp_it_a_0001` for weeks. The canonical fingerprint is **length-ratio collapse** (`hyp_words / ref_words < 0.85`).
92
+
93
+
**Cost:**~5–20 s per clip on M-series MPS, ~3 GB Whisper-large-v3 download on first use. Real Azure render to populate the cache is an additional ~$0.05–$0.20 per scene.
94
+
95
+
**Policy — for the foreseeable future, this check is _local-only_, not in CI:**
96
+
97
+
1.**Unit tests** for the cap logic (in `tests/unit/test_effective_prosody_cap.py`) run on every PR. They cover ~90 % of regressions with no Azure / Whisper cost.
98
+
2.**`qa-report --asr` (Tier-3)** is run **locally**, **once**, **after all PR reviews and fixes are done**, **and only for PRs that might affect audio rendering** (anything touching `synthbanshee/tts/`, `synthbanshee/script/`, `synthbanshee/augment/`, or the speaker / scene / acoustic / project YAML configs). PRs that only touch labels, packaging, docs, or tests skip it.
99
+
3.**Cost discipline:** use this command **very sparingly** — each invocation incurs Azure spend (~$1–$2 per real-render run). Prefer running against a directory of already-cached clips when possible; only re-render when a code change invalidates the SSML cache.
100
+
4.**Result documentation:** when run, paste the `qa-report --asr` output (table of any clips with `length_ratio < 0.85`, plus the green-tick "no asr warnings" line if all clean) into the PR body's "Test plan" section under a "Tier-3 ASR sanity (local)" sub-heading.
101
+
102
+
**Future CI consideration:** see GH issue tracking the addition of a scheduled / labeled Whisper-based CI workflow, including repo-secret management for `AZURE_TTS_KEY` and Azure spend caps. Re-evaluate when scene volume increases enough that local manual runs become impractical.
@@ -153,3 +171,5 @@ When addressing PR review comments (Copilot, human, or bot):
153
171
- Don't generate clips shorter than 3.0 s
154
172
- Don't add a new typology to `taxonomy.yaml` without adding it to `_TYPOLOGY_INTENSITY_MAP`
155
173
- Don't pin `pr-agent-context` workflow references to a specific patch version — keep `@v4` floating
174
+
- Don't relax the #87 effective-prosody cap (`_EFFECTIVE_PITCH_*`, `_EFFECTIVE_RATE_*` in `synthbanshee/tts/renderer.py`) without a paired listening test + `qa-report --asr` Tier-3 run; the cap defends both naturalness (May-3 listening test) and Whisper transcription
175
+
- Don't merge a PR that touches audio rendering (`synthbanshee/tts/`, `synthbanshee/script/`, `synthbanshee/augment/`, or speaker / scene / acoustic / project YAMLs) without running `qa-report --asr` locally first and pasting the result into the PR's test plan (see "ASR sanity check policy")
0 commit comments