Skip to content

Commit 588cf05

Browse files
shaypal5claude
andauthored
fix(quality_gates): #80 replace per-sample click detector with DC-step detector (#85)
* fix(quality_gates): #80 replace per-sample click detector with DC-step detector The M15 click detector flagged every turn (164–1766 events vs threshold 3) because it triggered on Hebrew sibilants (/ʃ/ /s/ /z/), not on the DC-offset boundary clicks it was designed to catch. Diagnosis on sp_neu_a_0001 (12 turns, 8907 isolated events): 0 events near silence, 1464+ in continuous high-energy speech — the per-sample-diff heuristic cannot discriminate zero-mean fricative noise from a true baseline shift. Replace the heuristic with a running-mean step detector: compare the mean of the 40 ms window before each sample to the mean of the 40 ms window after, count peaks where |post - pre| > 0.08, collapse adjacent positions via non-maximum suppression. 40 ms covers ≥3 F0 cycles even at the male low end (80 Hz), so voiced AC swing averages out and only true DC shifts survive. Verified on the 12 captured sp_neu_a_0001 turns: all 12 pass (0 events each). Synthetic DC-step fixture (3 sustained 50 ms shifts) registers 6 events as expected (rising + falling edge of each). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(quality_gates): #80 self-review pass — off-by-one, semantics, real fixture Self-review of the previous commit raised 11 issues; addressing 10 here (skipping the float32-cumsum optimization — at 24 kHz × 30 s the float32 drift is ~0.085, on the order of the 0.08 step threshold; float64 is load-bearing for precision and now has a comment saying so). Code (synthbanshee/tts/quality_gates.py): - Fix off-by-one in candidate range: idx = arange(win, n - win) (was n - win - 1, dropping the last valid candidate position). - Rename CLICK_COUNT_THRESHOLD → CLICK_STEP_EVENT_THRESHOLD with a comment making it explicit the threshold counts edges, not clicks (one sustained DC offset = 2 events: rising + falling edge). - Hoist `from scipy.signal import find_peaks` to module top. - Type CLICK_STEP_WINDOW_MS as int (was float = 40.0 for an integer ms). - Surface the first 3 step-event timestamps in the failure detail string so an operator can seek directly to the alleged click. Tests (tests/unit/test_quality_gates.py + tests/fixtures/clicks/): - Parametrize test_dc_offset_steps_fail and test_sibilant_like_noise_passes over [16_000, 24_000] (gate runs at 24 kHz Azure native in production). - Tighten DC-step assertion: 5 ≤ count ≤ 8 instead of just `not passed`, locking in detector behaviour rather than just outcome. - Add test_real_tts_turn_passes with two checked-in fixtures: 2 s segments from real Azure he-IL output for sp_neu_a_0001 (turn 03 VIC/F, turn 08 AGG/M) at the highest pre-fix click density windows. The pre-#80 detector flagged 337/232 events on these segments respectively — they are now a real regression guard against the synthetic test's blind spots (white noise * Hanning ≠ Hebrew sibilant spectral content). - Document test_isolated_single_sample_spikes_pass as an intentional capability change, with the rationale (Azure he-IL does not produce isolated impulse noise in practice). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(quality_gates): #80 address Copilot review — boundary, 24 kHz, comment Three issues from Copilot's PR review on PR #85: 1. Boundary blind spot: pre-fix, idx = arange(win, n - win) excluded the first/last 40 ms of the signal, so a DC step at turn onset or final sample would never register. Now zero-pad the signal by `win` samples on each side before cumsum so every original sample gets a valid pre and post window. At the boundaries the offset is compared against silence, which is the right semantic — a real DC stitching artifact present from t=0 should fire. Added test_dc_offset_at_signal_start_caught and test_dc_offset_at_signal_end_caught. 2. Real-fixture coverage only at 16 kHz: gate runs at 24 kHz Azure native in production, so behaviour at the production rate must be tested with real Hebrew TTS audio. Extracted the same 2 s window directly from the captured 24 kHz native WAVs (no resample), committed as a second pair of fixtures, parametrized test_real_tts_turn_passes over both SRs (4 fixture cases total). 3. Stale "20 ms detector window" comment in test_dc_offset_steps_fail — detector is 40 ms; updated. All 32 click-gate tests pass; full unit + integration suite green (1731 / 1 skip); ruff + mypy clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(quality_gates): #80 second Copilot pass — find_peaks boundary, tighter tests Previous commit zero-padded the *samples* array so every original sample got a valid pre/post window, but find_peaks itself still couldn't return index 0 or n-1 of the step array — peaks need both neighbours by definition. Effect: a DC offset present at sample 0 (or persisting through the final sample) was still silently dropped. Fix: also pad the *step* array with zero sentinels before find_peaks, then shift returned indices back by 1. Sentinels are below threshold, so they're never selected; the only reachable change is that step[0] and step[n-1] become eligible peaks. Tests: redesigned both boundary tests so the boundary edge is *required* to push the event count above the threshold. - test_dc_offset_at_signal_start_caught: 3 sustained offsets, first one starting at sample 0. Without the fix: 5 events (boundary edge missed). With the fix: 6. Test asserts count >= 6 AND that the first peak timestamp is within one sample of t=0. - test_dc_offset_at_signal_end_caught: symmetric. Last offset persists through the final sample. Asserts count >= 6. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 94086a8 commit 588cf05

6 files changed

Lines changed: 221 additions & 56 deletions

‎synthbanshee/tts/quality_gates.py‎

Lines changed: 87 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,9 @@
99
Three gates implemented:
1010
1. Sustained-vowel detection — reject turns with >2.8 s single sustained voiced segment.
1111
2. F0 guardrails — reject if median F0 outside gender-specific range.
12-
3. Click detection — flag DC-offset jumps between SSML blocks.
12+
3. Click detection — flag DC-offset baseline shifts (#80: targets sustained
13+
step shifts, not per-sample amplitude jumps, so it does not fire on
14+
Hebrew sibilants).
1315
"""
1416

1517
from __future__ import annotations
@@ -18,6 +20,7 @@
1820
from typing import Literal
1921

2022
import numpy as np
23+
from scipy.signal import find_peaks
2124

2225
# ---------------------------------------------------------------------------
2326
# Thresholds (from research-synthesis.md §Quality Gates)
@@ -32,18 +35,28 @@
3235
FEMALE_F0_MIN_HZ: float = 150.0
3336
FEMALE_F0_MAX_HZ: float = 290.0
3437

35-
# Click detection: minimum absolute sample-to-sample jump (normalized to [-1,1])
36-
# that qualifies as a DC-offset click. Set at 0.15 to avoid false positives on
37-
# legitimate plosive transients (/p/, /t/, /k/) which typically peak around 0.1.
38-
# True DC-offset jumps from SSML block boundaries are typically 0.2+.
39-
CLICK_THRESHOLD: float = 0.15
40-
41-
# Minimum number of *isolated* click events to flag a turn.
42-
CLICK_COUNT_THRESHOLD: int = 3
43-
44-
# Number of neighboring samples that must be below threshold for a spike to
45-
# qualify as an isolated click (vs. a plosive burst which spans many samples).
46-
CLICK_ISOLATION_RADIUS: int = 3
38+
# Click detection: a real DC-offset click manifests as a *sustained* baseline
39+
# shift across the click sample — the running mean before and after differs
40+
# by CLICK_STEP_THRESHOLD or more. Using a baseline-shift criterion (rather
41+
# than a per-sample diff) is what distinguishes a true DC click from
42+
# zero-mean high-frequency content like Hebrew sibilants (/ʃ/ /s/), which
43+
# produce large per-sample diffs but no shift in the running mean. See #80
44+
# for the diagnosis that motivated this change.
45+
CLICK_STEP_THRESHOLD: float = 0.08
46+
47+
# Window width (each side, in milliseconds) over which the pre/post running
48+
# mean is computed for the step-shift comparison. 40 ms covers ≥3 F0 cycles
49+
# even at the male low end (80 Hz → 12.5 ms period), so the running mean
50+
# averages out the AC swing of a voiced segment and the step shift reflects
51+
# an actual DC change.
52+
CLICK_STEP_WINDOW_MS: int = 40
53+
54+
# Minimum number of distinct step events (after non-maximum suppression) to
55+
# flag a turn. A single sustained DC offset produces 2 events (rising +
56+
# falling edge), so this threshold is in *edges*, not in clicks: ≥3 events
57+
# corresponds to ≥2 sustained offsets (or ≥3 ramp-shaped artifacts), which
58+
# we treat as a systematic stitching problem.
59+
CLICK_STEP_EVENT_THRESHOLD: int = 3
4760

4861

4962
# ---------------------------------------------------------------------------
@@ -206,13 +219,25 @@ def check_f0_guardrails(
206219

207220

208221
def check_clicks(samples: np.ndarray, sr: int) -> GateResult:
209-
"""Detect DC-offset jump clicks in the audio.
210-
211-
Clicks manifest as sudden large sample-to-sample amplitude changes,
212-
typically at SSML block boundaries. To distinguish from legitimate
213-
plosive transients (which span many consecutive high-diff samples),
214-
only *isolated* spikes are counted — a diff event is a click only if
215-
the surrounding ±CLICK_ISOLATION_RADIUS samples are all below threshold.
222+
"""Detect DC-offset clicks via running-mean step shifts.
223+
224+
A DC-offset click is a sustained baseline shift: the audio level steps
225+
from one value to another and stays there for many samples. This is
226+
distinct from the per-sample amplitude swings produced by zero-mean
227+
high-frequency content like fricatives, which also produce large
228+
sample-to-sample diffs but leave the running mean unchanged.
229+
230+
For each candidate sample i, we compare the running mean of the
231+
``CLICK_STEP_WINDOW_MS`` ms window ending at i to the running mean of
232+
the equally-sized window starting after i. A baseline shift larger
233+
than ``CLICK_STEP_THRESHOLD`` is a step event. Adjacent positions are
234+
collapsed into single events via non-maximum suppression at distance
235+
``win`` so each step transition is counted once.
236+
237+
The signal is virtually zero-padded by ``win`` samples on each side
238+
so the detector covers every original sample, including the first
239+
and last 40 ms — a DC offset present from sample 0 (or sustained
240+
through the final sample) compares against silence and is flagged.
216241
217242
Args:
218243
samples: Float32 audio samples, mono, normalized to [-1, 1].
@@ -221,35 +246,53 @@ def check_clicks(samples: np.ndarray, sr: int) -> GateResult:
221246
Returns:
222247
GateResult indicating pass/fail.
223248
"""
224-
if len(samples) < 2:
249+
win = max(1, int(CLICK_STEP_WINDOW_MS * sr / 1000))
250+
n = len(samples)
251+
if n == 0:
225252
return GateResult(passed=True)
226253

227-
# Compute absolute sample-to-sample differences
228-
diffs = np.abs(np.diff(samples))
229-
candidates = np.where(diffs > CLICK_THRESHOLD)[0]
230-
231-
if len(candidates) < CLICK_COUNT_THRESHOLD:
232-
return GateResult(passed=True)
233-
234-
# Filter to isolated spikes: surrounding samples must be below threshold
235-
radius = CLICK_ISOLATION_RADIUS
236-
click_count = 0
237-
for idx in candidates:
238-
lo = max(0, idx - radius)
239-
hi = min(len(diffs), idx + radius + 1)
240-
# Count how many neighbors (excluding self) are above threshold
241-
neighborhood = diffs[lo:hi]
242-
n_above = int(np.sum(neighborhood > CLICK_THRESHOLD))
243-
# An isolated click has only 1-2 samples above threshold (the jump itself
244-
# and possibly the immediate return)
245-
if n_above <= 2:
246-
click_count += 1
247-
248-
if click_count >= CLICK_COUNT_THRESHOLD:
254+
# Zero-pad by win on each side so original sample i gets a valid pre and
255+
# post window, even at the boundaries. Cumulative sum then lets us
256+
# compute window means in O(1) per index. Promote to float64: at 24 kHz
257+
# × 30 s = 720k samples, plain float32 cumsum can accumulate ~0.085 of
258+
# drift, on the same order as the 0.08 step threshold; float64 keeps
259+
# rounding error well below the threshold.
260+
pad = np.zeros(win, dtype=np.float64)
261+
padded = np.concatenate((pad, samples.astype(np.float64), pad))
262+
cs = np.concatenate(([0.0], np.cumsum(padded)))
263+
# idx indexes original samples [0, n). In padded coordinates each
264+
# original sample i lives at position i + win, with pre window
265+
# padded[i:i+win] and post window padded[i+1+win:i+1+2*win].
266+
idx = np.arange(n)
267+
pre_mean = (cs[idx + win] - cs[idx]) / win
268+
post_mean = (cs[idx + 1 + 2 * win] - cs[idx + 1 + win]) / win
269+
step = np.abs(post_mean - pre_mean)
270+
271+
# Pad step with zero sentinels so step[0] and step[n-1] can be reported
272+
# as peaks — find_peaks requires both neighbours, so without padding it
273+
# silently drops boundary events. The sentinels are below threshold,
274+
# so they're never selected themselves; we shift indices back by 1.
275+
peaks_padded, _ = find_peaks(
276+
np.concatenate(([0.0], step, [0.0])),
277+
height=CLICK_STEP_THRESHOLD,
278+
distance=win,
279+
)
280+
peaks = peaks_padded - 1
281+
click_count = int(len(peaks))
282+
283+
if click_count >= CLICK_STEP_EVENT_THRESHOLD:
284+
# Surface the first three peak times in the detail so an operator
285+
# can seek directly to the alleged click in the rendered turn.
286+
# peaks are indices into step (= indices into the original signal).
287+
first_times_s = [(int(p) + 0.5) / sr for p in peaks[:3]]
288+
times_str = ", ".join(f"{t:.2f}" for t in first_times_s)
249289
return GateResult(
250290
passed=False,
251291
gate_name="click_detection",
252-
detail=f"Detected {click_count} isolated click events (threshold: {CLICK_COUNT_THRESHOLD})",
292+
detail=(
293+
f"Detected {click_count} DC-offset step events "
294+
f"(threshold: {CLICK_STEP_EVENT_THRESHOLD}); first at t=[{times_str}]s"
295+
),
253296
)
254297
return GateResult(passed=True)
255298

62.5 KB
Binary file not shown.
93.8 KB
Binary file not shown.
62.5 KB
Binary file not shown.
93.8 KB
Binary file not shown.

‎tests/unit/test_quality_gates.py‎

Lines changed: 134 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,10 @@
33
from __future__ import annotations
44

55
import io
6+
from pathlib import Path
67

78
import numpy as np
9+
import pytest
810
import soundfile as sf
911

1012
from synthbanshee.tts.quality_gates import (
@@ -17,6 +19,8 @@
1719

1820
SR = 16_000
1921

22+
FIXTURES_DIR = Path(__file__).resolve().parents[1] / "fixtures" / "clicks"
23+
2024

2125
# ---------------------------------------------------------------------------
2226
# Helpers
@@ -128,35 +132,153 @@ def test_clean_sine_passes(self) -> None:
128132
result = check_clicks(samples, SR)
129133
assert result.passed
130134

131-
def test_isolated_dc_jumps_fail(self) -> None:
132-
# Low-amplitude signal with isolated DC jumps (simulating SSML boundary clicks)
135+
@pytest.mark.parametrize("sr", [16_000, 24_000])
136+
def test_dc_offset_steps_fail(self, sr: int) -> None:
137+
# Three sustained baseline shifts — the failure mode the gate targets.
138+
# Each shift holds 50 ms (longer than the 40 ms detector window) so
139+
# the rising and falling edge of every shift register as separate
140+
# step events. Six events expected (3 shifts × 2 edges); we accept
141+
# 5–8 to allow for find_peaks dropping a single boundary edge.
142+
samples = np.zeros(sr, dtype=np.float32)
143+
shift_n = int(0.05 * sr) # 50 ms
144+
for start_s, amp in [(0.10, 0.20), (0.35, -0.18), (0.60, 0.25)]:
145+
i = int(start_s * sr)
146+
samples[i : i + shift_n] = amp
147+
result = check_clicks(samples, sr)
148+
assert not result.passed
149+
assert result.gate_name == "click_detection"
150+
# Tight bound on the count locks in detector behaviour, not just outcome.
151+
# The detail string is "...step events (threshold: 3); first at t=[...]s".
152+
assert result.detail is not None
153+
n = int(result.detail.split()[1])
154+
assert 5 <= n <= 8, f"expected 5-8 step events, got {n}: {result.detail}"
155+
156+
def test_isolated_single_sample_spikes_pass(self) -> None:
157+
# Single-sample spikes are NOT DC clicks — they're impulse noise that
158+
# leaves the running mean unchanged. Pre-#80 the gate caught these
159+
# alongside any high-frequency content (Hebrew sibilants). This is
160+
# an intentional capability change: the new detector targets only
161+
# sustained baseline shifts, since Azure he-IL output does not
162+
# produce isolated impulse noise in practice (see #80 diagnosis).
133163
samples = np.zeros(16000, dtype=np.float32)
134-
# Insert isolated spikes well-separated (>CLICK_ISOLATION_RADIUS apart)
135164
for pos in [1000, 3000, 5000, 7000]:
136-
samples[pos] = 0.8 # Isolated single-sample spike
165+
samples[pos] = 0.8
137166
result = check_clicks(samples, SR)
138-
assert not result.passed
139-
assert result.gate_name == "click_detection"
167+
assert result.passed
140168

141169
def test_plosive_burst_passes(self) -> None:
142-
# A plosive burst spans many consecutive high-diff samples — not isolated
170+
# A 20-sample plosive burst at 0.5 leaves running mean ~0 either side
171+
# of the burst; not a baseline shift.
143172
samples = np.zeros(16000, dtype=np.float32)
144-
# Simulate a plosive: multiple consecutive high-amplitude samples
145-
samples[5000:5020] = 0.5 # 20-sample burst — NOT an isolated click
173+
samples[5000:5020] = 0.5
146174
result = check_clicks(samples, SR)
147175
assert result.passed
148176

149-
def test_few_clicks_passes(self) -> None:
150-
# Only 1-2 isolated clicks — below count threshold of 3
177+
@pytest.mark.parametrize("sr", [16_000, 24_000])
178+
def test_sibilant_like_noise_passes(self, sr: int) -> None:
179+
# Synthetic sibilant proxy: envelope-shaped white Gaussian noise.
180+
# Real /ʃ/ /s/ have spectral peaks at 5–8 kHz; this fixture is
181+
# weaker than that but exercises the zero-mean property the
182+
# step detector relies on. The real regression coverage lives
183+
# in test_real_tts_turn_passes (Azure-rendered fixture).
184+
rng = np.random.default_rng(42)
185+
n = int(1.5 * sr)
186+
envelope = np.zeros(n, dtype=np.float32)
187+
for start_s in (0.3, 0.9):
188+
i = int(start_s * sr)
189+
burst_n = int(0.2 * sr)
190+
ramp = np.hanning(burst_n).astype(np.float32)
191+
envelope[i : i + burst_n] = ramp
192+
noise = rng.standard_normal(n).astype(np.float32)
193+
samples = (noise * envelope * 0.30).astype(np.float32)
194+
result = check_clicks(samples, sr)
195+
assert result.passed, f"sibilant-like noise wrongly flagged: {result.detail}"
196+
197+
@pytest.mark.parametrize(
198+
"fixture_name",
199+
[
200+
# 16 kHz pair (post-mixer rate)
201+
"sp_neu_a_0001_turn_03_female_2s_16k.wav",
202+
"sp_neu_a_0001_turn_08_male_2s_16k.wav",
203+
# 24 kHz pair (Azure provider native rate — what the gate sees in production)
204+
"sp_neu_a_0001_turn_03_female_2s_24k.wav",
205+
"sp_neu_a_0001_turn_08_male_2s_24k.wav",
206+
],
207+
)
208+
def test_real_tts_turn_passes(self, fixture_name: str) -> None:
209+
# Regression guard for #80: 2 s segments captured from real Azure
210+
# he-IL output for sp_neu_a_0001 (one VIC/F, one AGG/M) at the
211+
# window with the highest pre-fix click density. Both the 24 kHz
212+
# native (gate-time) rate and the 16 kHz post-mixer rate are
213+
# exercised because behaviour depends on win = win_ms * sr / 1000.
214+
# Pre-#80 the per-sample-diff detector logged 100s of false events
215+
# on these segments; the step-shift detector must report passed=True.
216+
fixture_path = FIXTURES_DIR / fixture_name
217+
samples, sr = sf.read(fixture_path, dtype="float32")
218+
if samples.ndim > 1:
219+
samples = samples[:, 0]
220+
result = check_clicks(samples, sr)
221+
assert result.passed, f"real Hebrew TTS turn wrongly flagged: {result.detail}"
222+
223+
def test_few_steps_pass(self) -> None:
224+
# One sustained shift produces 2 edge events (rising + falling) —
225+
# below the threshold of 3.
151226
samples = np.zeros(16000, dtype=np.float32)
152-
samples[5000] = 0.8
227+
samples[5000:5800] = 0.20
153228
result = check_clicks(samples, SR)
154229
assert result.passed
155230

156231
def test_empty_passes(self) -> None:
157232
result = check_clicks(np.array([], dtype=np.float32), SR)
158233
assert result.passed
159234

235+
def test_short_audio_passes(self) -> None:
236+
# All-zero short audio — virtual zero padding gives step ≈ 0 everywhere.
237+
samples = np.zeros(100, dtype=np.float32)
238+
result = check_clicks(samples, SR)
239+
assert result.passed
240+
241+
def test_dc_offset_at_signal_start_caught(self) -> None:
242+
# Three sustained offsets; the first starts at sample 0. Without
243+
# the boundary fix the t=0 rising edge is silently dropped and we
244+
# see 5 events; with the fix we see 6. Asserting count ≥ 6 and
245+
# that the first detected peak is at t≈0 actually proves the
246+
# boundary edge was counted.
247+
samples = np.zeros(int(0.5 * SR), dtype=np.float32)
248+
samples[: int(0.06 * SR)] = 0.20 # offset starts AT sample 0
249+
samples[int(0.15 * SR) : int(0.20 * SR)] = -0.18
250+
samples[int(0.30 * SR) : int(0.35 * SR)] = 0.22
251+
result = check_clicks(samples, SR)
252+
assert not result.passed
253+
assert result.gate_name == "click_detection"
254+
assert result.detail is not None
255+
n = int(result.detail.split()[1])
256+
assert n >= 6, f"sample-0 edge dropped: only {n} events; expected ≥6"
257+
# Detail format: "...; first at t=[0.00, 0.06, 0.15]s" — the boundary
258+
# edge must appear as the first detected peak (within one sample of t=0).
259+
first_t_str = result.detail.split("[")[1].split(",")[0]
260+
assert float(first_t_str) < 1.0 / SR, f"sample-0 edge not the first peak: {result.detail}"
261+
262+
def test_dc_offset_at_signal_end_caught(self) -> None:
263+
# Symmetric: a DC offset that persists through the final sample.
264+
# Without the boundary fix the trailing edge at sample n-1 is
265+
# dropped (5 events); with the fix it's caught (6 events) and the
266+
# last detected peak time lands within one sample of the final.
267+
n_samples = int(0.5 * SR)
268+
samples = np.zeros(n_samples, dtype=np.float32)
269+
samples[int(0.10 * SR) : int(0.15 * SR)] = 0.20
270+
samples[int(0.25 * SR) : int(0.30 * SR)] = -0.18
271+
samples[int(0.44 * SR) :] = 0.22 # offset persists to last sample
272+
result = check_clicks(samples, SR)
273+
assert not result.passed
274+
assert result.gate_name == "click_detection"
275+
assert result.detail is not None
276+
n = int(result.detail.split()[1])
277+
assert n >= 6, f"sample-(n-1) edge dropped: only {n} events; expected ≥6"
278+
# The detail surfaces the FIRST 3 peak timestamps, not the last —
279+
# so we re-run with the same rationale and check the count alone
280+
# proves the boundary edge was counted (5 events without fix, 6 with).
281+
160282

161283
# ---------------------------------------------------------------------------
162284
# Composite gate runner

0 commit comments

Comments
 (0)