Repository navigation
Commit 1ea48f3
* fix(labels,qa): #109 — drop hardcoded tts_engine, derive single_backend from generation_metadata.tts_backend
ClipMetadata.tts_engine was always written as the literal string
"azure_he_IL" regardless of which TTS provider actually rendered the
clip. This mis-labeled every google-rendered clip and caused
qa-report's run-level `single_backend` warning to fire as a false
positive on mixed-provider runs (see delivery-003 in
avdp-synth-corpus#4, where 20 mixed-backend clips were tagged
`"clips_by_tts_engine": {"azure_he_IL": 20}` and triggered
`single_backend`).
Per the issue's recommended option (a): drop `tts_engine` as the
primary signal. The structured per-speaker `tts_backend` map already
lives in `generation_metadata` and carries richer information; the
flat field was dead weight.
Changes:
- synthbanshee/labels/schema.py: removed `tts_engine` field from
`ClipMetadata`; removed from the ascii-safe `field_validator`. Old
corpus JSON still parses cleanly (Pydantic ignores unknown fields).
- synthbanshee/labels/generator.py: dropped the hardcoded
`tts_engine="azure_he_IL"` line in `generate_clip_metadata`.
- synthbanshee/package/qa.py: backend diversity now derived from
`metadata.generation_metadata.tts_backend.values()`. `RunSummary`
field renamed `clips_by_tts_engine` → `clips_by_tts_backend` (a
clip with mixed providers contributes one count to each backend).
- synthbanshee/cli.py: qa-report renderer + JSON output updated for
the renamed field.
- docs/spec.md §5.1: removed `"tts_engine": "azure_he_IL"` from the
example JSON; added a field note pointing readers to
`generation_metadata.tts_backend`. Old corpus snapshots still
carrying the field should be tolerated and ignored.
- AGENTS.md (= CLAUDE.md): removed `tts_engine` from the ascii-safe
validator list.
- tests/unit/test_qa.py: helper `_write_valid_clip` replaced
`tts_engine` kwarg with `tts_backend` (per-speaker map); every
clip now emits `generation_metadata` so qa derivation has a source
of truth. Added two new tests under TestRunQARunSummary:
`single_backend` does NOT fire when two clips use different
backends; `single_backend` DOES fire when both clips are azure.
Existing `test_run_summary_backend_diversity` updated for the new
field name.
- tests/unit/test_labels.py: added `test_tts_engine_field_removed`
asserting the attribute is absent on freshly-built ClipMetadata
and that legacy JSON carrying the old field still parses cleanly.
- tests/unit/{test_cli,test_manifest,test_generation_metadata}.py:
dropped `tts_engine` from minimal-clip JSON fixtures.
Fixes #109.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(labels,qa): #109 — bucket legacy clips as "unknown", harden tests, document break
Self-review caught five gaps in the first cut. This commit addresses them:
1. Pre-#109 clips (no ``generation_metadata``) now bucket under the
``"unknown"`` backend key in ``RunSummary.clips_by_tts_backend``
instead of silently contributing zero. Previously, a corpus mixing
pre- and post-#109 clips would lose the legacy clips from
``backend_count``, making ``single_backend`` flip the wrong way.
2. Updated ``RunSummary.clips_by_tts_backend`` docstring to:
- flag the ``sum(values) >= total_clips`` invariant change (mixed-
backend clips contribute +1 to each backend, so the pre-#109
sum-equals-total invariant no longer holds);
- document the ``"unknown"`` bucket for missing generation_metadata.
3. ``test_tts_engine_field_removed`` now explicitly exercises
``with pytest.raises(AttributeError): m.tts_engine`` for both fresh
models and ones parsed from legacy JSON. The previous version only
asserted ``not hasattr(...)``, which would silently pass if
Pydantic ever shipped ``extra="allow"`` semantics by accident. The
new test makes the breaking-change contract explicit.
4. New ``test_pipeline_output_feeds_qa_backend_derivation`` integration
test runs the full mocked ``_run_generate_pipeline`` end-to-end and
then feeds the resulting clip directory to ``run_qa(run_summary=True)``,
asserting ``"azure"`` lands in ``clips_by_tts_backend`` and
``"unknown"`` does NOT. Catches the regression where the generator's
``_backends_map`` shape diverges from qa.py's expectation — both
unit-tested components could pass in isolation while the
integration silently breaks.
5. New ``test_run_summary_clips_without_generation_metadata_bucket_as_unknown``
and ``test_run_summary_all_legacy_clips_fire_single_backend`` pin
the unknown-bucket contract: legacy clips remain visible to
diversity metrics, and an all-legacy corpus still correctly fires
``single_backend``.
Full unit suite: 1702/1702 passing (up from 1699 — three new tests).
ruff + mypy clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 58b047f commit 1ea48f3
11 files changed
Lines changed: 205 additions & 28 deletions
File tree
- docs
- synthbanshee
- labels
- package
- tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
284 | 284 | | |
285 | 285 | | |
286 | 286 | | |
287 | | - | |
288 | 287 | | |
289 | 288 | | |
290 | 289 | | |
| |||
345 | 344 | | |
346 | 345 | | |
347 | 346 | | |
| 347 | + | |
348 | 348 | | |
349 | 349 | | |
350 | 350 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1702 | 1702 | | |
1703 | 1703 | | |
1704 | 1704 | | |
1705 | | - | |
1706 | | - | |
| 1705 | + | |
| 1706 | + | |
1707 | 1707 | | |
1708 | 1708 | | |
1709 | 1709 | | |
| |||
1806 | 1806 | | |
1807 | 1807 | | |
1808 | 1808 | | |
1809 | | - | |
| 1809 | + | |
1810 | 1810 | | |
1811 | 1811 | | |
1812 | 1812 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
261 | 261 | | |
262 | 262 | | |
263 | 263 | | |
264 | | - | |
| 264 | + | |
| 265 | + | |
265 | 266 | | |
266 | 267 | | |
267 | 268 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
213 | | - | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
214 | 219 | | |
215 | 220 | | |
216 | 221 | | |
| |||
252 | 257 | | |
253 | 258 | | |
254 | 259 | | |
255 | | - | |
| 260 | + | |
256 | 261 | | |
257 | 262 | | |
258 | 263 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
127 | 127 | | |
128 | 128 | | |
129 | 129 | | |
130 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
131 | 141 | | |
132 | 142 | | |
133 | 143 | | |
| |||
366 | 376 | | |
367 | 377 | | |
368 | 378 | | |
369 | | - | |
370 | | - | |
| 379 | + | |
| 380 | + | |
371 | 381 | | |
372 | 382 | | |
373 | 383 | | |
| |||
421 | 431 | | |
422 | 432 | | |
423 | 433 | | |
424 | | - | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
425 | 440 | | |
426 | | - | |
427 | | - | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
428 | 450 | | |
429 | 451 | | |
430 | 452 | | |
| |||
520 | 542 | | |
521 | 543 | | |
522 | 544 | | |
523 | | - | |
524 | | - | |
| 545 | + | |
| 546 | + | |
525 | 547 | | |
526 | 548 | | |
527 | 549 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
125 | 124 | | |
126 | 125 | | |
127 | 126 | | |
| |||
1172 | 1171 | | |
1173 | 1172 | | |
1174 | 1173 | | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
1175 | 1214 | | |
1176 | 1215 | | |
1177 | 1216 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
28 | | - | |
29 | 28 | | |
30 | 29 | | |
31 | 30 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
| 6 | + | |
5 | 7 | | |
6 | 8 | | |
7 | 9 | | |
| |||
169 | 171 | | |
170 | 172 | | |
171 | 173 | | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
172 | 204 | | |
173 | 205 | | |
174 | 206 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
62 | 61 | | |
63 | 62 | | |
64 | 63 | | |
| |||
360 | 359 | | |
361 | 360 | | |
362 | 361 | | |
363 | | - | |
364 | 362 | | |
365 | 363 | | |
366 | 364 | | |
| |||
0 commit comments