Add Irodori-TTS v4 support

[yoshphys]

Adds support for Irodori-TTS v4-Small, and fixes several divergences from the reference implementation that were found while validating it.

v4 support

v4-Small unifies the previous base and VoiceDesign models into one checkpoint. Compared to v3 the architecture change is narrow: the two scratch-trained text/caption encoders are replaced by a single pretrained ModernBERT-ja-310m backbone feeding separate projectors, and speaker_patch_size goes from 1 to 4. The DiT, adaLN, speaker encoder and duration predictor are unchanged.

• New modernbert.py: MLX port of ModernBERT — fused Wqkv, GeGLU MLP, alternating global / sliding-window attention with per-layer-type RoPE theta, and layer 0's identity attn_norm.
• PretrainedTextBackbone + PretrainedConditionProjector, used when text_encoder_type is "pretrained". v1–v3 keep the scratch encoders.
• The tokenizer bundled with the converted weights is preferred over the upstream repo, so inference needs no extra download.
• Multi-clip reference audio: clips are encoded separately and concatenated, then trimmed to the checkpoint's ref_max_seconds (120s for v4, 30s before). This matches how v4 was trained.

Converted models: mlx-community/Irodori-TTS-v4-Small-fp16 and -8bit.

Fixes that also change v1–v3 behaviour

Validating v4 against the reference implementation surfaced pre-existing bugs. normalize_text had drifted in ways that change what the model is asked to read, so existing checkpoints will sound different (closer to the reference):

• Sentence-final 。 and 、 were stripped. Upstream keeps them, and they carry prosody — dropping the 。 shortened predicted durations and clipped endings.
• ASCII spaces (U+0020) were removed, so "hello world" became "helloworld". Only the ideographic space U+3000 should go.
• .. / ... were not folded to ….
• NFKC normalization was missing, leaving ㈱ or Ⅲ untouched.
• Outer-bracket stripping ran once and ignored nesting depth, so 「前半」と「後半」 lost its outer quotes.

Also fixed, affecting every caption-conditioned checkpoint (v2 VD, v3 VD, v4):

• An absent caption still tokenizes to a BOS token and its mask was left set, so the DiT and duration predictor saw a one-token caption instead of none. Upstream zeroes the mask.
• The trailing-silence trim was applied unconditionally, so a silence point of 0 would have produced zero-length audio.
• use_cfg ignored has_caption_cfg, and has_caption_cfg did not check that the caption mask is non-empty.

And one v4-only bug: PretrainedConditionProjector coerced its input to self.projector.weight.dtype. After nn.quantize that weight is packed uint32, so activations were truncated to integers — a no-op in fp16, but it put the 8-bit projector output 79% off.

Verification

Each stage was compared against the reference PyTorch implementation on the real v4 checkpoint:

Stage	Agreement
ModernBERT encoder (fp32, random weights, vs transformers)	4.8e-7
normalize_text (3035 strings incl. 3000 fuzz)	exact
Tokenization (text/caption, truncation, padding)	exact
Duration features (14-dim)	exact
Duration prediction	exact (0.0000%)
text_state (backbone + projector + norm)	0.09%
DiT forward (v_pred)	0.46%

The two non-exact rows are fp16-vs-fp32 weight error.

Known model behaviour (not a port issue)

With a caption but no reference audio, v4 roughly doubles the predicted duration for short texts (under ~7 tokens) and reads the sentence twice to fill the window. Running the reference PyTorch sampler produces the same frame counts and the same repetition, so this is upstream model behaviour. It is documented in the model README along with the workarounds (pass reference audio, or set duration_scale / seconds).

Tests

79 Irodori tests pass (13 new for v4, 4 for caption/tail-trim handling, 8 for normalization including a differential check against a local upstream checkout when present). Full test_models.py: 441 passed, 1 pre-existing unrelated failure (TestSparkTTSModel::test_init).

🤖 Generated with Claude Code

[lucasnewman]

@yoshphys Very nicely done, thank you! Can you sign your commits and force push so we can clear the merge requirements?