Skip to content

feat: add CosyVoice3 MLX implementation - #861

Open
walkoncross wants to merge 1 commit into
Blaizzy:mainfrom
walkoncross:cosyvoice3
Open

feat: add CosyVoice3 MLX implementation#861
walkoncross wants to merge 1 commit into
Blaizzy:mainfrom
walkoncross:cosyvoice3

Conversation

@walkoncross

Copy link
Copy Markdown

Summary

Adds a full MLX port of CosyVoice3 (mlx_audio/tts/models/cosyvoice3/), following the existing model contract used by other TTS models in this repo (e.g. chatterbox, spark).

  • Pipeline: text → LLM (Qwen2 speech-token backbone) → speech tokens → flow (DiT-based CausalMaskedDiffWithDiT / CFM) → mel → HiFT vocoder (causal) → waveform.
  • Generation modes: zero-shot voice cloning, cross-lingual synthesis, and instruction-based (instruct2) generation, matching the reference CosyVoice3 Python implementation's behavior.
  • Speaker presets: spk2info system for caching/reusing zero-shot speaker prompts (--add_spk, --save_spkinfo, --load_spkinfo).
  • Text frontend: ported text normalization pipeline (Chinese blank/bracket handling, English number spelling) and full CosyVoice3Tokenizer special-token registration for fine-grained control tokens.
  • Weight conversion: sanitize/routing logic wired into convert.py so official CosyVoice3 checkpoints can be converted to MLX weights.
  • Correctness fixes along the way (each validated against the PyTorch reference under /CosyVoice/cosyvoice): causal HiFT rewrite, DiT RoPE pairing/application order, CFM noise seeding to remove pitch bias, sample-rate handling for timbre cloning, prompt-feature/silence-token alignment with the reference, and a defensive post-hoc fix for a rare stochastic LLM-sampling tail-hiss artifact (acoustic-signature detection + fade, since the exact triggering token sequence could not be reliably reproduced after 100+ attempts — see README.md under this model's ## Status section for the full investigation).
  • Docs: CLI usage examples (zero-shot, cross-lingual, instruct2, speaker presets) with bundled reference audio assets, and a ## Status section tracking known issues/investigations.

45 commits, squash-mergeable if preferred. See individual commit messages for a detailed history of the implementation and each fix.

Test plan

  • pytest mlx_audio/tts/tests/test_cosyvoice3.py — 31 passed (2 pre-existing failures unrelated to this branch, confirmed via git stash against main: TestLLM::test_autoregressive_decode, TestModelIntegration::test_generate_end_to_end_with_mock_frontend, both AttributeError on an incomplete MockFrontEnd test double)
  • End-to-end generate() validated against real CosyVoice3-0.5B checkpoint weights for zero-shot, cross-lingual, and instruct2 modes
  • HiFT/flow determinism verified (fixed mel → bit-identical waveform across repeated calls)
  • Tail-hiss fix verified: real reported bug file -28.1dB → -120dB after fade; synthetic clean-tail audio left bit-identical (no false positives)

component (v2 uses a UNet ConditionalDecoder).
3. HiFT : CausalHiFTGenerator vocoder (mel -> waveform, NSF + ISTFT).

Reference (PyTorch): /Users/admin/CosyVoice/cosyvoice

@lucasnewman lucasnewman Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove the implementation-specific comments here and elsewhere? We only want comments that improve the understanding of the code, not porting notes.

import mlx.core as mx


def nucleus_sampling(logits: mx.array, top_p: float = 0.8, top_k: int = 25) -> int:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just use make_sampler here or is there something specific needed for these?

Comment thread mlx_audio/tts/generate.py Outdated
streaming_interval: float = 2.0,
save: bool = False,
use_zero_spk_emb: bool = False,
cross_lingual: bool = False,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be passed through to the model as extra kwargs, we don't want these in the standard generate interface.

Comment thread .gitignore Outdated
claude.md
.claude/site/
.generated/legacy-docs-generated/

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert these.

@lucasnewman

Copy link
Copy Markdown
Collaborator

@walkoncross Thanks for the submission! This needs some cleanup, I left a few comments inline. Also, the examples should just live in the README.md, we don't want model-specific artifacts in there. If you can make those changes we can consider this.

@lucasnewman lucasnewman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments above.

Remove CosyVoice3 porting notes from runtime code, move model-specific arguments out of the shared generate interface, and align tests and examples with the reviewer feedback.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants