How to Choose an Open-Source Voice Cloning Model, Measured

Open-source voice cloning models now copy a voice from just 3-10 seconds of reference audio, so the choice turns not only on speed and quality but on watermarking and license. For commercial, real-time, and safety-conscious use, Chatterbox (MIT, PerTh watermarking built in by default, about 10 seconds) stands out; for multilingual and cross-lingual, CosyVoice 3; and the technical benchmark is XTTS v2 (6 seconds, 17 languages), though it is non-commercial. The open TTS stack we operate (Fish Speech, MOSS-TTS, Higgs Audio, Qwen3-TTS) is judged by the same yardstick - similarity, naturalness, latency, watermark, and license.

In one line: voice cloning molds a voice from a short sample. A few seconds is enough to speak new sentences in that person's voice, which is exactly why it is powerful and why it is dangerous.

What decides cloning quality?#

Three things: reference length, speaker similarity, and naturalness. Modern zero-shot models follow the path VALL-E opened, capturing a person's voice and intonation from about 3 seconds. XTTS v2 covers 17 languages from 6 seconds, and Chatterbox Multilingual clones across 23 languages cross-lingually (record once, speak in other languages). CosyVoice 3 uses supervised semantic tokens to reach top marks in content consistency, speaker similarity, and prosody. Quality comes down to how close it sounds (speaker similarity) and how human it sounds.

Longer references raise similarity, but not without limit - as below, the gain usually flattens near 10 seconds (saturation).

Why are watermarking and consent the crux?#

Because they are the last line of defense against misuse. Self-hosted open tools usually have no watermark, no consent verification, and no audit log, so they are "traceable to no one." Chatterbox is the exception, embedding a PerTh watermark by default in every output: inaudible to humans and designed to survive MP3, Opus, telephony codecs, and editing (though it is only a binary "made by Chatterbox" signal, not version or tenant attribution). Commercial tools verify consent with a consent clip, but a 2025 study found four of six relied only on a checkbox self-attestation.

Plot the models on two axes - license and watermark - and the quadrant that is safe for commercial release is narrower than you would expect.

How is regulation changing?#

It becomes mandatory in 2026. The EU AI Act Article 50 requires AI-generated audio to be marked in a machine-readable format from August 2026, and the US TAKE IT DOWN Act adds traceability duties for synthetic media. The threat is real: in 2025 an AI-generated voice impersonated a senior U.S. official to contact other officials. So built-in watermarking is shifting from nice-to-have to required.

How do you measure it yourself (and responsibly)?#

Measure with your own voice, with consent. Before you pick, following the single branching question below already narrows the field.

• Use 10-30 seconds of your own consented voice to compare speaker similarity and naturalness across models, blind.
• If you need cross-lingual, record in one language and check intonation retention in another.
• Do not clone anyone's voice without consent. Keep a watermark on outputs, and check license and local regulation before commercial release.

Reference links

• CosyVoice (repo)
• Chatterbox (repo)
• F5-TTS (repo)
• OpenVoice (repo)
• Coqui TTS / XTTS v2 (repo)

Note: model, reference-second, and language figures follow each project's public materials (2025-2026) and vary by version. Measure real quality on your own consented voice with the method above. Voice cloning carries high fraud and impersonation risk, so always honor consent, watermarking, and local regulation. Reviewed quarterly.