Open Voice Cloning: VRAM and RAM Requirements, Measured

The VRAM of open voice-cloning models is, unlike other models, not a wall - because the models are small. Three measured essentials: (1) Chatterbox is 2-3GB and XTTS-v2 is 4-6GB, fitting most GPUs, and ultralight ones like Kokoro are under 1GB and run even on CPU (only Bark at 8-12GB and Dia at 10-14GB are heavy). (2) So the real budget is not storage but real-time factor (RTF) and latency - XTTS-v2 measures RTF 0.3 (on a 4090) with 150-400ms first audio, and Chatterbox hits sub-300ms when optimized. (3) And concurrency is the real bottleneck: real-time is effectively one request per GPU, so you scale horizontally by adding instances, not with one big GPU. That is, the question shifts from "does it fit?" to "is it fast enough, and how many streams?"

In one line: voice cloning is like a voice-over booth. The booth (model) is small, so several fit in a room, but two people recording in one booth at once clash (the concurrency limit). So when demand grows you add booths (GPU instances), not enlarge one booth.

Why is voice-cloning VRAM not a wall?#

Because TTS models are small, hundreds of millions to ~2B params. The pipeline is text encoder + acoustic model (AR backbone, flow matching) + neural codec/vocoder, each piece small, summing to 2-3GB for Chatterbox and 4-6GB for XTTS. That is a different scale from LLMs (tens of GB) or image models (Flux 24GB). So "does the VRAM fit" is rarely the issue, and 8GB is enough to self-host with light concurrency. It runs on CPU too, but too slowly - RTF passes 1 and real-time breaks - so if real-time is the goal, a GPU is the answer.

Put the models' VRAM side by side as bars and it is obvious how different a league cloning models are from LLMs and image models.

So what is the real budget?#

Real-time factor (RTF) and first-audio latency. RTF is generation time versus audio length, and it must be below 1 to stream in real time - XTTS-v2 measures about 0.3 (little headroom, later first chunk), while Kokoro is 0.03, making 10 seconds in 0.3. First-audio latency matters too: XTTS cloning is 150-400ms by GPU (190ms on a 3090, 135ms on a 5090), and Blackwell is 30-40% lower than Ampere. Conversational voice agents target sub-200ms first audio, where GPU generation and streaming design, not VRAM, decide success.

Set RTF 1 as the baseline and it is clear why CPU breaks real time while a GPU keeps headroom - below, a bar must sit under the dashed line (RTF = 1) to stream.

Why is concurrency the bottleneck?#

Because real-time TTS is effectively one active request per GPU instance. Even with a small model, two parallel requests wreck latency (Chatterbox degrades beyond two). So the scaling rule differs - add GPU instances rather than one big GPU, and let a queue absorb spikes. Streaming buffers at sentence boundaries, playing the first sentence while the LLM makes the rest, cutting perceived latency. So voice-service sizing is done in concurrent streams x GPU instances, not "VRAM."

The open TTS stack we operate (Fish Speech, MOSS-TTS, Higgs Audio, Qwen3-TTS) is sized the same way - whichever model, we count one stream per GPU and scale out with instances, letting a queue absorb spikes.

How much does a voice-agent stack cost?#

The key is that TTS is the smallest piece in the stack.

• TTS alone: 8GB for light concurrency, 16-24GB for multiple voices and higher throughput.
• Full stack (ASR+LLM+TTS): Whisper Large (5-8GB) + Llama 11B (~22GB) + XTTS (~6GB) is about 33-36GB, tight for 24GB (an H100 80GB is recommended). The LLM dominates VRAM, and TTS is a bonus.
• Rule: design by RTF, concurrency, and GPU generation, not model choice, and measure exact throughput on your own text and load.

Stack up the full-stack VRAM and it is clear why worrying about TTS misses the point - as below, most of the bulk is the LLM and TTS is the thin top layer.

Related reading: How we operate open voice cloning - a pipeline retro, The open voice-cloning pipeline: hands-on impressions and limits

Related reading: Local TTS open models - VRAM and RAM requirements, measured, Ollama/LM Studio/llama.cpp runners - VRAM and RAM, measured

Reference links

• Chatterbox (open cloning, MIT)
• Coqui XTTS (multilingual cloning)
• Chatterbox-TTS-Server (self-hosting, concurrency)
• Kokoro (ultralight TTS)
• F5-TTS (flow-matching cloning)

Note: VRAM, RTF, and latency figures are 2026 public benchmarks (many vendor-measured) and estimates, and vary by GPU, text length, and concurrency (not permanent numbers; Chatterbox sub-200ms and similar are self-reported optimized cases). Load-test exact throughput and latency on your own hardware and load (these numbers are only a start). XTTS is community-maintained after Coqui's shutdown, so check the license. The TTS stack moves fast, so this is reviewed quarterly.