Coqui XTTS v2 (now community-maintained as the original Coqui company shut down) is the strongest open-weight voice-cloning TTS. Multilingual, voice cloning from a 6-second reference clip, ~5× real-time on a 5090. For voice assistants where the same speaker needs to come through every interaction, it’s the default choice.
For a voice assistant: Whisper Large-v3 + Llama 3.1 8B FP8 + XTTS v2 on a single GPU. Fits comfortably on a RTX 5090 32 GB; tight on a RTX 5060 Ti 16 GB. Sub-1-second end-to-end achievable.
Why XTTS for a voice assistant
- Voice cloning from a 6-second reference. No fine-tuning required.
- 17 languages on the same model. No per-language model swap.
- ~5× real-time on a 5090 — fast enough for live conversation.
- 4 GB VRAM — fits comfortably alongside an LLM and Whisper.
- Open weights (CPML license, non-commercial unless you self-host on dedicated hardware).
The pipeline
- User speaks — audio captured by client
- VAD detects end of speech (Silero)
- Whisper transcribes (faster-whisper)
- LLM generates response (Llama 3.1 8B FP8 or similar)
- XTTS synthesises with cloned voice
- Audio streamed back to client
Pipecat or LiveKit Agents handle the orchestration. See our voice agent hosting page for deployment.
VRAM math
| Component | VRAM |
|---|---|
| Whisper Large-v3 (faster-whisper INT8) | ~3 GB |
| Llama 3.1 8B FP8 | ~8 GB |
| XTTS v2 | ~4 GB |
| KV cache @ 4K context, 8 concurrent | ~3 GB |
| Speaker reference cache | ~1 GB |
| Total | ~19 GB |
~19 GB peak. Fits a 32 GB RTX 5090 comfortably. Tight on a 24 GB 4090 (workable). Doesn’t fit a 16 GB 5080.
Latency benchmarks
| Stage | 5090 latency | 4090 latency |
|---|---|---|
| Whisper (3s utterance) | 180 ms | 220 ms |
| LLM TTFT (1K context) | 120 ms | 180 ms |
| XTTS first chunk | 350 ms | 450 ms |
| End-to-end | ~650 ms | ~850 ms |
Verdict
XTTS v2 is the right TTS for a voice assistant where speaker identity matters. RTX 5090 is the right GPU. RTX 4090 works tight; smaller cards do not have enough VRAM for the full stack.
Bottom line
For a voice cloning + multilingual voice assistant: XTTS v2 on a 5090, paired with Whisper + Llama 3.1 8B. Sub-1-second end-to-end. For Bark / Kokoro alternatives see best GPU for TTS.
