Table of Contents
A real-time voice agent feels human under 1 second end-to-end. Achievable, but every component is on the critical path.
TL;DR
Sub-1s end-to-end voice agent on a 5090: Silero VAD (120 ms) + faster-whisper Large-v3-Turbo (140 ms) + Mistral 7B FP8 with prefix cache (120 ms TTFT) + Kokoro TTS (60 ms first audio) = ~440 ms typical.
Latency budget
| Stage | Target latency |
|---|---|
| VAD endpointing | 120 ms |
| STT (streaming) | 180 ms |
| LLM TTFT | 120 ms |
| TTS first audio chunk | 60 ms |
| Network return | 50 ms |
| Total | ~530 ms |
Component picks
- VAD: Silero VAD on CPU, aggressive endpointing
- STT: faster-whisper Large-v3-Turbo with sliding window
- LLM: Mistral 7B or Llama 3.1 8B FP8 with prefix caching
- TTS: Kokoro for speed; XTTS for voice cloning
- Orchestrator: Pipecat (Python) or LiveKit Agents
- Hardware: RTX 5090 32 GB
Verdict
Sub-1s voice agents are achievable on a single 5090. Each component matters; cutting one corner breaks the budget.
Bottom line
For voice that feels human, latency wins over model quality. See latency optimisation guide.
