Benchmark Overview
Voice agents must respond within 1-2 seconds to feel conversational. This total round-trip includes speech-to-text (STT), LLM inference, and text-to-speech (TTS). We benchmarked each stage across GPU models to determine the minimum hardware needed for natural-feeling voice conversations on dedicated GPU hosting.
Test Configuration
STT: Faster-Whisper Large-V3 (GPU). LLM: Llama 3 8B INT4 via vLLM, generating 80-token responses. TTS: Kokoro v1.0 (GPU). Input audio: 5-second user utterance (approximately 15-20 words). All components co-located on a single GPU server. See token speed benchmarks for LLM-only data.
Round-Trip Latency Breakdown
| Stage | RTX 5090 | RTX 6000 Pro | RTX 6000 Pro 96 GB | RTX 6000 Pro |
|---|---|---|---|---|
| STT (5s audio, Faster-Whisper) | 280ms | 310ms | 240ms | 160ms |
| LLM First Token (8B INT4) | 45ms | 52ms | 38ms | 25ms |
| LLM Generation (80 tokens) | 850ms | 920ms | 680ms | 420ms |
| TTS (80 tokens, Kokoro) | 190ms | 210ms | 165ms | 110ms |
| Total Round-Trip | 1,365ms | 1,492ms | 1,123ms | 715ms |
| Time to First Audio | 515ms | 572ms | 443ms | 295ms |
Stage Analysis
LLM generation dominates at 55-62% of total round-trip time. STT accounts for 18-22%, and TTS for 14-15%. Optimising the LLM stage (faster GPU, smaller model, speculative decoding) delivers the largest improvement. The RTX 5090 achieves 1.37-second round-trip with an 8B model, which is conversational but not seamless. The RTX 6000 Pro at 715ms feels genuinely real-time. Check GPU comparison for cost-performance data.
70B Model Voice Agent Feasibility
Replacing the 8B model with Llama 3 70B INT4 increases total round-trip to 3.8s on RTX 5090 and 1.9s on RTX 6000 Pro. Only the RTX 6000 Pro keeps 70B models under the 2-second conversational threshold. For voice agents requiring higher-quality responses, RTX 6000 Pro hardware or multi-GPU setups with tensor parallelism are necessary.
Concurrent Voice Sessions
Each concurrent voice session requires GPU time for STT, LLM, and TTS. At 5 simultaneous sessions on an RTX 6000 Pro, round-trip latency increases to approximately 2.1 seconds (8B model). At 10 sessions, it exceeds 3 seconds. Scaling voice agents beyond 5 concurrent sessions per GPU requires dedicated GPUs per pipeline stage. Review LLM hosting and tutorials for voice agent architecture.
Recommendations
For sub-1.5-second voice agent round-trips, deploy 8B models on RTX 6000 Pro or better hardware. For sub-1-second targets, RTX 6000 Pro is necessary. Use streaming TTS to begin audio playback before the full LLM response completes, reducing perceived latency by 200-400ms. Deploy voice agents on GigaGPU dedicated servers with private hosting. Explore the benchmarks section and infrastructure blog for scaling patterns.
