Benchmark Overview
Two distinct metrics define how fast an LLM feels: time-to-first-token (TTFT) and streaming throughput (tokens per second after the first). A fast TTFT with slow streaming feels responsive but sluggish. A slow TTFT with fast streaming feels delayed then bursts. We benchmarked both metrics independently across GPU models to help you optimise the right one for your application on dedicated GPU hosting.
Test Configuration
Models: Llama 3 8B INT4, Llama 3 70B INT4, Qwen 2.5 72B INT4. GPUs: RTX 5090, RTX 6000 Pro 96 GB, RTX 6000 Pro. Engine: vLLM. Input: 512 tokens. Output: 256 tokens. Single user. TTFT = time from request to first generated token. Streaming throughput = tokens per second during generation phase. See token speed benchmarks for extended data.
Time-to-First-Token (TTFT)
| Model | RTX 5090 | RTX 6000 Pro 96 GB | RTX 6000 Pro |
|---|---|---|---|
| Llama 3 8B INT4 | 52ms | 42ms | 28ms |
| Llama 3 70B INT4 | 195ms | 128ms | 78ms |
| Qwen 2.5 72B INT4 | 205ms | 135ms | 85ms |
Streaming Throughput (Tokens per Second, Single User)
| Model | RTX 5090 | RTX 6000 Pro 96 GB | RTX 6000 Pro |
|---|---|---|---|
| Llama 3 8B INT4 | 95 tok/s | 120 tok/s | 185 tok/s |
| Llama 3 70B INT4 | 35 tok/s | 48 tok/s | 78 tok/s |
| Qwen 2.5 72B INT4 | 33 tok/s | 45 tok/s | 72 tok/s |
What Drives Each Metric
TTFT is dominated by the prefill phase: processing the entire input prompt in a single forward pass. Longer prompts increase TTFT linearly. GPU compute throughput (TFLOPS) determines prefill speed. The RTX 6000 Pro achieves sub-100ms TTFT for 70B models because its FP8 tensor cores process the 512-token prompt in one efficient pass.
Streaming throughput is determined by memory bandwidth. Each generated token requires reading all model weights from HBM once. The RTX 6000 Pro’s 3.35 TB/s HBM3 bandwidth produces 78 tok/s for 70B INT4, while the RTX 6000 Pro’s 2 TB/s HBM2e produces 48 tok/s. Check GPU specifications for bandwidth comparisons. Compare engines at vLLM vs Ollama.
Application-Specific Priorities
Chatbots: TTFT matters most. Users notice the delay before the first word appears. Target sub-200ms TTFT. Streaming at 30+ tok/s is fast enough to read comfortably. Configure via the vLLM production guide.
Code completion: TTFT is critical since completions must appear within 300ms. Streaming speed is less important as completions are short (15-30 tokens).
Batch processing: Streaming throughput dominates total processing time. TTFT is negligible across thousands of requests on LLM hosting. Use Ollama for simpler single-user setups.
Recommendations
Optimise TTFT for interactive applications: use faster GPUs, shorter prompts, and prefix caching. Optimise streaming throughput for batch workloads: use larger batch sizes and higher memory bandwidth GPUs. Deploy on GigaGPU dedicated servers with private hosting. Visit the benchmarks section for more performance data.
