Benchmark Overview
Batch size is the single largest lever for GPU throughput. A GPU processing one request at a time runs at 10-30% utilisation. At batch size 32, utilisation jumps to 80-95%. We measured throughput scaling from batch size 1 to 128 on RTX 6000 Pro and RTX 6000 Pro GPUs to map the diminishing returns curve for production inference on dedicated GPU hosting.
Test Configuration
Model: Llama 3 70B INT4 via vLLM with continuous batching. GPUs: RTX 6000 Pro 96 GB, RTX 6000 Pro 96 GB. Each request: 256 input tokens, 128 output tokens. Batch sizes: 1, 4, 8, 16, 32, 64, 128 simultaneous requests. Throughput measured as total tokens generated per second across all requests. See vLLM production setup for configuration.
Throughput by Batch Size (Llama 3 70B INT4)
| Batch Size | RTX 6000 Pro Total tok/s | RTX 6000 Pro GPU Util | RTX 6000 Pro Total tok/s | RTX 6000 Pro GPU Util |
|---|---|---|---|---|
| 1 | 48 | 18% | 78 | 15% |
| 4 | 185 | 42% | 305 | 38% |
| 8 | 350 | 62% | 580 | 58% |
| 16 | 620 | 78% | 1,020 | 75% |
| 32 | 980 | 88% | 1,650 | 86% |
| 64 | 1,280 | 93% | 2,180 | 92% |
| 128 | 1,420 | 95% | 2,480 | 94% |
Per-Request Latency by Batch Size
| Batch Size | RTX 6000 Pro per-request (128 tokens) | RTX 6000 Pro per-request (128 tokens) |
|---|---|---|
| 1 | 2.7s | 1.6s |
| 8 | 2.9s | 1.8s |
| 32 | 4.2s | 2.5s |
| 64 | 6.4s | 3.8s |
| 128 | 11.5s | 6.6s |
Diminishing Returns Analysis
Throughput increases sub-linearly with batch size. Going from batch 1 to 32 provides a 20x throughput increase on RTX 6000 Pro. Going from 32 to 128 provides only a 1.45x increase. The GPU compute is nearly saturated at batch 32, and further batching primarily fills memory bandwidth gaps. Per-request latency increases linearly beyond batch 32 as requests queue. See token benchmarks for more configurations and GPU comparison data.
For latency-sensitive applications (chatbots, real-time APIs), batch sizes of 8-32 offer the best throughput-latency balance. For offline batch processing (document summarisation, data extraction), batch 64-128 maximises throughput at the cost of higher per-request latency. Explore multi-GPU clusters for scaling beyond single-GPU batch limits.
VRAM as the Batch Limit
At batch 128 with 256-token contexts, KV cache consumes approximately 30GB on RTX 6000 Pro. Combined with 38GB model weights, total VRAM reaches 68GB, leaving only 12GB headroom. Larger contexts or larger batches will hit OOM. FP8 KV cache doubles the maximum batch size. See LLM hosting for memory optimisation strategies.
Recommendations
Target batch sizes of 16-32 for interactive workloads and 64-128 for offline processing. Configure vLLM’s max_num_seqs parameter to match your target concurrency. Deploy on GigaGPU dedicated servers with private hosting. Visit the benchmarks section and infrastructure blog for scaling guidance.
