LLaMA 3 8B Tokens/sec by GPU (Full Benchmark)

Test Methodology

We benchmarked Meta’s LLaMA 3 8B across six GPUs available on GigaGPU dedicated servers. Tests used vLLM as the primary inference engine with continuous batching enabled. Each benchmark generates 512 output tokens from a 256-token prompt. All GPUs run the same software stack: Ubuntu 22.04, CUDA 12.4, Python 3.11, vLLM latest. For interactive exploration, visit our tokens/sec benchmark tool.

Single-User Throughput (Batch Size 1)

Single-user throughput determines the response speed when one user queries the model at a time. This is the most relevant metric for development, testing, and low-traffic applications.

The RTX 5090 delivers 2.2x the throughput of the RTX 3090 and 7.7x the RTX 3050. Memory bandwidth is the primary bottleneck for autoregressive generation. For comparisons with other GPUs, see RTX 3090 vs RTX 5090 and our GPU comparisons tool.

Concurrent User Throughput

Concurrent throughput measures how the GPU handles multiple users simultaneously. vLLM’s continuous batching significantly improves total throughput under concurrent load.

Total throughput scales 3-7x from 1 to 16 users depending on GPU. Individual per-user speed decreases, but total system throughput increases substantially. For more on serving engines, see our vLLM vs TGI vs Ollama comparison.

Quantisation Impact on Throughput

Quantisation reduces VRAM usage and can increase throughput on memory-bandwidth-limited GPUs. We tested FP16, AWQ 4-bit, and GPTQ 4-bit on the RTX 3090.

AWQ 4-bit quantisation increases single-user throughput by ~26% on the RTX 3090 while using 62% less VRAM. This makes quantised LLaMA 3 8B viable on 8 GB GPUs. For a deeper dive into quantisation formats, see our GPTQ vs AWQ vs GGUF guide.

Cost per Million Tokens

We calculated the cost to generate one million output tokens at sustained throughput on each GPU.

Budget GPUs offer the lowest cost per million tokens. Compare these self-hosted costs with API pricing in our GPU vs OpenAI cost breakdown and cost calculator.

Inference Engine Comparison

LLaMA 3 8B on the RTX 3090 across three inference engines:

vLLM delivers the highest throughput, especially under concurrent load. Ollama is simpler to deploy but lacks continuous batching. For full engine comparison, see our vLLM vs TGI vs Ollama guide. For Ollama hosting, single-user performance is adequate for development.

GPU Recommendations for LLaMA 3 8B

Best overall: RTX 3090. At 62 tok/s (FP16) and $2.02 per million tokens, the RTX 3090 provides the best balance of speed and cost for LLaMA 3 8B. The 24 GB VRAM fits the model in FP16 with room for KV cache and concurrent users.

Best for production: RTX 5090. At 138 tok/s the 5090 delivers sub-4-second responses for 512-token generations. The 32 GB VRAM handles larger context windows and more concurrent users.

Best budget: RTX 4060. With AWQ 4-bit quantisation, the 4060 achieves 52 tok/s at just $1.07 per million tokens. Viable for development and low-latency-tolerance applications.

Best mid-range: RTX 5080. At 85 tok/s the 5080 delivers fast single-user responses. The 16 GB VRAM fits FP16 LLaMA 3 8B with limited concurrent user capacity.

For other model benchmarks, see our Mistral 7B benchmark, DeepSeek benchmark, and best GPU for LLM inference overview. For RAG applications, see best GPU for RAG pipelines.