Table of Contents
Gemma 2 9B (Google) is the "safety-tuned, reasoning-strong" option in the 9B class. The 18 GB FP16 footprint puts it just over a 16 GB card, but FP8 brings it into reach.
TL;DR
Gemma 2 9B at FP8 fits the RTX 5060 Ti 16 GB with ~6 GB headroom for KV cache. ~620 tok/s aggregate, ~58 tok/s single-stream. AWQ-INT4 is even more efficient.
VRAM fit
| Precision | Weights | KV pool | Concurrent users |
|---|---|---|---|
| FP16 | 18 GB | Does not fit | n/a |
| FP8 | 9 GB | ~6 GB | ~20 |
| AWQ-INT4 | 6 GB | ~9 GB | ~30 |
Benchmark results
| Precision | Aggregate tok/s | Single-stream | TTFT |
|---|---|---|---|
| FP8 | 620 | 58 | 230 ms |
| AWQ-INT4 | 720 | 64 | 210 ms |
Verdict
Gemma 2 9B on a 5060 Ti is a credible deployment at FP8 or INT4. For applications that benefit from Gemma's safety tuning (customer-facing chatbots, regulated workloads), it's a reasonable pick.
Bottom line
Gemma 2 9B FP8 on a 5060 Ti at £119/mo. See best GPU for Gemma.
