RTX 4090 24GB for Gemma 2 27B: Google’s Flagship Open Model on a Single Card

Gemma 2 27B is Google’s flagship open-weight model and the largest member of the Gemma 2 family. FP16 weights are 54 GB and FP8 is 27 GB – both miss a 24 GB card cleanly. AWQ INT4 brings weights down to roughly 14 GB, which fits the RTX 4090 24GB with comfortable KV headroom for the model’s native 8k context. The headline result: at conversational benchmarks (MT-Bench 8.84) it competes with Llama 3 70B at less than half the parameters, while running on a single consumer GPU. This guide on our UK dedicated GPU hosting covers the architecture (46 layers, 16 KV heads, 128-dim heads, sliding-window attention), the AWQ deployment path that makes the fit work, throughput, the seven gotchas, and the decision logic against Qwen 32B AWQ and Llama 3 70B INT4.

Contents

• Why Gemma 2 27B fits a 24 GB card and 70B does not
• Architecture, sliding-window attention and KV math
• VRAM accounting and concurrency budgets
• Throughput, latency and batched performance
• Quality benchmarks vs alternatives
• vLLM deployment and code
• Production gotchas (seven items)
• When to pick this over alternatives
• Verdict

Why Gemma 2 27B fits a 24 GB card and 70B does not

Three architectural facts make Gemma 2 27B AWQ work where 70B-class models do not. First, the parameter count is moderate enough that AWQ INT4 brings weights to 14 GB – a comfortable fit. Second, sliding-window attention bounds the per-layer KV cache to 4k tokens regardless of full context length, which means concurrent long-context users do not blow the KV budget the way they would on full-attention models. Third, the 16 KV heads with 128-dim heads (vs Gemma 2 9B’s 8 KV heads with 256-dim heads) give a more conventional KV footprint that vLLM’s PagedAttention handles efficiently.

Compared with Llama 3 70B, Gemma 2 27B is roughly 60% the inference cost per token at comparable conversational quality. That is the core value proposition: GPT-3.5-class chat fluency on a single consumer GPU instead of two. For chat-heavy SaaS applications where MT-Bench is the headline metric, this is a meaningfully better ROI than running Llama 3 70B INT4 across two cards.

Architecture, sliding-window attention and KV math

Gemma 2 27B-it is a 46-layer transformer with 32 query heads, 16 key/value heads (a 2:1 GQA ratio), 128-dim heads, an SwiGLU MLP at 36,864 intermediate dim (very wide), RMSNorm, and the same 256,128-token vocabulary as the 9B sibling. Native context is 8,192. License is Gemma Terms (commercial use permitted with restrictions; not Apache 2.0).

The alternating attention pattern is identical to the 9B: every other layer uses full attention, the layers in between use sliding-window attention with a 4,096-token window. Logit soft-capping at 50 (attention) and 30 (final) is also retained. KV cache per token at FP16: 16 KV heads x 128 dim x 23 full layers x 2 (K+V) x 2 bytes = 188 KB for the full layers, plus 188 KB/token for SWA layers capped at 4k = 768 MB max. At FP8 KV: 94 KB/token full + capped 384 MB SWA. So 8k context = 0.75 GB FP8 full layers + 0.38 GB SWA = 1.13 GB per sequence.

VRAM accounting and concurrency budgets

SWA caps KV growth – even at the full 8k native context, KV is bounded by the 4096-token SWA window for half the layers. This is what makes the multi-stream concurrency work better than a comparable full-attention 27B would.

Throughput, latency and batched performance

vLLM 0.6.3, AWQ Marlin, FP8 KV, FlashAttention 2.6+, eager mode, prompt 1024, generate 256:

Slightly faster than Qwen 2.5 32B AWQ (~65 t/s) thanks to fewer parameters and SWA-bounded KV – see our Qwen 32B benchmark for the comparison axis. The aggregate plateau at batch 8 is the ceiling – past that, KV pressure forces preemption.

Quality benchmarks vs alternatives

Gemma 2 27B is competitive with Llama 3 70B on conversational quality (MT-Bench 8.84 vs 8.95) despite being a third of the size. For coding-heavy or math-heavy work, Qwen 2.5 32B AWQ is materially better. The gap on HumanEval (51.8 vs 88.4) is the most stark – if any meaningful fraction of your traffic involves code, Gemma is the wrong pick.

vLLM deployment and code

python -m vllm.entrypoints.openai.api_server \
  --model google/gemma-2-27b-it-AWQ \
  --quantization awq_marlin \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.93 \
  --enforce-eager \
  --enable-prefix-caching

Three flags are critical. --enforce-eager avoids the 1.5 GB CUDA-graph memory premium – essential when you are at 20 GB used. --gpu-memory-utilization 0.93 pushes vLLM to use the full envelope short of the OOM danger zone. --max-num-seqs 8 caps concurrency safely.

If Google have not yet published an official AWQ build, use the community casperhansen/gemma-2-27b-it-awq checkpoint, or build your own with AutoAWQ:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "google/gemma-2-27b-it"
quant_path = "gemma-2-27b-it-awq"
quant_config = {"zero_point": True, "q_group_size": 128,
                "w_bit": 4, "version": "GEMM"}

model = AutoAWQForCausalLM.from_pretrained(model_path, **{"low_cpu_mem_usage": True})
tokenizer = AutoTokenizer.from_pretrained(model_path)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
# Calibration uses pile-c4 by default; takes ~45 min on a single 4090

The AWQ quantization guide documents the calibration trade-offs. Our vLLM setup notes cover the FlashAttention 2.6+ pinning that Gemma 2 requires for soft-cap correctness.

Production gotchas (seven items)

• Eager mode is mandatory at 8k+: CUDA graphs add 1.5 GB. Cannot afford them at 17+ GB used. Throughput penalty is ~6%.
• FlashAttention 2.6+ required: Gemma 2’s 50/30 logit soft-capping needs FlashAttention 2.6 or newer. Older versions silently produce degraded outputs – the model still generates, just worse. Pin in requirements.
• vLLM SWA support requires 0.5.4+: older vLLM falls back to full attention for the SWA layers, doubling KV memory and breaking the fit.
• No system role in chat template: prepend system instructions to the first user message or your prompts produce degraded output.
• 8k context is a hard ceiling: do not extend with YaRN/LongRoPE – quality collapses. Use chunk-summarisation for longer documents.
• Cap max-num-seqs at 8: above 8 streams the KV pressure forces preemption and aggregate throughput plateaus or degrades.
• Power and thermal at sustained batch 8: 4090 draws 440-450 W. Cap to 410 W with nvidia-smi -pl 410 for stable throughput, see power draw analysis.

When to pick this over alternatives

Pick Gemma 2 27B over Gemma 2 9B when 9B’s MMLU 71.3 is not enough quality, or when MT-Bench 8.84 vs 8.36 matters for your headline conversational metric. Pick it over Qwen 2.5 32B AWQ when MT-Bench wins specifically and you have no code traffic – otherwise Qwen 32B is materially better on every quantitative benchmark. Pick it over Mistral Small 3 24B when MT-Bench matters and you accept the code regression.

Step up to Llama 3.1 70B INT4 across two cards if you need MMLU 83+ or MATH 68+. Step laterally to Yi-34B for Chinese-centric workloads. Step down to Gemma 2 9B if 8-stream concurrency is too few for your traffic.

Verdict

Gemma 2 27B AWQ on the RTX 4090 24GB is the conversational-quality leader at the 27B-30B class on a single consumer GPU. 75 t/s decode, MT-Bench 8.84 (within striking distance of Llama 3 70B), 8-stream concurrency on short prompts, and SWA-bounded KV that makes the multi-stream behaviour better than a comparable full-attention model. The trade-offs are weak code (HumanEval 51.8), 8k context ceiling, eager mode mandatory, FlashAttention 2.6+ required, and Google’s Gemma Terms licence with commercial restrictions worth checking. For a chat-heavy SaaS where conversational fluency dominates the user experience and code is not in scope, this is a default 4090 choice. For mixed reasoning + code workloads, prefer Qwen 2.5 32B AWQ. Compare with the RTX 5090 32GB for FP8 headroom or the upgrade decision page.

See also: Gemma 2 9B, Qwen 2.5 32B AWQ, Mistral Small 3 24B, Yi-34B, Mixtral 8x7B, 4090 vs 5090, monthly cost.