Run Gemma 2 on a Dedicated GPU Server

GPU Selection for Gemma 2

Google’s Gemma 2 is an open model family available in 2B, 9B, and 27B parameter sizes. The 9B variant is particularly notable for matching or exceeding many 13B models on standard benchmarks. Here is the GPU mapping for Gemma 2 hosting on a dedicated GPU server:

The 9B model at FP16 fits on an RTX 3090 with 6 GB to spare for KV cache. At INT4, even the 27B model runs on a 16 GB RTX 4060 Ti. For a head-to-head comparison with LLaMA, see our Gemma vs LLaMA 3 comparison.

Install and Serve with vLLM

# Install vLLM
pip install vllm

# Serve Gemma 2 9B Instruct
python -m vllm.entrypoints.openai.api_server \
  --model google/gemma-2-9b-it \
  --dtype float16 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --port 8000

# Test the endpoint
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-2-9b-it",
    "messages": [{"role": "user", "content": "Explain how attention mechanisms work in transformers."}],
    "max_tokens": 512
  }'

For serving framework trade-offs, read our vLLM vs Ollama guide.

Quick Start with Ollama

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run Gemma 2 9B
ollama run gemma2:9b

# Serve as API
ollama serve &
curl http://localhost:11434/api/generate \
  -d '{"model": "gemma2:9b", "prompt": "What are the benefits of open-source AI models?"}'

Performance Benchmarks

Benchmarked with vLLM, 512-token input, 256-token output. See the tokens-per-second benchmark tool for live data.

Gemma 2 9B at AWQ 4-bit on the RTX 4060 delivers 105 tok/s, making it one of the fastest mid-size models for budget deployments. The 27B variant at 42 tok/s on the RTX 3090 is usable for interactive chat applications.

Optimisation Tips

• Use the 9B variant as the default choice. It matches many 13B models on reasoning while being significantly faster.
• AWQ 4-bit for the 27B variant makes it accessible on 16-24 GB GPUs with good quality retention.
• Enable sliding window attention (built into Gemma 2) for efficient long-context inference without exploding KV cache size.
• Run the 2B model on edge GPUs like the RTX 3050 for cost-effective lightweight inference.
• Pair Gemma 2 with RAG using ChromaDB for document question answering pipelines.

Estimate running costs with the cost calculator. Browse all deployment guides in the model guides section.

Next Steps

Gemma 2 offers excellent quality-per-parameter for self-hosting. For multilingual needs, compare with Qwen 2.5. For the strongest open-weight English model, see LLaMA 3 hosting. Use the GPU comparisons tool to find the right hardware for your workload.