LLM Inference on RTX 5070: vLLM, LlamaCPP, and Blackwell CUDA Guide

The CUDA Inference Stack

The RTX 5070 runs every major LLM inference framework without modification:

Model VRAM Table

FP8 is the recommended production precision on Blackwell. It halves VRAM vs FP16, enabling models that technically wouldn’t fit at FP16 to fit at FP8, and Blackwell’s 5th Gen Tensor Cores run FP8 natively with no performance penalty versus INT8.

LlamaCPP CUDA Setup

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

# Run Llama 3.1 8B Q4_K_M (all 32 layers on GPU)
./bin/llama-cli \
  -m /models/llama-3.1-8b-instruct-q4_k_m.gguf \
  -ngl 32 \
  -c 8192 \
  --temp 0.7 \
  -p "You are a helpful assistant." \
  --interactive

# HTTP server (OpenAI-compatible)
./bin/llama-server \
  -m /models/llama-3.1-8b-instruct-q4_k_m.gguf \
  -ngl 32 \
  -c 8192 \
  --host 0.0.0.0 --port 8080

Set -ngl to the total number of transformer layers in your model (32 for 7–8B models, 40 for 13B). All layers go to GPU — CPU offload is only needed if the model exceeds 12 GB, which at Q4_K_M means models larger than ~20B.

vLLM Setup

vLLM’s PagedAttention and continuous batching are well-supported on Blackwell. Install and run:

pip install vllm

# Serve Llama 3.1 8B at FP8 (Blackwell-optimised)
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
  --dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --host 0.0.0.0 --port 8000

# Or serve at Q4 via GGUF
vllm serve /models/llama-3.1-8b-instruct-q4_k_m.gguf \
  --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct \
  --max-model-len 4096 \
  --host 0.0.0.0 --port 8000

For Qwen 2.5 14B at Q4 (8.7 GB weights), use --gpu-memory-utilization 0.85 to leave headroom for KV cache:

vllm serve Qwen/Qwen2.5-14B-Instruct \
  --quantization gptq \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --host 0.0.0.0 --port 8000

OpenAI-Compatible Serving

Both llama.cpp server and vLLM expose OpenAI-compatible endpoints. To call from a client:

from openai import OpenAI

# Point at your local server (llama.cpp or vLLM)
client = OpenAI(base_url="http://localhost:8080/v1", api_key="unused")

response = client.chat.completions.create(
    model="llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "What is Blackwell FP8?"}],
    temperature=0.7,
    max_tokens=512
)
print(response.choices[0].message.content)

FP8 Inference on Blackwell

Blackwell’s 5th Gen Tensor Cores natively compute FP8 matrix operations. For LLM inference this means:

• Llama 3.1 8B FP8: ~8 GB VRAM (vs 16 GB at FP16) — fits on 12 GB with 4 GB for KV cache at 8K context
• Mistral 7B FP8: ~7 GB — comfortable
• Gemma 2 9B FP8: ~9 GB — fits with ~3 GB KV budget
• Quality loss vs FP16: typically <1% on standard benchmarks

FP8 is superior to INT8 for inference quality on Blackwell because it maintains a floating-point exponent — key for the wide dynamic range in transformer attention layers. Use --dtype fp8 in vLLM or the AutoFP8 quantisation library for on-the-fly FP8 quantisation.

Throughput Estimates

RTX 5070 vs RTX 3090 for LLM

Interestingly, the RTX 5070 is faster per token than the RTX 3090 on smaller models at Q4 — Blackwell’s architecture and CUDA improvements outpace Ampere despite the lower raw bandwidth. But the 3090’s 24 GB VRAM wins decisively for 13B+ models at Q4 and any FP16 inference. See the full RTX 5070 vs RTX 3090 comparison.