Can RTX 5090 Run a 70B Model in FP16?

Can RTX 5090 Run 70B in FP16?

No, the RTX 5090 cannot run a 70B parameter model in FP16. A 70B model at FP16 requires approximately 140 GB of VRAM for weights alone, and the RTX 5090 has 32 GB of GDDR7. You would need roughly 4.4 RTX 5090s to hold the model weights. However, the 5090 can run 70B at 4-bit quantization with limited context, making it the best single consumer GPU for this task on a dedicated GPU server.

The RTX 5090 is NVIDIA’s flagship Blackwell consumer GPU with 32 GB of GDDR7 at approximately 1,792 GB/s bandwidth. This makes it the highest-VRAM consumer card available, but 70B in FP16 remains firmly in data center GPU territory.

The VRAM Math: 32 GB vs 140 GB

Here is a clear breakdown of why FP16 does not work:

Even INT8 quantization (which preserves very high quality) needs 73 GB, more than double the 5090’s capacity. Standard 4-bit quantization at ~41 GB also exceeds 32 GB, though aggressive 3-bit quantization can squeeze the model in. See our LLaMA 3 VRAM requirements guide for details on each precision level.

What 70B Configurations Fit on 32 GB?

The RTX 5090 can fit 70B at 2-3 bit quantization. GGUF IQ3_XXS is the best balance, offering roughly 86% of FP16 quality with a short context window. Q2_K fits more comfortably but quality drops further. Read our quantization format guide for details on each format.

Performance at Reduced Precision

Expected performance for 70B models on the RTX 5090:

At 12-17 tok/s with extreme quantization, the 5090 delivers a usable but compromised experience for 70B. The Blackwell architecture’s high bandwidth helps significantly compared to older cards. Check our tokens per second benchmark for real-time comparisons.

What Models Can Run in FP16 on RTX 5090?

The 5090’s 32 GB excels at FP16 inference for models up to about 14-15B parameters:

For models up to 14B, the RTX 5090 is exceptional. 32 GB gives you FP16 quality plus room for long context and batching. See related pages on Qwen VRAM requirements and Phi VRAM requirements.

Setup Commands

70B at Extreme Quantization

# Ollama with 70B (will auto-quantize)
ollama run llama3:70b

# llama.cpp with IQ3_XXS for best quality that fits
./llama-server -m llama-3-70b-IQ3_XXS.gguf \
  -ngl 80 -c 1024 --host 0.0.0.0 --port 8080

8B-14B at FP16 (Recommended for 5090)

# vLLM serving LLaMA 3 8B at FP16 with batching
pip install vllm
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --dtype float16 --max-model-len 8192 \
  --max-num-seqs 8 --gpu-memory-utilization 0.90

For deployment guides, visit our Ollama hosting and vLLM hosting pages.

GPUs That Can Run 70B in FP16

If you absolutely need 70B in FP16, here are the options:

Running 70B in true FP16 requires data center GPUs. For most practical purposes, INT8 or 4-bit quantization on consumer GPUs delivers 94-98% of FP16 quality at a fraction of the cost. Explore our multi-GPU cluster options or compare costs using the LLM cost calculator. Also see our RTX 3090 70B analysis and best GPU for LLM inference guide.