The most consequential trend in self-hosting this year is not a new card – it is a new number format. FP4 (4-bit floating point) inference is going mainstream in 2026, and it quietly changes the GPU you need to buy. A model that demanded a 32GB card last year may now fit on a 16GB one with negligible quality loss. Here is what is happening and how to use it on dedicated GPU hosting.
Table of Contents
A Quick Precision Primer
Model weights are stored as numbers, and the number of bits per weight sets the memory footprint. FP16 (16-bit) was the old default. FP8 (8-bit) became mainstream on Hopper and Blackwell hardware and roughly halved memory needs. FP4 takes the next step – 4 bits per weight – cutting the footprint again. The progression FP16 → FP8 → FP4 has been the single biggest driver of “bigger models on smaller cards” over the last three years.
What FP4 Does to Your VRAM Budget
| Model | FP16 | FP8 | FP4 |
|---|---|---|---|
| 8B params (weights) | ~16GB | ~8GB | ~4GB |
| 32B params (weights) | ~64GB | ~32GB | ~16GB |
| 70B params (weights) | ~140GB | ~70GB | ~35GB |
The practical effect: a 70B model that needed a 96GB card or a multi-GPU rig at FP16 fits on a single 32GB RTX 5090 or Radeon AI Pro R9700 in 4-bit (weights; budget extra for context). That is a tier-and-a-half drop in the hardware you have to pay for. See real numbers in the tokens per second benchmark.
Which Hardware Supports It
FP4 acceleration is a Blackwell-generation feature on the NVIDIA side – the RTX 50-series and RTX 6000 PRO have the tensor-core support to run it at full speed rather than emulating it. Older cards like the RTX 3090 can still run 4-bit quantised models (via GPTQ/AWQ-style integer quantisation) but without native FP4 acceleration. For open-source LLM hosting, this is a genuine reason to prefer a Blackwell card if low-precision throughput matters.
Run 70B Models on a 32GB Card
FP4-ready Blackwell GPUs on dedicated servers. Fit bigger models on smaller hardware.
Browse GPU ServersThe Quality Question
Does 4-bit hurt quality? Less than intuition suggests. Modern quantisation methods are calibration-aware and preserve most of a model’s capability for the majority of tasks; the gap between FP4 and FP16 on standard benchmarks is small for many models and shrinking as techniques improve. The honest caveat: the most precision-sensitive work – hard reasoning, exact code generation, long multi-step chains – can still show measurable degradation, so test on your own task before committing.
The Takeaway for Buyers
Low-precision inference means you should size hardware to the quantised footprint of your model, not its FP16 size. For most teams that is a meaningful cost saving – often a full price tier. Pair an FP4-capable card with a calibrated 4-bit checkpoint and you get yesterday’s flagship workload on today’s mid-tier budget.
Keep up with the format shifts in the news section, see method comparisons in the benchmarks section, and weigh the economics with the GPU vs API cost comparison.
