RTX 4090 24GB vs A100 80GB: Consumer Ada FP8 vs Ampere Datacentre HBM2e

The A100 80GB remains one of the most-deployed accelerators in production AI infrastructure. It pairs HBM2e (2.0 TB/s) with 80 GB of capacity, NVLink at 600 GB/s and MIG support — but stops at FP16/BF16. There is no native FP8. The RTX 4090 24GB brings native FP8 to a smaller VRAM envelope and roughly half the bandwidth, plus a 12x larger L2 cache. On UK GPU hosting the choice depends entirely on whether your workload is bandwidth-bound, capacity-bound, or precision-bound — and the answer is rarely obvious. This post explains which workloads pay back the A100 premium and which are better served by the cheaper, FP8-native consumer card.

Contents

• Spec sheet side by side
• The FP8 question — Ampere’s missing trick
• 80GB vs 24GB — capacity advantage
• Throughput across nine workloads
• Per-token economics
• Per-workload winner table
• vLLM serving examples
• Production gotchas
• Verdict

Spec sheet side by side

The interesting numbers: the 4090 has 79% higher clock, 80% larger L2, and native FP8 — but only half the bandwidth and a third of the VRAM. The A100 has the bigger, faster pool of memory but cannot use FP8, so its effective throughput on FP8-native workloads is closer than the bandwidth ratio suggests.

The FP8 question — Ampere’s missing trick

The A100 has no FP8 tensor instruction. When vLLM is asked for FP8 on an A100, it falls back to FP16 — halving the effective throughput per tensor-core op. There is no software fix. This is the single most important point of the comparison: for any modern LLM workload that can run FP8 (Llama, Mistral, Qwen, Phi, Gemma all have FP8 weights available), the 4090 has a precision-format advantage that partially offsets its bandwidth deficit.

For BF16 workloads (training, some research), the A100 wins because it has 2x the bandwidth and the format is its native sweet spot. For FP8 inference, the comparison is much closer than headline specs suggest. AWQ INT4 sits in between: bandwidth-dominated, so the A100’s HBM2e is decisive — Llama 70B AWQ on an A100 sustains ~52 t/s decode versus the 4090’s 22-24 t/s.

80GB vs 24GB — capacity advantage

Throughput across nine workloads

The 4090 wins decisively when FP8 is on the path. The A100 wins on bandwidth-dominated AWQ at large models, BF16 workloads, and anything needing 80GB.

Per-token economics

For workloads the 4090 can run, it is dramatically cheaper per token. For Llama 70B and beyond, the A100’s larger VRAM and bandwidth shift the picture.

Per-workload winner table

vLLM serving examples

# RTX 4090 — Llama 3 8B FP8, native, 32-way batch
docker run --rm --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --quantization fp8 --kv-cache-dtype fp8_e4m3 \
  --max-model-len 16384 --max-num-seqs 32 \
  --gpu-memory-utilization 0.92

# A100 — same model, AWQ INT4 path (FP8 emulates to FP16, slower)
docker run --rm --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
  --quantization awq_marlin \
  --max-model-len 32768 --max-num-seqs 64 \
  --gpu-memory-utilization 0.92

# A100 — Llama 70B AWQ at 32k context, the model 4090 cannot serve well
docker run --rm --gpus all -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4 \
  --quantization awq_marlin \
  --max-model-len 32768 --max-num-seqs 16 \
  --gpu-memory-utilization 0.92

Production gotchas

• FP8 silently falls back on A100. vLLM accepts --quantization fp8 on A100 but runs FP16 underneath at half the throughput. Use AWQ Marlin for 8-bit equivalent perf.
• A100 SXM is HGX-only. The PCIe variant exists at 1.94 TB/s vs SXM’s 2.04 TB/s. Both are HGX-class deployments.
• A100 cooling is a real cost. 400W per card x 4-8 in an HGX node needs serious airflow.
• NVLink topology matters. 4-way and 8-way NVLink configurations give different all-reduce performance. Plan workload to match.
• MIG partitioning is per-boot. Repartitioning requires GPU reset.
• A100 used market is active. Ex-cloud A100 SXM modules appear regularly at 40-60% of new price; verify warranty status.
• 4090 is not warranted in datacentre use. Strictly speaking, NVIDIA does not warrant 4090 for server deployment; the A100 (and 6000 Pro) is the supported choice.

Verdict

• Pick the RTX 4090 24GB if your model fits in 24GB at FP8 or AWQ; you serve fewer than ~50 concurrent users; you want the lowest £/token; or you need on-prem UK hosting.
• Pick the A100 80GB if you need 70B AWQ at high concurrency, Mixtral 8x22B, multi-tenant MIG isolation, or are doing BF16 training/fine-tuning at scale.
• Pick neither if you need native FP8 at 80GB capacity — go to H100 80GB instead. The H100 closes the gap.

For a 200-MAU SaaS, the 4090 is the right answer. For a research lab fine-tuning Llama 70B BF16 on a 4-card NVLink node, the A100 is the established workhorse.

See also: vs H100 80GB, vs MI300X 192GB, vs RTX 6000 Pro 96GB, vs RTX 3090 24GB, FP8 tensor cores on Ada, RTX 4090 spec breakdown, 2026 tier positioning.