RTX 4090 24GB Fine-Tuning Throughput: LoRA, QLoRA, Unsloth

Fine-tuning is one of the strongest reasons to own a RTX 4090 24GB. With 24 GB of GDDR6X you can LoRA-tune up to 8-9B densely in BF16 and QLoRA-tune as far as 70B in 4-bit on a single card. This benchmark catalogues throughput we measure on a stock UK dedicated 4090 with the modern toolchain (PEFT, bitsandbytes, Unsloth) across LoRA, QLoRA and the Unsloth-accelerated variants. Numbers are reproducible, sequence-length and batch-size variants are included, and the seven gotchas at the end cover the failure modes that bite teams during their first multi-day training run.

Contents

• Why benchmark fine-tuning at all
• Setup and methodology
• LoRA throughput
• QLoRA throughput
• Unsloth gains
• VRAM accounting and OOM avoidance
• Job runtime estimates
• Production gotchas

Why benchmark fine-tuning at all

Inference benchmarks tell you what your service will feel like; fine-tune benchmarks tell you what your iteration loop will cost. A run that takes 12 hours feeds back the same day; a run that takes 36 hours costs you a day. Multiplied across 10-30 fine-tune iterations during the lifetime of a model, the difference between an 18,000 t/s LoRA and a 32,000 t/s Unsloth-accelerated LoRA is the difference between weekly and biweekly model updates. The 4090 is the entry-level card where the full LoRA/QLoRA matrix fits cleanly; below it (5060 Ti, 4070) you start hitting OOMs at sequence 4096 and above 13B model size.

Setup and methodology

All numbers come from a single 4090 (Founders Edition, 450 W TDP, no power cap) on the standard test rig: Ryzen 9 7950X, 64 GB DDR5-5600, Samsung 990 Pro 2 TB Gen 4 NVMe, Ubuntu 24.04 LTS, NVIDIA driver 560.x, CUDA 12.6. Software stack: PyTorch 2.5, transformers 4.45, peft 0.13, bitsandbytes 0.43, Unsloth 2024.10. Sequence length 2048 unless noted. Optimiser is paged AdamW 8-bit. Mixed precision BF16. Gradient checkpointing on for QLoRA, off for LoRA where it fits without. Tokens per second is measured as forward + backward + optimiser tokens completed per wall-clock second, averaged over 200 steps after a 50-step warmup.

# Reference LoRA launch (Llama 3 8B, Unsloth)
from unsloth import FastLanguageModel
model, tok = FastLanguageModel.from_pretrained(
    "meta-llama/Llama-3.1-8B", load_in_4bit=False, dtype=torch.bfloat16,
    max_seq_length=2048)
model = FastLanguageModel.get_peft_model(
    model, r=16, target_modules=["q_proj","k_proj","v_proj","o_proj",
                                 "gate_proj","up_proj","down_proj"],
    lora_alpha=32, use_gradient_checkpointing="unsloth")

LoRA throughput

LoRA loads the base model in BF16 and adds rank-16 adapters on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj. Throughput numbers below are vanilla PEFT; Unsloth-accelerated numbers are in the dedicated table below.

Llama 3 8B at 18,000 tokens/s LoRA-tunes a 50M-token instruction dataset in roughly 47 minutes. Mistral 7B is slightly faster at 21,000 t/s due to its smaller intermediate FFN. Qwen 14B in dense BF16 LoRA is feasible only at batch 2 and barely fits; for any serious 14B work, use QLoRA below.

Effect of sequence length

Activation memory grows linearly with sequence and batch. Doubling sequence from 2048 to 4096 forces batch from 8 to 4 (constant VRAM), and throughput drops from 18,000 to 14,000 t/s for Llama 3 8B because per-token compute is identical but optimiser overhead pays a smaller fraction. Past 4096, gradient checkpointing becomes mandatory and throughput drops a further 25%.

QLoRA throughput

QLoRA loads the base model in NF4 (4-bit) via bitsandbytes and trains BF16 LoRA adapters on top. This unlocks 14-70B at the cost of ~30% throughput versus dense LoRA on the same model.

QLoRA on Llama 3 70B fits, just, with batch size 1 and sequence 2048. Effective batch size is reached via gradient accumulation; we typically run 16 accumulation steps for an effective batch of 16. Plan for ~36 hours per million tokens at this configuration without Unsloth, or about 19 hours with it. Paged AdamW is essential for 70B; the regular optimiser will OOM at the first backward.

Unsloth gains

Unsloth’s hand-tuned Triton kernels deliver a measurable speedup on the same hardware, mostly by fusing the LoRA forward pass and optimising the cross-entropy backward.

Unsloth uplift sits between 1.7x and 1.9x across the matrix, and the VRAM savings often unlock a longer sequence length or a larger batch on the same card. We use Unsloth as the default for new training jobs on the 4090.

VRAM accounting and OOM avoidance

For a rough sizing rule on a 4090: dense BF16 fits up to ~10B parameters with optimiser state for LoRA training; QLoRA NF4 stretches that to ~70B at small batch. Activation memory grows linearly with sequence times batch; gradient checkpointing roughly halves it for a 25% throughput cost.

Both configurations sit within 1 GB of the 24 GB ceiling. If you need any more headroom (longer eval batches, larger checkpoint cache), turn on gradient checkpointing or use Unsloth’s optimised path. The full recipe is documented in the LoRA fine-tune guide and QLoRA fine-tune guide.

Job runtime estimates

Rough wall-time estimates for typical instruction-tune corpora using Unsloth-accelerated paths:

Production gotchas

• Paged AdamW is mandatory above 32B. Without it, the optimiser allocates contiguous state and OOMs at the first backward. Set optim="paged_adamw_8bit" in transformers TrainingArguments.
• Gradient accumulation does not help VRAM, only effective batch. If you OOM at batch 8, you will OOM at batch 1 with 8 accumulation steps too. Reduce sequence length or enable checkpointing instead.
• Bitsandbytes NF4 silently changes tokeniser dtype. Some versions cast embedding to FP32 internally; check VRAM after first forward, not at model load.
• Save checkpoints to NVMe, not network storage. A 70B checkpoint is 140 GB; saving to slow storage stalls the training step for minutes.
• Watch power on long runs. A 24-hour LoRA at 430 W draws 75 GBP at UK industrial rates. Cap with nvidia-smi -pl 380 to drop power 12% for 4% throughput cost; see the power draw post.
• Don’t co-host inference and training on one card. Latency variance is unacceptable for either. If you must, use CUDA_VISIBLE_DEVICES isolation across two cards.
• Validate the adapter loads. A botched save can leave you with adapter weights but no base reference; always test load-and-generate before declaring a run complete.

Verdict

The 4090 is the right card for serious single-GPU fine-tuning across the 7B-70B range. With Unsloth you get 32k t/s on Llama 3 8B LoRA, 16k t/s on Qwen 14B QLoRA, and 3.3k t/s on Llama 70B QLoRA, all on a card you can rent for a few hundred GBP per month on UK dedicated hosting. For 70B+ work or long-context (8k+) you will eventually want a 5090 or 6000 Pro for the larger VRAM; see the 4090 vs 5090 decision piece. For lighter iteration the fine-tuning use case overview frames the choice.

See also: LoRA fine-tune guide, QLoRA fine-tune guide, fine-tuning use case, Qwen 14B on 4090, Qwen 32B on 4090, 70B INT4 deployment, tokens per watt, spec breakdown.