Table of Contents
The 4090's 24 GB is generous for fine-tuning. Comfortable QLoRA on 13B, plain LoRA on 7B FP16.
TL;DR
RTX 4090 fits QLoRA on 7B-13B comfortably, plain LoRA on 7B FP16. Full SFT on 7B is tight but possible with paged optimiser. Training time for typical 10K-sample SFT: ~5 hours.
Methods
| Method | 7B peak VRAM | 13B peak VRAM | Fits 24 GB? |
|---|---|---|---|
| QLoRA r=64 | 12 GB | 18 GB | Yes |
| LoRA (BF16 base) | 20 GB | OOM | 7B yes, 13B no |
| Full SFT BF16 | ~75 GB | ~140 GB | No |
| Full SFT + paged AdamW | ~22 GB | OOM | 7B tight, 13B no |
Throughput
- Mistral 7B QLoRA: ~3,800 fine-tune tok/s
- Llama 3.1 8B QLoRA: ~3,500 fine-tune tok/s
- 10K samples × 2K seq len: ~5 hours
Verdict
The 4090 is a solid fine-tuning host. For 13B QLoRA workloads it's the sweet spot. For larger models, step up to 5090 or 6000 Pro.
Bottom line
QLoRA fine-tuning on a 4090 is comfortable. See QLoRA guide on 5060 Ti — same recipe applies.
