Table of Contents
Most fine-tuning starts with SFT (supervised fine-tuning) and stops there. The preference-based methods (DPO, ORPO) can push quality further when you have preference data.
TL;DR
SFT for general adaptation. DPO when you have pairwise preferences (preferred vs rejected). ORPO when you want SFT + preference in a single training pass. Hardware-wise, all three fit similar VRAM budgets via QLoRA.
The three methods
- SFT (Supervised Fine-Tuning): train on (input, target) pairs. Standard, well-understood.
- DPO (Direct Preference Optimization): train on (input, preferred, rejected) triples. Aligns model with human preference without RL.
- ORPO (Odds Ratio Preference Optimization): combines SFT + preference in one training run. Newer, cheaper to run.
Comparison
| Aspect | SFT | DPO | ORPO |
|---|---|---|---|
| Data needed | Demonstrations | Preference pairs | Both (mixed) |
| Training stages | 1 | 2 (SFT then DPO) | 1 |
| Compute cost | Baseline | ~2× (two passes) | ~1.2× SFT |
| Quality on alignment | Good | Strong | Strong |
| Maturity | Highest | High | Medium |
| VRAM (7B QLoRA) | ~12 GB | ~14 GB (ref model) | ~13 GB |
Hardware needs
All three fit a single RTX 5090 32 GB for 7B-class models via QLoRA. DPO needs slightly more because of the reference model. ORPO is the cheapest preference-based method.
Verdict
- Have demonstrations only: SFT
- Have preference pairs: DPO (after SFT) or ORPO (instead of SFT)
- Want simplest preference-based pipeline: ORPO
Bottom line
Most teams should start with SFT. Move to DPO or ORPO once you have preference data. See fine-tuning pipeline guide.
