Table of Contents
Fine-tuning is rarely a one-shot job. Production fine-tuning is a pipeline — data prep, training, eval, decision, deploy. Self-hosting it on dedicated hardware is straightforward.
TL;DR
For production fine-tuning: data prep with HF datasets, training with TRL + PEFT, eval with lm-evaluation-harness, deploy via vLLM with merged weights or multi-LoRA. Single 5090 handles 7B–14B QLoRA; 6000 Pro for 32B+.
Pipeline flow
- Data prep: clean, deduplicate, format as ChatML or Alpaca. ~10K samples typical.
- Training: QLoRA r=64 on a 7B base model. Overnight on a 5060 Ti.
- Eval: standard benchmarks + custom 200-prompt set scored by LLM judge.
- Decision: deploy if >3% improvement on custom set, no >1% regression on standards.
- Deploy: merge LoRA back to base, push to vLLM. Or serve via vLLM's multi-LoRA path.
Tooling
- TRL: SFTTrainer, DPOTrainer for the training step
- PEFT: LoraConfig, prepare_model_for_kbit_training
- bitsandbytes: 4-bit quant for QLoRA
- WandB: experiment tracking
- lm-evaluation-harness: standard eval
- vLLM: deploy
Hardware tier
| Model size | Method | Hardware | Wall time |
|---|---|---|---|
| Phi-3 Mini | QLoRA | RTX 5060 Ti | ~3 hours |
| Mistral 7B / Llama 3.1 8B | QLoRA | RTX 5060 Ti / 5090 | ~6 hours |
| Qwen 2.5 14B | QLoRA | RTX 5090 | ~10 hours |
| Llama 3.3 70B | QLoRA | RTX 6000 Pro | ~36 hours |
| Llama 3.3 70B | Full SFT | Multi-GPU cluster | ~5 days |
Verdict
Self-hosted fine-tuning is mature. The pipeline is well-trodden. Hardware is straightforward. Most teams hesitate because of dataset prep — that’s the actually hard part.
Bottom line
Most fine-tuning failures are dataset failures, not training failures. See QLoRA guide for the training-step detail.
