DPO Training on a Dedicated GPU Server

DPO (Direct Preference Optimisation) aligns a language model to preferred outputs using pairs of (chosen, rejected) responses. No reward model, no RL loop. On our dedicated GPU hosting it is the practical alignment step after SFT for teams without a full RLHF pipeline.

Contents

• Dataset format
• Training config
• Memory considerations
• Practical tips

Dataset

DPO expects a dataset with three columns: prompt, chosen, rejected. Typically ~5,000-50,000 pairs. Quality matters more than quantity – AI-generated preference pairs often work well, validated by a small human sample.

{"prompt":"...", "chosen":"better response", "rejected":"worse response"}

Config

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

ds = load_dataset("json", data_files="pairs.jsonl", split="train")
tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype="bfloat16",
    device_map="cuda",
)

trainer = DPOTrainer(
    model=model,
    args=DPOConfig(
        output_dir="./dpo-out",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        learning_rate=5e-6,
        beta=0.1,
        num_train_epochs=1,
        bf16=True,
    ),
    train_dataset=ds,
    tokenizer=tok,
    peft_config=LoraConfig(r=16, lora_alpha=32),
)
trainer.train()

beta=0.1 is the KL constraint weight. Lower beta allows bigger policy shift; higher keeps closer to the reference model.

Memory

DPO holds two models in memory (policy and reference). With LoRA you effectively share the base – only the LoRA adapter is trainable so peak memory stays reasonable. Without LoRA, budget roughly 2x the memory of a standard SFT on the same base model.

Tips

• Use a low learning rate (1e-6 to 5e-6). DPO is sensitive.
• One epoch is usually enough. More epochs frequently degrade.
• Monitor reward margin on a held-out set; stop when it plateaus.
• SFT first, then DPO. DPO on a base model without SFT rarely works well.

See ORPO vs DPO for a single-stage alternative.