DPO — Direct Preference Optimization

The big picture

Chapter 5's PPO works, but it is heavy: 4 models in memory (policy, reference, reward model, value head), complex RL infrastructure, and notorious instability. DPO asks a disarming question: can we skip the RL and learn directly from preferences? The key insight — the optimal policy under the RLHF objective (reward maximization + KL penalty) has a closed-form solution, so a supervised loss can implicitly optimize the same objective.

Stop 1 — The derivation: how RL disappears

Four steps. (1) Start from the RLHF objective: max_π E[r(x,y)] − β D_KL[π ‖ π_ref]. (2) Its optimal solution is known in closed form: π*(y|x) = (1/Z(x)) π_ref(y|x) exp(r(x,y)/β), with Z(x) the partition function. (3) Rearrange to express the reward in terms of the policy: r(x,y) = β log(π*(y|x)/π_ref(y|x)) + β log Z(x). (4) Substitute into the Bradley-Terry preference model P(y_w ≻ y_l) = σ(r(y_w) − r(y_l)) — and Z(x) cancels, leaving a loss over preference pairs alone:

L_DPO(θ) = −E_(x,yw,yl) log σ( β log [π_θ(y_w|x)/π_ref(y_w|x)] − β log [π_θ(y_l|x)/π_ref(y_l|x)] ) (6.1)

Stop 2 — Gradient analysis and the worked example

Differentiating eq. 6.1 decomposes the gradient into a direction and a weight:

∇_θL = −β · σ(−r̂_w + r̂_l) · [ ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ] (6.2)

The bracket raises the chosen response's probability and lowers the rejected one's. The weight σ(−r̂_w + r̂_l) is largest when the model currently prefers the wrong answer — DPO automatically focuses learning on the confusing pairs.

Stop 3 — Full token-level mechanics

The key quantity is the log-probability of an entire response — computed as a sum, not an average, over response tokens:

log π_θ(y|x) = Σ_t=1^T log π_θ(y_t | x, y_<t) (6.3)

One training example flows through a 7-step forward pass: (1) concatenate [x; y_w] and [x; y_l], pad within the batch; (2) run both through the policy, collecting logits at every response position; (3) take log-softmax of the actual token at each position; (4) sum over response tokens → logp_chosen, logp_rejected (6.7–6.8); (5) subtract the reference log-probs (pre-computed or a second forward pass) → ratio_w, ratio_l (6.9–6.10); (6) loss = −log σ(β·(ratio_w − ratio_l)); (7) backprop through steps 5→2.

Label masking decides what gets gradient: prompt tokens — no (context only, their logits don't enter the sum); padding — no (masked out); all chosen and rejected response tokens — yes, pushed up and down respectively. And the sum-not-average choice has a subtle consequence: longer sequences naturally score lower (more non-positive terms), so raw-sum DPO implicitly penalizes verbosity — length normalization (used by SimPO, stop 7) fixes the bias but can hurt instruction-following quality; standard unnormalized DPO is more common in production.

def dpo_loss(model, ref_model, batch, beta=0.1):
    logps_c = get_sequence_logprob(model, batch["chosen"])
    logps_r = get_sequence_logprob(model, batch["rejected"])
    with torch.no_grad():
        ref_c = get_sequence_logprob(ref_model, batch["chosen"])
        ref_r = get_sequence_logprob(ref_model, batch["rejected"])
    rewards_c = beta * (logps_c - ref_c)
    rewards_r = beta * (logps_r - ref_r)
    return -F.logsigmoid(rewards_c - rewards_r).mean()

Stop 4 — When DPO fails, and choosing β

Five failure modes, all consequences of skipping the RL loop: distribution shift (preference data from an old model — the loss optimizes irrelevant examples), no exploration (can't discover behaviors absent from the dataset — stuck in a local optimum), reference collapse (too strong a reference: the policy can't move; too weak: no regularization), data quality (PPO averages over many samples; DPO memorizes individual pairs, so noisy labels poison training), and pair diversity (pairs must cover the full quality spectrum — pairs differing in approach, not just quality, teach richer distinctions).

The β selection guide:

Stop 5 — Batch size and scaling

DPO's pairwise loss changes the memory math. The empirical target for the global batch is:

B_global ∈ [32, 128] (6.12)  ·   B_global = B_micro × N_GPUs × K_accum (6.13)  ·   T_sequences = 2 × B_micro (6.14)

Below 32: severe gradient noise in the implicit reward — the policy oscillates between alignment goals (helpfulness vs safety). Above 128: diminishing returns and communication overhead. The pairing multiplier is the gotcha: each instance carries chosen and rejected, so a micro-batch of B_micro pairs is really 2×B_micro sequences — and with two model copies resident (policy + reference), models >7B on 80GB GPUs at 4–8K context are pinned to B_micro ∈ [1, 2]. The book's profiles for B_global = 64: single GPU — B_micro 2, K_accum 32 (slow, sequential); 8-GPU node — B_micro 2, K_accum 4 (high parallel throughput).

Stop 6 — TRL implementation

Minimal working DPO with HuggingFace TRL — dataset format {"prompt", "chosen", "rejected"}, config values verbatim from the book:

dpo_config = DPOConfig(
    beta=0.1,                          # KL regularization strength
    learning_rate=5e-7,                # very low LR for stability
    loss_type="sigmoid",               # standard DPO loss
    max_length=2048,
    max_prompt_length=1024,            # truncation for prompts
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,     # effective batch = 16
    num_train_epochs=1,                # DPO overfits fast — 1 epoch!
    warmup_ratio=0.1,
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,                  # with LoRA, ref = base model (no copy!)
    args=dpo_config,
    peft_config=lora_config,         # r=64, alpha=16, dropout=0.05, all projections
    ...
)

Stop 7 — The variant zoo

Eight extensions, each fixing one specific weakness of vanilla DPO. The map first, details in the deep-dives:

Stop 8 — When to use which

Chapter test