Chapter 8 · Book pages 180–187

Preference Optimization Variants

Main read ≈ 15 minutes · dotted words have hover/tap definitions · ▸ boxes go deeper · finish with the test (100% to complete)

The big picture

Chapter 6's DPO is simple and cheap — and it has specific, nameable weaknesses. This chapter is the repair catalog: each variant fixes exactly one limitation of standard offline DPO. Distribution shift → Online DPO. The paired-data requirement → KTO. Overfitting to noisy labels → IPO. The reference model's memory cost → ORPO. Training complexity itself → Best-of-N.

This is the chapter where Part II's triangle (PPO's machinery, DPO's simplicity, GRPO's verifiable rewards) becomes a map. The final stop's table 8.1 and decision tree are the payoff: eight methods, one choice per constraint set. When someone asks "should we just use PPO?", this chapter is the answer.

Stop 1 — Online DPO: fresh pairs, same loss

Standard DPO's primary limitation: the preference data came from a different model — an older checkpoint, even another model family. As training progresses the policy generates text that looks nothing like the training pairs, and the loss optimizes an irrelevant distribution. Online DPO's fix: generate fresh preference pairs from the current policy at every step, judge them with a reward model, apply the DPO loss. Five steps: (1) generate K responses per prompt from the current π_θ; (2) score all with the reward model r_φ; (3) highest-scoring = chosen, lowest = rejected; (4) DPO loss on these fresh pairs; (5) repeat — new generation every step.

Best of both worlds: from DPO — simple supervised loss, no value function, no GAE, stable optimization; from PPO — on-policy data, self-improvement beyond the dataset, no distribution shift. The key difference from GRPO: Online DPO uses the pair-based DPO loss instead of per-sample advantages. The trade-off: it needs a reward model (DPO doesn't) but no value head (PPO does) — middle-ground complexity.

	Data	Models	Loss	Best for
Offline DPO	Static pairs	2 (policy + reference)	DPO	Quick alignment, limited compute
Online DPO	Fresh from πθ	3 (+ reward model)	DPO	When DPO plateaus, need exploration
PPO	Fresh from πθ	4 (+ reward model + value head)	PPO clip	Max quality, complex reasoning

Go deeper: the TRL config

OnlineDPOConfig: learning_rate=5e-7, beta=0.1 (same meaning as standard DPO), num_generations=4 (K responses per prompt), per_device_train_batch_size=4, gradient_accumulation_steps=4, max_new_tokens=512, temperature=0.7, num_train_epochs=1. The reward model in the book's listing is RLHFlow/ArmoRM-Llama3-8B-v0.1 loaded as a sequence classifier; the trainer takes reward_model directly alongside the policy. Book §8.1.3, p. 180–181.

Stop 2 — KTO: preferences without pairs

DPO requires matched pairs — same prompt, one good and one bad response. Most real feedback is unpaired: a thumbs up/down on an individual response, nothing to compare it against. Prospect theory (from behavioral economics) supplies the fix: humans feel losses more strongly than gains, so a thumbs-down should produce a stronger gradient than a thumbs-up.

L_KTO = E_{y_w}[λ_w(1 − v(x, y_w))] + E_{y_l}[λ_l · v(x, y_l)], v(x,y) = σ(β log [π_θ(y|x)/π_ref(y|x)] − z_ref) (8.1)

with z_ref the expected KL divergence — a running baseline. Desirable responses get utility with diminishing returns (already likely? don't push harder); undesirable ones get the loss-aversion weight — defaults λ_w = λ_l = 1.0, set λ_l > λ_w to strengthen it.

Each example is independent — data format is just {"prompt", "completion", "label": bool} instead of DPO's chosen/rejected pairs. Usable sources: production thumbs up/down, forum upvotes/downvotes, binarized star ratings (4–5 good, 1–2 bad) — any per-response quality signal. Choose KTO when you have binary feedback at scale, when one class dominates (90% good / 10% bad — KTO handles imbalance better), or for rapid iteration with noisy labels (more robust than DPO to noise). TRL: KTOConfig(beta=0.1, desirable_weight=1.0, undesirable_weight=1.0, learning_rate=5e-7).

Go deeper: reading eq. 8.1

For a desirable example the loss is λ_w(1 − v): it vanishes as v → 1, so already-likely good responses stop attracting gradient — the diminishing-returns curve. For an undesirable one the loss is λ_l·v: it vanishes as v → 0. z_ref (a running KL baseline) centers the sigmoid so "desirable" means better than the reference-adjusted average, not better than zero. The reference model is still present (or None with the LoRA trick from chapter 6). Book §8.2.1–8.2.4, p. 181–182.

Stop 3 — IPO: a finite target instead of an infinite margin

DPO has a degenerate solution: it can reach zero loss by driving the margin between chosen and rejected to infinity — in practice, pushing chosen probability to 1 and rejected to 0, memorizing the training data. IPO swaps the saturating log-sigmoid for a squared loss with a specific target:

L_IPO = E[ ( log [π_θ(y_w|x)/π_ref(y_w|x)] − log [π_θ(y_l|x)/π_ref(y_l|x)] − 1/(2β) )² ] (8.2)

DPO: σ(margin) → 1 optimally, margin → ∞, no natural stopping point. IPO: margin → 1/(2β) optimally — the squared loss penalizes both too-small and too-large margins. Result: bounded influence for mislabeled pairs (robust to noisy labels) and better generalization, because it doesn't memorize. (Note the echo of chapter 6: DPO's margin "stabilizes around 1/(2β)" in practice; IPO makes that same quantity the explicit target.)

Interactive · Margin-target visualizer

One margin axis, two losses: DPO's keeps falling as the margin grows; IPO's bottoms out at 1/(2β). Slide β.

TRL: IPO is just a DPO loss variant — DPOConfig(loss_type="ipo", beta=0.1), everything else unchanged. Choose IPO with noisy preference data (crowdsourced, AI-judged with errors), when you observe DPO overfitting (train loss → 0 while eval degrades), when you want more conservative robust alignment, or when you need multiple epochs (DPO degrades after epoch 1; IPO is more stable).

Go deeper: why a target margin regularizes

The log-sigmoid's gradient never quite reaches zero — there is always more margin to chase, and chasing it means memorizing pairs. The squared loss turns the margin into a regression target: overshooting 1/(2β) hurts as much as undershooting, so a mislabeled pair (whose "true" margin is huge and wrong) exerts bounded influence instead of an endless pull. Book §8.3.1–8.3.4, p. 183.

Stop 4 — ORPO: no reference model at all

Every method so far needs a reference model — a separate copy (double memory) or the LoRA trick. ORPO eliminates it entirely by fusing SFT and preference alignment into one loss:

The SFT term trains the model to generate the chosen response (standard language modeling); the odds-ratio term pushes chosen over rejected. Why is no reference needed? The SFT loss already anchors the model to reasonable text — it serves the KL-to-reference role. One model, one forward pass, one loss: 50% less memory. TRL: ORPOConfig(beta=0.1) — here beta is λ, the odds-ratio weight — no ref_model, DPO's data format.

Choose ORPO when memory-constrained (saves 70–140GB for 70B), when starting from a base model (it does SFT simultaneously), or when you want the simplest possible pipeline with good preference data from the start. Limitations: less studied than DPO/PPO (fewer proven recipes at 70B+ scale); the SFT component needs high-quality chosen responses, not just relative preference; and the two loss components can conflict, making debugging harder. See also SimPO (chapter 6, §6.9.8) — the other reference-free method, built on length-normalized log-probability.

Go deeper: why odds, not probabilities

The odds P/(1−P) stretch differences near the extremes: a response moving from 0.9 to 0.95 probability changes odds from 9 to 19 — the same relative jump as 0.5 → 0.67. The log-odds-ratio between chosen and rejected is thus a sensitive contrast signal that needs no reference baseline; the SFT term supplies the anchor the reference used to provide. Book §8.4.1–8.4.4, p. 184–185.

Stop 5 — Best-of-N: alignment without training

Sometimes the simplest approach wins: generate N candidates, score them all, return the best — y* = arg max_{y_i} r_φ(x, y_i) (8.4). Typically N = 4–64. At inference, no weights change: with N = 64, win-rate improves 10–20% over greedy — sometimes matching or exceeding PPO. As a training method it becomes RFT (rejection sampling fine-tuning): generate many, select the best, SFT on them, repeat — how many production models are trained: simpler than PPO, almost as effective, completely stable. The theoretical connection: Best-of-N implements an implicit KL-constrained policy, π_BoN(y|x) ∝ π_θ(y|x)^1−1/N · r(x,y)^1/N.

Interactive · Best-of-N explorer

Slide N along the book's scaling table — win-rate gain against cost multiplier — and watch a live argmax over canned candidates. (Candidate scores illustrative.)

The baseline rule: always compare your RL method against Best-of-N at the same compute budget. If PPO with 64 GPU-hours doesn't beat Best-of-N with 64 GPU-hours of generation, your PPO has a bug.

Go deeper: the scaling table and the RFT loop in code

N = 1: baseline, 1× cost · N = 4: +5–8% win-rate, 4× (minimum useful, good cost/quality ratio) · N = 16: +10–15%, 16× (strong, often matches PPO quality) · N = 64: +15–20%, 64× (diminishing returns start) · N = 256: +18–22%, 256× (critical applications only). The book's inference helper: best_of_n(prompt, n=16, temperature=0.8) returning the argmax-scored candidate; the RFT loop generates 16 candidates at temp 0.9, applies a quality-gate threshold, then SFTs (SFTConfig(learning_rate=2e-5, num_train_epochs=2)) and repeats. Book §8.5.2–8.5.4, p. 185–186.

Stop 6 — Choosing a method: table 8.1 and the decision tree

Method	Models	Data	Compute	Stability	Best for
PPO	4	Online (gen)	Very high	Low	Max quality, complex reasoning
GRPO	2 (no critic)	Online (gen)	High	Medium	Math/code (verifiable rewards)
DPO	2	Offline pairs	Low	High	Style/safety, limited compute
Online DPO	3	Online (gen)	Medium	Medium-High	DPO without distribution shift
KTO	2	Unpaired binary	Low	High	Production feedback, thumbs up/down
IPO	2	Offline pairs	Low	Very high	Noisy labels, anti-overfitting
ORPO	1	Offline pairs	Very low	High	Memory-limited, SFT+align combined
Best-of-N	1+RM	Online (gen)	Medium	Perfect	Strong baseline, data generation

The book's fig 8.1 idea — quality vs compute; methods above the SFT ceiling beat supervised fine-tuning alone. Positions are illustrative and model-dependent (the book's own caveat), not measurements (p. 187).

The book's decision tree: (1) verifiable rewards (math/code)? → GRPO · (2) need max quality on complex tasks? → PPO · (3) paired preferences? → DPO (or IPO if noisy) · (4) only unpaired binary feedback? → KTO · (5) memory-limited, starting from a base model? → ORPO · (6) DPO plateauing, want on-policy? → Online DPO · (7) need a strong baseline quickly? → Best-of-N / RFT.

Interactive · Decision-tree walkthrough

Answer the tree's questions about your constraints; it walks you to the recommended method with its table-8.1 row.

The whole part in one look: every row of table 8.1 is a chapter you have read. PPO's "Low" stability is chapter 5's step-size problem; GRPO's "2 (no critic)" is chapter 7's deletion; DPO's "Offline pairs" is chapter 6's deal with the devil; Best-of-N's "Perfect" stability is what you get when you never update weights at all. Method selection is constraint matching, not fashion.