Chapter 4 · Book pages 139–141

RL Foundations for Language Models

Main read ≈ 10 minutes · dotted words have hover/tap definitions · ▸ boxes go deeper · finish with the test (100% to complete)

The big picture

Supervised fine-tuning teaches a model to imitate demonstrations — but imitation has a ceiling: the model can never exceed the quality of its training data. Reinforcement learning breaks that barrier. By generating novel text, receiving reward feedback, and updating toward higher-reward behaviors, an RL-trained model can discover strategies no human demonstrator ever wrote — outputs that are more helpful, more accurate, and better aligned with human preferences.

This is the mechanism behind every frontier model: GPT-4, Claude, Llama-3, and DeepSeek-R1 all apply RL after SFT as the critical step that turns a capable but unsteered model into an aligned assistant. This short bridge chapter wires chapter 3's RL machinery (MDP, policy gradients, actor-critic) to language models — everything in Part II (chapters 5–11) builds on the six stops ahead.

Stop 1 — Two paradigms for RL in LLMs

Paradigm 1: alignment via human preferences. The original motivation for RL on LLMs was alignment — making models helpful, harmless, and honest. RLHF (Reinforcement Learning from Human Feedback) trains a reward model from pairwise human judgments ("which response is better?"), then optimizes the policy to maximize that learned reward. DPO simplifies this by eliminating the reward model entirely, converting preferences directly into a supervised loss.

Paradigm 2: capability enhancement via verifiable rewards. More recently RL has been used to teach new capabilities — reasoning, mathematics, code generation. Here the reward comes from verifiable outcomes: did the model produce the correct answer? Did the code pass all tests? DeepSeek-R1 demonstrated that GRPO with rule-based rewards (format correctness + answer accuracy) can train sophisticated chain-of-thought reasoning without any human preference data. This paradigm — RLVR, RL from Verifiable Rewards — is now the dominant approach for building reasoning models and agentic systems.

Interactive · Paradigm comparator

Pick a dimension and see how RLHF/DPO and RLVR answer it side by side.

Same machinery, different judges: RLHF's judge is a learned reward model of human taste; RLVR's judge is a rule that checks reality (correctness, passing tests). Alignment polishes behavior; RLVR grows capability.

Go deeper: the book's framing of both paradigms

Both RLHF and DPO produce aligned assistants that follow instructions and respect safety constraints; DPO trades flexibility for simplicity by skipping RL entirely (chapter 6 details the conversion). RLVR's rule-based rewards need no preference data at all — which is why it scales to domains like math and code where correctness is cheap to check but human taste is not the point. Book §4.1, p. 139; §4.4, p. 141.

Stop 2 — The shared foundation

Despite their different goals, both paradigms run on the same four pieces of machinery:

A policy π_θ (the LLM itself) that generates text autoregressively · a reward signal r(x, y) (learned from preferences, or computed from verification) · a KL constraint against a reference policy to prevent degenerate solutions · policy gradient optimization (PPO or GRPO) to update the model toward higher reward.

Notice the division of labor: the paradigms differ only in where the reward signal comes from. Everything else — the LLM-as-policy, the KL anchor, the PPO/GRPO update — is chapter 3's actor-critic machinery wearing a language-model costume. The chapters of Part II each zoom into one component.

Go deeper: why a KL constraint at all?

Without an anchor, a policy chasing a learned reward can drift into degenerate solutions — text that games the reward model while no longer resembling language. The KL constraint keeps the trained policy close to the reference (the SFT model), buying safety at the cost of reduced exploration — a trade-off stop 5 revisits. Book §4.1, p. 139; §4.3, p. 140–141.

Stop 3 — Text generation as an MDP

The key insight that makes RL applicable to language models is recasting autoregressive generation as a Markov Decision Process. Think of the LLM as an agent writing a response one token at a time: it looks at everything written so far (the state), chooses the next word (the action), and the page grows by one token (the transition). When the response is complete, a judge scores it (the reward).

The mapping, formally: state s_t = (x, y₁, …, y_t−1), the prompt plus all tokens so far · action a_t = the next token from a vocabulary of 32K–128K options · transition: deterministic — just append the token, no environment stochasticity · reward: typically only at the end of generation (sparse) — the reward model score for RLHF, answer correctness for RLVR · policy π_θ(a_t|s_t): the LLM's next-token distribution — exactly what the softmax output already computes · discount γ = 1.0, because episodes are finite (one response), so no discounting is needed.

Generation as an MDP: the LLM already is the policy — its softmax output defines πθ(aₜ|sₜ) at every state (book p. 140).

Watch one tiny canned generation step through this loop:

Interactive · Generation as an MDP, token by token

Step a toy response through state → action → deterministic append, with reward arriving only at the end. (Illustrative example, not from the book.)

The mapping is powerful because the LLM already is a policy — no separate policy network needs building. RL training simply adjusts the weights θ so the model assigns higher probability to token sequences that earn higher reward.

Go deeper: reading each MDP element in LLM terms

Compare with chapter 3's generic 5-tuple: S = all token sequences (infinite in practice), A = the vocabulary, P = append (deterministic — the one trivial part of language RL), R = a judge's score delivered once per episode, γ = 1.0. This is why stop 5 can say LLM RL is model-free almost by accident: the dynamics never needed modeling in the first place. Book §4.2, p. 140.

Stop 4 — The RLHF pipeline

The classic RLHF pipeline has four stages: (1) SFT — train a base model on high-quality demonstrations to produce a policy π_SFT that can follow instructions; (2) reward model training — collect human preference comparisons (y_w ≻ y_l for the same prompt) and train a reward model R_φ(x, y) using the Bradley-Terry objective; (3) RL optimization — optimize the policy via PPO or GRPO against the reward model's signal, subject to a KL constraint against π_SFT; (4) evaluation and iteration — evaluate the aligned model, collect new failure cases, and iterate.

For RLVR, stages 1–2 are replaced — the SFT model trains on reasoning traces, and a verifier (e.g. checking mathematical correctness) replaces the reward model — while stage 3 stays exactly the same. Explore both versions:

Interactive · RLHF pipeline explorer

Click through the four stages, then flip the RLVR toggle to see stages 1–2 swapped out.

Stage 3 is the stable core: whatever the reward source, PPO or GRPO optimization against a KL anchor does the actual learning. That is why chapters 5 (PPO) and 7 (GRPO) carry the most weight in this part — and why stage 2 splits into chapter 6 (DPO, which deletes the stage) and chapter 9 (reward modeling, which perfects it).

Go deeper: where each pipeline stage gets its own chapter

Stage 1 → chapter 10 (SFT best practices: packing, templates, data mixing — and how SFT quality determines the RL ceiling). Stage 2 → chapter 9 (Bradley-Terry models, process vs outcome rewards, rule-based rewards for RLVR). Stage 3 → chapters 5 and 7 (PPO, GRPO). The iteration loop of stage 4 is where evaluation plugs in (site cross-link, not from this chapter: evaluation gets its own treatment in chapter 14). Book §4.3–4.4, p. 140–141.

Stop 5 — How LLM RL differs from classical RL

Five differences separate this setting from the game-playing RL of chapter 3:

Deterministic transitions — the next state is just concatenation, no stochastic environment · sparse reward — feedback typically once at the end (outcome reward) or at key steps (process reward) · massive action space — 32K–128K tokens at every step, with exploration implicit via temperature sampling · KL anchor — the policy is constrained to stay close to the SFT policy, preventing reward hacking at the cost of reduced exploration · no value function needed — GRPO eliminates the critic network entirely, using group-relative normalization of rewards instead.

Together these explain why PPO and GRPO dominate over DQN-style approaches for LLMs — the same conclusion chapter 3 reached from the action-space side.

Go deeper: connecting the differences back to chapter 3

Deterministic transitions kill the need for a world model (model-free by default). The 32K–128K action space kills value-based methods (no tractable max over Q). The KL anchor is the practical answer to reward hacking from chapter 3, stop 10. And GRPO's group normalization is a baseline computed from group statistics — the critic's job done without a critic. Book §4.3, p. 140; ch. 7 preview p. 141.

Stop 6 — Roadmap of Part II

Seven chapters build the complete RL-for-LLMs toolkit:

Ch 5, PPO — clipped surrogate objective, GAE, the critic network, the full RLHF loop; the workhorse behind GPT-4 and Claude · Ch 6, DPO — preferences converted into a contrastive supervised loss; simpler but less flexible than online RL · Ch 7, GRPO — DeepSeek's critic-free algorithm with group-level reward normalization; the method behind DeepSeek-R1 and the dominant choice for reasoning models · Ch 8 — preference optimization variants: Online DPO, KTO, Best-of-N, method selection · Ch 9 — reward modeling: Bradley-Terry, process vs outcome rewards, rule-based rewards, multi-objective combinations · Ch 10 — SFT best practices: sequence packing, chat templates, data mixing, and how SFT quality determines the RL ceiling · Ch 11 — systems engineering: parallelism strategies, generation–training decoupling, infrastructure for hundreds of GPUs.

Read the roadmap as the pipeline made flesh: chapter 10 is stage 1, chapter 9 is stage 2, chapters 5–8 are stage 3's algorithm zoo, and chapter 11 is the hardware reality (chapter 2's economics) that makes stage 3's constant regeneration affordable. Every one of them assumes the six stops you just finished.

Go deeper: the book's own chapter summaries

PPO is described as the workhorse behind GPT-4 and Claude; DPO as bypassing RL entirely via a contrastive supervised loss; GRPO as the dominant choice for reasoning model training. Chapter 8 adds guidance on method selection across Online DPO, KTO, and Best-of-N; chapter 11 covers distributed training at scale. Book §4.4, p. 141.