GRPO — Group Relative Policy Optimization

The big picture

Chapter 5's PPO needs a critic — a value network predicting expected reward — and for language that critic has three problems: memory (the value head shares the policy backbone — 140GB for 70B — and doubles memory if separate), accuracy (predicting reward for a partial sequence is extremely hard; a wrong value means a wrong advantage means a wrong gradient direction), and training (early in RL the critic's noisy predictions destabilize policy learning). GRPO's insight: don't learn V(s) at all — estimate it empirically from a group of samples.

Stop 1 — The algorithm

Four steps. (1) For each prompt x, sample G completions {y₁,…,y_G} from the current policy. (2) Score each: r_i = R(x, y_i). (3) Normalize within the group: Â_i = (r_i − µ_G)/σ_G, where µ_G and σ_G are the group's mean and standard deviation. (4) Apply a PPO-style clipped update with these advantages, plus a KL term:

Â_i = (r_i − µ_G) / σ_G,   L = E[ min(r_t(θ)Â_i, clip(r_t(θ), 1±ε)Â_i) ] − β D_KL[π_θ ‖ π_ref] (7.1)

Stop 2 — TRL implementation

Minimal GRPO with HuggingFace TRL (values verbatim from the book):

grpo_config = GRPOConfig(
    num_generations=8,               # G = group size
    temperature=1.0,                 # high temp for diversity within group
    max_completion_length=2048,
    beta=0.04,                       # KL penalty coefficient
    learning_rate=1e-6,
    per_device_train_batch_size=2,   # prompts per device (×8 gens = 16 responses)
    gradient_accumulation_steps=8,
    num_train_epochs=2,
    max_grad_norm=0.5,
    use_vllm=True,                   # critical for GRPO due to 8× generation
    vllm_gpu_memory_utilization=0.7,
)

def reward_fn(completions, prompts, **kwargs):
    """1.0 if correct, 0.0 if wrong."""
    ...

def format_reward_fn(completions, **kwargs):
    """Bonus for proper LaTeX formatting."""
    return [0.5 if "\\boxed{" in c else 0.0 for c in completions]

trainer = GRPOTrainer(model=model, args=grpo_config,
    reward_funcs=[reward_fn, format_reward_fn],  # multi-objective!
    ...)

Stop 3 — Group size: the signal-vs-compute dial

Stop 4 — Diversity: keeping the group alive

Without diversity pressure, RL-trained LLMs mode-collapse: one "template" answer per question type, entropy dropping fast, reward hacking made easier (narrow outputs are easier to exploit), and memorized reward patterns instead of reasoning. The KL penalty is the primary defense but not sufficient alone. Within-group diversity is the GRPO-specific lever: high temperature (τ = 0.8–1.0) for varied responses, large N (8–16) so both good and bad approaches appear, and DAPO's "no repeat" penalty rejecting duplicate responses inside a group. All-identical responses → zero advantage; too-random responses → noisy reward, slow learning. The sweet spot: distinct approaches that stay on-topic.

Stop 5 — The clipping and aggregation variants

The variants each fix one observed failure mode: pretraining bias diluting gradients (Dr. GRPO, stop 6), symmetric clipping limiting exploration (DAPO), wasteful large groups (2-GRPO, stop 6), reward-scale imbalance (GDPO, stop 7). First, the four that re-engineer the clip and the loss aggregation.

Stop 6 — The data and bias variants

Stop 7 — Multi-reward and ranking variants

GDPO fixes advantage collapse: with multiple rewards (correctness + format), standard GRPO normalizes the combined reward — and if one reward has much higher variance, it dominates, drowning the other to near-zero gradient. GDPO normalizes each reward independently within the group, then combines with user weights: Â⁽ⁱ⁾_n = (r⁽ⁱ⁾_n − µ_n)/(σ_n + ϵ), then Â⁽ⁱ⁾ = Σ_n w_nÂ⁽ⁱ⁾_n. TRL: multi_objective_aggregation="normalize_then_sum", reward_weights=[1.0, 0.5].

GOPO starts from an epistemics point: reward models are trained on pairwise comparisons, so only the rank order of their scores is trustworthy — a 0.6-point gap might mean genuine quality in one region and nothing in another. GOPO discards magnitudes entirely: rank the group (1 = worst, N = best) and set Â_i = f(rank(o_i)/N) with f monotonic (e.g. linear to [−1, 1]) (eq. 7.2). Invariant to any monotonic reward transformation; empirically (book's report): reward curves above GRPO throughout, better LLM-judged win-rates at most checkpoints, markedly faster convergence, and the advantage grows as the reward model gets noisier.

Stop 8 — When to use which

Chapter test