Every preference-optimization method answers the same question: given a base policy that can already produce fluent tokens, how do you push its distribution toward outputs humans (or a verifier) prefer, without shredding the capabilities the base model already has? The differences between PPO, DPO and GRPO are not cosmetic. They are different answers to where the reward signal comes from, whether you estimate a value function, and how you keep the policy tethered to a reference. Get the choice wrong and you pay in compute, instability, or a model that games your metric while getting worse at the task.
This is an engineering map, not a survey. We optimize models for a living - Beacon SI 2.5 was trained SFT+DPO, and our efficient-reasoning work leans on budget-forced reasoning rather than sprawling RL - so the framing here is about what actually breaks in production and how each objective's gradient behaves when it does. We don't rent AI. We engineer it, and that starts with understanding the loss you are descending.
The RLHF-with-PPO pipeline: reward model plus KL-constrained policy
Classic RLHF is three stages. Supervised fine-tuning gives you an initial policy. A reward model is trained on pairwise human preferences under a Bradley-Terry likelihood - the reward model scores a completion, and the loss maximizes the log-sigmoid of the reward gap between the chosen and rejected response. Then you run reinforcement learning to maximize expected reward under that model, minus a per-token KL penalty against the SFT reference.
The optimized objective is the expected reward of sampled completions minus beta times the KL divergence between the current policy and the frozen reference. PPO maximizes this with a clipped surrogate: you compute a probability ratio between the new and old policy for each token, multiply by an advantage estimate, and clip that ratio to a trust region so a single update can't move the policy too far. The advantage comes from a learned value function via GAE - which means you are training a second network the same size as the policy, purely to reduce the variance of your gradient estimate.
Why it's unstable and expensive
- Four models in memory at once: policy, reference, reward model, and critic. For large reasoning models this is the dominant cost of the whole program.
- The value function is a moving target trained on-policy from sparse, noisy returns. Early in training it is wrong, so advantages are wrong, so the policy chases noise.
- KL is a penalty, not a hard constraint. Set beta too low and the policy runs away from the reference to exploit the reward model; too high and it never learns anything.
- Reward-model error is baked in. The policy will happily find the regions of input space where the reward model is miscalibrated, because that is literally what maximizing its output rewards.
PPO's clipping bounds how far you step, not where you are stepping toward. If the reward model points the wrong way, a stable optimizer just walks you off the cliff more carefully.
DPO: the reward model was implicit all along
Direct Preference Optimization starts from an observation about the KL-constrained objective: it has a closed-form optimal policy. The reward-maximizing policy under a KL penalty is the reference policy reweighted by the exponentiated reward over the partition function. Invert that relationship and the reward is expressible as beta times the log-ratio of the optimal policy to the reference, plus a term that depends only on the prompt.
Substitute that implicit reward back into the Bradley-Terry preference likelihood and the prompt-only partition term cancels between the chosen and rejected completions. You are left with a supervised objective over preference pairs: maximize the log-sigmoid of beta times the difference of log-ratios for the winning and losing responses. No reward model, no sampling loop, no value function. It is a classification loss you can run with the same machinery as SFT.
What the DPO gradient actually does
The gradient weights each pair by how wrong the implicit reward model currently is - the sigmoid of the reward gap it assigns in the wrong direction. High weight on pairs the model gets backwards, low weight where it already agrees with the preference. It increases the log-likelihood of the chosen completion and decreases the rejected one, but crucially both are measured relative to the reference, so the beta and the reference model are doing the KL regularization implicitly.
Where DPO drifts and over-optimizes
- It is off-policy. The preference pairs are fixed data, not samples from the current policy, so DPO optimizes preferences on a distribution the policy is actively leaving. The further the policy moves, the more stale the signal.
- The objective can raise the margin by pushing the rejected log-prob down faster than it raises the chosen - probability mass leaks to unseen, out-of-distribution tokens. You can see chosen-response likelihood fall in absolute terms while the loss still improves.
- Beta is the only KL knob and it is global. There is no per-token trust region, so on long completions the reference tether is weak exactly where reasoning models generate the most tokens.
- With mediocre or off-distribution preference data, DPO over-optimizes a proxy just as readily as PPO over-optimizes a reward model - the reward model is still there, just implicit.
We used SFT+DPO for Beacon SI 2.5 precisely because the preference structure was well-defined and we could curate on-distribution pairs. DPO shines when your preferences are clean, your completions are not extremely long, and you would rather not stand up a full RL rig. It is the right tool far more often than the literature's PPO-centric framing suggests.
GRPO: throw away the value function
Group Relative Policy Optimization keeps the on-policy PPO-style surrogate but deletes the critic. Instead of a learned value baseline, it samples a group of completions for each prompt, scores them, and computes each completion's advantage as its reward standardized within the group - subtract the group mean, divide by the group standard deviation. That normalized score is the advantage used in the clipped policy-gradient update, broadcast to every token in the completion.
The insight is that the value function's only job was variance reduction via a baseline, and for a batch of completions to the same prompt the group mean is an unbiased, essentially free baseline. You lose the per-token credit assignment a critic provides, but for tasks where the reward lands at the end of a trajectory anyway, per-token credit was mostly fiction. The KL term is often kept as an explicit penalty against the reference, and clipping still bounds the step.
Why it suits verifiable reasoning and its relation to RLVR
GRPO's structure is a natural fit for RLVR - reinforcement learning from verifiable rewards. If a math answer, a unit test, or a proof checker gives you a clean binary or graded signal, you don't need a reward model at all; the verifier is the reward. Within a group of sampled solutions, some pass and some fail, and the group-relative advantage directly encodes 'be more like the ones that verified.' No reward-model miscalibration to hack, because the reward is ground truth.
- Half the models: no critic to train, no reward model if you have a verifier. Memory and wall-clock drop sharply versus PPO.
- The baseline is exact for the sampled group, so early-training advantage estimates aren't corrupted by a half-trained critic.
- It rewards relative improvement, which keeps a learning signal alive even when absolute success rates are low - useful for hard reasoning problems where most samples fail.
When the reward is a verifier, reward hacking has nowhere to hide. That is the real reason GRPO took over reasoning training - not the missing critic, the honest reward.
GRPO's own failure modes
- Group-relative standardization inflates advantages when all completions have near-identical rewards - dividing by a tiny standard deviation amplifies noise. Guard the denominator or drop degenerate groups.
- With only a verifier, the policy optimizes exactly what the verifier measures. If the test suite is weak, the model reward-hacks the tests, not the task.
- No reward model means no signal for open-ended quality (tone, helpfulness, safety). GRPO is for verifiable objectives; it does not replace preference learning for subjective ones.
- Length and format biases creep in when the verifier is lenient - models learn to pad reasoning because longer traces correlate with passing, not because the reasoning is better.
Reward hacking and the role of KL
Reward hacking is the same phenomenon across all three methods: the policy maximizes the measured proxy rather than the intended objective, because that is the only thing the gradient sees. In PPO it is the reward model's blind spots. In DPO it is the implicit reward's off-distribution extrapolation. In GRPO it is the verifier's incompleteness. KL regularization is the shared defense - it caps how far the policy can travel from a reference you trust, which bounds how deep into proxy-exploiting territory it can go before the penalty dominates.
But KL is a blunt instrument. It penalizes all divergence equally, including the divergence you want. The art is choosing a reference and a beta such that the policy has room to improve on the true objective while the proxy-exploiting regions stay expensive. A useful discipline: monitor KL against the reference as a first-class metric, and treat a sudden KL spike with a flat or rising reward as the signature of hacking, not progress.
On-policy versus off-policy, in one breath
PPO and GRPO are on-policy: they learn from completions the current policy generates, so the signal always matches the distribution being optimized, at the cost of a generation loop every step. DPO is off-policy: it learns from a fixed preference dataset, which is cheap and stable but goes stale as the policy moves away from the data's distribution. This is the deepest axis of the whole comparison. On-policy methods pay compute to keep their signal fresh; off-policy methods trade freshness for simplicity and risk optimizing a distribution they've already left.
# The three objectives, stripped to their cores.
# PPO: clipped surrogate on a learned-critic advantage, KL to reference.
ratio = pi(a|s) / pi_old(a|s)
adv = gae(rewards, value_fn(s)) # needs a critic
L_ppo = -min(ratio*adv, clip(ratio,1-e,1+e)*adv) + beta*kl(pi, ref)
# DPO: closed-form preference loss, reference is the only KL tether.
lr_w = beta*(logp(pi, y_w) - logp(ref, y_w)) # implicit reward, chosen
lr_l = beta*(logp(pi, y_l) - logp(ref, y_l)) # implicit reward, rejected
L_dpo = -logsigmoid(lr_w - lr_l) # no critic, no RM, off-policy
# GRPO: group-relative advantage, no value function.
r = [verify(y) for y in group] # verifier or reward model
adv = (r - mean(r)) / (std(r) + 1e-8) # baseline = group mean
L_grpo = -min(ratio*adv, clip(ratio,1-e,1+e)*adv) + beta*kl(pi, ref)How to choose
The decision is driven by three questions: do you have a verifier, how clean are your preferences, and how long are your completions. Answer those honestly and the method mostly picks itself.
- Verifiable task (math, code, tool-use with a checker): GRPO / RLVR. The verifier is your reward, on-policy sampling keeps the signal honest, and you skip both the critic and the reward model. This is where reasoning models get their gains.
- Subjective quality with clean, on-distribution preference pairs and moderate lengths: DPO. Cheapest stable path to a well-shaped policy - how we did Beacon SI 2.5. Curate the pairs; that is where the quality actually lives.
- Subjective quality, high stakes, and budget for infrastructure: PPO. When you need an explicit reward model to generalize beyond your pair dataset and you can afford the four-model rig and the tuning, PPO still gives the most control over the reward surface.
- Efficiency-constrained reasoning: consider budget-forced reasoning before reaching for RL at all. Constraining or capping the thinking budget can buy most of the reasoning gains at a fraction of the training cost - we lean on it precisely to avoid paying for RL we don't need.
None of these are mutually exclusive. SFT then DPO then a GRPO pass on the verifiable slice is a completely reasonable stack. The mistake is treating preference optimization as a single lever. It is three different gradients with three different failure modes, and choosing well means knowing which one your task's reward structure actually admits.