In short: RLHF and DPO both teach a model human preferences. RLHF needs three stages and a separately trained reward model, optimised through a reinforcement learning loop. DPO folds that reward into one supervised loss — cheaper, far more stable. ORPO and KTO go further: a single training stage, and feedback data you can afford to collect.
The history of preference alignment reads as a list of deletions: first the separately trained judge, then the frozen reference copy, then the assumption that feedback arrives in tidy A/B pairs. Each deletion trades a slice of generalisation for a large gain in accessibility. Which method fits you depends less on which paper is newest than on what feedback data already sits in your database.
Why did teaching a model human preferences ever need three separate stages?
Picture a student who writes fluently but has no idea which of two essays a reader would rather receive. Supervised fine-tuning fixes the fluency and the format. It cannot fix taste, because taste lives in comparisons — this answer over that one — and a next-token objective has no way to represent a comparison.
So the original recipe hired a grader. Humans rank some tens of thousands of answer pairs, a second neural network learns to imitate those rankings, and that network then grades the model's output millions of times during training. Human labellers cannot sit inside a gradient loop; a proxy can. Three stages existed because the preference signal and the optimisation that consumed it were built as separate artifacts.
The bill arrives as memory. Several models sit on the GPU at once, and the size of that resident footprint — not vague expense — decides whether an alignment run fits the hardware you have. Compute is now the infrastructure layer under all of this, and its price sets the boundary of what a small team can attempt. Frontier labs still run the full pipeline at scale, where its generalisation across enormous preference datasets appears to justify the overhead. Almost everyone else has moved on.
What does RLHF actually do, step by step?
Reinforcement learning from human feedback is a pipeline, not an algorithm.
- Supervised fine-tuning. Train the base model on curated demonstrations so it answers in the shape you want.
- Reward modelling. Show humans pairs of candidate answers, record which one they prefer, and fit a separate network that maps any answer to a scalar score approximating human judgement.
- Reinforcement learning, usually PPO. Sample answers from the policy, score them with the reward model, and push the policy toward higher scores while a frozen reference copy holds it back from drifting into gibberish.
Count the models: a policy being trained, a reference copy enforcing the KL constraint, a reward model doing the grading, and in most PPO implementations a value head as well. All resident. All consuming memory on every step.
The optimisation is temperamental in a way supervised training is not. Small changes to the KL coefficient or the learning rate can send a run somewhere useless, and the whole procedure is known to be hyperparameter-sensitive. Worse is reward hacking: the policy learns the grader rather than the task. Ask for helpfulness as scored by a proxy and you get answers that hit whatever surface features that proxy rewards — length, hedging, a confident tone — while the underlying quality quietly slides. The grader was always an approximation, and a determined optimiser finds where approximations break.
How does DPO collapse the reward model into a single loss?
The intuition first. If you already know the exact mathematical relationship between a reward function and the policy that would be optimal under it, you do not need to build the reward function and then go searching for that policy. You can rewrite the whole problem in terms of the policy and solve it once.
That is the move in Direct Preference Optimization: Your Language Model is Secretly a Reward Model, published at NeurIPS 2023. A change of variables expresses the standard preference loss directly over the policy, which turns alignment into plain binary classification: given a chosen answer and a rejected answer, raise the relative log-probability of the chosen one. The objective is binary cross-entropy. No sampling from the model mid-training, no reward network to fit first, no RL loop to stabilise.
Now the correction, because this is the method's most common misreading. DPO does not remove the reward model. It removes the separately trained, explicit reward model — the reward function is still there, implicit in the policy's own log-probabilities. The paper's subtitle is not a flourish. Your language model is secretly a reward model; DPO stops making you train the secret one a second time.
What you get is two stages instead of three, and two models in memory instead of four: the policy and a frozen reference. The paper reports the method as stable, performant and computationally lightweight, free of heavy hyperparameter tuning, matching or exceeding RLHF on sentiment control, summarisation and single-turn dialogue. For most teams fine-tuning open weights, that is the entire argument.
What did ORPO merge, and why does that matter on a single GPU?
DPO still runs two passes and still keeps a frozen twin in memory. ORPO questions both.
ORPO: Monolithic Preference Optimization without Reference Model (EMNLP 2024) drops the reference model entirely and collapses the remaining two stages into one. Rather than aligning after supervised fine-tuning, it attaches an odds-ratio penalty to the SFT objective itself. The model is trained toward the chosen response and, in the same gradient step, penalised on the odds of producing the disfavoured style.
The second misreading to clear up: ORPO does not skip supervised fine-tuning. The paper revisits SFT and argues the opposite — that a minor penalty on the disfavoured style, applied during SFT, is sufficient for alignment. Alignment is folded into the pass you were already running instead of bolted on after it. One stage, one model resident, no frozen copy.
The results are not confined to toy scales. ORPO was validated from 125M up to 7B parameters, and Phi-2 (2.7B), Llama-2 (7B) and Mistral (7B) fine-tuned on UltraFeedback alone surpassed larger state-of-the-art instruction-tuned models on AlpacaEval 2.0, IFEval and MT-Bench. One dataset, one pass, no reward model, no reference model. When your alignment budget is a single 24GB card, that gap is the difference between running the job and not running it.
How does KTO align a model on plain thumbs-up, thumbs-down data?
Every method so far demands paired comparisons: two answers to the same prompt, one marked better. That data is expensive, and your production app almost certainly does not collect it. What your app does collect, assuming you shipped the buttons, is a pile of unpaired thumbs.
KTO: Model Alignment as Prospect Theoretic Optimization — Ethayarajh, Xu, Muennighoff, Jurafsky and Kiela, from Stanford and Contextual AI — takes that pile as its input. Each output needs a single bit: desirable or undesirable. No pairing, no ranking, no second candidate to generate or buy.
The name comes from Kahneman and Tversky. Their prospect theory describes how people actually value outcomes, and its central asymmetry is loss aversion: a loss hurts more than an equivalent gain pleases. KTO builds that asymmetry into the training objective through a class of losses the authors call Human-Aware Losses, or HALOs, using a value function that weights an undesirable example more heavily than a desirable one. The model learns what to avoid faster than it learns what to imitate — which, for anyone who has read their own product's thumbs-down queue, matches how the signal behaves in practice.
KTO also tolerates badly imbalanced feedback pools, where nine positives arrive for every negative, and practitioners report that on a strong enough base model it can sometimes be applied without a preceding SFT stage. Both of those come from field reports rather than a published benchmark. Treat them as reasons to run an experiment, not as guarantees.
Which method should you actually use?
Start from the data you hold, not the paper you read most recently. These four differ far more in what they demand from your feedback pipeline than in what they promise at the top of a results table.
| Method | Training stages | Models resident in memory | Feedback data required | What it deletes from the stack |
|---|---|---|---|---|
| RLHF (PPO) | Three: SFT, reward model, RL | Four: policy, reference, reward, value head | Human A/B preference pairs, plus a labelled set to fit the reward model | Nothing — this is the baseline everything else subtracts from |
| DPO | Two: SFT, then DPO | Two: policy and frozen reference | Chosen/rejected pairs | The separately trained reward model and the RL sampling loop |
| ORPO | One: SFT carrying an odds-ratio penalty | One: the policy | Chosen/rejected pairs | The reference model and the standalone alignment stage |
| KTO | One or two: SFT sometimes optional | Two: policy and reference | Unpaired binary labels — desirable or undesirable, one per output | The requirement for paired comparisons |
Pick this if:
- RLHF — you run a frontier lab with a very large preference corpus and staff who can babysit a PPO run; at that scale the generalisation still appears to pay for the instability.
- DPO — you have a curated paired dataset and want the best-documented, lowest-risk route from it to an aligned model.
- ORPO — same paired dataset, but one GPU and one training window; folding alignment into SFT removes both the second pass and the reference copy.
- KTO — your feedback is thumbs from a live product, arriving unpaired and lopsided, and you would rather train on data you already own than commission comparisons you do not.
What the matrix cannot tell you is whether to fine-tune at all. If the gap you are closing is factual rather than stylistic, retrieval is usually the cheaper instrument, and the engineering lessons from real RAG deployments apply long before a preference dataset does. Preference tuning shapes how a model behaves. It does not install knowledge the model never had.
Alignment is also one layer rather than a shield. A preference-tuned model still invents citations, which is why teams that care about accuracy stack it with retrieval, verification and refusal behaviour — the seven-layer defence stack against hallucination treats preference tuning as a single tier of a much taller structure. The trade running through the whole progression stays consistent: each deleted component buys accessibility and costs a little generalisation. For most teams that is a good deal, because the alternative was never a better-aligned model. It was no aligned model at all.
Key takeaways
- RLHF needs three stages because the preference signal and the optimiser consuming it were separate artifacts; the cost shows up as four models resident in memory and a hyperparameter-sensitive RL loop.
- DPO removes the separately trained reward model, not the reward itself — the paper's subtitle says the language model was secretly a reward model all along.
- ORPO removes the reference model and the standalone alignment stage by attaching an odds-ratio penalty to SFT. It does not skip SFT.
- KTO removes the need for paired data, using a prospect-theory loss that weights undesirable examples more heavily than desirable ones.
- Choose by the feedback you already collect: paired and curated points to DPO or ORPO; unpaired thumbs from a live product points to KTO.
FAQ
Is DPO better than RLHF?
For most fine-tuning work outside a frontier lab, yes. It is cheaper, markedly more stable, and competitive on quality — matching or beating RLHF on sentiment control, summarisation and single-turn dialogue in the original paper's evaluations. At very large scale the answer softens: labs training frontier models still run full RLHF pipelines, where the reward model's generalisation across enormous preference datasets appears to earn its keep. Decide by use case and dataset size rather than by verdict.
Does DPO get rid of the reward model?
No. It gets rid of the separately trained, explicit reward model — the network you previously had to fit on its own before RL could start. The reward function itself remains, implicit in the policy's log-probabilities relative to the reference model. That is exactly what the paper's subtitle claims: your language model is secretly a reward model.
What is the difference between ORPO and DPO?
DPO is a second stage that runs after supervised fine-tuning and keeps a frozen reference copy in memory to anchor each update. ORPO is monolithic and reference-model-free: the odds-ratio penalty applies during SFT itself, so you run one pass with one model resident instead of two passes with two. Same paired chosen/rejected data, roughly half the training machinery.
When should I use KTO instead of DPO?
When your feedback is unpaired binary signal rather than curated A/B comparisons — thumbs-up and thumbs-down events from a live product, tickets marked resolved or not, moderation flags. DPO needs two candidate answers to the same prompt with a winner marked; KTO needs one label on one output. KTO also tolerates lopsided ratios, so a stream that is mostly positive with a thin tail of complaints remains usable.
Do I still need supervised fine-tuning first?
For RLHF and DPO, yes — both assume a model already tuned into the right response format before preference training starts. ORPO does not remove SFT either; it folds the alignment signal into that pass, so you are still doing SFT, once, with a penalty term attached. KTO is the only one practitioners report skipping it for, and only on a base model strong enough to carry the format unaided.
The other route to firing the human grader was replacing them with a written rulebook instead of a cheaper loss function — read how Constitutional AI hands the model its own rulebook for the other half of this story.