Preference tuning (RLHF, DPO)
A second training stage that teaches a model which of two answers people prefer, rather than a single target answer. RLHF uses a reward model + reinforcement learning; DPO does it directly and cheaply.
What it is
After supervised fine tuning, you collect pairs of model outputs labelled "better / worse". RLHF trains a separate reward model on those labels, then uses RL (PPO) to push the model toward high-reward outputs - powerful but fiddly and compute-heavy. DPO skips the reward model and RL loop: a single loss function moves probability toward the preferred answer and away from the rejected one. Similar results, a fraction of the cost and complexity.
Why it matters
This is the stage that makes a model feel "helpful" rather than just fluent. For a self-hosted model, DPO on a lora adapter is now within a hobbyist budget; RLHF rarely is.