In this lesson: Describe each training stage, what data it needs, and what capability it adds.
A deployed assistant is the product of three distinct stages. They differ enormously in cost, in data, and in what they change.
Stage 1: pretraining
Next-token prediction over trillions of tokens of text and code. Cross-entropy loss, one pass or slightly more over the corpus, thousands of accelerators for weeks or months. This is where essentially all knowledge and capability is acquired, and where essentially all of the money goes.
Data work dominates quality far more than architecture does:
- Deduplication — repeated documents cause memorisation and waste compute.
- Quality filtering — classifier-based selection of well-formed text.
- Decontamination — removing evaluation benchmarks from training data. Skipping this produces impressive published scores and a model that disappoints in use.
- Mixture design — how much code, how much multilingual text, how much mathematics. Code in the mixture measurably improves reasoning on non-code tasks.
Stage 2: supervised fine-tuning
The pretrained model continues text; it does not answer. SFT trains it on tens of thousands to a few hundred thousand curated demonstrations of instruction following, formatted as conversations, with loss computed only on the assistant turns. It is cheap next to pretraining — a rounding error in compute — and it is what converts a text continuer into an assistant. It adds format and behaviour, not knowledge.
Stage 3: preference optimisation
SFT teaches one acceptable answer per prompt. Preference training teaches ranking: given two responses, which is better? This is where helpfulness, honesty and harm avoidance are actually installed.
The RLHF pipeline
- Sample several responses per prompt from the SFT model.
- Humans rank them.
- Train a reward model to predict those rankings.
- Optimise the policy against the reward model with reinforcement learning (typically PPO), with a KL penalty against the SFT model to stop it drifting into degenerate high-reward text.
DPO and the simplification
Direct Preference Optimisation removes the separate reward model and the RL loop, deriving a loss that can be optimised directly on preference pairs. Far simpler and more stable; the trade is less control over the exploration that online RL provides. Both approaches are in production use across the industry.
AI feedback
Human ranking is the bottleneck: slow, expensive, and inconsistent between annotators. Constitutional AI replaces much of it with a written set of principles and a model that critiques and revises its own outputs against them, generating preference data at scale. The advantages are cost and — importantly — auditability: the principles are a document you can read and argue with, rather than the implicit aggregate judgement of a labelling workforce.
Where reasoning training fits
Recent capability gains come substantially from training models to produce extended reasoning before answering, with reinforcement learning against verifiable outcomes — mathematics that can be checked, code that can be run, tasks with a ground-truth result. Because correctness is machine-checkable, the reward signal is far cleaner than human preference, and models can be trained on much more of it.
What this means for you as a practitioner
- Model behaviour is a training artefact, not a fixed property. Two models with similar benchmark scores can behave very differently on your task; test on your data.
- Being agreeable is partly a preference-training side effect — humans rate agreement highly. Design for it: ask for critique explicitly rather than trusting a model that agrees with you.
- A knowledge cutoff is a pretraining property. No amount of prompting recovers information the model never saw; that is what retrieval is for.
Try it yourself
Take a task with a clear right answer and give a model a wrong premise stated confidently ("since the algorithm is O(n), …" when it is not). See whether it corrects you or builds on your error. Then re-run with a system prompt that explicitly rewards challenging false premises. The gap you measure is preference training showing through.