Step 2: The Reward Model, Teaching a Model to Judge

This is the lesson where the "human feedback" in RLHF actually enters the picture. After supervised fine-tuning the assistant speaks fluently but has no way to tell a better answer from a worse one. The fix is to build a second model whose only job is to judge: given a prompt and a reply, output a score for how much a person would like that reply.

That second model is the reward model. Understanding it is the key to understanding both why aligned chatbots feel so agreeable and why they sometimes go wrong.

What You'll Learn

• What a reward model is and what it takes in and puts out
• How preference data is collected from human raters
• Why comparisons are turned into a single score
• What raters are asked to judge, and why that choice shapes the assistant's personality
• Where AI feedback and written principles fit in

What a reward model is

A reward model is a language model that has been retrained to do one narrow thing. Instead of generating text, it reads a prompt plus a candidate reply and produces a single number. Higher means "a person would prefer this." Lower means "a person would not."

In the vocabulary of lesson one, this is the missing reward signal. Games had a scoreboard. Conversations did not. The reward model is a learned scoreboard for "helpful, honest, and harmless," built from human opinions.

It is usually started from a copy of the SFT model, because a model that already understands language is a much better starting point for judging language than a blank one. The final generation layer is swapped for a layer that produces a score, and then it is trained on preference data.

Collecting preferences

The training data for a reward model is made of comparisons, and collecting it looks like this.

1. Take a prompt, often sampled from real user requests.
2. Have the SFT model generate several different replies to it, say four.
3. Show a human rater the prompt and the replies, and ask them to rank the replies from best to worst.
4. Break the ranking into pairs: reply A beat reply B, A beat C, B beat C, and so on.

Each pair is one training example: this reply was preferred over that one, for this prompt. Labs collect hundreds of thousands to millions of these pairs. Raters follow detailed guidelines that define what "better" means, covering accuracy, following the instructions, tone, safety, and honesty about uncertainty.

Why rank several replies rather than rate one? Because, as lesson two explained, people are much more consistent at relative judgments than absolute ones. "This reply is a 7 out of 10" varies wildly from rater to rater. "This reply is better than that one" does not.

From comparisons to a score

Here is the clever part. The reward model never sees a score in its training data, only pairs. So how does it learn to output a number?

The training rule is simple. For each pair, the reward model scores both replies. If it gives the preferred reply a higher score than the rejected one, good. If not, its parameters are nudged so that next time the gap goes the right way. Do this over millions of pairs and a consistent scale emerges on its own. Replies that tend to win comparisons collect high scores. Replies that tend to lose collect low ones.

The scale is relative and a little arbitrary. A score of 3.2 means nothing by itself. What matters is that 3.2 beats 1.9. That is enough for reinforcement learning, which only needs to know which direction is "better."

The judge's taste becomes the assistant's personality

This is the point in the course worth slowing down for. In step three, the assistant will be trained to produce whatever the reward model scores highly. So whatever the raters preferred becomes what the assistant does, including preferences the raters never consciously intended.

None of these are bugs in the code. They are the raters' taste, faithfully learned. If raters slightly prefer a reply that opens with "Great question!", the reward model learns to score that opening higher, and the assistant learns to say it. Much of what feels like a chatbot's personality is the averaged preference of a few thousand people, compressed into a scoring function.

The final lesson returns to the failure modes on this list. For now the takeaway is that the reward model is where the assistant's values come from, and its values are only as good as the comparisons it was trained on.

Who is doing the judging: humans, AI, and principles

Early RLHF used human raters for every comparison. That is slow and expensive, and it does not scale to the volume modern training needs. Two developments changed the picture.

AI feedback. A strong existing model can compare replies too, and it is fast and consistent. Using an AI model as the rater is called RLAIF (reinforcement learning from AI feedback). Labs now mix the two: humans set the guidelines and check a sample, and AI raters produce the bulk of the comparisons. The reward model is then trained on the combined data.

Written principles. Instead of hoping raters absorb a values guideline, you can hand the AI rater an explicit list of principles and ask it to judge replies against them: "choose the reply that is less likely to cause harm," "choose the reply that is more honest about what it does not know." Anthropic's Constitutional AI is the best-known version. The written constitution becomes the standard the judge applies, which makes the assistant's values inspectable in a way that a pile of anonymous human comparisons is not.

Both approaches still end in the same place: a reward model that scores replies, which step three then optimizes against.

A note on verifiable rewards

For some tasks you do not need a learned judge at all. If the prompt is a math problem with one right answer, or code that either passes its tests or does not, the reward can be checked automatically. Training with these verifiable rewards is a big part of how the recent generation of "reasoning" models learned to think through problems step by step. It is reinforcement learning without the human-preference part, and it works precisely because those tasks have the scoreboard that conversation lacks.

Real assistants are trained with a blend: verifiable rewards where an answer can be checked, and a preference-based reward model everywhere else.

Key Takeaways

• A reward model is a trained judge: it reads a prompt and a reply and outputs a score for how much a person would prefer that reply.
• It is trained on comparisons: raters rank several candidate replies, and each pair becomes an example of "this beat that."
• The reward model learns a score scale from pairs alone by being nudged until preferred replies score higher than rejected ones.
• Whatever the raters preferred, including unintended habits like length or flattery, becomes the assistant's behavior in the next step. The reward model is where an assistant's values come from.
• Modern training mixes human raters, AI raters (RLAIF), written principles (Constitutional AI), and verifiable rewards for tasks with checkable answers.