Step 3: Optimizing With PPO, and the Simpler DPO Shortcut

You now have an assistant that speaks the right way (from supervised fine-tuning) and a judge that scores replies the way people would (the reward model). The last step connects them. The assistant generates replies, the judge scores them, and the assistant is adjusted to produce higher-scoring replies. This is the reinforcement learning in RLHF, and it is where the dog-training loop from lesson one finally runs on a language model.

This lesson explains the classic way to do it, an algorithm called PPO, the guardrail that keeps it from going off the rails, and a newer shortcut called DPO that many teams now use instead. No math. Just the shape of what happens.

What You'll Learn

• The RLHF training loop, step by step
• What PPO is and why "proximal" is the important word
• The KL penalty: why the model is tied to its starting point
• What DPO does differently and why it became popular
• Which method the models you use were trained with (roughly)

The loop

Map the four words from lesson one onto a chatbot.

• The agent and its policy are the assistant model.
• An action is generating a reply, token by token.
• The environment is the prompt plus the reward model waiting to score the result.
• The reward is the reward model's score for the finished reply.

Then repeat, with a new prompt, tens of thousands of times. Each pass nudges the model's parameters so that the kinds of replies that scored well become slightly more likely and the kinds that scored badly become slightly less likely. No reply is ever labeled "correct." The model is finding, through trial and reward, what the judge likes.

Notice the randomness in step two. This is the exploration from lesson one. If the assistant always produced its single most likely reply, every score would be about the same and there would be nothing to learn from. Sampling several varied replies gives the loop contrast.

PPO: adjust, but not too much

The specific algorithm the original RLHF recipe used is Proximal Policy Optimization, or PPO. It was developed for games and robotics and borrowed for language models. You do not need its internals, but the name carries the one idea that matters.

Proximal means "nearby." PPO limits how much the policy can change in a single update. It takes the reward signal, works out which direction to move the model, and then clips the step so the model after the update stays close to the model before it.

Why be so cautious? Because rewards are noisy and the reward model is imperfect. If a batch of lucky high scores were allowed to yank the model a long way in one step, it could land somewhere strange, and the next batch might yank it somewhere else. Small steps keep the process stable. Think of it as turning a dial a notch at a time rather than spinning it.

PPO in practice is famously fiddly. It runs four models at once (the assistant being trained, a frozen copy of it, the reward model, and a helper model that estimates how good a partial reply is), it has many settings to tune, and it uses a lot of memory. That cost is the reason the shortcut later in this lesson caught on.

The KL penalty: staying anchored

Even with small steps, there is a deeper danger. The assistant is optimizing against a learned judge, not against real human opinion. If it pushes hard enough, it can find replies the reward model loves but that no human would, the same way a student can learn to game a specific teacher's grading quirks rather than learn the subject. Lesson six is about this problem in detail.

The main defense is built into the training objective. Alongside the reward, the assistant is charged a penalty for drifting away from where it started, meaning the SFT model from lesson three. The technical name for the measure of drift is the KL divergence, so this is called the KL penalty.

The two forces pull against each other on purpose. The reward says "go where the score is." The penalty says "but do not wander far." The result is a model that is meaningfully better than the SFT model at what humans prefer, while still writing like a sensible language model. Tune the leash too loose and you get a model that produces bizarre, repetitive, judge-pleasing text. Too tight and alignment barely changes anything. Getting this balance right is a large part of the craft.

DPO: skipping the reward model

In 2023, researchers at Stanford showed something neat. With a bit of algebra, the whole PPO setup, including the reward model, the sampling loop, and the KL penalty, could be collapsed into a single, ordinary supervised training step that works directly on the preference pairs. They called it Direct Preference Optimization, or DPO.

The recipe becomes:

1. Take each human comparison: prompt, preferred reply, rejected reply.
2. Train the assistant to raise the probability of the preferred reply and lower the probability of the rejected one, relative to the frozen SFT model.
3. That is it. No separate reward model. No generating fresh replies during training. No four-model juggling act.

DPO's appeal is obvious. It is far simpler, cheaper, and more stable, and for many purposes it produces results close to PPO. It quickly became the default for open-source models and for teams without a large infrastructure budget.

Its limitation sits in the last row. Because DPO only ever looks at the comparisons that were collected, it cannot discover a great reply that no rater ever saw. PPO can, because it keeps generating and scoring new replies during training. The online nature of PPO, learning from the model's own fresh attempts, is why the largest labs still use PPO-style methods, or variants of them, for their flagship models, while DPO and its many descendants dominate elsewhere. Most serious pipelines today mix elements of both.

What this means for the model you talk to

Whichever method was used, the model you chat with is the policy that came out the end of this loop. Its parameters are then frozen, as How LLMs Actually Work explained. When you use it, no reward model is running and no learning is happening. Everything you experience as helpfulness, tone, caution, and personality was baked in during these training runs, by pushing the model toward what a judge preferred and holding it close to its starting point.

That is the whole mechanism. Three steps, one loop, two forces in tension. The last lesson looks at what happens when the tension is not quite right.

Key Takeaways

• The RLHF loop is: sample a prompt, have the assistant generate several replies, score them with the reward model, and adjust the assistant toward the high scorers. Repeat tens of thousands of times.
• PPO is the classic algorithm. "Proximal" means each update is clipped to a small step, which keeps noisy rewards from destabilizing the model.
• The KL penalty charges the model for drifting from the SFT starting point. Reward and penalty pull against each other by design, which stops the model gaming the judge.
• DPO folds the reward model and the loop into one supervised step on the preference pairs. It is simpler and cheaper, but it cannot explore beyond the collected comparisons.
• The assistant you use is the frozen policy that came out of this loop. Its personality was set here, not while you chat.