Why a Base Model Is Not a Helpful Assistant

Here is a fact that surprises most people. If you could chat with a freshly pre-trained language model, the raw thing that just finished reading a big slice of the internet, it would not be helpful. It would not answer your question so much as continue it. Ask "How do I fix a leaking tap?" and you might get back "How do I fix a leaking toilet? How do I fix a dripping shower?" because on the internet, a question is often followed by more questions.

This lesson is about the gap between that raw base model and the polished assistant you actually use, and why closing that gap turned out to need reinforcement learning rather than just more of the same training.

What You'll Learn

• What a base model does well and what it does badly
• The three things an assistant needs that pre-training does not provide
• Why "just write the rules" fails
• Why humans find comparing answers far easier than writing perfect ones
• The three-step recipe the rest of the course walks through

What a base model is

Pre-training, covered in How LLMs Actually Work, teaches a model one skill: given some text, predict the most likely next token. Done at enormous scale, that single skill soaks up grammar, facts, reasoning patterns, coding conventions, and the style of thousands of kinds of writing.

The result is impressive and strange at the same time. A base model is a superb text continuer. Give it the first half of a news article and it writes a plausible second half. Give it three examples of a pattern and it produces a fourth. What it does not have is any notion that it is talking to a person who wants something.

None of the right-hand column is present in a base model. It has to be added. Understanding how is the whole point of this course.

The three things an assistant needs

Researchers usually describe the target with three words. An assistant should be:

• Helpful. It actually does what you asked, in a useful form, and asks when your request is unclear.
• Honest. It tells the truth as best it can, flags uncertainty, and does not invent facts or sources.
• Harmless. It refuses to cause harm, whether that is writing malware, helping with violence, or producing abusive content.

Pre-training does not aim at any of these. The internet contains helpful answers and unhelpful ones, honest claims and confident lies, kind words and cruel ones. A next-token predictor learns to reproduce all of it, because all of it is "likely text." The training objective never says which kind you want.

Why "just write the rules" does not work

The obvious fix is to write rules. Be polite. Answer the question. Never help with weapons. Engineers tried variations of this for years and kept hitting the same wall: good behavior is easy to recognize and nearly impossible to specify.

Try writing a rule for "a helpful answer." How long should it be? It depends on the question. How formal? Depends who is asking. Should it include caveats? Sometimes. Should it refuse? Only for some topics, and the line is fuzzy. Every rule you write has exceptions, and the exceptions have exceptions. You would need a rulebook the size of the internet.

Reinforcement learning offers a way around this. You do not need to describe good behavior. You need a reward signal that scores it. Then the model finds behaviors that earn reward, the same way the dog in lesson one discovered sitting without ever being told what sitting is.

That raises the real question: where does the reward for "a good answer" come from? In a game it is the score. In a conversation, there is no score.

The key insight: people can compare better than they can create

The answer researchers landed on is simple and a little humbling. Ask a person to write the perfect reply to a tricky question and they will struggle, take a long time, and disagree with the next person. But show that same person two replies and ask "which is better?" and they will answer in seconds, with far more consistency.

Comparisons are fast, cheap, and reasonably consistent across people. Collect enough of them and you have something a computer can learn from: a picture of what humans prefer. Turn that picture into a number, and you have your reward.

This single insight, that human preference can stand in for a score, is what the H and F in RLHF are about. Human feedback becomes the reward signal that reinforcement learning needs.

The three-step recipe

Putting it together, the standard recipe for turning a base model into an assistant has three steps. The best-known write-up of it came from the 2022 research behind the first ChatGPT, and every major lab uses some version of it today.

1. Supervised fine-tuning. Show the model a few thousand examples of good conversations, written by people, so it learns the shape of being an assistant. This is the subject of lesson three.
2. Train a reward model. Collect human comparisons between candidate replies and train a separate model to predict which reply a person would prefer. This is the subject of lesson four.
3. Optimize with reinforcement learning. Let the assistant generate replies, score them with the reward model, and adjust the assistant to produce higher-scoring replies. This is lesson five, along with a popular shortcut called DPO.

Each step builds on the one before. Skip step one and the model never learns the format. Skip step two and there is no reward. Skip step three and the reward is never used.

A useful way to hold all this

Pre-training gives the model capability. Alignment gives it direction. A base model can write a beautiful, honest, careful answer. It can also write a lazy, wrong, or dangerous one. It has no reason to prefer either. RLHF gives it a reason.

That framing also explains a lot of what you will meet in the final lesson. When alignment goes slightly wrong, it does not make the model less capable. It makes the model point in a slightly wrong direction, toward flattery, or toward refusing too much, or toward answers that look good to a judge. Those are direction problems, and once you can see them you can work around them.

Key Takeaways

• A base model is a text continuer. It is knowledgeable but has no idea it is helping a person, so it may answer, ramble, or continue the question.
• An assistant needs to be helpful, honest, and harmless, and pre-training aims at none of these because the internet contains every kind of text.
• Writing rules fails because good behavior is easy to recognize and impossible to fully specify.
• Humans are bad at writing perfect answers but good at comparing two answers, and those comparisons can become a reward signal.
• The three-step recipe is supervised fine-tuning, then a reward model, then reinforcement learning. Pre-training gives capability; alignment gives direction.