Step 1: Supervised Fine-Tuning, Learning From Demonstrations

Before any rewards are involved, the base model needs to learn what a conversation with an assistant even looks like. That is the job of supervised fine-tuning, usually shortened to SFT. It is the least glamorous step in the pipeline and one of the most important, because everything after it builds on the habits it creates.

This lesson explains what SFT is, where the training examples come from, what it changes in the model, and, just as important, what it cannot fix on its own.

What You'll Learn

• What supervised fine-tuning is and why it comes first
• Where demonstration data comes from and what a good example looks like
• The difference between learning a format and learning good judgment
• Why SFT alone produces an assistant that is fluent but not reliable
• How SFT relates to the "fine-tuning" businesses do on their own data

The idea: show, don't score

Remember the three learning styles from lesson one. SFT is plain supervised learning: examples with correct answers, and the model learns to copy them. No rewards yet. The "correct answer" is simply a reply that a person wrote.

A single training example looks like a short conversation:

• User: Can you explain what a mortgage is in two sentences?
• Assistant: A mortgage is a loan you take out to buy a home, using the home itself as security for the lender. You repay it in monthly installments over many years, and if you stop paying, the lender can take the property.

The model is trained on thousands of conversations like this, predicting the assistant's tokens the same way pre-training predicted the next token in web text. The difference is the data. Instead of everything on the internet, it now sees only examples of the behavior we want.

Where the demonstrations come from

The examples have to be written by someone, and their quality caps the quality of the result. Labs get them from a few sources.

• Hired writers and domain experts. People are paid to write the ideal reply to a prompt, following a style guide that spells out tone, formatting, and when to refuse. For specialist topics such as law, medicine, or code, labs hire specialists.
• Real user prompts. Prompts are often sampled from the kinds of questions actual users ask, so the examples match real demand rather than a researcher's imagination.
• Existing models. Once you have a good assistant, it can draft demonstrations that humans then review and edit. This is much cheaper than writing from scratch, and it is now standard practice. A small amount of careful human editing goes a long way.

The surprising lesson from the early research was how little data this step needs. Pre-training uses trillions of tokens. SFT works with tens of thousands of high-quality conversations. A well-chosen ten thousand examples beat a sloppy million, because the model is not learning knowledge here, it is learning a style of behaving.

What SFT actually changes

It helps to be precise about what this step does and does not do.

After SFT the model reliably behaves like an assistant. It answers the question you asked. It uses paragraphs and bullet points when they help. It says "I" and addresses "you." It has learned the format by imitation, the way an actor learns a role from a script.

What it has not learned is judgment. Imitation teaches the model what a good answer looks like on the surface. It does not teach the model to prefer a correct answer over a fluent wrong one, or to know when to say "I am not sure," because the demonstrations mostly show confident, complete replies. An SFT-only model is famously good at sounding right.

Why imitation runs out

Think about how you would train a new customer-support hire using only written examples of great support emails. They would quickly pick up the greeting, the structure, the sign-off. They would not learn what to do when a customer asks something the examples never covered, or when the polite-sounding reply is actually wrong.

SFT has the same limits, for three reasons.

1. Coverage. No set of demonstrations covers every situation. When the model meets a new one, it falls back on pre-training habits, which include making things up.
2. The ceiling is the demonstrator. The model can only get as good as the people writing the examples. It cannot exceed them, because it is copying.
3. No signal about bad answers. Every example is a good answer. The model never sees a bad answer labeled as bad, so it learns nothing about what to avoid.

That third point is what the next two lessons fix. A reward model learns from comparisons, which contain exactly the missing information: this reply is better than that one. Reinforcement learning then uses that signal to push the model past its demonstrations.

SFT in the wild: the fine-tuning you hear about

The phrase "fine-tuning" shows up in business contexts too. A company fine-tunes a model on its own support tickets, its legal templates, or its product catalog. That is the same technique as SFT: further supervised training on a focused set of examples. It teaches the model a domain's format and vocabulary, and it is one of three common ways to adapt a model, alongside prompt engineering and retrieval. If you want the practical comparison, read RAG vs Fine-Tuning vs Prompt Engineering.

The distinction to keep in mind: the SFT in this course is done by the lab before release, to create the assistant. Business fine-tuning happens afterward, on top of an already-aligned model, to specialize it.

Why SFT comes first

You might wonder whether the reward-based steps could run directly on the base model and skip this one. In practice, no. A base model produces such a wide spread of outputs, many of them not even attempts at answering, that a reward model would have little useful to compare. SFT narrows the model's behavior to "reasonable attempts at helping," which gives the reinforcement learning step a sensible starting point and a much easier job.

It also sets the reference point. In lesson five you will see that reinforcement learning is deliberately kept from drifting too far from where it started. The SFT model is that anchor.

Key Takeaways

• Supervised fine-tuning (SFT) trains the base model on thousands of human-written example conversations, so it learns the format of being an assistant.
• SFT needs surprisingly little data compared with pre-training, because it teaches a style of behaving rather than knowledge. Quality matters far more than quantity.
• After SFT the model answers questions in a consistent style, but it has no judgment: it cannot tell a correct answer from a fluent wrong one, and it never sees examples of what to avoid.
• Imitation is capped by the demonstrators and by coverage; reward-based training is what lets the model go beyond its examples.
• SFT comes first because it gives reinforcement learning a sensible starting point and an anchor to stay close to.