Reinforcement Learning: Learning by Trial, Error, and Reward

If you have used ChatGPT, Claude, or Gemini, you have noticed something odd. These tools are built to predict the next word, yet they answer questions politely, decline to help with dangerous requests, admit when they are unsure, and stay on task. None of that comes from reading the internet. It comes from a second kind of training, one where the model is rewarded for good behavior and nudged away from bad behavior.

That second kind of training is built on reinforcement learning. This lesson explains what reinforcement learning is, using no code and no math, so that the rest of the course, which explains how it is applied to chatbots, makes complete sense.

What You'll Learn

• The one idea behind reinforcement learning: learn from rewards, not from answers
• The four words you need: agent, environment, action, reward
• What a "policy" is, and why it is the thing that actually gets trained
• The exploration versus exploitation trade-off
• Why this course is about chatbots, not games or robots

Three ways a machine can learn

Most machine learning falls into one of three styles. You only need the difference at a high level.

Supervised learning is studying with an answer key. You show the system thousands of emails already marked spam or not spam, and it learns to copy those labels. The pre-training of a large language model, where it predicts the next token and gets corrected, is a form of this. The correct answer is simply whatever token came next in the real text.

Unsupervised learning has no answer key at all. The system looks for structure on its own, like grouping customers who behave similarly.

Reinforcement learning is different from both. Nobody hands the system a correct answer. Instead the system tries something, gets a score, and adjusts so that it earns a higher score next time. That is the style this course is about.

The four words: agent, environment, action, reward

Every reinforcement learning setup has the same four parts. Picture teaching a dog to sit.

• The agent is the learner. In our examples it will be a chatbot.
• An action is something the agent can do. For a dog, sit or bark. For a chatbot, write a particular reply.
• The environment is everything the agent acts on and gets feedback from. For the dog, it is you and the room. For a chatbot, it is the conversation and whoever is judging the reply.
• The reward is a number that says how good that action was. A treat is a high reward. Being ignored is zero. A scolding is negative.

The loop runs over and over. Act, get a reward, adjust, act again. Nobody ever tells the dog "the correct move is to lower your hindquarters." The dog works that out from what earns treats. That is the whole trick: reinforcement learning turns a score into behavior.

The policy: the thing that actually learns

Inside the agent there is a rule for choosing actions. Given the situation, which action should I take? That rule is called the policy.

At the start the policy is close to random. The dog tries everything. Over time, actions that earned rewards become more likely and actions that did not become less likely. The policy gets sharper. "When I hear the word sit, sitting is the move."

This word matters because when you get to the chatbot chapters, the policy is the language model itself. A language model is a rule that, given the conversation so far, picks the next token. Training it with reinforcement learning means adjusting that rule so the replies it tends to produce earn higher rewards. The knowledge stays. What changes is what the model chooses to say.

Exploration versus exploitation

There is a tension built into every reinforcement learner. Should it repeat the action that has worked before (exploitation), or try something new that might work even better (exploration)?

A dog that only ever sits will never discover that rolling over earns a bigger treat. A dog that only ever tries new things never collects the easy rewards it already knows about. Good learners do both, exploring a lot early and exploiting more as they get confident.

You will see this again with chatbots. During alignment training the model generates many different replies to the same prompt, some safe and some unusual, and learns from which ones scored well. If it only ever produced its single favorite reply, it would have nothing to compare and nothing to learn.

Where you have already seen reinforcement learning

Reinforcement learning has been around for decades, and its most famous wins were in games and control.

• Board and video games. Systems that learned Go, chess, and classic arcade games by playing millions of rounds against themselves, with winning as the reward, reached superhuman skill. Nobody programmed the strategies. The reward signal found them.
• Robotics. Robots learn to walk or grasp objects by trying, falling, and being rewarded for progress.
• Recommendation and ads. Systems that pick what to show you next are often tuned by rewards like clicks and watch time.

Games are a clean fit because the reward is obvious: you won or you lost. The hard part of applying reinforcement learning to a chatbot is that there is no scoreboard for "a good answer." The next lesson is about that gap, and the rest of the course is about how researchers filled it.

Why this course is about chatbots

There are whole degree programs on reinforcement learning theory. This is not one of them. You are here because you want to understand why the AI tools you use every day behave the way they do. So this course keeps to the version of reinforcement learning that shaped ChatGPT and its rivals: reinforcement learning from human feedback, usually shortened to RLHF.

If you took How LLMs Actually Work, you saw the three-stage pipeline of pre-training, fine-tuning, and alignment in one lesson. This course opens that last stage up and shows you the machinery inside. You do not need to have taken it first, but it helps to know that the model starts out as a next-token predictor trained on huge amounts of text.

Key Takeaways

• Reinforcement learning is learning from a score, not from an answer key: try an action, receive a reward, adjust, repeat.
• The four parts are the agent (learner), actions (what it can do), the environment (what it acts on), and the reward (a number for how good the action was).
• The policy is the agent's rule for choosing actions, and it is the thing that gets trained. For a chatbot, the policy is the language model itself.
• Learners balance exploration (trying new things) against exploitation (repeating what works), and chatbots are trained on many candidate replies for exactly this reason.
• Games have an obvious reward. A "good answer" does not, and closing that gap is what RLHF is for.