Gradient Descent and Backpropagation: Learning by Nudging

The network has made a prediction and scored its loss. Now comes the hard part: it has millions of weights, and it needs to work out which ones to change, in which direction, and by how much.

This is the step that makes neural networks possible at all. Two ideas do the work. Gradient descent is the strategy for improving by small steps. Backpropagation is the method that makes the strategy affordable. Both are usually taught with calculus, and neither actually needs it to make sense.

What You'll Learn

• Why brute force cannot possibly work here
• The downhill-in-fog picture behind gradient descent
• What the learning rate controls and what happens when it is wrong
• How backpropagation traces blame backward through the layers

Why you cannot just try things

Start with the obvious approaches and watch them fail, because that shows what the real method has to solve.

Try every combination of weights. Even with a small network and just a handful of possible values per weight, the number of combinations is larger than anything that could be checked in the lifetime of the universe. Not slow. Impossible.

Change one weight, see if loss improved, keep or discard. More sensible, and it does work in principle. But with a million weights, a single improvement step needs a million separate test runs, and training needs millions of steps. Also far too slow.

What is needed is a way to know, for every weight at once, both the direction to move it and roughly how much it matters, without testing them one at a time. That is exactly what these two ideas deliver.

Walking downhill in fog

Picture the loss as a landscape. Every possible combination of weights is a location, and the height at that location is how much loss the network has there. High ground means bad predictions. Valleys mean good ones. Training is the search for low ground.

The catch is that you cannot see this landscape. It has millions of dimensions and no one has ever viewed it. You only know one thing: your current position, and the slope beneath your feet.

That turns out to be enough. Stand on a hillside in thick fog and you can still feel which way is downhill. So take a small step that way. Feel the slope again. Step again. Repeat enough times and you end up somewhere low, without ever having seen the landscape.

That is gradient descent. The gradient is just the slope, and for each individual weight it answers one question: if I nudge this weight up a little, does the loss go up or down, and how sharply?

• Slope says loss rises if I increase this weight, so decrease it.
• Slope says loss falls if I increase this weight, so increase it.
• Slope is nearly flat, so this weight barely matters right now, leave it roughly alone.

Every weight gets its own answer, and every weight is nudged accordingly. Then the network takes the next example and does it again.

Step size matters more than you would think

How big should each step be? That setting is called the learning rate, and it is one of the most consequential dials in all of training.

Too large and the network overshoots. It leaps across the valley, lands high on the far slope, leaps back, and bounces around without ever settling. Loss jumps erratically or stops improving.

Too small and it creeps. Progress is real but so slow that training takes impractically long, and it can get stuck in a shallow dip that a larger step would have carried it out of.

There is no universally correct value, which is why finding a good learning rate is one of the standard jobs when training a model. Most modern training also adjusts it as it goes, taking bolder steps early and finer ones as things settle.

Backpropagation: passing the blame backward

Gradient descent tells you what to do with the slopes. It does not tell you how to get them, and getting them is the expensive part. This is where backpropagation comes in, and it is the reason deep networks are trainable in practice.

Think of a kitchen with several stations in a line. Ingredients enter at one end, each station does something and passes the result on, and a finished dish comes out. The dish is too salty. Who is responsible?

You start at the end and work backward. The final station knows what it received and what it did, so it can work out how much of the saltiness came from its own actions and how much was already there when the dish arrived. It keeps its share of the blame and passes the rest back to the station before it. That station does the same. The responsibility flows backward down the line until every station knows its own contribution to the problem.

That is backpropagation. The error is measured at the output layer, then passed backward layer by layer, with each layer working out how much its own weights contributed and handing the remaining responsibility further back. By the time the signal reaches the first layer, every weight in the network has its own slope.

The efficiency is the point. One forward pass to make a prediction, one backward pass to assign blame, and you have the direction for all million weights. Compare that to testing each weight separately and you can see why this single idea unlocked the field.

What this method does not promise

Two honest caveats, because gradient descent is often described as though it finds the best answer.

It does not. It finds a good answer. Walking downhill in fog gets you to low ground, but not necessarily to the lowest point in the whole landscape. You might settle in a perfectly decent valley while a deeper one sits over the next ridge, and you would never know. In practice this matters far less than it sounds like it should, because in these enormous landscapes there are many good valleys and most of them perform similarly well.

It also explains why training the same model twice gives different results. Different random starting positions mean different paths downhill and different resting places, all of them reasonable, none of them identical.

Key Takeaways

• Brute force is impossible; training needs the direction for every weight at once, which is what these two ideas provide.
• Gradient descent is walking downhill in fog: feel the slope, take a small step, repeat.
• The learning rate sets the step size, and getting it wrong means either bouncing around or crawling.
• Backpropagation sends the error backward through the layers so each weight learns its own share of the blame, in a single efficient pass.
• The result is a good set of weights, not a perfect one, which is why two training runs never land in exactly the same place.