The Optimization Landscape
When gradient descent runs, it navigates a high-dimensional surface defined by the loss function. The shape of this surface — its hills, valleys, saddle points, and plateaus — determines whether training succeeds, how quickly it converges, and what solution it finds.
Convex vs. Non-Convex Functions
A convex function has a single global minimum, like a bowl. Any downhill direction eventually leads to the bottom.
Convex (bowl): Non-convex (landscape):
* * * *
/ \ / \ / \ / \
/ \ / \/ \ / \
/ \ / local \/ \
global minima global
minimum minimum
Linear regression has a convex loss surface — gradient descent always finds the best solution. Neural networks have non-convex loss surfaces — gradient descent may find a local minimum, not the global one.
Local Minima
A local minimum is a point lower than all nearby points, but not necessarily the lowest overall.
Loss
^
| *
| / \ *
| / \ / \
| / local\ / \
|/ min \ / local
| \/ min
| global
| minimum
+────────────────> parameters
In early deep learning research, local minima were feared. The worry: gradient descent might get stuck in a poor local minimum. Modern research has shown this fear is mostly unfounded for large networks — local minima in high dimensions tend to have loss values close to the global minimum.
Saddle Points
A saddle point is a critical point (∇L = 0) that is a minimum in some directions and a maximum in others. It looks like a horse saddle.
up in this direction
↗
────────*────────
↙
down in this direction
In high dimensions, saddle points are far more common than local minima. A random critical point in 100-dimensional space is almost certainly a saddle point, not a minimum.
Why Saddle Points Are Problematic
At a saddle point, the gradient is zero, so basic gradient descent stalls. Momentum-based optimizers like SGD with momentum and Adam can escape saddle points because accumulated velocity carries the optimizer through the flat region.
Plateaus and Flat Regions
Plateaus are regions where the loss surface is nearly flat — the gradient is close to zero over a large area.
Loss
^
| *
| \
| \
| \_____________
| plateau \
| \___
+─────────────────────────> parameters
Training slows dramatically on plateaus because the gradient signal is weak. Adaptive optimizers (Adam, RMSProp) help by amplifying the effective learning rate in flat regions.
Ravines and Ill-Conditioning
A ravine is a narrow valley where the surface curves sharply in one direction and gently in another.
w₂
^
| ╭──────────────╮
| ╭┤ ├╮
| ╭┤ ╭──────────╮ ├╮
| │ ╭─┤ * ├╮ │ elongated contours
| ╰┤ ╰──────────╯ ├╯ = ravine
| ╰┤ ├╯
| ╰──────────────╯
+────────────────────────> w₁
In a ravine, the gradient points mostly across the valley (the steep direction) rather than along it (toward the minimum). This causes oscillation across the valley with slow progress along it. Momentum dampens the oscillation, and adaptive learning rates reduce steps in the steep direction.
The Role of Curvature
The second derivative (Hessian) describes the curvature of the loss surface:
- Positive curvature (∂²L/∂w² > 0): the surface curves upward — a valley, stable for gradient descent
- Negative curvature (∂²L/∂w² < 0): the surface curves downward — a hill, unstable
- Zero curvature (∂²L/∂w² = 0): the surface is flat — no information about the minimum
The condition number (ratio of largest to smallest curvature) determines how elongated the contours are. A high condition number means a ravine, making plain gradient descent inefficient.
| Condition Number | Surface Shape | Training Behavior |
|---|---|---|
| Close to 1 | Circular contours | Fast, smooth convergence |
| 10-100 | Moderately elongated | Slower, some oscillation |
| 1000+ | Extreme ravine | Very slow, heavy oscillation |
Why Neural Network Landscapes Are Special
Large neural networks have interesting loss surface properties:
- Many equivalent minima: Permuting neurons in a hidden layer gives different parameters with the same loss. So there are many global minima, connected by symmetries.
- Wide vs. narrow minima: Wide minima (flat-bottomed valleys) tend to generalize better than narrow, sharp minima. SGD with small batches naturally favors wide minima.
- Loss barriers decrease with overparameterization: Larger networks have smoother loss surfaces with lower barriers between minima.
Practical Implications
| Landscape Feature | Problem | Solution |
|---|---|---|
| Local minima | Suboptimal convergence | Mostly a non-issue for large networks |
| Saddle points | Training stalls (zero gradient) | Momentum, Adam |
| Plateaus | Very slow progress | Adaptive learning rates (Adam) |
| Ravines | Oscillation across, slow along | Momentum, learning rate scheduling |
| Sharp minima | Poor generalization | Small batch SGD, weight decay |
Escaping Poor Regions
Several techniques help the optimizer explore the landscape more effectively:
- Learning rate warm-up: Start small to avoid divergence, then increase to enable exploration
- Cyclical learning rates: Periodically increase the learning rate to escape local minima
- Stochastic noise: Mini-batch SGD naturally adds noise that helps escape sharp minima
- Weight initialization: Careful initialization places the starting point in a favorable region
Summary
- The loss surface is the high-dimensional landscape that gradient descent navigates
- Convex functions have one global minimum (linear regression); neural networks are non-convex
- Saddle points (flat in some directions) are more common than local minima in high dimensions
- Plateaus slow training because gradients are near zero
- Ravines cause oscillation; momentum and adaptive rates mitigate this
- The second derivative (curvature) determines how well gradient descent can navigate the surface
- Large neural networks have favorable landscape properties: many equivalent good minima and smoother surfaces
- Choice of optimizer, learning rate schedule, and batch size all interact with the loss landscape
The next lesson covers regularization — techniques that modify the loss function to prevent overfitting and improve generalization.