Cost Functions and Gradient Descent

Now we know linear regression finds the best line. But how does it find it? The answer involves two key concepts: cost functions and gradient descent.

The Cost Function

A cost function measures how wrong your model is. The lower the cost, the better the model.

Why Squared Error?

Visualizing the Cost Function

Think of the cost function as a landscape. We want to find the lowest point.

Gradient Descent: Finding the Minimum

Gradient descent is an optimization algorithm that finds the minimum of the cost function step by step.

The Learning Rate

The learning rate controls how big each step is.

Gradient Descent Intuition

Putting It Together

Key Takeaways

• Cost function measures how wrong the model is (MSE for regression)
• Gradient descent finds parameters that minimize the cost
• The gradient tells us which direction reduces cost
• Learning rate controls step size (too small = slow, too large = unstable)
• Training = iteratively updating parameters until cost is minimized
• This is how neural networks and many ML algorithms learn!

Next, we'll implement linear regression with sklearn and build a real model!