Understanding Gradient Descent for Linear Regression

Gradient descent is a pivotal optimization technique utilized in linear regression to determine the optimal line that best fits the data. It operates by incrementally adjusting the line's slope and intercept to minimize the difference between the actual and predicted values. This method significantly enhances the model's predictive accuracy by reducing errors iteratively.

Gradient Descent in Linear Regression

The illustration above depicts two graphs. On the left, house prices are plotted against size, showcasing errors as measured by the cost function. The right graph demonstrates how gradient descent navigates the cost curve, minimizing error by updating parameters step by step.

Importance of Gradient Descent in Linear Regression

Linear regression aims to identify the best-fit line for a dataset by minimizing the error between actual and predicted values, usually quantified using the Mean Squared Error (MSE). The objective is to ascertain the model parameters, namely the slope ( m ) and the intercept ( b ), that minimize this cost function.

While simple linear regression can employ formulas like the Normal Equation to directly find parameters, these methods become computationally burdensome for large datasets or high-dimensional data due to:

• Extensive matrix computations
• Memory constraints

In scenarios such as polynomial regression, where cost functions are notably complex and non-linear, analytical solutions become impractical. Here, gradient descent is invaluable, especially for:

• Large datasets
• Complex, high-dimensional problems

How Gradient Descent Operates in Linear Regression

The process of gradient descent in linear regression involves several steps:

1. Initializing Parameters: Begin with random starting values for the slope (m) and intercept (b).

2. Calculate the Cost Function: Compute the error using Mean Squared Error (MSE):

   [ J(m, b) = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - (mx_i + b) \right)^2 ]

3. Compute the Gradient: Determine the rate of change of the cost function concerning m and b.

   • For slope ( m ):

     [ \frac{\partial J}{\partial m} = -\frac{2}{n} \sum_{i=1}^{n} x_i (y_i - (mx_i + b)) ]

   • For intercept ( b ):

     [ \frac{\partial J}{\partial b} = -\frac{2}{n} \sum_{i=1}^{n} (y_i - (mx_i + b)) ]

4. Update Parameters: Adjust m and b to reduce the error:

   • For slope ( m ):

     [ m = m - \alpha \cdot \frac{\partial J}{\partial m} ]

   • For intercept ( b ):

     [ b = b - \alpha \cdot \frac{\partial J}{\partial b} ]

   Here, ( \alpha ) is the learning rate, determining the magnitude of each update.

5. Repeat: Continue iterating steps 2–4 until there is no significant decrease in error.

Implementing Gradient Descent in Linear Regression

Let's explore implementing linear regression step by step. Initially, a simple linear regression is constructed without gradient descent to observe its outcomes.

Python code using libraries such as NumPy, Pandas, Matplotlib, and Scikit-learn:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_regression

X, y = make_regression(n_samples=100, n_features=1, noise=15, random_state=42)
y = y.reshape(-1, 1)
m = X.shape[0]

X_b = np.c_[np.ones((m, 1)), X]

theta = np.array([[2.0], [3.0]])

plt.figure(figsize=(10, 5))
plt.scatter(X, y, color="blue", label="Actual Data")
plt.plot(X, X_b.dot(theta), color="green", label="Initial Line (No GD)")
plt.xlabel("Feature")
plt.ylabel("Target")
plt.title("Linear Regression Without Gradient Descent")
plt.legend()
plt.show()

The initial model's predictions are not precise, and the line does not fit the data well due to unoptimized parameters. Applying gradient descent improves the model by optimizing these parameters.

learning_rate = 0.1
n_iterations = 100

for _ in range(n_iterations):
    y_pred = X_b.dot(theta)
    gradients = (2 / m) * X_b.T.dot(y_pred - y)
    theta -= learning_rate * gradients

plt.figure(figsize=(10, 5))
plt.scatter(X, y, color="blue", label="Actual Data")
plt.plot(X, X_b.dot(theta), color="red", label="Optimized Line (With GD)")
plt.xlabel("Feature")
plt.ylabel("Target")
plt.title("Linear Regression With Gradient Descent")
plt.legend()
plt.show()

The model with Gradient Descent illustrates how it progressively learns to adjust the line, minimizing differences between predicted and actual values by updating parameters iteratively.