Linear Regression from Scratch: Univariate & Multivariate Examples

This project contains two Python scripts that implement linear regression from the ground up using NumPy for calculations and Matplotlib for visualization. These scripts serve as a practical guide to understanding the core mechanics of training a linear model via gradient descent.

1. Multivariate Linear Regression: Models a relationship with two input variables (e.g., $y = w_1x_1 + w_2x_2 + b$) and visualizes the result as a 3D plane.
2. Univariate Linear Regression: Models a relationship with a single input variable (e.g., $y = wx + b$) and visualizes the result as a 2D line.

---

🧠 Core Concepts

Both implementations are built upon the same foundational principles of linear regression.

Model Representation

The goal is to find a linear model that best fits the data. The model's prediction, $f_{w,b}(x)$, is given by: $$f_{w,b}(x) = w \cdot x + b$$

• For univariate regression, $w$ and $x$ are scalars.
• For multivariate regression, $w$ and $x$ are vectors.

Cost Function (Mean Squared Error)

To measure the model's accuracy, we use the Mean Squared Error (MSE) cost function. It calculates the average of the squared differences between the model's predictions and the actual target values. The objective of training is to find the parameters w and b that minimize this cost. $$J(w, b) = \frac{1}{2m} \sum_{i=1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)})^2$$

Gradient Descent

Gradient descent is the optimization algorithm used to minimize the cost function. It works by iteratively taking steps in the opposite direction of the cost function's gradient. The update rules for the parameters are: $$w_j := w_j - \alpha \frac{\partial J(w,b)}{\partial w_j}$$ $$b := b - \alpha \frac{\partial J(w,b)}{\partial b}$$ where $\alpha$ is the learning rate.

---

📈 Part 1: Multivariate Linear Regression

This script trains a model to learn the relationship $y = 3a + 5b + 7$ from a given dataset. It then visualizes the training data, test predictions, and the learned regression plane in a 3D space.

Output

Running the multivariate script will:

1. Print Training Progress: The console will display the decreasing cost and updating w and b values at each iteration, showing that the model is learning parameters close to w = [3, 5] and b = 7.
2. Generate a 3D Plot: A Matplotlib window will appear with a 3D visualization.
   • Blue dots are the original training data points.
   • Red dots are the model's predictions on the test data.
   • The orange surface is the final regression plane that the model has learned, representing the best fit for the data.

---

📉 Part 2: Univariate Linear Regression

This script trains a simpler model to learn the relationship $y = 2x + 3$. It visualizes the final best-fit line against the training data points in a 2D plot.

Output

Running the univariate script will:

1. Print Training Progress: The console will show the cost and the values of w and b being optimized. The final parameters will be very close to the true values of w = 2 and b = 3.
2. Generate a 2D Plot: A Matplotlib window will show:
   • Black dots (k) representing the training data points.
   • A blue line (b) with circle markers representing the learned regression line, which shows the model's predictions for a range of x-values.

---

⚙️ How to Run

Prerequisites

• Python
• NumPy
• Matplotlib

1. Install Dependencies

Open your terminal or command prompt and install the required libraries:

pip install numpy matplotlib

2. Save the Code

Because the file contains two separate, complete scripts, you should save them as two different files.

• Save the first part as multivariate_regression.py.
• Save the second part as univariate_regression.py.

3. Execute the Scripts

Run each script individually from your terminal to see its output:

# To run the multivariate example
python multivariate_regression.py

# To run the univariate example
python univariate_regression.py