Linear Regression: Ordinary Least Squares
The first model everyone learns — and the one everything else is built on. Watch a line hunt for the best fit, drag your own residuals, and race gradient descent against the closed-form solution.
What Does “Fitting a Line” Actually Mean?
You have data: hours studied vs exam score. Linear regression draws the straight line that comes closest to all points at once — is the slope (how much the score rises per extra hour) and is the intercept (the baseline score).
But “closest” needs a definition. For every point we measure the residual — the vertical gap between truth and prediction. We square each gap (so negatives don't cancel positives) and add them up:
In the animation, the orange line is a candidate fit sweeping through angles — too steep, too flat, too steep again. The red dashed segments are the residuals it leaves behind. The best line is the one minimizing the total squared length of those red segments — Ordinary Least Squares (OLS).
The dashed ink line never moves — OLS already found it.
Sandbox: Steal the Line Yourself
Drag the slope and intercept and watch the squared error react. Can you beat the computer's answer before you press the snap button?
Notice that reads as “the fraction of variance the line explains” — 0 means the line is no better than predicting the mean, means a perfect fit through every point.
Red dashed hairlines are the errors . OLS minimizes the sum of their squares.
Normal Equations vs Gradient Descent
The error is a bowl over the parameter space — every pair is a point on its surface, and the height is the MSE. There are two classic ways to reach the bottom:
One matrix inversion teleports you straight to the bottom. Exact — but cost makes it impractical for huge feature counts.
Take small steps downhill, opposite the gradient. Cheap per step, iterative — and the learning rate is everything: too small crawls, too large overshoots and zigzags (watch the ink ball).
Both balls want the same minimum — only the journey differs.
Metrics & the Fine Print
Squaring makes outliers scream — a single point 10× off hurts 100× more. RMSE returns to the target's units.
Every error counts linearly — robust to outliers. Pick MAE when a few wild points shouldn't dominate the score.
Variance explained vs a mean-only baseline. Comparable across datasets — but a high R² never implies causation.
- Linearity — the relationship really is straight.
- Independence — errors don't correlate (no time leakage).
- Homoscedasticity — error spread is constant, not funnel-shaped.
- Normal residuals — needed for valid confidence intervals.
Six Assumptions Holding the Line Up
OLS will hand you coefficients no matter how badly the data misbehaves. These six assumptions are the conditions under which those coefficients mean what the textbook says they mean. Read each one as a question you can ask of a residual plot.
Do the features and the target still mean what the decision needs?
Would a fresh case from deployment look at home in your training rows?
Plot residuals against fitted values — a curve hiding there breaks the straight-line claim.
Do residuals drift with collection order, time, or group?
Does the residual cloud fan out as predictions grow?
Read a QQ plot — this one buys valid confidence intervals, not better predictions.
The cloud fans open and snaps shut — variance is not constant.
The ideal the assumption asks for — same width everywhere.
A violated assumption is not an automatic failure. In machine learning the test is narrower: does the model generalize to unseen data? A predictor that scores well on a held-out set is still useful even when its errors are lumpy or its residuals are not Gaussian. Statistical purity matters most when you are making inferences — confidence intervals, p-values, causal claims — and far less when you are shipping predictions. So start by asking whether the assumption even applies to your question.
Logs or square roots tame a fanning residual cloud and straighten a multiplicative relationship.
Add interaction or polynomial terms, or move to something that bends — splines, GAMs, trees.
Widen the sample until it represents the population you actually deploy to.
Narrow the claim or the input range until the data can honestly support it.
MSE, RMSE, MAE & R² — Four Lenses on One Error
All four scores are computed from the same residuals. They differ only in how they weigh a miss — and that choice is a statement about what you care about.
Units: squared target units — score², ₹², whatever is measured in.
Failure mode: punishes outliers quadratically — an error twice as large contributes four times as much.
Units: the target's own units — is why RMSE is the score you report out loud.
Failure mode: the square is still inside the root, so one wild point can set your headline number.
Units: target units — directly readable as “the average miss”.
Failure mode: robust to outliers, but is non-differentiable at , and a fat tail of medium errors hides inside the same average.
Units: dimensionless — is perfect, is no better than the mean, negative is worse than the mean.
Failure mode: always rises as you add features, even pure noise — never compare R² across models with different feature counts. Use adjusted R² or a held-out set.
RMSE (5.27) sits above MAE (4.31) — that gap is the outlier signature. When the two converge, errors are evenly sized; when RMSE runs away, a handful of points are carrying the score.