03 / Resources/Foundations · Bias & Variance

Interactive Diagnostic Essay · Topic 11

Bias & Variance: Two Ways To Be Wrong

A model can miss the target by aiming at the wrong place, or by spraying its shots everywhere — usually both. This is the essay that separates the two, proves they add up to your error, and hands you a slider to feel the tradeoff with your own hands.

01 / Two Ways To Be Wrong

The Archer, the Aim and the Spread

Picture an archer. The bullseye is the truth $f(x)$ — the real relationship hiding in your data. Every model you train is one shot at it. Score the arrows and two different failures jump out.

Bias · the aim is off

Arrows land in a tight little group — consistently, in the same wrong place. Retrain on a fresh dataset and the group barely moves; it is still off. That systematic offset is bias.

Variance · the spread is wide

Now the group is centred correctly but sprayed across the whole board. Retrain and the arrows land somewhere completely new. That erratic scatter is variance.

Watch the loop flip between the two: first a steady aim landed far from the bullseye, then a perfect aim that cannot hold still. A model is routinely wrong both ways at once — which is why we need to price each one separately.

THROWS ON A TARGET · 7s LOOPbias ↔ variance

the crosshair is the aim · the ring is the spread · the gap from the bullseye is the bias

02 / The Decomposition

Expected Error Splits Into Exactly Three Pieces

The archer's two failures are not a metaphor that fades out when the maths starts. For squared error they are the maths. Take a fixed point $x$ , imagine retraining your model on many independent datasets, and average the squared error you would get.

The Most Important Formula in This Article

\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\text{Bias}^2[\hat{f}(x)]}_{\text{systematic offset}} + \underbrace{\text{Var}[\hat{f}(x)]}_{\text{erratic scatter}} + \underbrace{\sigma^2}_{\text{irreducible noise}}

Bias² · how far off the aim is

\text{Bias}[\hat{f}(x)] = \mathbb{E}[\hat{f}(x)] - f(x)

The average prediction across all those hypothetical datasets, minus the truth. A property of your model class and features, not of any one lucky fit. It cannot be averaged away by collecting more data.

Variance · how much the aim wobbles

\text{Var}[\hat{f}(x)] = \mathbb{E}\Big[\big(\hat{f}(x) - \mathbb{E}[\hat{f}(x)]\big)^2\Big]

How widely predictions jump around their own average as the training set changes. This one does shrink when you feed the model more data or hold it back from chasing noise.

σ² · the floor nobody can dig through

\sigma^2 = \mathbb{E}\big[(y - f(x))^2\big]

The label $y$ is itself random around $f(x)$ . Even a model that recovers $f$ perfectly still eats this error. It is irreducible — a hard lower bound on achievable loss.

Sit with $\sigma^2$ for a second, because it is the term people forget. A validation loss of $0.00$ is not a goal you are failing to reach — it is a goal that does not exist. If the labels carry noise, no architecture, no dataset size and no amount of compute drives the expected error below it. Everything else in this article is a fight over the first two terms, and the only lever you truly own is how you split your budget between them.

03 / Underfitting & Overfitting

Too Simple, Too Complex, and the Sweet Spot Between

Complexity is the dial that trades one error for the other. Turn it down and bias takes over; turn it up and variance does. The same story is told by polynomial degree, tree depth, the number of neighbours in KNN, or how long you train.

Underfitting · high bias, low variance

A straight line forced through a curve. The model is too rigid to bend, so it misses the trend the same way every time — low variance, because a new dataset barely changes the answer. Both train and validation error sit high, and uncomfortably close together.

Overfitting · low bias, high variance

A degree-25 polynomial threaded through every point — including the noise. Training error collapses toward zero while validation error climbs. Change one training row and the whole curve lurches. Low bias on paper, useless in practice.

SWEET SPOT: enough flexibility to follow the signal, not enough to memorise the noise.

ERROR vs MODEL COMPLEXITYthe U-shaped curve

solid line = total error · it is the sum of the two dashed ones

04 / KNN: The Knob in Plain Sight

Drag $k$ and Watch Bias Trade For Variance

K-Nearest Neighbours makes no assumptions at all — it predicts by averaging the labels of the $k$ training points closest to the query:

\hat{f}_k(x) = \frac{1}{k}\sum_{i \in \mathcal{N}_k(x)} y_i

NEIGHBOURS (k)3

k = 1 · memorisek = 25 · average everything

TRAIN MSE0.0040

SMOOTHNESS · Σ|Δ|1.63

NOW: low bias · high variance

At $k = 1$ every training point predicts itself: train MSE is exactly $0$ and the curve is a spike for every noisy label. That is pure variance on display. Push $k$ up and the wiggles flatten as the neighbourhood swallows more and more of the signal — until the curve is nearly straight and train MSE balloons. That is bias, bought honestly.

1-D KNN FIT · FIXED NOISY DATASETk = 3

orange = the fit · red dashes = training residuals · drag the slider

05 / Regularization as Deliberate Bias

Paying a Little Bias to Buy a Lot of Calm

If variance is the disease, one cure is to stop the coefficients from growing wild. Regularization adds a penalty on the size of $\beta$ to the loss, so the optimizer must trade fit against restraint:

LASSO · L1 penalty

\hat{\beta} = \arg\min_{\beta} \Big\{ \sum_{i=1}^{n} (y_i - x_i^\top \beta)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \Big\}

The absolute value has a sharp corner at zero, so weak coefficients snap exactly to zero — the model selects features and shrinks at once.

RIDGE · L2 penalty

\hat{\beta} = \arg\min_{\beta} \Big\{ \sum_{i=1}^{n} (y_i - x_i^\top \beta)^2 + \lambda \sum_{j=1}^{p} \beta_j^2 \Big\}

A smooth bowl instead of a corner: coefficients shrink toward zero but rarely reach it. Every feature keeps a small say in the answer.

Read $\lambda$ as the same dial as $k$ in Section 04, just wearing a different costume:

\lambda \uparrow \; \Rightarrow \; \text{Bias} \uparrow, \; \text{Var} \downarrow

Why would you ever accept bias on purpose? Because variance usually collapses faster than bias grows, so total error falls. A model that is slightly wrong in a predictable direction beats one that is right on average but unrecognisable from run to run — especially when you only get one run in production.

CONSTRAINT GEOMETRY · WHY L1 GOES SPARSEcorners vs arc

dashed rings = contours of the fit loss · the constrained optimum is where they first meet

06 / Diagnostics

Reading the Gap Between Train and Validation

You never get to see bias or variance directly — you only see two error numbers. Their relationship tells you which failure you are staring at, and therefore which fix is worth your afternoon.

Variance signaturetrain ≪ validation

Training error is low and validation error is far above it. The model has learned the training set, noise included, and cannot repeat the trick on rows it has not seen. The width of that gap is the variance bill.

Try, in order

· More data — the highest-leverage fix, if you can get it
· Regularize: raise λ, lower complexity, early stopping
· Simplify: fewer features, shallower tree, larger k
· Ensemble: bagging and forests average the wobble out

Bias signaturetrain ≈ validation, both high

Both errors are high and — this is the tell — they sit close together. More data will not help, because the model is already stable; it is stably wrong. The ceiling here is the model class itself, not the sample size.

Try, in order

· Better features — engineer what the model cannot infer
· A richer model class: interactions, splines, depth
· Train longer, or loosen over-aggressive regularization
· Check the labels: σ² is a floor, but a broken target is not

The one-line heuristic

If validation is much worse than training, attack variance. If both are bad together, attack bias. And remember the third term from Section 02: whenever your validation error flattens out onto a floor that neither more data nor a bigger model moves, you have probably met $\sigma^2$ — and the honest move is to stop spending compute, not to keep tuning.

Do not tune on test

Every decision you make by peeking at a score leaks information into the model. Keep a validation split for the bias-variance decisions above, and touch the test set once, at the end — otherwise your variance estimate is really just the memory of your own tuning.

← Topic 10: Random Forests Topic 12: Train, Test & Validation →