Double Descent
For decades the rule was simple: too much capacity and you overfit. Then models with far more parameters than data points started winning. This essay follows the error curve past the point it was supposed to explode — and finds it falling again.
A Curve With One Bottom
Topic 11 gave us the classic tradeoff. Sweep model capacity from tiny to large and test error traces a single U: a stiff model misses real structure (bias), a flexible one chases noise (variance), and somewhere between them sits the bottom of the bowl.
The story comes with a warning attached. Increase capacity far enough and you can drive training error to exactly zero — the model has enough freedom to memorise every label you gave it. Classical theory treats that moment as the worst place to stand. Fit the noise perfectly and you should generalise terribly.
Once a model can interpolate the training set — zero training error — test error should spike and stay high. Everything to the right of that point is a model you should not ship.
That claim is where the interesting part begins, because for modern model classes it turns out to be false.
Take The Capacity Dial
Drag the dial from a model with a handful of parameters to one with far more than it has training rows. Watch three things at once: the training error pinning to zero, the test error spiking at the threshold — and then, against every classical instinct, falling again.
The Marker Walks The Curve
Same curve, on a loop. The accent dot traces test error from a stiff model to an enormous one; the dashed ink line draws the training error in behind it. Watch the dot climb into the spike at the threshold — then keep going and come back down.
Left of the threshold the classical rule holds · right of it, it stops predicting anything useful
Not All Zero-Training-Error Models Are Equal
The classical claim assumed that “fits the training set perfectly” picks out one model. It does not. Past the threshold there is an entire family of parameter settings that all drive training error to zero — so the training loss alone no longer says which one you got.
What breaks the tie is the optimiser. Gradient descent started from a small initialisation does not wander into an arbitrary interpolating solution; it converges toward the one with the smallest parameter norm — the “closest” interpolating fit to where it started. That implicit preference is a form of regularisation nobody wrote down, and it is doing real work:
Among all the fits that nail the training data, prefer the smoothest. A minimum-norm interpolating fit is typically much less wiggly than the pathological memoriser the classical picture imagined — and a smoother fit is exactly what generalises.
There is a second effect stacked on top. In the over-parameterised regime the model has far more directions to move in than it has data constraints, so many of those extra directions stay near their initialisation and never get shaped by noise at all. Capacity you cannot steer is capacity that cannot overfit.
Cannot fit the signal. Train and test error both high.
Just enough freedom to interpolate, and the optimiser has almost no spare directions to be smooth in. Worst generalisation.
Plenty of interpolating solutions, and the optimiser lands on a smooth one. Test error falls again.
Bias-variance was never wrong — it was describing a different axis. What matters is not raw parameter count but effective capacity, which the optimiser and the architecture decide as much as the size does.
The Fine Print
Double descent shows up clearly for some model families and optimisers, weakly or not at all for others. It is an observed phenomenon with conditions, not a law that replaces the U-curve everywhere. Do not assume it applies to your problem because it applied to someone else's.
The spike at the threshold is real and it is the worst place to be. If your model sits anywhere near it, the second descent is not a rescue — you have to travel all the way through the bad region to reach the good one.
That over-parameterised models generalise well does not mean they cost nothing. They need the data, the compute and the engineering to train. The useful reading is “large models can be worth it”, never “larger is always better”.
The only way to know where your curve turns is to sweep capacity and plot the result. That plot is a validation-set exercise — see Topic 05 for how to get an honest estimate.
The Practical Read
Keep the classical instinct — it is still the right default for the small, deliberately constrained models most tabular problems want. A regularised linear model or a shallow tree lives nowhere near the threshold, and the U-curve governs it completely.
Update the part that said zero training error is automatically fatal. It is a warning sign, not a verdict: what matters is how the zero was reached. Reached by memorising, it generalises badly. Reached by a smooth minimum-norm fit in a hugely over-parameterised space, it can generalise very well indeed.
And keep the habit that both pictures agree on: judge a model on data it has not seen. Every curve on this page was measured on a held-out set — that discipline is the one thing double descent never overturns.
- • Test error can rise at the interpolation threshold and then fall again.
- • Past the threshold, training loss stops identifying a unique model — the optimiser picks among many.
- • Minimum-norm bias is implicit regularisation you get for free.
- • Bias-variance still holds; the axis is effective capacity, not parameter count.
- • Sweep and measure. Never infer the shape of your own curve from someone else's.
Adapted from MLU-Explain (Amazon, CC BY-SA 4.0).