03 / Resources/Foundations · Train, Test & Validation

Interactive Diagnostic Essay · Topic 12

Train, Test & Validation: The Honest Split

A model graded on the rows it memorised always looks brilliant. This essay shows why one dataset must become three — and how a single careless peek can quietly turn your final number into a fiction.

01 / The Cardinal Sin

Never Grade The Exam You Taught

A model earns its keep by being right about data it has never seen. Train and evaluate it on the same rows and that question silently disappears — you are no longer asking whether it generalises, only whether it can recognise what it was already handed.

The degenerate extreme makes this obvious. A 1-nearest-neighbour classifier simply stores every training point. Ask it to predict a stored point and its nearest neighbour is itself, at distance zero:

\text{Train accuracy}(k{=}1) = \frac{1}{n}\sum_{i=1}^{n} \mathbb{1}\!\left[\hat{y}(x_i) = y_i\right] = 1

A degree-12 polynomial threaded through 13 points is the same trick in a different costume: zero training error, and wild oscillation in the gaps between the points. Neither model has learned anything about cats, dogs, or digits. Each has only learned the answer key.

The tell

If your score keeps climbing every time you add capacity, that is not progress — it is the definition of memorisation. The only honest question is what happens off-book, on rows the model has never had the chance to store.

02 / The Three Sets

Three Jobs, Three Datasets

So we split the data. Not into two parts but three, each with a different job and a different rule for when it may be read. The discipline is deliberately asymmetric: training and validation are working sets, while the test set is a sealed envelope opened once.

Training Set60–80%

The model fits its parameters here. Read it as often as you like — it is the only set whose feedback is allowed to change the model.

Validation Set10–20%

Used to compare models and choose hyperparameters. Consulted many times; after each look you may change your mind. This is the set that does model selection.

Test Set10–20%

Locked away until the very end. Read exactly once, after every choice is frozen. The number you report comes from here.

ONE DATASET → THREE SETS8s loop

TRAIN

VAL

TEST

fit parametersconsult freelyread once

trainvalidation · many lookstest · one reading

validation flickers — it is consulted repeatedly · test stays dim until the final flash

03 / Why Validation Is Not Optional

Two Kinds Of Parameters

A model holds two kinds of parameters. The first — weights and biases — the optimiser fits for you by descending the loss on the training set. The second you choose by hand before training begins: tree depth, the regularisation strength $\lambda$ , the number of neighbours $k$ , the number of layers. Call these hyperparameters.

Now ask how you pick them. If you compare candidates on the test set and keep the winner, the test set has been used as a training signal. Each comparison teaches you something about its noise, and the number you report becomes the best of many tries rather than an unbiased reading of one frozen model.

Select On Validation, Report On Test

\theta^\star = \arg\min_{\theta \in \Theta} \text{Err}_{\text{val}}(h_\theta), \qquad \text{report } \text{Err}_{\text{test}}(h_{\theta^\star})

This is the core of the article: choosing hyperparameters by test performance quietly promotes the test set into a second training set. The final number then measures how well you fitted the test set, not how well the model will travel.

Fit by the optimiser

Weights and biases, learned by gradient descent. The training set exists to shape these.

Chosen by you

Depth, $\lambda$ , $k$ , learning rate. The validation set exists to rank these — never the test set.

Every configuration you try on the test set leaks a little information out of it. Try enough of them and the best test score is optimistic by construction.

04 / Splitting in Practice

Random — Until It Must Not Be

The common conventions are a 60 / 20 / 20 split for large datasets and 70 / 15 / 15 when data is tighter. The exact ratio matters less than keeping the test set big enough for a stable estimate and the validation set big enough to tell candidates apart. And the split must be random — except in the three cases below, where randomness is exactly the bug.

Stratifiedclass imbalance

With a rare class — fraud, disease — a plain random split can leave almost none of it in validation. Sample each class in proportion within every split.

Groupedrows are not independent

Several visits per patient, many sessions per user. Keep every group entirely inside one split, or the model recognises the patient instead of the pattern.

Time-orderedtemporal data

Shuffle nothing. Train on the past, validate and test on the future. A random split lets the model peek at tomorrow to predict yesterday.

The rule behind all three: your split must mimic how the model will meet the world in production. If the future never appears in training once deployed, it must not appear in training here either.

05 / Leakage

The Doors Information Sneaks Through

Even with three clean sets, information leaks. Leakage is any path by which knowledge of the test data reaches training. It almost always inflates the score, and it is almost always invisible until production.

Scaling before splitting

Computing the mean and standard deviation over the whole dataset leaks test statistics into training. Fit the scaler on the training split, then apply that same transform to validation and test.

Target leakage

A feature that secretly encodes the label — account_closed_date predicting churn, or a field filled in only after the outcome was known.

Duplicate rows

The same record lands in both train and test, so the model has effectively already seen the test point. De-duplicate before you split.

Tuning on the test set

Trying many hyperparameter settings and reporting the best test score. Each peek nudges the model toward the test set and the reported number upward.

The scaling rule

z = \frac{x - \mu_{\text{train}}}{\sigma_{\text{train}}}

$\mu$ and $\sigma$ come from training rows only.

A useful audit for any pipeline: could a single test row have influenced anything in training — a mean, a feature, a hyperparameter, a duplicate? If yes, the estimate is optimistic.

06 / When Data Is Scarce

Borrowing From Yourself

A single split carries a hidden cost: the model never trains on the validation rows, and you never learn how sensitive the score is to which rows landed where. On small datasets that waste is real, and the estimate is noisy.

Cross-validation answers this by rotating the job. Split the data into $k$ folds; train on $k-1$ and validate on the held-out fold; repeat $k$ times so every row takes a turn as validation. The average is a more stable estimate, and the spread across folds is one you can actually quote.

Rotate The Validation Role

\text{CV}_k = \frac{1}{k}\sum_{i=1}^{k} \text{Err}_{\text{val}}^{(i)}

The test set still stays sealed — cross-validation replaces the validation role, not the final exam. Topic 05 works through the mechanics, the choice of $k$ , and the grouped and time-series variants that keep the folds honest.

Rule of thumb

Small data, many candidates → cross-validation. Large data, few candidates → a single 60 / 20 / 20 split is simpler and just as honest. Either way, the test set is read once.

Adapted from MLU-Explain (Amazon, CC BY-SA 4.0).

← Topic 11: Bias-Variance Tradeoff Topic 13: Double Descent →