03 / Resources/Validation · Cross-Validation

Interactive Diagnostic Essay · Topic 05

K-Fold Partitioning & Generalization

One test split is one lottery ticket. K-fold makes every data point take a turn as the examiner — watch the holdout rotate forever, then drive the machine yourself.

01 / The Rotating Examiner

Five Splits, One Honest Score

A single train/test split suffers from sample variance: a lucky split makes a weak model look brilliant; an unlucky one buries a strong model. The fix is to hold a tournament:

1. Chop the dataset into $K$ equal folds.
2. Train on $K-1$ folds, validate on the one left out.
3. Rotate the holdout fold until every fold has been the examiner once.
4. Average the $K$ scores — now every point was validated exactly once, by a model that never saw it.

In the loop on the right, watch the orange validation fold march down the diagonal. Each flash is a fresh model trained from scratch — nothing about the model carries over between rounds, which is exactly why the average is trustworthy.

FIVE-FOLD MEAN ERROR:14.76%

SPREAD ACROSS FOLDS:13.8% – 16.0%

TrainValidation

auto-looping · 5s cycle

SPLIT #1

VAL

13.8%

SPLIT #2

VAL

15.2%

SPLIT #3

VAL

14.1%

SPLIT #4

VAL

16.0%

SPLIT #5

VAL

14.7%

CV score = mean of the five flashes = 14.76% ± 0.39%

02 / Take The Controls

Change K, Watch the Variance

FOLDS (K):

Switch between $K = 3, 5, 10$ . Larger $K$ means more training data per round (less biased estimate) but higher variance between folds and more compute. The trade-off ends at Leave-One-Out ( $K = N$ ): maximal data, maximal variance, maximal cost.

ACTIVE ITERATION:ROUND 1 OF 5

MEAN CV ERROR:14.76%

STANDARD ERROR (SE):±0.393%

Train Folds (4)Validation Holdout (1)

Fold 1

SPLIT #1Holdout Loss: 13.8%

VAL

SPLIT #2Holdout Loss: 15.2%

VAL

SPLIT #3Holdout Loss: 14.1%

VAL

SPLIT #4Holdout Loss: 16.0%

VAL

SPLIT #5Holdout Loss: 14.7%

VAL

03 / The Rulebook

Variants, Formulas & the 1-SE Rule

STRATIFIED K-FOLD

Preserves class ratios inside every fold — the default choice for imbalanced data. Without it, a fold may contain almost no positives and report nonsense.

SHUFFLE FIRST

If data is ordered (by time, by class, by source), unshuffled folds become systematically different. Always shuffle — unless order is information, as in time series.

NESTED CV

Tuning hyperparameters on the same CV you report is leakage. Nested CV runs an inner loop for tuning inside an outer loop for honest evaluation.

The overall cross-validation generalization score aggregates the out-of-fold validation loss across all $K$ splits:

CV Error Aggregation

\text{CV}_{(K)} = \frac{1}{K} \sum_{k=1}^K \mathcal{L}_k = \frac{1}{K} \sum_{k=1}^K \left( \frac{1}{N_k} \sum_{i \in \mathcal{C}_k} \ell(y_i, \hat{f}^{(-k)}(x_i)) \right)

The One-Standard-Error Rule: when comparing models, don't blindly take the minimum. Pick the simplest model whose error sits within one standard error of the best — it will generalize more reliably:

1-SE Selection Boundary

\text{Error}_{\text{candidate}} \le \text{Error}_{\text{best}} + \text{SE}(\text{Error}_{\text{best}})

This estimator feeds every experiment in Topics 06–09: the tree depths, forest sizes, and network capacities ahead are all chosen by exactly this loop.

04 / The Degenerate Limit

Leave One Out, or K = N

Put the single hold-out set back under the lamp. One partition, one trained model, one number — and that number is only as stable as the split. Move a handful of rows between train and validation and the measured error can swing by more than the real gap between two competing models. It is a high-variance estimate because it is built from a single draw.

Leave-One-Out Cross-Validation pushes the dial to its limit: set $K = N$ , one fold per observation. Each round trains on $N-1$ points and validates on the single point left out. Rotate until every point has had a turn as the examiner, then average the $N$ scores. Nothing is held back — every model sees almost the entire dataset.

The appeal is genuine: the estimate is nearly unbiased. Each training set is the full dataset minus one point, so LOOCV scores the model you would actually ship, not one starved of 20% of the data. But the $N$ validation scores are not independent. Consecutive models differ by a single training point, so their errors move together. Correlated scores do not average down — the standard error barely shrinks — and the estimate stays noisy. You bought that noise with $N$ full fits.

That is why nobody ships LOOCV for large $N$ : $N$ rounds at $N \times$ the compute of one fit, for an estimate no steadier than a cheap $K$ would give. It survives only where the arithmetic changes — tiny datasets where $N$ is small, and models such as linear or ridge regression where every leave-one-out residual can be recovered from a single fit.

LOOCV · Cost Ledger

ROUNDS TO FITN

TRAIN SET PER ROUNDN − 1 points

VALIDATION PER ROUND1 point

BIAS OF ESTIMATENEAR ZERO

VARIANCE OF ESTIMATEHIGH

TOTAL COMPUTEN × one fit

N = 24 POINTS · ONE HELD OUT PER ROUNDN rounds

ROUND 1HOLDOUT = POINT 4

ROUND 2HOLDOUT = POINT 12

ROUND 3HOLDOUT = POINT 20

…and so on for all N rounds. Every trained model sees N − 1 points; every score rests on exactly one.

K at a glance · rounds, training share, bias and variance direction

K = 3

ROUNDS3

TRAIN SHARE66.7%

BIASHIGH ↓

VARIANCELOWER ↑

K = 5

ROUNDS5

TRAIN SHARE80.0%

BIASMODERATE ↓

VARIANCEMODERATE ↑

K = 10

ROUNDS10

TRAIN SHARE90.0%

BIASLOW ↓

VARIANCEHIGHER ↑

K = N

ROUNDSN

TRAIN SHARE(N − 1) / N

BIASNEAR ZERO ↓

VARIANCEHIGHEST ↑

05 / The Practical Rule

Choosing K

$K$ is a dial with a failure mode at each end. Turn it down and every training set is small: the model you score is systematically worse than the one you will actually ship, so the estimate reads pessimistic and lurches between folds. Turn it up and each training set approaches the full dataset: the estimate turns optimistic and stable, but you paid for $K$ fits to measure a model that is already almost the shipped one.

Plot measured error against $K$ and you get a rough U: bias falls monotonically as training sets grow, variance climbs as the rounds become near-copies of one another, and the sum dips in between. The useful choice is the flat bottom of that U, not either wall.

Between roughly 5 and 10 the bottom is flat enough that moving stops paying. Five folds train on 80% of the data and cost five fits; ten folds train on 90% and cost ten. Past ten the estimate barely moves and you are mostly buying arithmetic — which is why $K = 5$ and $K = 10$ are the defaults in essentially every library, tutorial and paper.

ESTIMATE ERROR vs K9s loop · bias fades as variance grows

Three splits that break the naive shuffle

STRATIFIED K-FOLD

Random folds can starve a fold of the minority class: with a 2% positive rate, some folds land near zero positives and report a meaningless score. Stratified K-fold samples each fold so its class ratio mirrors the dataset. Reach for it whenever the target is imbalanced.

GROUPED K-FOLD

Rows are not always independent. Ten scans from one patient scattered by a random split means the model trains on nine and is validated on the tenth — a near-copy it has already memorized. Grouped K-fold keeps every group wholly on one side, so the score reflects unseen patients, not unseen rows.

TIME-SERIES SPLITS

Ordered data must never be shuffled: validating on Tuesday with a model trained through Wednesday leaks the future backwards into the past. Rolling and expanding-window splits always train on what came before and validate on what comes next, fold by fold.

← Topic 04: ROC & AUC Topic 06: Decision Trees & Random Forests →