ROC & AUC: Diagnostic Power

One threshold is one opinion. The ROC curve is every opinion at once — watch the threshold sweep the class distributions and trace the entire curve on loop, then drive the sweep yourself.

01 / The Autopilot

Sweeping the Threshold Traces the Curve

On the left, two class score distributions overlap. The sweeping $\tau$ line decides: everything to its right is predicted positive. On the right, that same sweep — on the same clock — traces the ROC curve point by point. Where the distributions barely overlap, the curve climbs steeply; that climb is the model's diagnostic power.

Class 0 (Negative)Class 1 (Positive)

8s loop · τ sweep ↔ ROC trace, synced

τ far left → (FPR 1, TPR 1) corner · τ far right → origin (0,0)

02 / Take The Controls

Thresholding Class Distributions

A binary classifier produces continuous risk scores $S(x) \in [0, 1]$ . To assign a discrete category ( $\hat{y} \in \{0, 1\}$ ), we apply a decision threshold $\tau$ :

\hat{y} = \begin{cases} 1 & \text{if } S(x) \ge \tau \\ 0 & \text{if } S(x) < \tau \end{cases}

Moving $\tau$ left captures more positive instances (higher sensitivity) at the cost of admitting false alarms. Moving it right reduces false alarms but causes false negatives. Topic 03's confusion matrix is computed at one threshold — ROC zooms out to all of them.

THRESHOLD (τ)0.50

DISTRIBUTION SEPARATION (d')1.80

Class 0 (Negative)Class 1 (Positive)

τ = 0.50

TRUE POSITIVE RATE89.5%

FALSE POSITIVE RATE14.7%

SPECIFICITY (1-FPR)85.3%

AREA UNDER CURVE0.948

03 / Parametric Space

The Receiver Operating Characteristic

The ROC curve plots $\text{TPR}$ against $\text{FPR}$ across every continuous choice of threshold $\tau \in [0, 1]$ .

Area Under Curve (AUC)

\text{AUC} = \int_{0}^{1} \text{TPR}(\tau) \, d\text{FPR}(\tau) = P(S(X^+) > S(X^-))

Probabilistic Interpretation: the AUC equals the probability that the classifier ranks a randomly chosen positive observation higher than a randomly chosen negative one. 0.5 is coin-flipping; 1.0 is perfect ranking.

Increasing the separation $d'$ pushes the curve toward the upper-left corner $(0, 1)$ — the holy grail where every true positive is caught with zero false alarms.

ROC vs Precision-Recall (Topic 03)

ROC's FPR divides by the huge negative count, so it stays optimistic under heavy imbalance — a 1% FPR can hide thousands of false alarms. For rare-event detection (fraud, disease), the precision-recall curve tells the harsher, more honest story.

04 / Reading the Shape

What Makes a Good ROC Curve?

Before comparing models, learn the four shapes a ROC curve can take. Every classifier you will ever train lands somewhere between the first and the last of them.

PerfectAUC 1.00

TPR = 1 at every FPR — every positive ranked above every negative.

UsefulAUC 0.88

Bows toward the top-left corner. The real-world result worth shipping.

RandomAUC 0.50

TPR = FPR everywhere: the classifier is guessing, not ranking.

Worse Than RandomAUC 0.35

Sags below the diagonal — invert its scores and it improves.

The 0.5 baseline. A curve that tracks the dashed diagonal has $\text{TPR} = \text{FPR}$ at every threshold: the score carries no information, so AUC is 0.5. Below the diagonal means the ranking is systematically inverted — AUC drops under 0.5, and simply negating the scores would push it back above. An AUC of exactly 0.5 on real data is suspicious; an AUC of exactly 1.0 usually means leakage, not genius.

The probabilistic reading. AUC is the probability that a randomly chosen positive is scored above a randomly chosen negative: $\text{AUC} = P(S(X^{+}) > S(X^{-}))$ . This is why it is called a measure of ranking and not of accuracy. It also gives a cheap estimator — tally the fraction of positive-negative pairs the model orders correctly — which is exactly the Wilcoxon–Mann–Whitney U statistic.

The geometric reading. Literally, AUC is the area under the curve: $\int_{0}^{1} \text{TPR} \, d(\text{FPR})$ , an average of sensitivity across the entire false-positive range. One number, no threshold commitment — the convenience that also becomes its blind spot in the next section.

05 / The Fine Print

When AUC Misleads

AUC is a single number squeezed out of a whole curve — and any single number can hide the thing you care about. Three failures recur in practice.

SAME AREA · TWO DIFFERENT MODELS8s loop · area trades shape, never size

Both curves enclose AUC 0.67 · identical area, opposite strengths

(a) Imbalance Flatters ROC

FPR divides by the entire negative class, so when negatives dominate, a flood of false alarms still reads as a tiny rate. Take 1,000,000 transactions with 100 fraudulent ones: a model that catches 90 (recall 0.90) while raising 1,000 false alerts sits at FPR = 1,000/999,900 ≈ 0.001. The ROC point (0.001, 0.90) looks near-perfect. Precision is 90/1,090 ≈ 8.3% — 92% of the alerts are noise. The PR curve is the honest view; this is exactly the bridge built in Topic 03, section 07.

(b) Ranking ≠ Calibration

AUC only checks the order of the scores, never their magnitude. A model that emits 0.51 for every positive and 0.49 for every negative has AUC 1.0 and probabilities that are worthless: a policy acting above 0.30 never fires, and a risk score that never exceeds 0.51 cannot feed an expected-loss calculation. Whenever a decision needs real probabilities — pricing, triage bands, cost models — read a reliability diagram and the Brier score. AUC is blind to calibration.

AUC averages TPR across the whole false-positive range, including regions your workflow never visits. One triage model can be excellent only inside the 10–20% risk band where your team actually acts and poor everywhere else; another can be uniformly mediocre. Both can land at AUC 0.80. If your application lives in a narrow operating band, the global average hides it — inspect the partial AUC, or zoom the PR curve to that band.

← Topic 03: Confusion Matrix, Precision-Recall & F1 Topic 05: K-Fold Cross-Validation →