03 / Resources/Classification · Logistic Regression

Interactive Diagnostic Essay · Topic 02

Logistic Regression & The Sigmoid Curve

Despite the name, it is a classifier. Watch a straight line get bent into an S, see how one threshold splits the world into two decisions, and learn the loss that keeps probabilities honest.

01 / From Line to Probability

The Sigmoid Transformation

Topic 01 predicted numbers. But “spam or not?” needs a probability — and probabilities must live in $[0, 1]$ , while a straight line $z = wx + b$ runs off to $\pm\infty$ .

The fix is a squashing function. Feed the unbounded line score $z$ into the sigmoid and it gets bent into a soft S that saturates at exactly 0 and 1:

\sigma(z) = \frac{1}{1 + e^{-z}} \quad\quad P(Y=1|x) = \sigma(wx + b)

In the animation, a prediction point slides along the input axis: far left the model is almost sure class 0, far right almost sure class 1, and near the center — where $z = 0$ — it genuinely cannot decide and outputs $0.5$ .

σ(z) SELF-DRAWING · DOT TRAVERSES FOREVERP(y=1|x)

Flat ends = confident. Steep middle = everything is decided here.

02 / The Sigmoid Lab

Steer the Curve, Fit the Data

The weight $w$ controls how steep the S is (how quickly the model becomes confident), and the bias $b$ slides the decision boundary left or right. Fit the 10 observations below by hand.

WEIGHT (w) · SLOPE SHARPNESS1.20

BIAS (b) · HORIZONTAL SHIFT-2.40

DECISION BOUNDARY (z=0):x = 2.00

LOG LOSS (BCE):0.1459

Lower the loss to win. The boundary sits where $wx + b = 0$ , i.e. $x^* = -b/w$ — a hyperplane in feature space.

Class 0 (y=0)Class 1 (y=1)

P(y=1|x)

03 / Drawing the Border

The Decision Boundary & Its Casualties

2D CLASSIFICATION · BLINKING = MISCLASSIFIEDP > 0.5 ?

A linear boundary can never be perfect — those two blinking points are where Topic 03 begins.

The sigmoid outputs a continuous probability, but the world wants verdicts. Applying a threshold (default $0.5$ ) turns the S-curve into a linear decision boundary — everything on one side is class 0, everything on the other is class 1.

In 2D that boundary is a line; in 3D a plane; in 100 dimensions a hyperplane. Same idea, same equation $w \cdot x + b = 0$ .

Why Not Just Use MSE (Like Topic 01)?

Squared error over a sigmoid creates a non-convex, wavy loss surface — gradient descent gets stranded in fake minima. Cross-entropy keeps the bowl-shaped surface that made Topic 01's descent work, and it punishes confident wrong answers without mercy: predicting $0.99$ on a true $0$ costs $-\ln(0.01) \approx 4.6$ per point and grows without bound.

Those two blinking misclassified points are exactly the FP / FN cells of the confusion matrix — Topic 03 teaches you to price them properly.

04 / The Betting Interpretation

Odds, Logits & Binary Cross-Entropy

Probability asks “how likely?”. Odds ask “how many times more likely?” — a 75% event has odds 3:1. Take the log and you get the logit, and something beautiful happens:

Log-Odds (Logit) Form

\ln \left( \frac{p}{1 - p} \right) = w \cdot x + b

The log-odds is linear in the features — logistic regression is linear regression running in odds-space, then translated back by $\sigma$ . Each weight multiplies the odds by $e^{w}$ per unit of $x$ .

To train it, maximize the likelihood of the observed labels — equivalently minimize binary cross-entropy:

Cross-Entropy Loss

\mathcal{L}(w, b) = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \ln(\hat{y}_i) + (1 - y_i) \ln(1 - \hat{y}_i) \right]

Each term of the loss is a curve, and the ball riding it tells you what the model feels: when a true-1 point gets probability near 1, the $-\ln(p)$ cost flattens to zero; let it drift toward 0 and the cost rockets to infinity.

PER-POINT COST −ln(p)

y = 1y = 0

Because the loss is convex in (w, b), gradient descent always finds the global minimum.

Multi-class? Stack sigmoids and normalize — that is softmax regression, the last layer of every neural network in Topics 07–09.

05 / Reading The Coefficients

Weights Multiply Odds, Not Probabilities

A weight is not a nudge on the probability dial. It acts on the odds — the ratio of an event happening to it not happening.

\text{odds} = \frac{p}{1-p} \qquad \text{logit}(p) = \ln \!\left( \frac{p}{1-p} \right) = w \cdot x + b

Because the logit is linear, adding one unit of $x$ adds $w$ to the log-odds — and exponentiating turns that addition into a multiplication: the odds scale by $e^{w}$ . A weight of $\ln 2 \approx 0.69$ doubles the odds; $w = 1.2$ multiplies them by $e^{1.2} \approx 3.3$ .

That multiplicative reading is the whole reason logistic regression counts as interpretable: each coefficient is a self-contained odds ratio, legible without inspecting the rest of the model.

The Scale Trap

Ranking raw coefficients across features is meaningless when the features live on different scales — a weight of 2 on a centimetre axis is not the same as 2 on a kilometre axis. Standardize first (mean 0, unit variance), then compare. Same model, comparable numbers.

ONE-UNIT PUSH · START AT p = 0.5Δ logit = w

0.50 before0.769 after

WEIGHT (w) · LOG-ODDS SHIFT1.20

ODDS MULTIPLIER e^w:×3.320

p AFTER x + 1 (FROM 0.5):0.769

WRONG · 0.5 + w1.70

Slide w negative to shrink the odds toward zero · positive to inflate them · the probability never moves additively

06 / Log Loss

Why Cross-Entropy, Not MSE

Training maximizes the likelihood of the labels you actually saw — equivalently, it minimizes log loss, one $-\ln(\hat{p})$ term per point, scored only against the truth:

\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \ln \hat{y}_i + (1 - y_i) \ln(1 - \hat{y}_i) \right]

The logarithm is the entire point. $-\ln(p)$ is nearly flat near $p = 1$ , so a confident correct answer costs almost nothing — but it plunges toward infinity as $p \to 0$ . Being confidently wrong is not merely penalized; it is catastrophically penalized.

Asymmetry · Cost In Nats

p̂	cost \| y=0	cost \| y=1
0.99	4.61	0.01
0.90	2.30	0.11
0.50	0.69	0.69
0.10	0.11	2.30
0.01	0.01	4.61

Predicting $0.99$ on a true $0$ costs $-\ln(0.01) \approx 4.61$ . Flip the blunder — predict $0.01$ on that same true $0$ — and you pay only $-\ln(0.99) \approx 0.01$ . Confidence is cheap when it is right and ruinous when it is wrong.

That explode-to-infinity edge is not a bug to be capped. It is the gradient reporting, in exact proportion, how far the model has strayed.

Watch the cost as the marker sweeps $p$ . One bar tracks the penalty for a true 1, the other for a true 0 — and at either edge one of them races off the top of the frame.

CONFIDENCE PENALTY · MARKER WALKS pLOOP · 8s

y = 1 · cost −ln py = 0 · cost −ln(1 − p)

p = 00.5p = 1

Bar height ∝ −ln p, clipped at 3 nats · one bar fills the frame as the other vanishes

Cross-Entropy + SigmoidCONVEX

A single bowl with one global minimum — gradient descent cannot get stranded, no matter where it starts.

MSE + SigmoidNON-CONVEX

A wavy surface with flat plateaus. Squared error saturates the sigmoid, vanishing gradients stall training, and the descent can settle in a fake minimum.

Pair the sigmoid with log loss and the surface stays convex; pair it with squared error and it does not. That single compatibility is why every modern classifier — up to the softmax heads of Topics 07–09 — trains on cross-entropy.

← Topic 01: Linear Regression Topic 03: Confusion Matrix, Precision-Recall & F1 →