Logistic Regression

It classifies, despite the name. A linear score goes in, a sigmoid squeezes it down into a probability, and the log-odds come out straight in the features.

Scores of minus 1.2, 0 and 1.2 become probabilities 0.23, 0.50 and 0.77 through the sigmoid with the boundary at zero

Why Does This Exist?

You need a yes or no with a number for how sure. Not this subscriber cancels, but 78 percent chance they cancel next month, because 78 and 51 deserve different calls.

Carry one case. Forty thousand subscribers, one feature each for days since last login, and last month outcome of stayed or cancelled.

A straight line fails here. Fit days against a 0 or 1 label and you get 1.4 and minus 0.31, which are not probabilities. Clipping hides the cause. Squared error keeps chasing already correct far points, so a few idle subscribers tilt the line where uncertain ones live.

Two words first. Odds restate probability as a ratio. Probability $0.8$ is $4$ to $1$ . Log odds take the log of that ratio. Probabilities live in $[0, 1]$ , odds in $[0, \infty)$ , log odds across the whole line, room enough for a straight fit. The name is honest once. Logistic regression regresses log odds linearly.

This page covers score, squash, and loss. The linear starting point lives in linear regression. Scoring yes or no lives in classification metrics. Fixing the numbers after fitting lives in probability calibration.

Think of It Like This

A tug of war with a fixed pit

Two teams pull a rope. Each feature is one puller. Strength is feature value times weight, and sign picks the side. Add everyone and you get net pull, any number at all, an inch or a landslide.

Watch the flag in the middle. It cannot leave the pit. Even pull puts it centre. A small lead drifts it. A huge lead jams it against the far ditch, where one more strong puller moves it almost none. Net pull is the linear score, flag position is probability, centre line is the boundary, and the jam is why extra evidence pays little once you are sure.

Where it stops: rope and pit explain saturation but not learning. Real fitting needs a loss that keeps pushing on confident mistakes, which rope tension alone would not do.

How It Actually Works

Start with a linear score. $w$ holds one weight per feature, $x$ holds one subscriber row, $b$ is the intercept:

z = w^\top x + b

For any subscriber $z$ is one unbounded number. Squash it with the sigmoid:

\sigma(z) = \frac{1}{1 + e^{-z}}

Here $e$ is $2.718$ . Large positive $z$ drives $e^{-z}$ to zero and $\sigma(z)$ to $1$ . Large negative $z$ drives it to $0$ . At $z = 0$ it is exactly $0.5$ . Inverted, $\log \frac{p}{1-p} = z$ where $p$ is predicted probability. Log odds linear in features. All bending happens in the squash.

Worked example with three scores

Take $z = -1.2$ , $0$ , $1.2$ . Sigmoid gives $1 / (1 + e^{1.2}) \approx 0.23$ , $0.50$ , $1 / (1 + e^{-1.2}) \approx 0.77$ . Boundary sits at $z = 0$ with $p = 0.50$ . In subscriber terms, idle $20$ days maps to $-1.2$ and $0.23$ , idle $30$ days to $0$ and $0.50$ , idle $40$ days to $1.2$ and $0.77$ . Ten idle days multiplies odds by $e^{1.2} \approx 3.32$ . That multiplier is the diagram headline: three scores in, three valid odds out.

Why squared error fails twice

Through a sigmoid squared error is no longer convex in $w$ . It grows local dips where gradient descent can stick. Convexity would guarantee one bottom, and you gave it up.

Worse, it quits when you need it. Its gradient carries $\sigma^\prime(z) = \sigma(z)(1 - \sigma(z))$ , near zero at both ends. Predict $0.999$ for a stayer and you are badly wrong with almost no correction. Binary cross entropy fixes both:

L = -\big[\, y \log p + (1 - y) \log (1 - p) \,\big]

Here $y$ is $0$ or $1$ . Predict $0.999$ when truth is $0$ and $-\log(0.001) \approx 6.91$ charges hard, running to infinity as wrong confidence grows. The $\sigma^\prime$ term cancels with the log, leaving gradient $(p - y)\,x$ . Prediction minus truth times input.

Reading coefficients

One unit in feature $j$ adds $w_j$ to log odds, so it multiplies odds by $e^{w_j}$ . Weight $1.2$ triples odds at $3.32$ . It never adds a fixed probability. Tripling $0.02$ gives $0.06$ . Tripling $0.50$ gives $0.75$ . Boundary is flat at $z = 0$ . The sigmoid bends probability, not the boundary.

Code

Fit one weight by hand and confirm $0.77$ and $3.32$ .

1import math2
3def sigmoid(z: float) -> float:4    return 1.0 / (1.0 + math.exp(-z))5
6scores: list[float] = [-1.2, 0.0, 1.2]7probs: list[float] = [sigmoid(z) for z in scores]8odds_mult: float = math.exp(1.2)9
10print(f"{probs[0]:.2f} {probs[1]:.2f} {probs[2]:.2f}")  # -> 0.23 0.50 0.7711print(f"{odds_mult:.2f}")  # -> 3.32

The printout matches the hand numbers. Change $1.2$ to $2.0$ and probability moves only to $0.88$ . That jam is the pit wall in code.

Watch Out For

Separable data sends weights to infinity

Add a feature that separates classes cleanly. Loss keeps falling and never settles. Coefficients pass hundreds into thousands with longer runs.

Symptom: huge weights plus huge or missing standard errors on cleanly split data. Fix: shrink weights with ridge regression rather than iterating longer, and check for leakage first. A survey timestamp present only for cancellers separates beautifully and predicts nothing.

Treating raw output as calibrated odds

On a representative sample unregularised logistic output is close to calibrated, which is why credit teams use the number itself. Reweight for imbalance and the intercept now describes a population that never existed.

Symptom: ranking holds while every expected value calc runs high, often near $0.30$ predicting $0.04$ in the wild. Fix: bucket predictions against observed rates and correct the intercept or fit a map from probability calibration.

The Quick Version

Logistic regression is a classifier. Log odds are linear in features.
Scores minus 1.2, 0, and 1.2 become probabilities 0.23, 0.50, and 0.77.
Ten idle days multiplies cancel odds by about 3.32 in our example.
Squared error through a sigmoid is non convex and stalls when confidently wrong.
Cross entropy gradient is prediction minus truth times input.
The boundary is flat at z equals 0. Curvature lives in probability, not the cut.