Logarithms for Machine Learning

A logarithm asks which power turns the base into the number, compressing thousand-fold ranges into small readable steps. Entropy and log-loss both run on it.

The marks 1, 10, 100 and 1000 sit evenly spaced on a log axis because each step adds one power of ten, compressing a thousand-fold range into three equal hops

Why Does This Exist?

Some quantities refuse to sit on a straight axis. A probability of $10^{-9}$ and a learning rate of $10^{-2}$ differ by seven orders of magnitude, and a linear plot puts one of them at zero. Logarithms compress multiplication into addition, so thousand-fold gaps become readable steps, loss curves straighten, and products of tiny probabilities become sums a float can actually hold.

That last one is load-bearing. Multiplying a thousand probabilities underflows to zero; adding their logs stays in range. Entropy is defined as expected log-surprise, and cross-entropy is the loss that results, so this page is the prerequisite both of them assume.

The one idea you need first. An exponent counts repeated multiplication: $2^{10} = 1024$ . The logarithm asks the question backwards: 10 is the power that turns 2 into 1024. If exponents and roots feel shaky, read that page first and this one becomes vocabulary.

Think of It Like This

Counting digits instead of the number

Nobody reads 1,000,000 by counting zeros one at a time; you notice it has 7 digits. The digit count is a base-10 logarithm in disguise: 10 has 2 digits, 100 has 3, each extra digit is one more multiplication by 10.

That is what a log axis does to a plot. The marks 1, 10, 100, 1000 sit evenly spaced because each step multiplies by 10, and a curve climbing them steadily is growing exponentially. You read the growth in the spacing instead of the raw heights.

Where it stops: digit counts are whole numbers, while logarithms are continuous, $\log_{10}(500) \approx 2.7$ . The analogy shows the compression; the arithmetic below shows the smooth version ML actually uses.

How It Actually Works

The definition: $\log_b(x) = y$ exactly when $b^y = x$ , with $x > 0$ and base $b > 0$ , $b \neq 1$ . In plain words: the log is the power the base needs. $\log_2(1024) = 10$ because $2^{10} = 1024$ . $\log_{10}(1000) = 3$ because $10^3 = 1000$ . Three bases dominate: base 2 for bits, base 10 for human scales, and base $e \approx 2.718$ for calculus, written $\ln$ . Change between them with $\log_b(x) = \ln(x)/\ln(b)$ .

Three rules do all the work, and each mirrors an exponent law. Products split into sums: $\log(ab) = \log(a) + \log(b)$ . Quotients split into differences: $\log(a/b) = \log(a) - \log(b)$ . Powers drop their exponent in front: $\log(a^n) = n\log(a)$ . That last rule is the underflow fix: a product of a thousand probabilities becomes a sum of a thousand logs.

Worked example: bits, digits and one natural log

Bits: $\log_2(1024) = 10$ , since $2^{10} = 1024$ . A fair coin flip carries $-\log_2(0.5) = 1$ bit of surprise, because $2^{-1} = 0.5$ .

Digits: $\log_{10}(1000) = 3$ , the three multiplications hiding in those zeros.

Natural: $\ln(e^2) = 2$ by the power rule, since $\ln(e) = 1$ . And a product check: $\log_{10}(2 \times 500) = \log_{10}(2) + \log_{10}(500) \approx 0.301 + 2.699 = 3.000$ , which matches $\log_{10}(1000) = 3$ . The rules agree with direct computation.

Code

1import math2
3print(math.log2(1024))          # -> 10.04print(math.log10(1000))         # -> 3.05print((1e-300) ** 2)            # -> 0.06print(round(math.log(1e-300) * 2, 2))  # -> -1381.55

The third line is the catastrophe: squaring a tiny probability underflows to exactly zero, and every log after that is negative infinity. The fourth line does the same work in log space, where the answer $-1381.55$ fits comfortably. Multiply probabilities, add log-probabilities.

Watch Out For

Taking the log of zero or a negative

$\log(0)$ is not zero, it is negative infinity, and the log of a negative is undefined over the reals. A softmax output that underflows, or a count matrix with empty cells, hands exactly these inputs to the loss. NumPy returns -inf with a warning and keeps going, so the NaN surfaces ten steps later in the gradients.

Add a small epsilon inside the log: math.log(p + 1e-12). Clamp model outputs away from exact 0 and 1 before any log touches them.

Mixing bases without converting

$\ln(1000) \approx 6.91$ while $\log_{10}(1000) = 3$ ; same input, different answer, and the ratio is always $\ln(10) \approx 2.303$ . An entropy reported in nats against a baseline in bits looks like a different model, and a hand-rolled learning-rate schedule in the wrong base decays 2.3 times too fast.

Name the base in code (math.log2, math.log10, math.log for natural) and in prose. When comparing two numbers, convert first with the change-of-base rule.

The Quick Version

$\log_b(x)$ is the power the base needs: $b^y = x$ , defined only for positive $x$ .
Products become sums, quotients become differences, powers drop their exponent in front.
Base 2 measures bits, base 10 measures human scale, base $e$ is the calculus default.
Add log-probabilities instead of multiplying probabilities; products of tiny values underflow to zero.
Logs of zero or negatives are undefined: clamp with epsilon before the log, not after the NaN.