Naive Bayes

Treat every word as independent evidence for a class, multiply the per-word chances, and pick the largest product. The chances come out wrong but the winner is usually right.

Three ticket words score 0.40 times 0.24 times 0.0125 for billing and 0.026 times 0.026 times 0.256 for outage, so log sums minus 6.74 beats minus 8.69 only after smoothing

Why Does This Exist?

A support inbox receives 300 tickets a day and a human has routed 40,000 of them to billing or outage. The team wants routing before anyone reads the message.

Bayes theorem gives the exact answer: the chance a ticket is billing given its words is proportional to how often billing occurs times how likely those words are in billing. The second term needs the count of the exact full message, which is zero for almost every future ticket, since the space of messages grows faster than any inbox. More labels never fix that sparsity.

Naive Bayes replaces the full-message chance with a product over single words, each counted separately. This page covers that product, why the independence claim is false, and the two repairs that keep it working. The count-based text baseline it pairs with is bag of words. Its discriminative counterpart is logistic regression. How winners are graded lives in classification metrics.

One grounding term: the prior $P(C)$ means how often class $C$ occurs before seeing words, estimated by counting tickets per class.

Think of It Like This

Eight witnesses, one rumour

A panel must decide which of two contractors damaged a wall. Eight people name the first contractor and four name the second, which sounds decisive at eight against four. Then the panel learns all eight stood in one room and heard it from the same foreman: one observation counted eight times.

Was the strength real? No, the confidence is inflated. Was the verdict still right? Quite possibly, because the foreman saw something and every copy pushed the same way. That is the independence assumption: correlated evidence counted twice, wrecking the number while usually leaving the winner standing. Where it stops: witnesses can be cross-examined about their source, while word counts cannot, so the model needs smoothing and logs added by hand.

How It Actually Works

The product rule

Call the class $C$ , meaning billing or outage. Call $w_i$ the $i$ -th word, meaning invoice or charged. Call $P(C)$ the prior, meaning the share of tickets in $C$ . Call $P(w_i \mid C)$ the likelihood, meaning how often $w_i$ appears in $C$ . The score is:

P(C \mid w_1, \ldots, w_n) \propto P(C) \prod_{i=1}^{n} P(w_i \mid C)

Training is counting: tally tickets per class for $P(C)$ and word occurrences per class for each $P(w_i \mid C)$ . No optimizer and no learning rate. The multinomial variant counts occurrences, the Bernoulli variant counts presence or absence, and the Gaussian variant fits a mean and variance per continuous feature.

Worked example: invoice, charged, timeout

Vocabulary has six words. Billing counts are $[31, 24, 18, 1, 0, 0]$ and outage counts are $[1, 2, 1, 27, 19, 22]$ , in the order invoice, refund, charged, crash, timeout, stacktrace. The ticket holds invoice, charged, and timeout. Add one pseudo-count to every word in every class. Billing total becomes $74 + 6 = 80$ . Outage total becomes $72 + 6 = 78$ .

Billing likelihoods:

32/80 = 0.40, \quad 19/80 = 0.2375 \approx 0.24, \quad 1/80 = 0.0125

Outage likelihoods:

2/78 \approx 0.0256 \approx 0.026, \quad 2/78 \approx 0.026, \quad 20/78 \approx 0.2564 \approx 0.256

Add logs rather than multiplying. Billing log sum is $\log 0.40 + \log 0.24 + \log 0.0125 \approx -0.916 - 1.427 - 4.382 = -6.725 \approx -6.74$ . Outage log sum is $\log 0.026 + \log 0.026 + \log 0.256 \approx -3.650 - 3.650 - 1.362 = -8.662 \approx -8.69$ . Billing wins by about two log units. Without smoothing the third billing factor is $0/74 = 0$ , its log is minus infinity, and outage wins at $-9.89$ by default. Those are the diagram numbers.

Why wrong numbers still rank correctly

Invoice and charged travel together, so the model double-counts one piece of evidence and pushes billing from a deserved 0.70 to 0.9999. Ranking only asks which class scores higher, and the extra push lands on the already-winning side rather than flipping the order. That split explains the reputation: respectable classifier, poor probability estimator. Fix the numbers afterward with probability calibration when a threshold needs them.

Code

The point is the check: score the same ticket unsmoothed and smoothed, and confirm minus infinity flips to billing winning by two log units.

1import numpy as np2
3vocab = ["invoice", "refund", "charged", "crash", "timeout", "stacktrace"]4counts = np.array([[31.0, 24.0, 18.0, 1.0, 0.0, 0.0],5                   [1.0, 2.0, 1.0, 27.0, 19.0, 22.0]])6ticket = [0, 2, 4]7
8
9def log_scores(alpha: float) -> np.ndarray:10    totals = counts.sum(axis=1, keepdims=True) + alpha * counts.shape[1]11    with np.errstate(divide="ignore"):12        log_probs = np.log((counts + alpha) / totals)13    return log_probs[:, ticket].sum(axis=1)14
15
16for alpha in (0.0, 1.0):17    billing, outage = log_scores(alpha)18    winner = "billing" if billing > outage else "outage"19    print(f"alpha={alpha:.0f} billing {billing:.2f} outage {outage:.2f} -> {winner}")20# -> alpha=0 billing -inf outage -9.89 -> outage21# -> alpha=1 billing -6.74 outage -8.69 -> billing

The printout matches the hand arithmetic. Change the pseudo-count to 0.5 and billing still wins, by a smaller margin. Drop the log and multiply raw chances on a 200-word ticket and the product underflows toward zero, which is why numerical overflow matters here.

Watch Out For

Trusting the 0.9999 as a real chance

The model returns 0.9999 and a router auto-approves above 0.95. Correlated words push scores toward the ends, and longer tickets push harder than short ones, so confidence histograms pile at 0 and 1 while accuracy sits near 0.88.

Symptom: high-confidence bucket accuracy far below its label, growing worse with ticket length. Fix: use only the argmax for routing, and calibrate on held-out data or switch to logistic regression when a downstream step needs a number it can price.

Shipping without smoothing or an unknown-word rule

Training and validation share a vocabulary, so no likelihood is ever zero during testing and the gap stays hidden. Then a customer writes a new product name, every class scores minus infinity, and argmax returns index zero by array order.

Symptom: a burst of misroutes after a launch or rename, all decided by position rather than evidence. Fix: smooth every class with a pseudo-count and decide the out-of-vocabulary rule up front: dropping the unknown word is defensible, letting it zero the row is never a plan.

The Quick Version

Naive Bayes scores a class as prior times the product of per-word likelihoods, trained by counting.
The independence assumption is false and usually harmless for ranking: double-counted evidence inflates the winner without flipping it.
Our ticket scores minus 6.74 for billing against minus 8.69 for outage after smoothing, so billing wins by two log units.
Without smoothing the unseen timeout zeroes billing to minus infinity and outage wins by default.
Use multinomial counts for text, Bernoulli presence for short fields, and Gaussian means for continuous features.
The output is a ranking signal, not a priceable chance, and every class needs smoothing plus an unknown-word rule.