Information Theory

Information theory prices uncertainty in bits. This hub links the entropy, coding, and tradeoff pages without re-teaching them.

Bits sit at the center with spokes to the entropy family, coding limits, bottleneck compression, and rate-distortion tradeoffs

Why Does This Exist?

Entropy, cross-entropy, KL, and mutual information already teach their own mechanisms elsewhere on this site. A residual hub is still needed for what none of them owns: shared units, chain rules across variables, the data-processing limit, and how coding, bottleneck, MDL, and rate-distortion divide the territory. Without it readers re-derive units per page and miss the shared ceiling.

This page is that hub by contract: it links components, never re-teaches them. Component owners: entropy for surprise averaging, cross-entropy for coding with the wrong model, KL divergence for the extra-bit price, mutual information for shared bits. Residual pages below are the only new mechanisms here.

The one idea you need first

One bit halves uncertainty: a fair coin carries 1 bit, four equiprobable symbols carry 2. Nats use $\\ln$ instead of $\\log_2$ (1 nat $\\approx 1.44$ bits). Every formula here is a bit price; unit mistakes scale every number.

Think of It Like This

A currency board, not a mint

This hub is a currency board: it fixes the exchange (bits versus nats), posts member banks' rates (entropy, KL, MI pages), and charters only the shared roads (chain rules, processing limits, coding theorems).

Minting new coins here would duplicate the members. Where it stops: exchange boards do not lend. Quantitative claims live on member pages; this page routes.

How It Actually Works

Residual mechanisms owned here, each on its own page:

Joint and conditional entropy: $H(X,Y) = H(X) + H(Y\\mid X)$ chains staged prediction; see joint and conditional entropy.
Jensen-Shannon: symmetric capped midpoint of KLs for comparing distributions; see Jensen-Shannon divergence.
Data processing: $I(X;Z) \\leq I(X;Y)$ on chains $X \\to Y \\to Z$ ; post-processing cannot invent bits, see data processing inequality.
Bottleneck: minimize $I(X;Z) - \\beta I(Z;Y)$ to compress while keeping label bits; see information bottleneck.
Source and channel coding: entropy floors compression, capacity ceilings transmission; see source and channel coding.
MDL and rate-distortion: model bits plus data bits pick models; bitrate-error curves price lossy compression, see minimum description length and rate-distortion theory.

Worked example

Two independent fair coins: each carries $H = 1$ bit, joint $H(X,Y) = 2$ bits, conditional $H(Y\\mid X) = 1$ bit, chain $1 + 1 = 2$ checks. Correlate them so $Y = X$ always: joint drops to 1 bit, $H(Y\\mid X) = 0$ , MI $I(X;Y) = 1$ . Same margins, dependence decides the totals. The arithmetic belongs to the joint page; the routing belongs here.

Watch Out For

Re-deriving units per page

Mixing bits and nats across entropy, KL, and bottleneck pages scales answers 1.44x silently. A "0.69 threshold" in nats is "1.0" in bits; comparing them raw misfires every gate.

Fix units at the hub: state bits or nats per formula, convert explicitly. If a number looks $\\ln 2$ off, it is units.

Expecting post-processing to add information

No classifier, embedding, or prompt trick beats the data-processing ceiling: $I(X;Z)$ never exceeds the upstream $I(X;Y)$ . Teams stack heads hoping signal appears; the hub says buy better $Y$ instead.

Audit upstream information before downstream architecture. If $I(X;Y)$ is capped, collect features, not layers.

The Quick Version

This hub routes; entropy, cross-entropy, KL, and MI pages teach their mechanisms.
Bits halve uncertainty; nats differ by $\\ln 2$ , about 1.44x.
Two fair coins total 2 bits; perfect correlation drops the joint to 1.
Chain, processing, bottleneck, coding, MDL, and rate-distortion live on residual pages.
State units per formula and convert explicitly.
Post-processing never beats its upstream information.