Algebraic Geometry of Neural Networks

Piecewise-linear networks cut input space into polyhedral regions, and counting those regions with polynomial tools explains what depth buys.

Bent hyperplanes cut the input plane into polygonal linear regions, and each new layer subdivides every existing region again

Why Does This Exist?

A ReLU network is globally a puzzle of flat pieces: on each polyhedral region of input space it equals one affine map, and kinks sit on bent hyperplanes where units switch. Counting those regions turns depth-versus-width intuition into theorems: depth multiplies regions, width adds them. Algebraic geometry (through its tropical, piecewise-linear shadow) supplies the counting tools: activation patterns as varieties, region bounds from hyperplane arrangements, and exact formulas for small nets. This page gives the region picture with counts; training on those regions is loss geometry, and symmetry structure is group theory.

The one idea you need first

One ReLU unit splits input space along the hyperplane $w^T x + b = 0$ : active on one side, zero on the other. An activation pattern records which units are on per region; each feasible pattern is one linear piece of the whole network.

Think of It Like This

Folding maps with crease patterns

An origami crease pattern cuts paper into facets: each facet stays flat, folds live on creases, and adding creases subdivides existing facets rather than appending beside them. Two crease directions give rectangles; diagonal creases split each rectangle again.

ReLU hyperplanes are creases in input space. Each layer's creases bend at earlier layers' creases (they are piecewise, not straight), so subdivision compounds: ten well-placed bent creases cut thousands of facets where ten straight cuts manage eleven. Counting facets from creases is exactly region counting from units.

How It Actually Works

A layer with $n$ units in general position cuts $\mathbb{R}^d$ into at most $\sum_{j=0}^{d} \binom{n}{j}$ regions (hyperplane-arrangement bound). Composition multiplies: $L$ layers of width $n$ reach $O(n^{dL})$ -scale counts in the best arrangements (Montufar bounds), exponential in depth. Tropical geometry rewrites ReLU nets as tropical rational maps (max-plus polynomials), so region vertices are tropical hypersurface points and counting becomes lattice geometry.

Feasibility constrains the maximum: not every on-off pattern occurs (patterns must be realizable by nested half-spaces), so exact counts need arrangement-specific computation, and typical trained nets use far fewer regions than the maximum.

Worked example

Two inputs, first layer $3$ units in general position: regions $\le \binom{3}{0} + \binom{3}{1} + \binom{3}{2} = 1 + 3 + 3 = 7$ . Check by drawing: three lines through the plane in general position (no two parallel, no three concurrent) cut $7$ regions (six unbounded wedges plus one central triangle). Add a second layer with $3$ units whose bent boundaries subdivide: upper bounds multiply toward $7 \times 7 = 49$ feasible subregions in the best arrangement, while a wide single layer with $6$ units caps at $1 + 6 + 15 = 22$ . Depth's $49$ beats width's $22$ with equal units: the geometric form of exponential separation.

Code

1def max_regions(n: int, d: int) -> int:2    from math import comb  # arrangement bound: sum of binomial terms3    return sum(comb(n, j) for j in range(min(n, d) + 1))4
5
6print(max_regions(3, 2))  # -> 77print(max_regions(6, 2))  # -> 228print(7 * 7)  # -> 49 deep upper reach with two layers of 3

Three lines cut $7$ ; six cut $22$ ; two bent layers reach toward $49$ .

Watch Out For

Quoting maximum regions as used regions

Trained nets activate a thin sliver of feasible patterns: measured region counts near training data run orders of magnitude below maxima. Capacity arguments from maxima overstate what the fitted function uses. Count regions on your data manifold (sample activation patterns) instead of citing the bound.

Assuming general position in trained nets

Dead units, collinear weights, and correlated biases collapse arrangements far from general position: coincident hyperplanes add zero regions. If your region count stalls while width grows, inspect for dead and duplicated units before invoking theory.

The Quick Version

ReLU nets are piecewise-affine over polyhedral regions defined by activation patterns.
One layer of $n$ units in $d$ dimensions cuts at most $\sum_{j \le d} \binom{n}{j}$ regions.
Three lines cut $7$ regions; six cut $22$ ; two bent layers of $3$ reach toward $49$ .
Tropical geometry rewrites the pieces as max-plus polynomials for exact counting.
Trained nets use far fewer regions than maxima; count patterns on real data.
Dead and duplicated units collapse arrangements below general-position bounds.