Compactness

Closed and bounded sets trap sequences and force continuous functions to reach their minimum, which is why minimizers exist.

A compact set traps every sequence inside itself with a convergent subsequence, so minima are attained

Why Does This Exist?

Optimizers promise to find a minimizer, but a continuous loss on an open or unbounded set need not have one: $f(x) = e^x$ on the reals has infimum $0$ never attained, and $1/x$ on $(0,1]$ dives without bottom. Something must guarantee the bottom exists before algorithms chase it.

Compactness is that guarantee. In $\mathbb{R}^n$ it means closed plus bounded. A continuous function on a compact set attains its minimum and maximum, and every sequence inside has a convergent subsequence staying inside.

This page states the finite-dimensional picture used in ML. Metric background is in metric spaces and convex structure in convex sets and functions.

The one idea you need first

Closed means containing all limit points; bounded means fitting inside some large ball. The interval $[0,1]$ is both. The interval $(0,1]$ is not closed, and $[0,\infty)$ is not bounded.

Think of It Like This

A fenced yard with no exits

Sheep wander the yard. Closed means the fence has no gaps: no sheep can stand at the boundary trickling out. Bounded means the fence exists at finite distance. Together the yard is compact.

Any infinite line of sheep has a cluster piling up somewhere inside the yard. That cluster is the convergent subsequence. Where the analogy stops: in infinite dimensions closed plus bounded is not enough; compactness needs a stronger finite-cover property.

How It Actually Works

Heine-Borel in $\mathbb{R}^n$ : a set is compact exactly when it is closed and bounded. Bolzano-Weierstrass: every bounded sequence in $\mathbb{R}^n$ has a convergent subsequence. Extreme value theorem: a continuous $f$ on compact $K$ attains $\min_K f$ and $\max_K f$ .

Finite subcover view: $K$ is compact when every open cover has a finite subcover. Three small discs can cover the ellipse in the diagram; infinitely many are never needed.

Worked example

Minimize $f(x) = (x - 2)^2 + 1$ on $K = [0, 4]$ . Values: $f(0) = 5$ , $f(2) = 1$ , $f(4) = 5$ . Since $K$ is closed and bounded, a minimizer exists; calculus gives $x = 2$ with value $1$ . On the open set $(0, 4)$ the same point still works, but minimize $g(x) = 1/x$ on $(0, 1]$ : values $1, 2, 10, 100$ at $x = 1, 0.5, 0.1, 0.01$ dive toward no minimum because the endpoint $0$ is missing.

Watch Out For

Assuming compactness in infinite dimensions

Symptom: treating the unit ball of a function space as compact and claiming convergent subsequences of networks. Fix: in infinite dimensions closed unit balls are not compact; use weak topologies or explicit finite-dimensional sieves.

Optimizing over an unbounded set without coercivity

Symptom: loss drifts to infinity in parameter norm while values keep improving, so no minimizer exists. Fix: add a norm constraint to compactify, or prove the loss grows at infinity so sublevel sets are bounded.

The Quick Version

In finite dimensions compact means closed plus bounded.
Every sequence in a compact set has a convergent subsequence staying inside.
Continuous functions on compact sets attain minima and maxima.
$1/x$ on $(0,1]$ shows what fails without closedness: no bottom point.
Constrain or prove boundedness before claiming a minimizer exists.