KKT Optimality Conditions

Constrained optima balance the objective pull against active constraint pushes. KKT writes that balance as four checkable equations.

Minimizing x minus 3 squared left of x equals 1 balances the objective pull against the bound price of 4

Why Does This Exist?

Unconstrained optima set the gradient to zero. Constrained optima usually sit on the boundary where the gradient is not zero: the objective pulls outward while the wall pushes back. Declaring "gradient nonzero so not optimal" misses every boundary solution, which is most of them.

Karush-Kuhn-Tucker writes the standoff as four conditions on $(x^\\star, \\lambda^\\star, \\nu^\\star)$ : stationarity, primal feasibility, dual feasibility, complementary slackness. Under convexity plus constraint qualification they certify global optimality. It sharpens Lagrange multipliers and duality into the checker for convex optimization.

The one idea you need first

A constraint $g(x) \\leq 0$ is active when $g(x^\\star) = 0$ (on the wall) and inactive when strictly inside. Only active walls push, priced by multipliers $\\lambda \\geq 0$ . Inactive walls price zero.

Think of It Like This

A dog on a leash at the fence

A dog runs toward a squirrel (objective pull) until the leash snaps taut at the fence (active constraint). At rest, pull plus leash tension balance: stationarity. The leash only pulls when taut (slackness), never pushes the dog outward (dual feasibility $\\lambda \\geq 0$ ), and the dog stays inside the yard (primal feasibility).

A loose leash prices zero. Where it stops: leashes are single and smooth, while real problems stack many walls with kinks needing constraint qualifications.

How It Actually Works

For minimize $f_0$ subject to $f_i \\leq 0$ , $h_j = 0$ with differentiable functions: stationarity $\\nabla f_0 + \\sum \\lambda_i\\nabla f_i + \\sum \\nu_j\\nabla h_j = 0$ ; primal $f_i \\leq 0, h_j = 0$ ; dual $\\lambda_i \\geq 0$ ; slackness $\\lambda_i f_i = 0$ . SVMs read off support vectors as active constraints with $\\lambda_i > 0$ . Constraint qualifications (Slater, LICQ) rule out degenerate wall tangles where multipliers misbehave.

Worked example

Minimize $(x-3)^2$ subject to $x \\leq 1$ (write $g(x) = x - 1 \\leq 0$ ). Lagrangian $L = (x-3)^2 + \\lambda(x-1)$ . Stationarity: $2(x-3) + \\lambda = 0$ . Guess active: $x^\\star = 1$ , then $2(-2) + \\lambda = 0$ gives $\\lambda^\\star = 4 \\geq 0$ . Checks: primal $1 - 1 = 0$ feasible, slackness $4 \\times 0 = 0$ , dual $4 \\geq 0$ . Certified constrained optimum with value $(1-3)^2 = 4$ and wall price 4. Inactive guess $\\lambda = 0$ would give $x = 3$ , violating $x \\leq 1$ : correctly rejected.

Code

1x, lam = 1.0, 4.02stat = 2 * (x - 3) + lam  # stationarity residual3print(round(stat, 6))  # -> 0.04print(x <= 1, lam >= 0, round(lam * (x - 1), 6))  # -> True True 0.0

Residual zero with feasibility, sign, and slackness all passing. Four numbers certify the point.

Watch Out For

Trusting KKT without constraint qualification

Degenerate constraints (duplicated walls, cusps) admit KKT points that are not optimal, or optima with no multipliers. Solvers return "KKT satisfied" on such models and teams stop.

Verify Slater (strictly feasible point for convex) or LICQ (independent active gradients). If walls are redundant or kinky, reformulate before certifying.

Reading inactive multipliers as influence

$\\lambda_i = 0$ means the wall did not push: removing it changes nothing locally. Teams regularize or price constraints by $\\lambda$ and chase zeros.

Rank active constraints by $\\lambda_i > 0$ only. Sensitivity $dp^\\star/db_i = -\\lambda_i$ holds at active walls under regularity, nowhere else.

The Quick Version

KKT balances objective pull against active wall pushes with four checks.
Example $(x-3)^2$ left of 1 certifies $x^\\star = 1$ , $\\lambda^\\star = 4$ , value 4.
Slackness pairs prices with walls: positive price means on the wall.
Convex plus qualified makes KKT sufficient for global optimality.
SVM support vectors are exactly the active set with positive multipliers.
Without qualification, KKT neither certifies nor prices reliably.