DARTS Searches Cells by Gradients

DARTS relaxes discrete ops to softmax weights, so architecture search becomes gradient descent over cells.

Softmax over 8 ops finds cells in 4 days not 2000

Why Does This Exist?

RL search trains thousands of nets for $2000$ GPU days. Most labs cannot pay that.

DARTS differentiates. It grounds efficient architectures, searches convolutional layers cells, and lifts image classification backbones.

Think of It Like This

Blending paints before picking one color

Instead of painting eight walls to pick best, you blend eight colors with adjustable ratios and slide ratios toward the winner.

Ops are colors, softmax weights are ratios. Where it stops: blends flatter winners, just as relaxation favors skips.

How It Actually Works

Edge $(i,j)$ mixes $8$ ops $o$ with $\bar{o} = \sum softmax(\alpha)_o o(x)$ , where $\alpha$ are architecture weights. Bi-level: inner trains $w$ on train split, outer trains $\alpha$ on val split. Discretize by argmax per edge.

Worked example

Weights $\alpha = [2.0, 1.0, 0.0]$ over skip, conv, pool. Softmax $e^2 = 7.39$ , $e^1 = 2.72$ , $e^0 = 1$ , sum $11.11$ , probs $[0.665, 0.245, 0.090]$ . Output blends $0.665$ skip plus $0.245$ conv. Search $4$ days versus $2000$ , a $500$ times cut ( $2000/4 = 500$ ).

Watch Out For

Collapsing to skip heavy cells

Softmax favors fast converging skips. Symptom: searched cells full of skips with weak test accuracy. Fix: regularize skips, early stop, or use robust variants before final training.

The Quick Version

DARTS relaxes $8$ ops per edge to softmax blends.
Bi-level trains weights inner and architecture outer.
Fixture probs $0.665, 0.245, 0.090$ blend skip plus conv.
Search $4$ days replaces $2000$ , 500 times cheaper.
Guard against skip collapse with regularization.