Depthwise Separable Convolutions

Split one expensive convolution into a cheap per-channel pass and a cheap channel-mixing pass, cutting the weight count by nearly an order of magnitude.

A 128 to 256 channel 3x3 layer needs 294,912 weights as one convolution but 33,920 as a depthwise plus pointwise pair, about 8.7 times fewer

Why Does This Exist?

A phone camera that blurs faces in the live preview has about 33 milliseconds per frame, no data center behind it, and a battery that notices waste. One convolutional layer reading 128 channels and writing 256 through a 3x3 kernel carries 294,912 weights. That is a single layer, and a real network stacks dozens of them, with every weight paid as a multiply-and-add at every position of every frame.

The obvious fixes hurt. Fewer layers or narrower layers run faster but give up the accuracy the task needs, and the phone cannot grow a bigger chip. So instead of shrinking the network, shrink the operation: keep the input and output shapes exactly where they are, but rebuild the layer so the same two jobs, filtering space and mixing channels, happen as two small steps rather than one combined one.

This page covers that split and its exact savings. The operation it replaces is in convolution operation. Networks assembled from these blocks are in CNN architectures. A different route to cheap inference, compressing a trained model instead of rebuilding its layers, is model quantization.

Think of It Like This

Two specialist crews instead of one generalist crew

A warehouse must move every box to a new bay and relabel each one with a fresh inventory code. One generalist crew does both jobs at once, box by box, so every worker needs a forklift ticket and a labeling rig. Costly to hire, costly to equip.

Split the job. One crew drives boxes and never touches a label. A second crew prints and sticks labels on boxes sitting still, and never touches a forklift. Neither crew needs the combined kit, so two narrow crews cost far less than one crew equipped for both jobs, while the warehouse ends up fully moved and fully relabeled either way.

The mapping: the generalist crew is the standard convolution, the driving crew is the depthwise pass (space, one channel at a time), the labeling crew is the pointwise pass (channels, no spatial reach), and the hiring bill is the weight count.

Where it stops: warehouse crews work in either order without interfering. Here the order is fixed, depthwise first and pointwise second, and the split cannot express a pattern that must read several channels jointly while filtering space.

How It Actually Works

What a standard convolution pays for

A standard 3x3 convolution reading $C_{in}$ channels and writing $C_{out}$ channels does two jobs inside one weight tensor. $C_{in}$ is the number of incoming channels, $C_{out}$ is the number of outgoing channels, and the 3x3 kernel reads a small spatial neighborhood around each position. It filters space and it blends every input channel into every output channel simultaneously. Its weight count multiplies both jobs together:

\text{standard} = 3 \times 3 \times C_{in} \times C_{out}

Both jobs are genuinely needed. Nothing says they must be bought in the same tensor.

Splitting the two jobs apart

A depthwise convolution handles only the spatial job. It applies one small kernel per input channel in isolation: a 3x3 filter over channel 1 alone, a separate one over channel 2 alone, and so on. No blending across channels happens here, so the output keeps exactly $C_{in}$ channels, and the cost has no $C_{out}$ factor:

\text{depthwise} = 3 \times 3 \times C_{in}

A pointwise convolution handles only the blending job. It is a 1x1 convolution, meaning no spatial neighborhood at all, just a weighted mix of the channels at each position, mapping $C_{in}$ channels to $C_{out}$ :

\text{pointwise} = C_{in} \times C_{out}

Chain them, depthwise first then pointwise, and the pair reads neighborhoods and mixes channels just as the standard layer did, in two small steps. The total is a sum where the standard cost was a product:

\text{separable} = 3 \times 3 \times C_{in} + C_{in} \times C_{out}

Worked example: 128 to 256 channels, computed

Take $C_{in} = 128$ , $C_{out} = 256$ , kernel $3 \times 3$ . Standard cost:

9 \times 128 \times 256 = 1152 \times 256 = 294{,}912

Separable cost, depthwise plus pointwise:

9 \times 128 + 128 \times 256 = 1{,}152 + 32{,}768 = 33{,}920

The ratio:

294{,}912 / 33{,}920 \approx 8.694 \approx 8.7

So identical input and output shapes cost about 8.7 times fewer weights. In general, for kernel size $k$ , the separable cost divided by the standard cost equals $1/C_{out} + 1/k^2$ . As $C_{out}$ grows, the first term fades and the saving approaches a full $k^2$ factor, which is 9 for 3x3 kernels. At 1024 outputs the same arithmetic lands near 8.9x, closing in on 9 from below rather than growing without bound.

What is actually lost

This is not free. Each depthwise output channel sees exactly one input channel, so a spatial pattern that only makes sense when read jointly across channels cannot be expressed at that step, and the pointwise step that follows mixes channels at a single position with no spatial context of its own. Families built from these blocks, MobileNet and EfficientNet among them, recover most of that gap through the surrounding architecture. The gap is not zero, which is why this pattern rules where the compute budget is tight rather than where accuracy is the only constraint.

Code

The point is the check: recompute the worked example instead of quoting it.

1def standard_conv_params(kernel: int, in_ch: int, out_ch: int) -> int:2    return kernel * kernel * in_ch * out_ch3
4
5def separable_conv_params(kernel: int, in_ch: int, out_ch: int) -> int:6    depthwise = kernel * kernel * in_ch  # spatial only, one filter per input channel7    pointwise = in_ch * out_ch  # 1x1 channel mix, no spatial extent8    return depthwise + pointwise9
10
11standard = standard_conv_params(3, 128, 256)12separable = separable_conv_params(3, 128, 256)13
14print(f"standard: {standard}")  # -> standard: 29491215print(f"separable: {separable}")  # -> separable: 3392016print(f"reduction: {standard / separable:.1f}x")  # -> reduction: 8.7x

Rerun with out_ch=1024 and the factor climbs toward 9 without ever passing it, exactly as the $1/C_{out} + 1/k^2$ form predicts.

Watch Out For

Quoting one network-wide shrink factor for every layer

The factor depends on the $C_{out}$ and kernel size of each layer, not a flat multiplier. A layer with few outputs keeps a proportionally larger pointwise term, so its saving is far smaller than a wide layer's. The symptom is a compression estimate that misses by a wide margin once summed over a real stack. Fix: compute the factor per layer from its actual channel counts instead of spreading the headline 8.7x across the whole net.

Reading fewer weights as a proportional speedup

Weight count and wall-clock latency are related but not equal. Depthwise passes run many small per-channel operations instead of one dense one, and some chips and kernels handle that pattern poorly. The symptom is a model with nearly 9 times fewer weights running only 2 to 3 times faster on the target phone. Fix: benchmark latency on the actual device before promising the parameter math as a frame-rate win.

The Quick Version

A standard 3x3 layer costs $9 \times C_{in} \times C_{out}$ weights because it filters space and mixes channels inside one tensor.
A depthwise pass filters space per channel for $9 \times C_{in}$ weights, then a pointwise 1x1 pass mixes channels for $C_{in} \times C_{out}$ .
At 128 to 256 channels that is 294,912 weights as one step against 1,152 plus 32,768, which is 33,920, about 8.7 times fewer.
The fractional cost is $1/C_{out} + 1/k^2$ , so wide layers approach a full 9 times saving for 3x3 kernels while narrow ones save less.
The split cannot express joint cross-channel spatial patterns at the filtering step, so it rules where budgets are tight, not where accuracy is the sole constraint.