EfficientNet

The Problem

Before EfficientNet, scaling up Convolutional Neural Networks for better accuracy was an arbitrary process. Researchers typically chose one dimension to scale: making the network deeper (ResNet), wider (Wide ResNet), or increasing the input image resolution. Scaling one dimension indefinitely yielded diminishing returns, and tuning all three manually was an impossible search space. There was no mathematical framework to ensure compute was being spent optimally.

The Idea

Google researchers proposed "Compound Scaling." They demonstrated that network width, depth, and resolution are strictly dependent on each other. If you increase the resolution of the image, the network needs to be deeper (to increase the receptive field to cover more pixels) and wider (to capture more fine-grained patterns). They derived a simple scaling heuristic that scales all three dimensions by a constant ratio.

How It Works

The authors first used Neural Architecture Search (NAS) to design a small, highly optimized baseline network called EfficientNet-B0. Then, they applied their compound scaling method:

Instead of arbitrarily picking numbers, they defined a compound coefficient $\phi$ that controls how many resources are available. Depth, width, and resolution are scaled as:

• Depth: $d = \alpha^\phi$
• Width: $w = \beta^\phi$
• Resolution: $r = \gamma^\phi$

Where $\alpha, \beta, \gamma$ are constants found via a small grid search on the baseline model (subject to $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$). By increasing $\phi$, they systematically generated a family of models from B1 to B7.

Why It Mattered

EfficientNet-B7 achieved state-of-the-art accuracy on ImageNet while being 8.4x smaller and 6.1x faster on inference than the previous best model. It became the default architecture for deployment on edge devices and mobile phones where compute efficiency is paramount.

What Came After

The EfficientNet family was widely adopted in industry. The authors later released EfficientNetV2, which incorporated training-aware NAS to design networks that were not only parameter-efficient but also faster to train on modern hardware by progressively resizing the image during training.