EfficientNetV2: Smaller Models and Faster Training

The Problem

The original EfficientNet paper was highly successful at achieving state-of-the-art accuracy with minimal FLOPs (floating-point operations) and parameters. However, relying purely on FLOPs or parameter counts as a proxy for efficiency ignored real-world hardware realities. Specifically, the original EfficientNet was slow to train and had high memory access costs in its early layers because depthwise convolutions are often memory-bandwidth bound on modern accelerators (like GPUs and TPUs). Furthermore, simply scaling up image resolution for larger models resulted in drastically slower training times without proportional accuracy gains.

The Idea

EfficientNetV2 proposed two major changes to address these training bottlenecks:

1. Fused-MBConv: Replacing the depthwise and 1x1 convolutions in the early layers of the network with a single dense standard convolution (Fused-MBConv), which modern hardware accelerates much better.
2. Progressive Learning: Dynamically increasing both the image size and the strength of regularization (like dropout and data augmentation) during training.

How It Works

Fused-MBConv The standard building block of EfficientNetV1 was the Mobile Inverted Bottleneck (MBConv), which used a 1x1 expansion convolution, a depthwise convolution, and a 1x1 projection. While parameter-efficient, depthwise convolutions have a low ratio of computation to memory access. In early layers, where spatial resolution is high, this memory bottleneck dominates. EfficientNetV2 replaces the first two operations (1x1 expansion and depthwise conv) with a single regular 3x3 convolution, creating a Fused-MBConv. Using neural architecture search (NAS), the authors found that replacing MBConv with Fused-MBConv in just the first few stages of the network achieved the best trade-off between training speed and parameter count.

Progressive Learning Training on very large images from the start is computationally expensive. The authors introduced an improved progressive learning approach:

• Start training with smaller image sizes and weak regularization (since smaller images require less regularization to prevent underfitting).
• Gradually increase the image size as training progresses.
• Crucially, simultaneously increase the regularization strength (e.g., Mixup, RandAugment, Dropout) to prevent the model from overfitting on the larger, more complex inputs.

Why It Mattered

EfficientNetV2 dramatically improved the training speed of large vision models. It demonstrated that FLOPs and parameter counts are not the only, or even the best, proxies for real-world efficiency. By optimizing for actual training latency and introducing adaptive regularization, EfficientNetV2 models trained up to 11x faster than V1 while being smaller and achieving better accuracy on ImageNet.

What Came After

EfficientNetV2 became a default baseline for many downstream vision tasks due to its excellent accuracy-to-compute ratio. Its insights into hardware-aware architecture search and progressive learning influenced subsequent CNN and Vision Transformer designs, emphasizing that memory bandwidth and real-world latency must be treated as first-class architectural constraints.