ImageNet Classification with Deep CNNs

The Problem

Before 2012, computer vision relied on hand-crafted features (like SIFT or HOG) fed into traditional machine learning models (like SVMs). These methods were hitting a performance ceiling. Neural networks were known to be powerful, but they were incredibly slow to train and prone to overfitting, making them impractical for large, high-resolution datasets.

The Idea

The authors realized that two things had changed: data and compute. The ImageNet dataset provided millions of labeled images, solving the data starvation problem. Modern GPUs provided the massive parallel compute needed to train deep networks. They combined these with a few key architectural tweaks to build a Deep Convolutional Neural Network (CNN).

How It Works

The network, later dubbed 'AlexNet', consisted of 5 convolutional layers and 3 fully connected layers.

To make this massive network train effectively, they introduced several novelties:

ReLU Nonlinearity: Instead of the standard tanh or sigmoid functions, they used Rectified Linear Units (ReLU). ReLU solved the vanishing gradient problem in shallow networks and accelerated training by 6x. Overlapping Pooling: They used max-pooling layers with strides smaller than the pool size, which slightly reduced overfitting. Multi-GPU Training: The network was too big to fit on a single GTX 580 GPU (which only had 3GB of memory). They split the network across two GPUs, with certain layers only communicating with the GPU they were housed on. Dropout: To reduce overfitting in the massive fully connected layers, they used dropout (randomly zeroing out neurons during training).

Why It Mattered

AlexNet was entered into the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC). It didn't just win; it obliterated the competition. It achieved a top-5 error rate of 15.3%, compared to the runner-up's 26.2% (which used traditional methods).

This massive gap proved that deep learning, powered by GPUs and big data, was vastly superior to hand-crafted features.

What Came After

This paper triggered the Deep Learning Revolution. Within a year, every serious computer vision lab abandoned traditional methods and switched to deep learning. The structural choices made in AlexNet (ReLUs, Dropout, GPU training) became standard practice for the next decade.