Playing Atari with Deep Reinforcement Learning (DQN)

The Problem

Before this paper, reinforcement learning (RL) agents struggled in high-dimensional environments like vision-based games. Traditional RL relied heavily on handcrafted features or simple linear representations. While deep learning was transforming supervised tasks like image classification, applying it to RL was notoriously unstable. The incoming data distributions in RL shift constantly as the agent learns new behaviors, and consecutive observations are highly correlated, which violated the assumptions needed for stable neural network training.

The Idea

The authors proposed the Deep Q-Network (DQN), which combined deep convolutional neural networks (CNNs) with Q-learning to process raw pixels directly into action values. To solve the instability problems caused by correlated and shifting data, they introduced a mechanism called "experience replay." By storing past transitions and sampling them randomly during training, the network was fed a more stable and diverse dataset, decoupling the immediate sequence of experiences from the network's updates.

How It Works

DQN acts as a function approximator for the Q-value function, estimating the expected future reward for taking a specific action in a given state. The state is represented by raw screen pixels.

• Raw Pixel Processing: A CNN processes a stack of recent frames (to capture motion) and outputs Q-values for all possible joystick actions.
• Experience Replay: As the agent interacts with the environment, its experiences (state, action, reward, next state) are saved in a replay buffer. The network is updated by sampling random minibatches from this buffer, breaking the correlation between consecutive frames.
• Target Network (in later updates): To further stabilize training, a separate target network was introduced in a follow-up paper, fixing the target values temporarily to prevent feedback loops.

Why It Mattered

DQN was a landmark achievement that launched the modern era of deep reinforcement learning. It demonstrated for the first time that a single algorithm, with no domain-specific knowledge or handcrafted features, could learn to achieve human-level or superhuman performance across a diverse suite of Atari 2600 games using only the screen pixels and the game score as inputs.

What Came After

DQN's success spurred an explosion of research into deep reinforcement learning. It led to improvements like Double DQN, Dueling DQN, and Prioritized Experience Replay. Its core philosophy of combining deep learning with RL laid the direct theoretical and practical foundation for more advanced agents, including AlphaGo, AlphaZero, and modern continuous-control algorithms like PPO and SAC.