Deep Q-Networks

Deep Q-Networks swap the Q table for a neural network, so one forward pass from pixels or sensors yields a value for every action.

One network maps a state to a Q value per action, replay breaks time correlation, and a frozen target network supplies a stable Bellman target

Why Does This Exist?

Tabular Q-learning stores one value per state action pair. That works for a small grid and collapses for camera input, where the number of possible frames dwarfs any table and most exact frames never repeat.

Early neural attempts kept failing. Consecutive frames correlate strongly, so online updates overwrote old skills. The target also moved with every weight change, so the network chased itself.

Deep Q-Networks, introduced by Mnih and colleagues in 2015 on Atari games, fixed both with two mechanisms. Experience replay stores transitions and samples them at random. A frozen target network supplies Bellman targets for thousands of steps before syncing. Together they made Q-learning from pixels train reliably.

This page covers the network layout, replay and the target. For the tabular rule it scales see Q-learning. For the buffer mechanism in depth see experience replay. For the policy family that avoids values see policy gradient.

The one idea you need first

The action value $Q(s, a)$ is the expected discounted return after taking action $a$ in state $s$ and acting well after. The Bellman target $y = r + \gamma \max_{a^{\prime}} Q(s^{\prime}, a^{\prime})$ says the value should equal immediate reward $r$ plus discounted best future value, with discount $\gamma$ weighting later rewards less.

Think of It Like This

A food critic who learned patterns, not a ledger

A ledger critic writes a score for every exact dish they ever tasted and freezes when a new fusion plate arrives. A trained critic instead learned patterns like aroma, texture and balance, so they score the new plate from familiar features.

The ledger is the Q table. The trained palate is the network with weights $\theta$ . Pixels are ingredients. Replay is rereading old tasting notes out of order so recent meals do not erase old lessons. The frozen target is scoring against last month notes instead of notes you rewrite mid bite.

How It Actually Works

The network takes state $s$ alone as input and outputs one $Q(s, a; \theta)$ per discrete action in a single forward pass. With three actions the output is a vector of three numbers and the greedy choice is the argmax. One pass replaces one pass per action, which matters when the policy must run at game frame rates.

Experience replay

The agent stores each $(s, a, r, s^{\prime})$ in a large buffer $\mathcal{D}$ and trains on random minibatches. Random sampling breaks the time correlation of consecutive frames, smooths the data distribution and reuses rare rewarding events many times. New transitions overwrite the oldest when the buffer fills.

Frozen target network

The loss compares the online prediction against a target built by a separate network with weights $\theta^{-}$ :

y = r + \gamma \max_{a^{\prime}} Q(s^{\prime}, a^{\prime}; \theta^{-})

\mathcal{L}(\theta) = \mathbb{E}_{(s,a,r,s^{\prime}) \sim \mathcal{D}}[(y - Q(s, a; \theta))^2]

Weights $\theta^{-}$ stay fixed for many updates, often ten thousand steps, then copy $\theta$ . Fixed targets turn a moving chase into ordinary regression for a while, which calms oscillations.

Worked example

State $s$ feeds the online net and yields:

Q(s) = [1.2, 4.5, 0.9]

Greedy action is $a_2$ with value $4.5$ . The target net reads next state $s^{\prime}$ and its best value is $\max_{a^{\prime}} Q(s^{\prime}, a^{\prime}; \theta^{-}) = 3.0$ . Reward $r = 1.0$ and $\gamma = 0.99$ .

Target:

y = 1.0 + 0.99 \times 3.0 = 1.0 + 2.9700 = 3.9700

Error on the taken action:

\delta = 3.9700 - 4.5 = -0.5300

Loss:

\mathcal{L} = (-0.5300)^2 = 0.2809

The negative error says the current estimate is too high, so gradient descent lowers $Q(s, a_2)$ toward $3.97$ .

Code

This snippet computes the Bellman target and loss for the numbers above with plain Python.

1def bellman_target(reward: float, gamma: float, max_next_q: float) -> float:2    """DQN target from a frozen target network max value."""3    return reward + gamma * max_next_q4
5
6def dqn_loss(pred_q: float, target: float) -> tuple[float, float]:7    """Return TD error and squared loss for the taken action."""8    td = target - pred_q9    return td, td**210
11
12q_online = [1.2, 4.5, 0.9]13greedy = max(range(len(q_online)), key=lambda i: q_online[i])14print([round(float(v), 4) for v in q_online])  # -> [1.2, 4.5, 0.9]15print(greedy)  # -> 116y = bellman_target(1.0, 0.99, 3.0)17print(f"{y:.4f}")  # -> 3.970018td, loss = dqn_loss(float(q_online[greedy]), y)19print(f"{td:.4f}")  # -> -0.530020print(f"{loss:.4f}")  # -> 0.2809

Argmax index $1$ selects the second action with value $4.5$ , and the printed target, error and loss match the hand arithmetic.

Watch Out For

Max over noisy values inflates the target

The target takes a max over estimated values. When estimates carry noise, the max picks the most positive noise, so targets drift upward. You see Q curves climbing while episode scores stall.

Fix it by splitting selection from evaluation. Use the online net to pick $a^{\star} = \arg\max_{a^{\prime}} Q(s^{\prime}, a^{\prime}; \theta)$ and the target net only to score $Q(s^{\prime}, a^{\star}; \theta^{-})$ . That split is Double DQN.

Correlated online stream without replay

Training on consecutive frames feeds the optimizer long runs from one room or level. The network fits the current corridor and forgets earlier ones, then oscillates as the agent moves. Loss looks smooth inside each run and jumps at each transition.

Fix it with a large buffer, often one million transitions, and random minibatches of 32. Keep the target frozen for thousands of steps. If memory is tight, prefer a smaller batch from a large buffer over a large batch from a tiny one.

The Quick Version

DQN outputs all action values in one forward pass from state features or pixels.
Replay stores $(s, a, r, s^{\prime})$ and samples at random to break correlation and reuse rare wins.
A frozen target network builds $y = r + \gamma \max Q(s^{\prime}; \theta^{-})$ and syncs periodically.
Worked numbers: values $[1.2, 4.5, 0.9]$ , target $3.97$ , error $-0.53$ , loss $0.2809$ .
Max over noisy estimates overstates value, so split selection and evaluation with Double DQN.
Greedy follows the argmax, while epsilon greedy exploration fills the buffer.