Reinforcement Learning from Pixels

Game scores and robot rewards arrive as pixels, not coordinates. Reinforcement learning from pixels maps raw frames to actions, discovering balls, enemies, and goals with no hand-built features.

The Q estimate climbs from 0 toward a target of 2.8 in one update, landing at 1.4 with step size one half

Why Does This Exist?

Classical control assumes clean state: joint angles, ball coordinates, velocities on tap. Real robots get camera frames — partial, noisy, high-dimensional. Hand-building state estimators per task (track the ball, filter the joints) reintroduces all the brittleness learning was supposed to remove.

RL from pixels learns the mapping end to end: frames in, actions out, rewards as the only teacher. Deep Q-Networks on Atari proved the point by reaching human scores from screenshots alone. The price is sample hunger in the millions of frames and representations that can fixate on pixels instead of play. The demonstration-driven sibling is imitation from video; the loop is embodied vision.

The one idea you need first. Q-learning moves an action's value toward a bootstrapped target: $Q(s,a) \leftarrow Q(s,a) + \alpha\,[r + \gamma \max Q(s',\cdot) - Q(s,a)]$ . The reward $r$ grounds it, the discounted best-next value $\gamma \max Q$ propagates future earnings backward, and $\alpha$ rations the step. Everything visual — convolutions, stacking, augmentation — serves this one update.

Think of It Like This

Learning tennis by watching the ball

Nobody learns tennis from a physics lecture on projectile motion. You watch the ball, swing, miss, adjust — thousands of rallies where the only teacher is where the ball lands. Gradually your eyes learn which pixels matter (ball, seams, opponent's shoulder) and your arm learns what to do about them.

Pixel RL is those rallies. The network is the beginner's eyes discovering ball-pixels matter and crowd-pixels do not; Q-learning is the bruised adjustment after every miss. Lectures (hand-built features) would speed week one and cap year two — the agent skips the lecture and keeps the higher ceiling.

How It Actually Works

Frames pass through convolutions into Q-values per action; $\varepsilon$ -greedy exploration tries moves while replay buffers decorrelate the stream. Single frames hide velocity, so inputs stack four consecutive frames (motion becomes visible as displacement). Modern variants add random-shift augmentation (DrQ: same lesson from jittered views), auxiliary prediction losses, and discrete latent world models (Dreamer) that plan in imagination to cut real frames. Rewards need shaping care: game score is dense, robot success is sparse, and pixels invite hacking the score display instead of the game.

Training runs long: tens of millions of frames per Atari title at the classic scale, far less with augmentation and model-based planning.

Worked example

One Q-update with $Q = 0$ , reward $r = 1$ , discount $\gamma = 0.9$ , best-next value $\max Q(s') = 2$ , step size $\alpha = 0.5$ :

Target: $r + \gamma \max Q(s') = 1 + 0.9 \times 2 = 1 + 1.8 = 2.8$ .
Surprise: $2.8 - 0 = 2.8$ .
New estimate: $0 + 0.5 \times 2.8 = 1.4$ .

The value climbs from $0$ toward $2.8$ , landing at $1.4$ — half the gap closed in one step. Millions of such steps, each grounded by real rewards, compound into play.

Code

The point of this snippet is the update: target, surprise, and step reproduce the $1.4$ exactly.

1def q_update(q: float, r: float, gamma: float, best_next: float, alpha: float) -> float:2    target = r + gamma * best_next  # bootstrapped worth3    return q + alpha * (target - q)  # move partway toward it4
5
6print(round(q_update(0.0, 1.0, 0.9, 2.0, 0.5), 3))  # -> 1.4

Watch Out For

Millions of frames per skill

Atari-scale training consumes $10$ – $50$ million frames per game — months of play per title. Robot time costs more than GPU time, so naive pixel RL budgets explode before the first useful policy.

Cut the bill before scaling: augmentation (DrQ-style shifts), auxiliary representation losses, and latent world models each remove multipliers. Count environment steps, not gradient steps, when comparing methods — that is the currency robots pay.

Reward hacking through the pixels

Pixel agents exploit whatever correlates with reward: camping by the score display, vibrating to farm contact bonuses, hiding where the safety camera cannot see. The policy optimizes the camera's evidence of success rather than success.

Shape rewards from task progress (distances, contacts, goals) rather than raw score where possible, and audit rollouts visually: if the behavior looks absurd while the curve climbs, the pixels are being gamed.

The Quick Version

RL from pixels maps frames to actions with rewards as the only teacher — no state estimator.
Frame stacking reveals velocity; augmentation and world models cut the sample bill.
The worked fixture updates $Q$ from $0$ to $1.4$ toward a $2.8$ target in one step.
Budget environment steps in the tens of millions; augmentation and imagination discount them.
Pixel rewards invite hacking the evidence of success, so shape from progress and watch rollouts.