Temporal Difference Learning

Temporal difference learning updates each value from the next state's estimate instead of waiting for an episode to end, so it learns at every step from guesses built on guesses.

One TD step turns the value 10 into 10.28 from a reward of 2 plus the discounted next estimate, without waiting for the episode to finish

Why Does This Exist?

Monte Carlo evaluation waits for an episode to finish, totals the return, and only then updates the states it visited. That fails twice: endless tasks never finish, and even finite ones make the agent idle through thousands of steps before a single lesson lands.

Temporal difference learning updates at every step from a target built out of the next state's current estimate: $R_{t+1} + \gamma V(S_{t+1})$ . Learning from a guess sounds circular, and in a sense it is, which is why the method is called bootstrapping. The payoff is online learning from incomplete episodes, the property nearly every modern value method inherits. This page covers the TD(0) update, its target and error. The algorithms built on it are Q-learning and SARSA-style updates like expected SARSA, and the head-to-head trade-offs sit in advantages of TD over MC and DP.

The one prerequisite is the value function itself: $V(s)$ is the expected discounted return from $s$ , introduced in MDP foundations. TD is simply the fastest way to estimate it from experience.

Think of It Like This

Revising a road-trip arrival time at every exit

You leave the city guessing arrival in 45 minutes. Ten minutes later you hit roadworks and crawl for two exits. A Monte Carlo traveler refuses to revise anything until the car is parked, then finally admits the first guess was wrong. A TD traveler revises at every exit: still crawling at exit 9, add 15 minutes on the spot and drive on with the new number.

The exit-by-exit revision is the TD update, the fresh delay you just saw is the reward, and the remaining route time you still believe in is the next state's estimate. The trip total is always a mix of road already seen and road still guessed.

Where it stops: your mental map of the remaining route is roughly right from years of driving. An agent's initial values can be wildly wrong everywhere, so early revisions compound errors instead of absorbing facts, which is exactly why step sizes stay small.

How It Actually Works

Each transition $(S_t, R_{t+1}, S_{t+1})$ produces a TD target, the immediate reward plus the discounted next estimate:

\text{target} = R_{t+1} + \gamma V(S_{t+1})

where $\gamma$ in $[0, 1)$ discounts the future and $V$ is the current value table or approximator. The TD error is how far the old estimate missed that target:

\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)

and the update nudges the old estimate toward the target by step size $\alpha$ :

V(S_t) \leftarrow V(S_t) + \alpha\,\delta_t

A positive error means things turned out better than expected, so the value rises; negative means worse, so it falls. Because the target uses the current $V(S_{t+1})$ , every update teaches one guess from another guess, and values propagate backward from rewarding states one step per visit.

Worked example

Current estimate $V(S_t) = 10.0$ , next estimate $V(S_{t+1}) = 12.0$ , reward $R_{t+1} = 2.0$ , $\gamma = 0.9$ , $\alpha = 0.1$ .

Target: $2.0 + 0.9 \times 12.0 = 2.0 + 10.8 = 12.8$ . Error: $12.8 - 10.0 = 2.8$ . Update: $10.0 + 0.1 \times 2.8 = 10.28$ . One transition moved the estimate from $10.00$ to $10.28$ with no episode in sight, and the $12.0$ it leaned on will get its own correction when its own transition arrives.

Code

1def td_update(v: float, v_next: float, reward: float,2              alpha: float = 0.1, gamma: float = 0.9) -> tuple[float, float]:3    """One TD(0) step; returns (td_error, new_value)."""4    target = reward + gamma * v_next5    td_error = target - v6    return td_error, v + alpha * td_error7
8
9err, new_v = td_update(10.0, 12.0, 2.0)10print(f"{err:.2f}")  # -> 2.8011print(f"{new_v:.2f}")  # -> 10.28

Watch Out For

Bootstrapping on garbage initial values with a large step size

Early in training every estimate is a guess, so a big $\alpha$ lets one bad target yank values far from sense, and the next updates bootstrap from the damage. The symptom is values that swing wildly in the first thousand episodes while Monte Carlo on the same task improves smoothly, which convinces people TD is broken rather than too hot.

The fix is a small step size early, exactly the Robbins-Monro decay habit, plus a quick check that TD and Monte Carlo agree on a tiny task before scaling up. If they disagree, distrust the step size before distrusting the method.

Combining bootstrapping with neural nets and off-policy data unchecked

TD plus function approximation plus off-policy updates form the deadly triad, and the combination can diverge where each piece alone converges. The symptom is values that grow without bound late in training while behavior looks reasonable, because the policy runs on relative preferences that survive even as absolute values blow up.

The fix is the standard stabilizer kit: a target network to freeze the bootstrap target and experience replay to de-correlate updates. Divergence that vanishes when either is added was triad divergence.

The Quick Version

TD learning updates values from the next state's estimate, learning at every step from incomplete episodes.
The TD target is immediate reward plus discounted next value; the TD error is target minus current estimate.
Positive errors raise values and negative errors lower them, propagating returns backward one step per visit.
The worked step moves V from 10.00 to 10.28 on a single reward-2 transition with no episode boundary.
Early guesses bootstrapping on guesses need small step sizes, or errors compound instead of draining.
With neural nets and off-policy data, TD needs target networks and replay to stay convergent.