Equivalence of Forward and Backward Views

Why Does This Exist?

In reinforcement learning theory, the forward view of $\text{TD}(\lambda)$ is philosophically compelling: it defines an ideal learning target, the $\lambda$-return $G_t^\lambda$, by blending all future $n$-step returns into a geometric mixture. However, the forward view is non-causal and acausal online—an agent standing at time $t$ cannot compute $G_t^\lambda$ because it requires rewards and states that have not yet occurred.

To implement learning in real time, practitioners rely on the backward view: a mechanistic algorithm that looks backward into the past using an eligibility trace vector $z_t \in \mathbb{R}^{|\mathcal{S}|}$. Whenever a local one-step prediction error $\delta_t$ occurs, it broadcasts that error backward to recently visited states.

This introduces a foundational theoretical question: Are these two formulations actually solving the same problem?

The mathematical equivalence between the forward view and backward view proves that the mechanistic, step-by-step backward algorithm is not merely an intuitive heuristic. In the offline setting (when updates are accumulated and applied at the end of an episode), the sum of all backward updates across the trajectory is mathematically identical to the sum of all theoretical forward-view updates.

Think of It Like This

How It Actually Works

The Telescoping Sum Decomposition of the λ-Return

Let an episode terminate at step $T$. At each step $k$, the standard one-step Temporal Difference error is:

$\delta_k = R_{k+1} + \gamma V(S_{k+1}) - V(S_k)$

Now consider the sequence of $n$-step returns starting from time $t$:

• $1$-step return: $G_t^{(1)} = R_{t+1} + \gamma V(S_{t+1})$
• $2$-step return: $G_t^{(2)} = R_{t+1} + \gamma R_{t+2} + \gamma^2 V(S_{t+2})$
• $n$-step return: $G_t^{(n)} = \sum_{i=1}^n \gamma^{i-1} R_{t+i} + \gamma^n V(S_{t+n})$

Notice that the difference between any $n$-step return and the initial value estimate $V(S_t)$ expands into a telescoping sum of one-step TD errors:

$G_t^{(1)} - V(S_t) = \delta_t$

$G_t^{(2)} - V(S_t) = [R_{t+1} + \gamma V(S_{t+1}) - V(S_t)] + \gamma [R_{t+2} + \gamma V(S_{t+2}) - V(S_{t+1})] = \delta_t + \gamma \delta_{t+1}$

In general, for any horizon $n \ge 1$:

$G_t^{(n)} - V(S_t) = \sum_{k=t}^{t+n-1} \gamma^{k-t} \delta_k$

The forward-view $\lambda$-return blends all $n$-step returns geometrically with weights $(1-\lambda)\lambda^{n-1}$:

$G_t^\lambda = (1 - \lambda) \sum_{n=1}^{T - t - 1} \lambda^{n-1} G_t^{(n)} + \lambda^{T - t - 1} G_t$

Subtracting $V(S_t)$ from both sides and substituting the telescoping expansion, the geometric weights group each $\delta_k$ into an exponential factor:

$G_t^\lambda - V(S_t) = \sum_{k=t}^{T-1} (\gamma \lambda)^{k-t} \delta_k$

This identity reveals that the forward $\lambda$-error is equal to the discounted sum of all future one-step TD errors from time $t$ to the end of the episode.

Swapping Summations: From Lookahead to Eligibility Traces

In the offline forward view, the value function is held fixed throughout the episode, and the total update applied to state $s$ at episode termination is the sum of forward updates over all visits:

$\Delta V_{\text{fwd}}(s) = \sum_{t=0}^{T-1} \alpha [G_t^\lambda - V(S_t)] \mathbf{1}_{S_t = s} = \sum_{t=0}^{T-1} \alpha \mathbf{1}_{S_t = s} \sum_{k=t}^{T-1} (\gamma \lambda)^{k-t} \delta_k$

This expression features a double summation over indices $0 \le t \le k \le T-1$. Reversing the order of summation—making $k$ the outer loop over time steps and $t$ the inner loop over past visits:

$\Delta V_{\text{fwd}}(s) = \sum_{k=0}^{T-1} \alpha \delta_k \sum_{t=0}^k (\gamma \lambda)^{k-t} \mathbf{1}_{S_t = s}$

Inspect the inner summation closely:

$z_k(s) \triangleq \sum_{t=0}^k (\gamma \lambda)^{k-t} \mathbf{1}_{S_t = s}$

This matches the recursive definition of the accumulating eligibility trace:

$z_k(s) = \gamma \lambda z_{k-1}(s) + \mathbf{1}_{S_k = s}, \quad z_{-1}(s) = 0$

Substituting $z_k(s)$ directly back into the reversed summation gives:

$\Delta V_{\text{fwd}}(s) = \sum_{k=0}^{T-1} \alpha \delta_k z_k(s) = \Delta V_{\text{bwd}}(s)$

This proves the exact offline equivalence: the forward view's future lookahead and the backward view's memory traces generate the identical cumulative weight update.

Worked numerical example

Consider a 2-step episodic trajectory: $S_0 \xrightarrow{R_1 = 3.0} S_1 \xrightarrow{R_2 = 4.0} S_T \quad (\text{terminal, } V(S_T) = 0.0)$

Hyperparameters:

• Initial estimates: $V(S_0) = 1.0$, $V(S_1) = 2.0$.
• Discount factor $\gamma = 0.9$, decay parameter $\lambda = 0.7$, learning rate $\alpha = 0.1$.
• Effective trace decay: $\gamma \lambda = 0.9 \times 0.7 = 0.63$.

1. Forward View Calculation

At step $t = 0$ for state $S_0$:

• 1-step return: $G_0^{(1)} = R_1 + \gamma V(S_1) = 3.0 + 0.9(2.0) = 4.800$
• 2-step full return: $G_0^{(2)} = R_1 + \gamma R_2 + \gamma^2 V(S_T) = 3.0 + 0.9(4.0) + 0 = 6.600$
• $\lambda$-return: $G_0^\lambda = (1 - \lambda) G_0^{(1)} + \lambda G_0^{(2)} = 0.3(4.800) + 0.7(6.600) = 1.440 + 4.620 = 6.060$
• Forward update for $S_0$: $\Delta V_{\text{fwd}}(S_0) = \alpha [G_0^\lambda - V(S_0)] = 0.1 \times [6.060 - 1.0] = 0.1 \times 5.060 = \mathbf{0.506}$

At step $t = 1$ for state $S_1$:

• Full return to terminal: $G_1^{(1)} = R_2 + \gamma V(S_T) = 4.0 + 0 = 4.000$
• $\lambda$-return: $G_1^\lambda = 4.000$
• Forward update for $S_1$: $\Delta V_{\text{fwd}}(S_1) = \alpha [G_1^\lambda - V(S_1)] = 0.1 \times [4.000 - 2.0] = 0.1 \times 2.000 = \mathbf{0.200}$

2. Backward View Calculation

One-step TD errors:

• $\delta_0 = R_1 + \gamma V(S_1) - V(S_0) = 3.0 + 0.9(2.0) - 1.0 = 3.800$
• $\delta_1 = R_2 + \gamma V(S_T) - V(S_1) = 4.0 + 0.0 - 2.0 = 2.000$

Eligibility trace vector across steps ($z_k = [z_k(S_0), z_k(S_1)]$):

• Step $0$: Visit $S_0 \implies z_0(S_0) = 1.000, \; z_0(S_1) = 0.000$
• Step $1$: Decay by $0.63$, visit $S_1 \implies z_1(S_0) = 0.63 \times 1.0 = 0.630, \; z_1(S_1) = 0.63(0) + 1.0 = 1.000$

Accumulated backward updates:

• For state $S_0$: $\Delta V_{\text{bwd}}(S_0) = \alpha \delta_0 z_0(S_0) + \alpha \delta_1 z_1(S_0) = 0.1(3.800)(1.0) + 0.1(2.000)(0.630) = 0.380 + 0.126 = \mathbf{0.506}$
• For state $S_1$: $\Delta V_{\text{bwd}}(S_1) = \alpha \delta_0 z_0(S_1) + \alpha \delta_1 z_1(S_1) = 0.1(3.800)(0.0) + 0.1(2.000)(1.0) = 0.000 + 0.200 = \mathbf{0.200}$

The offline forward updates and offline backward updates match identically to three decimal places.

Offline Equivalence vs. Online Drift and True Online TD(λ)

The mathematical derivation above assumes that value estimates $V$ remain frozen while accumulating updates until the episode ends. When practitioners run standard online $\text{TD}(\lambda)$, value estimates are updated immediately after every transition:

$V_{t+1}(s) \leftarrow V_t(s) + \alpha \delta_t z_t(s)$

Because $V$ changes on the fly, subsequent prediction errors $\delta_{t+1}$ are computed using $V_{t+1}$ rather than $V_0$. This causes standard online $\text{TD}(\lambda)$ to drift away from the forward-view $\lambda$-return, especially under large learning rates $\alpha$.

To resolve this discrepancy, van Seijen and Sutton (2014) introduced True Online $\text{TD}(\lambda)$. By employing Dutch traces:

$e_t = \gamma \lambda e_{t-1} + \left(1 - \alpha \gamma \lambda \mathbf{x}_t^\top e_{t-1}\right) \mathbf{x}_t$

and maintaining an auxiliary tracking scalar, True Online $\text{TD}(\lambda)$ achieves exact step-by-step equivalence to the forward view in an online, causal algorithm—strictly dominating standard $\text{TD}(\lambda)$ across tabular and linear approximation benchmarks.

Code

1import math2from typing import Dict, List, Tuple3
4
5def verify_forward_backward_equivalence(6    trajectory: List[Tuple[str, float]],7    initial_values: Dict[str, float],8    gamma: float = 0.9,9    lam: float = 0.7,10    alpha: float = 0.1,11) -> Tuple[Dict[str, float], Dict[str, float]]:12    """Compute and verify offline forward and backward TD(lambda) updates.13
14    Args:15        trajectory: Sequence of (state, reward) pairs ending in 'terminal'.16        initial_values: Initial state-value mapping V_0(s).17        gamma: Discount factor in [0, 1].18        lam: Trace decay parameter lambda in [0, 1].19        alpha: Learning rate step size.20
21    Returns:22        Tuple of (forward_updates, backward_updates) dictionaries.23    """24    states = [s for s, _ in trajectory[:-1]]25    t_max = len(states)26    unique_states = sorted(list(set(states)))27
28    # ---------------------------------------------------------29    # 1. Offline Forward View: Compute lambda-return G_t^lambda30    # ---------------------------------------------------------31    fwd_updates: Dict[str, float] = {s: 0.0 for s in unique_states}32
33    for t in range(t_max):34        s_t = states[t]35        # Collect all n-step returns from step t36        n_step_returns: List[float] = []37        for n in range(1, t_max - t + 1):38            # Sum discounted rewards up to step t + n39            return_val = 0.040            for i in range(n):41                reward_step = trajectory[t + i + 1][1]42                return_val += (gamma**i) * reward_step43
44            # Add bootstrapped terminal value of horizon45            next_state = trajectory[t + n][0]46            if next_state != "terminal":47                return_val += (gamma**n) * initial_values[next_state]48            n_step_returns.append(return_val)49
50        # Blend n-step returns with geometric weights (1 - lambda)*lambda^(n-1)51        g_lambda = 0.052        for n_idx, g_n in enumerate(n_step_returns[:-1]):53            weight = (1.0 - lam) * (lam**n_idx)54            g_lambda += weight * g_n55
56        # Final term receives remaining geometric tail weight lambda^(H-1)57        last_weight = lam ** (len(n_step_returns) - 1)58        g_lambda += last_weight * n_step_returns[-1]59
60        # Accumulate offline forward update61        fwd_updates[s_t] += alpha * (g_lambda - initial_values[s_t])62
63    # ---------------------------------------------------------64    # 2. Offline Backward View: Accumulating Eligibility Traces65    # ---------------------------------------------------------66    bwd_updates: Dict[str, float] = {s: 0.0 for s in unique_states}67    traces: Dict[str, float] = {s: 0.0 for s in unique_states}68
69    for t in range(t_max):70        s_t = states[t]71        next_s, r_t1 = trajectory[t + 1]72
73        # Decay traces and bump visited state74        for s in traces:75            traces[s] *= gamma * lam76        traces[s_t] += 1.077
78        # One-step TD error using initial values79        next_v = 0.0 if next_s == "terminal" else initial_values[next_s]80        delta_t = r_t1 + gamma * next_v - initial_values[s_t]81
82        # Accumulate backward update across all states83        for s in bwd_updates:84            bwd_updates[s] += alpha * delta_t * traces[s]85
86    return fwd_updates, bwd_updates87
88
89if __name__ == "__main__":90    # Trajectory: S0 ->(R=3)-> S1 ->(R=4)-> terminal91    episode: List[Tuple[str, float]] = [92        ("S0", 0.0),93        ("S1", 3.0),94        ("terminal", 4.0),95    ]96    v_init: Dict[str, float] = {"S0": 1.0, "S1": 2.0}97
98    fwd, bwd = verify_forward_backward_equivalence(99        episode, v_init, gamma=0.9, lam=0.7, alpha=0.1100    )101
102    print("=== Verification of Forward-Backward Equivalence ===")103    for state in sorted(fwd.keys()):104        f_val = fwd[state]105        b_val = bwd[state]106        is_equal = math.isclose(f_val, b_val, rel_tol=1e-9)107        print(108            f"State {state} | Forward Delta V: {f_val:.6f} | "109            f"Backward Delta V: {b_val:.6f} | Exact Match: {is_equal}"110        )111        assert is_equal, f"Mismatch found in state {state}!"112
113    print("\nAll offline updates match to machine precision.")114
115# Expected Output:116# === Verification of Forward-Backward Equivalence ===117# State S0 | Forward Delta V: 0.506000 | Backward Delta V: 0.506000 | Exact Match: True118# State S1 | Forward Delta V: 0.200000 | Backward Delta V: 0.200000 | Exact Match: True119#120# All offline updates match to machine precision.

Watch Out For

The Quick Version

• Theoretical vs. mechanistic: The forward view defines an ideal learning target ($G_t^\lambda$) by looking into the future, whereas the backward view implements a causal, online algorithm using decaying memory traces ($z_t$).
• The telescoping proof: Any multi-step error $G_t^{(n)} - V(S_t)$ telescopes into a discounted sum of one-step TD errors $\sum \gamma^k \delta_k$. Blending these into $G_t^\lambda - V(S_t)$ yields $\sum (\gamma \lambda)^k \delta_k$.
• Summation swap equivalence: By reversing the order of double summation across the episode, the future lookahead factors naturally regroup into accumulating eligibility traces: $\sum \Delta V_{\text{fwd}}(s) \equiv \sum \Delta V_{\text{bwd}}(s)$.
• Online drift and True Online TD: Exact equivalence holds strictly offline; standard online updating introduces drift, which True Online $\text{TD}(\lambda)$ resolves using Dutch traces.

What to Read Next

• Forward View of TD(lambda)
• Backward View of TD(lambda)
• SARSA(lambda)