Forward View of TD(lambda)

Why Does This Exist?

In value-based reinforcement learning, choosing how far into the future an agent should look before updating its value estimates represents a fundamental trade-off:

1. One-step Temporal Difference learning [TD(0)] looks ahead exactly one transition. It enjoys low target variance and enables real-time online updates, but suffers from high bootstrapping bias because early value predictions are inherently noisy and inaccurate.
2. Monte Carlo (MC) methods look ahead across the entire trajectory until termination. They have zero bootstrapping bias because they rely exclusively on realized rewards, but suffer from high target variance as stochastic state transitions and action choices compound multiplicatively over time.
3. Multi-step TD ($n$-step TD) compromises by looking ahead $n$ steps before bootstrapping. While effective, choosing a single fixed horizon $n$ is arbitrary: a 3-step return might perform best in short-horizon subgoals, while a 15-step return might excel in long corridors.

This leads to a central question: why restrict the learning target to an arbitrary single horizon $n$? Any convex combination of valid returns—where non-negative weights sum to one—produces a mathematically sound update target.

The forward view of $\text{TD}(\lambda)$ solves this problem by defining the $\lambda$-return ($G_t^\lambda$). Instead of picking a single $n$, it combines all future $n$-step returns into a single composite target weighted by an exponential decay parameter $\lambda \in [0, 1]$. Shorter, low-variance horizons receive the greatest weight, while longer, low-bias horizons receive progressively decaying contributions.

Crucially, the geometric weighting $(1 - \lambda)\lambda^{n-1}$ is not an arbitrary artistic curve: its unique memoryless property allows this seemingly impractical, forward-looking theoretical target to be computed causally, online, and backward in time via eligibility traces—the foundation of the backward view of $\text{TD}(\lambda)$.

Think of It Like This

How It Actually Works

Mathematical Formulation of the Compound $\lambda$-Return

To understand the forward view, we first construct the family of $n$-step returns and then aggregate them geometrically.

1. The $n$-Step Return Family

For a state visited at time $t$, the $n$-step return $G_{t:t+n}$ accumulates realized rewards for $n$ consecutive time steps and bootstraps on the estimated value of state $S_{t+n}$:

$G_{t:t+n} \doteq R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots + \gamma^{n-1} R_{t+n} + \gamma^n V_t(S_{t+n})$

where:

• $\gamma \in [0, 1]$ is the discount factor.
• $R_{t+k}$ is the reward received at step $t+k$.
• $V_t(S_{t+n})$ is the value estimate of state $S_{t+n}$ at step $t$.
• If $t + n \ge T$ (where $T$ is the terminal time step of an episode), all remaining rewards beyond termination are zero, and the return collapses to the complete Monte Carlo return $G_t$:

$G_{t:t+n} \doteq G_t = \sum_{k=1}^{T-t} \gamma^{k-1} R_{t+k} \quad (\text{for } t + n \ge T)$

2. The Continuing / Infinite-Horizon $\lambda$-Return

In an infinite-horizon task, the forward $\lambda$-return $G_t^\lambda$ is defined as the geometrically weighted sum of all $n$-step returns for $n \in \{1, 2, 3, \dots\}$:

$G_t^\lambda \doteq (1 - \lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n}$

where $\lambda \in [0, 1]$. Notice that the geometric weights sum exactly to one:

$\sum_{n=1}^{\infty} (1 - \lambda)\lambda^{n-1} = (1 - \lambda) \sum_{k=0}^{\infty} \lambda^k = (1 - \lambda) \cdot \frac{1}{1 - \lambda} = 1$

Because the weights form a valid probability distribution over horizons $n$, the $\lambda$-return is a true convex combination of valid return targets.

3. Episodic Truncation at Terminal Step $T$

In episodic tasks that terminate at time step $T$, there are only $T - t$ distinct transitions remaining. Every $n$-step return for $n \ge T - t$ is identical to the complete Monte Carlo return $G_t$:

$G_{t:t+n} = G_t \quad \text{for all } n \ge T - t$

Summing the infinite tail of geometric weights for all $n \ge T - t$:

$\sum_{n=T-t}^{\infty} (1 - \lambda)\lambda^{n-1} = (1 - \lambda) \lambda^{T-t-1} \sum_{k=0}^{\infty} \lambda^k = (1 - \lambda)\lambda^{T-t-1} \cdot \frac{1}{1 - \lambda} = \lambda^{T-t-1}$

This gives the practical episodic formula for the forward $\lambda$-return:

$G_t^\lambda \doteq (1 - \lambda) \sum_{n=1}^{T-t-1} \lambda^{n-1} G_{t:t+n} + \lambda^{T-t-1} G_t$

The finite sum weights $(1 - \lambda)\sum_{n=1}^{T-t-1} \lambda^{n-1} = 1 - \lambda^{T-t-1}$, and the terminal tail receives the remaining weight $\lambda^{T-t-1}$. Their sum is identically $1.0$.

4. Boundary Cases: The Spectrum of $\lambda$

• When $\lambda = 0$: $G_t^{\lambda=0} = (1 - 0) \cdot 0^0 \cdot G_{t:t+1} + 0^{T-t-1} \cdot G_t = G_{t:t+1} = R_{t+1} + \gamma V(S_{t+1})$ The $\lambda$-return reduces strictly to the standard one-step TD(0) return.
• When $\lambda = 1$: $G_t^{\lambda=1} = (1 - 1) \sum_{n=1}^{T-t-1} 1^{n-1} G_{t:t+n} + 1^{T-t-1} G_t = 0 + G_t = G_t$ The $\lambda$-return reduces strictly to the full trajectory Monte Carlo return.
• When $0 < \lambda < 1$: The $\lambda$-return smoothly interpolates between TD(0) and Monte Carlo, allowing the agent to capture long-range consequences while retaining the variance-damping stability of bootstrapping.

5. The Forward-View Value Update

The theoretical forward-view update rule updates the value of state $S_t$ toward the composite target $G_t^\lambda$:

$V(S_t) \leftarrow V(S_t) + \alpha \left[ G_t^\lambda - V(S_t) \right]$

where $\alpha \in (0, 1]$ is the step-size parameter.

Worked numerical example

Consider a 3-step episodic trajectory:

$S_0 \xrightarrow{R_1 = 2.0} S_1 \xrightarrow{R_2 = 1.0} S_2 \xrightarrow{R_3 = 4.0} S_3 (\text{terminal})$

Let the hyperparameter values be:

• Discount factor $\gamma = 0.9$
• Geometric blending factor $\lambda = 0.7$
• Step size $\alpha = 0.1$
• Terminal state value $V(S_3) = 0.0$

Suppose current state value estimates before this episode are: $V(S_0) = 1.0, \quad V(S_1) = 3.0, \quad V(S_2) = 5.0, \quad V(S_3) = 0.0$

We evaluate the forward $\lambda$-return $G_0^\lambda$ at time $t = 0$. Here, total episode length $T = 3$, so $T - t = 3$.

Step 1: Compute All Individual $n$-Step Returns from $t=0$

• 1-step return ($n = 1$): $G_{0:1} = R_1 + \gamma V(S_1) = 2.0 + (0.9 \times 3.0) = 2.0 + 2.70 = 4.7000$

• 2-step return ($n = 2$): $G_{0:2} = R_1 + \gamma R_2 + \gamma^2 V(S_2) = 2.0 + (0.9 \times 1.0) + (0.9^2 \times 5.0) = 2.0 + 0.90 + (0.81 \times 5.0) = 2.90 + 4.05 = 6.9500$

• 3-step terminal return ($n = 3$, Full Monte Carlo Return $G_0$): $G_{0:3} = G_0 = R_1 + \gamma R_2 + \gamma^2 R_3 + \gamma^3 V(S_3) = 2.0 + 0.90 + (0.81 \times 4.0) + 0.0 = 2.90 + 3.24 = 6.1400$

Step 2: Compute Geometric Weight Weights

For $T - t = 3$, the weights assigned to $G_{0:1}$, $G_{0:2}$, and $G_0$ are:

• Weight for $G_{0:1}$ ($n = 1$): $w_1 = (1 - \lambda)\lambda^0 = 1 - 0.7 = 0.30$
• Weight for $G_{0:2}$ ($n = 2$): $w_2 = (1 - \lambda)\lambda^1 = 0.30 \times 0.7 = 0.21$
• Weight for $G_0$ ($n = 3$, terminal tail): $w_3 = \lambda^{T-t-1} = 0.7^{3-0-1} = 0.7^2 = 0.49$

Check total weight: $w_1 + w_2 + w_3 = 0.30 + 0.21 + 0.49 = 1.00$

Step 3: Compute the Composite $G_0^\lambda$ Target

Multiply each $n$-step return by its geometric weight:

• Contribution from $G_{0:1}$: $0.30 \times 4.7000 = 1.4100$
• Contribution from $G_{0:2}$: $0.21 \times 6.9500 = 1.4595$
• Contribution from $G_0$: $0.49 \times 6.1400 = 3.0086$

Sum the weighted components: $G_0^{\lambda=0.7} = 1.4100 + 1.4595 + 3.0086 = 5.8781$

Step 4: Compare with the Extreme Boundaries

• Under $\lambda = 0$ (TD(0)): Target is $G_{0:1} = 4.7000$.
• Under $\lambda = 1$ (Monte Carlo): Target is $G_0 = 6.1400$.
• Under $\lambda = 0.7$: Target is $5.8781$, smoothly balancing the low 1-step target and the high full rollout return.

Step 5: Update State Value $V(S_0)$

Applying the forward-view update with step size $\alpha = 0.1$: $V(S_0) \leftarrow V(S_0) + \alpha \left[ G_0^\lambda - V(S_0) \right] = 1.0 + 0.1 \times [5.8781 - 1.0] = 1.0 + 0.48781 = 1.4878$

The state value $V(S_0)$ increases from $1.0000$ to $1.4878$.

Code

Below is a self-contained, type-hinted Python script that implements the forward-view $\lambda$-return calculation from trajectory logs and verifies boundary convergence across the entire $\lambda$ spectrum:

1from dataclasses import dataclass2from typing import Dict, List, Optional, Tuple3
4
5@dataclass(frozen=True)6class Transition:7    """Represents a single experienced transition step."""8    state: str9    reward: float10    next_state: str11
12
13def compute_n_step_return(14    trajectory: List[Transition],15    start_idx: int,16    n: int,17    gamma: float,18    values: Dict[str, float],19) -> float:20    """Compute the n-step return G_{t:t+n} starting from trajectory[start_idx].21    22    If start_idx + n reaches or exceeds the terminal step, bootstraps on 0.0.23    """24    total_steps = len(trajectory)25    horizon = min(start_idx + n, total_steps)26    accumulated_return = 0.027    discount = 1.028
29    # Sum discounted rewards along the n-step horizon30    for k in range(start_idx, horizon):31        accumulated_return += discount * trajectory[k].reward32        discount *= gamma33
34    # If the horizon did not reach terminal step T, bootstrap on V(S_{t+n})35    if start_idx + n < total_steps:36        bootstrap_state = trajectory[start_idx + n].state37        accumulated_return += discount * values.get(bootstrap_state, 0.0)38
39    return accumulated_return40
41
42def compute_forward_lambda_returns(43    trajectory: List[Transition],44    gamma: float,45    lam: float,46    values: Dict[str, float],47) -> List[float]:48    """Compute the forward lambda-return G_t^lambda for every time step t.49    50    Uses the episodic truncation formula:51    G_t^lambda = (1 - lam) * sum_{n=1}^{T-t-1} lam^(n-1) * G_{t:t+n} + lam^(T-t-1) * G_t52    """53    total_steps = len(trajectory)54    lambda_returns: List[float] = []55
56    for t in range(total_steps):57        steps_remaining = total_steps - t58        # Compute all valid n-step returns for n in 1 ... steps_remaining59        n_step_returns = [60            compute_n_step_return(trajectory, t, n, gamma, values)61            for n in range(1, steps_remaining + 1)62        ]63
64        # Blend n-step returns with geometric weights65        g_lambda = 0.066        for idx in range(steps_remaining - 1):67            n = idx + 168            weight = (1.0 - lam) * (lam ** (n - 1))69            g_lambda += weight * n_step_returns[idx]70
71        # The terminal tail accumulates all remaining weight72        terminal_weight = lam ** (steps_remaining - 1)73        g_lambda += terminal_weight * n_step_returns[-1]74
75        lambda_returns.append(g_lambda)76
77    return lambda_returns78
79
80def run_forward_td_demonstration() -> None:81    # 3-step trajectory matching the numerical example82    trajectory: List[Transition] = [83        Transition(state="S0", reward=2.0, next_state="S1"),84        Transition(state="S1", reward=1.0, next_state="S2"),85        Transition(state="S2", reward=4.0, next_state="ST"),86    ]87
88    values: Dict[str, float] = {"S0": 1.0, "S1": 3.0, "S2": 5.0, "ST": 0.0}89    gamma = 0.990    lam = 0.791    alpha = 0.192
93    print("--- Individual n-Step Returns at t=0 ---")94    for n in range(1, 4):95        g_n = compute_n_step_return(trajectory, 0, n, gamma, values)96        print(f"G_0:{n} = {g_n:.4f}")97
98    # Compute compound forward targets99    g_lambda_all = compute_forward_lambda_returns(trajectory, gamma, lam, values)100    print(f"\nG_0^lambda (lambda={lam}): {g_lambda_all[0]:.4f}")101    print(f"G_1^lambda (lambda={lam}): {g_lambda_all[1]:.4f}")102    print(f"G_2^lambda (lambda={lam}): {g_lambda_all[2]:.4f}")103
104    print("\n--- Lambda Spectrum at t=0 ---")105    for test_lam in [0.0, 0.5, 0.7, 1.0]:106        g_val = compute_forward_lambda_returns(trajectory, gamma, test_lam, values)[0]107        print(f"lambda={test_lam:.1f} -> G_0^lambda = {g_val:.4f}")108
109    # Offline Forward-View Value Updates110    updated_values = dict(values)111    for t, step in enumerate(trajectory):112        target = g_lambda_all[t]113        updated_values[step.state] += alpha * (target - updated_values[step.state])114
115    print("\n--- Updated Values after Forward Pass ---")116    for s in ["S0", "S1", "S2"]:117        print(f"V({s}): {values[s]:.4f} -> {updated_values[s]:.4f}")118
119
120if __name__ == "__main__":121    run_forward_td_demonstration()122
123# -> Expected output:124# -> --- Individual n-Step Returns at t=0 ---125# -> G_0:1 = 4.7000126# -> G_0:2 = 6.9500127# -> G_0:3 = 6.1400128# -> 129# -> G_0^lambda (lambda=0.7): 5.8781130# -> G_1^lambda (lambda=0.7): 4.8700131# -> G_2^lambda (lambda=0.7): 4.0000132# -> 133# -> --- Lambda Spectrum at t=0 ---134# -> lambda=0.0 -> G_0^lambda = 4.7000135# -> lambda=0.5 -> G_0^lambda = 5.6225136# -> lambda=0.7 -> G_0^lambda = 5.8781137# -> lambda=1.0 -> G_0^lambda = 6.1400138# -> 139# -> --- Updated Values after Forward Pass ---140# -> V(S0): 1.0000 -> 1.4878141# -> V(S1): 3.0000 -> 3.1870142# -> V(S2): 5.0000 -> 4.9000

Watch Out For

The Quick Version

• Compound Geometric Averaging: The forward view of $\text{TD}(\lambda)$ constructs a single composite target, the $\lambda$-return $G_t^\lambda$, by calculating a geometrically decaying average of all possible $n$-step returns with weights $(1 - \lambda)\lambda^{n-1}$.
• Unifying the Spectrum: $\lambda$ acts as an interpolation knob between pure 1-step temporal difference learning ($\lambda = 0$, low variance, high bias) and full-trajectory Monte Carlo evaluation ($\lambda = 1$, zero bias, high variance).
• Exact Convex Weights: For episodic tasks, all weights beyond horizon $T - t$ accumulate into the full return $G_t$ with weight $\lambda^{T-t-1}$, guaranteeing that weights always sum strictly to $1.0$.
• Theoretical Target vs. Online Implementation: Because the forward view requires future transitions, it cannot run causally online; it serves as the theoretical benchmark faithfully implemented by the backward view via eligibility traces.

What to Read Next

• n-Step TD Prediction
• Backward View of TD(lambda)
• Eligibility Traces