Hindsight Experience Replay (HER)

Even when an agent misses its intended objective, it successfully achieved something; by pretending in hindsight that what it reached was the target all along, it extracts valuable learning signals from failures in sparse-reward environments.

Hindsight Experience Replay converts unrewarded trajectories into successful demonstrations by substituting the original goal with actually achieved states.

Why Does This Exist?

In multi-goal reinforcement learning tasks—such as a robotic arm learning to place a block at an arbitrary 3D coordinate—rewards are naturally binary and sparse. The agent receives a reward of $0$ when the object settles within a tight threshold $\epsilon$ of the target goal $g$ , and $-1$ at every other time step. Under standard exploration, an untrained agent executing random motor babbling has a vanishingly small probability of reaching a distant target coordinate by chance.

Consequently, almost every rollout trajectory finishes with complete failure: the replay buffer fills with transitions where $r = -1$ across all steps. Because the value landscape is completely flat, temporal-difference updates have no non-zero reward signals to bootstrap from. Q-learning stalls indefinitely because all actions appear equally futile.

The conventional workaround was reward shaping: replacing binary rewards with negative Euclidean distances $\|s - g\|$ . However, engineered distance rewards introduce severe pitfalls:

They create unintended local optima (for example, hovering near an obstacle rather than reaching around it).
They alter the optimal policy objective.
They fail in non-Euclidean state spaces (such as maze navigation or complex manipulation).

Hindsight Experience Replay (HER), introduced by Marcin Andrychowicz et al. (2017), solves this bottleneck without modifying the true reward function. HER recognizes that every failed trajectory to target goal $g$ is simultaneously a perfect, successful trajectory to whichever state $s_T$ the agent actually managed to visit. By relabeling the transitions in hindsight with achieved states, the agent manufactures guaranteed success signals ( $r = 0$ ) out of complete failures.

Think of It Like This

The Archery Student and the Missed Target

Imagine a novice archer taking aim at a distant red bullseye on Target A. The archer draws the bow, releases, and the arrow flies wide, lodging deeply into the bark of an old oak tree thirty yards to the left.

Under strict tournament scoring, this shot is an absolute failure: zero points. If the archer discards the attempt as useless waste, they learn almost nothing about how their muscles, release velocity, and stance influenced arrow trajectory.

Now imagine a perceptive coach standing nearby who says: "Forget Target A for a moment. Pretend your intention all along was to hit that exact oak tree. Your stance, draw tension, and release angle were an absolute masterclass in striking that tree trunk! Let us record that technique into your muscle memory."

Once the student logs thousands of relabeled shots—learning how to deliberately strike oak trees, fence posts, rocks, and dirt mounds—they develop a comprehensive mental map of bow mechanics across the entire field. When asked once more to hit Target A, they simply consult their repertoire: Target A is merely another point in space they now know how to reach.

The analogy stops when considering physics: in real archery, an arrow's trajectory does not change based on what you wished to hit. Similarly, in reinforcement learning, the environment's transition dynamics $P(s_{t+1} \mid s_t, a_t)$ are strictly independent of the goal vector $g$ . This physical invariance is precisely what allows an agent to retroactively change the goal label in its replay memory without invalidating transition physics.

How It Actually Works

Goal-Conditioned Formulation and Universal Value Functions

HER operates within the framework of Goal-Conditioned Markov Decision Processes (Universal Value Function Approximators, or UVFAs). The state space $\mathcal{S}$ is paired with a goal space $\mathcal{G}$ , and there is a mapping $m: \mathcal{S} \to \mathcal{G}$ that extracts the achieved goal from an environmental state.

The policy $\pi(a \mid s, g)$ and action-value function $Q(s, a, g)$ take both the current state $s$ and the desired goal $g$ as inputs. The environment provides a sparse goal-conditioned reward:

r(s, a, g) = \begin{cases} 0 & \text{if } \|m(s') - g\| \le \epsilon \\ -1 & \text{otherwise} \end{cases}

where $s'$ is the next state and $\epsilon$ is a task tolerance threshold.

The Goal Relabeling Mechanism

During standard rollouts, an agent is assigned a desired goal $g$ and generates an episode trajectory:

\tau = \left( (s_0, a_0, r_0, s_1, g), (s_1, a_1, r_1, s_2, g), \dots, (s_{T-1}, a_{T-1}, r_{T-1}, s_T, g) \right)

Under standard replay storage, all $T$ transitions are stored in replay buffer $\mathcal{D}$ using original goal $g$ . If $g$ was never reached, $r_t = -1$ for all $t \in [0, T-1]$ .

HER augments the buffer by creating additional hindsight copies of each transition $(s_t, a_t, s_{t+1})$ . For each transition at step $t$ , HER selects a set of alternative goals $G'$ from states visited during the episode, recomputes the reward under $g' \in G'$ , and stores the augmented tuple:

(s_t, a_t, r', s_{t+1}, g') \quad \text{where} \quad r' = r(s_{t+1}, a_t, g')

Andrychowicz et al. evaluated four primary goal-sampling strategies:

final: Relabel using exclusively the final achieved state of the rollout: $g' = m(s_T)$ .
future: For transition at time step $t$ , randomly sample $k$ alternative goals from states visited later in the same episode ( $t' \in [t+1, T]$ ). This is the empirical gold standard ( $k \approx 4$ ).
episode: Sample $k$ alternative goals randomly from any time step across the current episode.
random: Sample $k$ alternative goals uniformly from arbitrary states encountered in prior episodes.

The future strategy outperforms all others because it preserves temporal coherence: it samples goals that were physically reachable within the remaining time horizon of that trajectory.

Why Off-Policy RL is Mandatory

HER is fundamentally incompatible with on-policy reinforcement learning algorithms like PPO or A2C. An on-policy algorithm requires that the actions $a_t$ were generated directly by the current policy conditioned on the exact evaluated goal: $a_t \sim \pi(\cdot \mid s_t, g')$ . Because the trajectory was collected under the original goal $g$ , conditioning post-hoc on $g'$ constitutes off-policy distribution shift.

Off-policy algorithms (such as DDPG, TD3, or SAC) evaluate transitions using the Bellman optimality equation:

y_t = r' + \gamma Q_{\text{target}}\left(s_{t+1}, \pi(s_{t+1}, g'), g'\right)

Because the physical environment transition $P(s_{t+1} \mid s_t, a_t)$ is invariant to the goal argument, the transition $(s_t, a_t, s_{t+1})$ remains completely valid under any goal $g'$ . The target network simply evaluates the value of reaching $g'$ from $s_{t+1}$ .

Worked numerical example

Consider a 1D continuous positioning task. An agent starts at coordinate $s_0 = 0.0$ and is tasked with reaching $g_{\text{target}} = 10.0$ with tolerance $\epsilon = 0.5$ . The discount factor is $\gamma = 0.90$ and the learning rate is $\alpha = 0.20$ .

The reward function is:

r(s', g) = \begin{cases} 0.0 & \text{if } |s' - g| \le 0.5 \\ -1.0 & \text{otherwise} \end{cases}

The agent executes constant velocity actions $a_t = +2.0$ for $T = 3$ steps:

Step 0: $s_0 = 0.0 \xrightarrow{a_0=2.0} s_1 = 2.0$ . Distance $|2.0 - 10.0| = 8.0 > 0.5 \implies r_0 = -1.0$ .
Step 1: $s_1 = 2.0 \xrightarrow{a_1=2.0} s_2 = 4.0$ . Distance $|4.0 - 10.0| = 6.0 > 0.5 \implies r_1 = -1.0$ .
Step 2: $s_2 = 4.0 \xrightarrow{a_2=2.0} s_3 = 6.0$ . Distance $|6.0 - 10.0| = 4.0 > 0.5 \implies r_2 = -1.0$ .

1. Standard Experience Replay Failure

Every transition in the buffer stores $r = -1.0$ . If prior values are uniformly initialized to $-5.0$ , every Bellman target evaluates to $-1.0 + 0.90(-5.0) = -5.50$ . The agent receives zero directional signal explaining how to navigate towards $10.0$ .

2. HER Goal Relabeling

We apply HER using the final strategy: we relabel the rollout with achieved goal $g' = s_3 = 6.0$ .

Relabeled Step 2 ( $s_2 = 4.0 \xrightarrow{a_2=2.0} s_3 = 6.0$ , goal $g'=6.0$ ):
- Check success condition: $|s_3 - g'| = |6.0 - 6.0| = 0.0 \le 0.5 \implies r'_2 = 0.0$ .
- State $s_3$ satisfies the goal, marking successful termination with respect to $g'$ . Terminal value $V(s_3, g') = 0.0$ .
- Bellman target: $y_2 = r'_2 + \gamma \cdot 0.0 = 0.0 + 0.0 = 0.0$
- Initial Q-estimate: $Q(s_2=4.0, a_2=2.0, g'=6.0) = -2.50$ .
- TD Error: $\delta_2 = y_2 - Q = 0.0 - (-2.50) = +2.50$
- Updated Q-value: $Q_{\text{new}}(s_2, a_2, g') = -2.50 + 0.20 \times (+2.50) = -2.00$
Relabeled Step 1 ( $s_1 = 2.0 \xrightarrow{a_1=2.0} s_2 = 4.0$ , goal $g'=6.0$ ):
- Check success condition: $|s_2 - g'| = |4.0 - 6.0| = 2.0 > 0.5 \implies r'_1 = -1.0$ .
- Next state $s_2$ is not terminal. It bootstraps from the newly updated downstream Q-value: $y_1 = r'_1 + \gamma Q_{\text{new}}(s_2, a_2, g') = -1.0 + 0.90 \times (-2.00) = -1.0 - 1.80 = -2.80$
- Initial Q-estimate: $Q(s_1=2.0, a_1=2.0, g'=6.0) = -4.00$ .
- TD Error: $\delta_1 = -2.80 - (-4.00) = +1.20$
- Updated Q-value: $Q_{\text{new}}(s_1, a_1, g') = -4.00 + 0.20 \times (+1.20) = -3.76$

Notice the transformation: the single hindsight success at $s_3$ immediately propagated a positive $+2.50$ gradient to Step 2 and a $+1.20$ gradient to Step 1. The agent has acquired concrete knowledge of how to reach coordinate $6.0$ . As multiple rollouts achieve varied endpoints, the Universal Value Function interpolates across the goal space, unlocking rapid convergence toward the true target $10.0$ .

Code

The following self-contained implementation demonstrates a complete HindsightExperienceReplayBuffer supporting both future and final relabeling strategies alongside original trajectory storage.

1from dataclasses import dataclass2from typing import Callable, List, Optional, Tuple3import numpy as np4
5
6@dataclass7class Transition:8    state: np.ndarray9    action: np.ndarray10    reward: float11    next_state: np.ndarray12    done: bool13    goal: np.ndarray14
15
16class HindsightExperienceReplayBuffer:17    """18    Experience Replay Buffer implementing Hindsight Experience Replay (HER).19    Stores rollout episodes and creates hindsight-relabeled transitions.20    """21
22    def __init__(23        self,24        capacity: int = 10_000,25        k_future: int = 4,26        strategy: str = "future",27        reward_fn: Optional[Callable[[np.ndarray, np.ndarray], float]] = None,28    ) -> None:29        self.capacity = capacity30        self.k_future = k_future31        self.strategy = strategy32        self.reward_fn = reward_fn or self._default_sparse_reward33        self.buffer: List[Transition] = []34        self.position = 035
36    @staticmethod37    def _default_sparse_reward(38        achieved_state: np.ndarray, goal: np.ndarray, threshold: float = 0.539    ) -> float:40        """Sparse binary reward: 0.0 if within threshold distance, else -1.0."""41        distance = float(np.linalg.norm(achieved_state - goal))42        return 0.0 if distance <= threshold else -1.043
44    def add_episode(45        self,46        trajectory: List[Tuple[np.ndarray, np.ndarray, np.ndarray, bool, np.ndarray]],47    ) -> None:48        """49        Stores an entire episode trajectory and augments it with HER relabeling.50
51        trajectory: List of (state, action, next_state, done, original_goal) tuples.52        """53        episode_length = len(trajectory)54        if episode_length == 0:55            return56
57        for t in range(episode_length):58            state, action, next_state, done, original_goal = trajectory[t]59
60            # 1. Store the original transition under original goal61            orig_reward = self.reward_fn(next_state, original_goal)62            self._store(63                Transition(64                    state=state,65                    action=action,66                    reward=orig_reward,67                    next_state=next_state,68                    done=done,69                    goal=original_goal,70                )71            )72
73            # 2. HER Relabeling74            if self.strategy == "future":75                # Sample k states observed after step t in the current episode76                candidate_indices = list(range(t, episode_length))77                sampled_indices = np.random.choice(78                    candidate_indices, size=self.k_future, replace=True79                )80                for idx in sampled_indices:81                    achieved_goal = trajectory[idx][2]  # next_state of step idx82                    relabelled_reward = self.reward_fn(next_state, achieved_goal)83                    relabelled_done = bool(relabelled_reward == 0.0)84
85                    self._store(86                        Transition(87                            state=state,88                            action=action,89                            reward=relabelled_reward,90                            next_state=next_state,91                            done=relabelled_done,92                            goal=achieved_goal,93                        )94                    )95
96            elif self.strategy == "final":97                # Relabel using strictly the final state of the episode98                final_achieved_goal = trajectory[-1][2]99                relabelled_reward = self.reward_fn(next_state, final_achieved_goal)100                relabelled_done = bool(relabelled_reward == 0.0)101
102                self._store(103                    Transition(104                        state=state,105                        action=action,106                        reward=relabelled_reward,107                        next_state=next_state,108                        done=relabelled_done,109                        goal=final_achieved_goal,110                    )111                )112
113    def _store(self, transition: Transition) -> None:114        if len(self.buffer) < self.capacity:115            self.buffer.append(transition)116        else:117            self.buffer[self.position] = transition118        self.position = (self.position + 1) % self.capacity119
120    def sample_batch(self, batch_size: int) -> List[Transition]:121        indices = np.random.choice(len(self.buffer), size=batch_size, replace=False)122        return [self.buffer[i] for i in indices]123
124    def __len__(self) -> int:125        return len(self.buffer)126
127
128# Demonstration129if __name__ == "__main__":130    np.random.seed(42)131
132    her_buffer = HindsightExperienceReplayBuffer(133        capacity=1000, k_future=4, strategy="future"134    )135
136    # Simulated rollout attempting to reach distant goal [10.0]137    target_goal = np.array([10.0])138    rollout = [139        (np.array([0.0]), np.array([2.0]), np.array([2.0]), False, target_goal),140        (np.array([2.0]), np.array([2.0]), np.array([4.0]), False, target_goal),141        (np.array([4.0]), np.array([2.0]), np.array([6.0]), True, target_goal),142    ]143
144    her_buffer.add_episode(rollout)145
146    orig_successes = sum(147        1 for tr in her_buffer.buffer if np.array_equal(tr.goal, target_goal) and tr.reward == 0.0148    )149    total_successes = sum(1 for tr in her_buffer.buffer if tr.reward == 0.0)150
151    print(f"Total stored transitions: {len(her_buffer)}")152    print(f"Successes under target goal [10.0]: {orig_successes}")153    print(f"Hindsight relabeled successes (r=0.0): {total_successes}")154
155    sample = her_buffer.sample_batch(batch_size=3)156    for i, tr in enumerate(sample):157        print(158            f"Sample {i}: s={tr.state[0]:.1f} -> s'={tr.next_state[0]:.1f} | "159            f"goal={tr.goal[0]:.1f} | reward={tr.reward:.1f}"160        )

Output:

1Total stored transitions: 152Successes under target goal [10.0]: 03Hindsight relabeled successes (r=0.0): 84Sample 0: s=0.0 -> s'=2.0 | goal=10.0 | reward=-1.05Sample 1: s=4.0 -> s'=6.0 | goal=6.0 | reward=0.06Sample 2: s=4.0 -> s'=6.0 | goal=6.0 | reward=0.0

Watch Out For

Sampling Over-Saturation: The k-Ratio Imbalance

A frequent trap when implementing HER is setting the hindsight replay parameter $k$ excessively high (for example, $k = 16$ or $k = 32$ ) in hopes of maximizing positive reinforcement.

When $k$ is too large, the replay buffer becomes overwhelmingly saturated with one-step transitions where the relabeled goal was reached almost instantly ( $s_{t+1} \approx g'$ ). The policy rapidly learns to stabilize locally around trivial nearby states but suffers from goal myopia: it never learns the extended temporal sequences necessary to cross long distances toward the original training goals. Conversely, setting $k = 1$ provides too sparse a hindsight signal, slowing down exploration.

The Fix: Stick to the empirical ratio established in the seminal paper: $k \approx 4$ for standard manipulation tasks (yielding an approximate ratio of 4 hindsight transitions for every 1 original transition, or an 80/20 mix during mini-batch sampling). If training on very long-horizon tasks, combine HER with prioritized experience replay (PER) so transitions with high temporal-difference error are sampled preferentially over trivial one-step arrivals.

The Quick Version

Core Intuition: In sparse-reward multi-goal RL, failed trajectories to target goal $g$ are successful trajectories to the states actually reached; relabeling them in hindsight extracts rich learning signals from failures.
Goal Relabeling Modes: The future strategy (sampling $k \approx 4$ states visited later in the same episode) outperforms final, episode, and random because it respects the reachable time horizon.
Off-Policy Synergy: HER strictly requires off-policy reinforcement learning (such as DDPG, TD3, or SAC) because changing the goal vector post-rollout introduces an off-policy distribution shift that invalidates on-policy methods like PPO.
Reward Shaping Alternative: HER solves the sparse-reward exploration plateau without the distortion, local optima, and task-specific engineering traps of hand-crafted distance rewards.