Reward Shaping

When the goal pays only at the end, shaping adds small step hints that point toward it. Done right the hints speed learning without changing the best path.

One step closer turns a silent zero into a shaping bonus of 1.09, while flat bonuses pay for looping

Why Does This Exist?

A maze that pays plus 1 only at the exit is honest and almost unlearnable. A random walker in a 10 by 10 grid may take thousands of episodes to stumble out once, and until that first success every update has nothing to learn from. The signal is correct but too rare to steer by.

Reward shaping adds hints along the way so every step teaches something. Move closer, get a small plus. That dense feedback turns a hopeless search into a hill the agent can climb. The danger is that the hints become the game, and the agent climbs the hints instead of reaching the exit.

This page covers how to add hints without changing the goal. The true goal signal lives in reward function. The update that consumes the dense signal lives in temporal difference learning. A different fix that relabels failed goals lives in hindsight experience replay.

The one idea you need first is the shaped reward. Write the true reward as $R$ and the hint as $F$ , with $s$ the current state, $a$ the action, $s^\prime$ the next state, and $\gamma$ (gamma) the discount. The agent learns from $R^\prime = R + F(s, a, s^\prime)$ . Here $R$ is the sparse truth and $F$ is your added hint. The rest of the page is how to choose $F$ so the best path stays the best path.

Think of It Like This

Teaching a puppy to fetch

You want the ball in your hand. If you reward only the final drop, the puppy naps. So you pay in stages: a crumb for glancing at the ball, a bigger crumb for picking it up, a jackpot for bringing it back. Each crumb is a hint that says closer.

That maps part for part. The hand is the sparse goal. The glance and the pickup are intermediate states. The crumbs are $F$ , small on purpose. The jackpot is $R$ , large on purpose. The puppy chaining crumbs into a full fetch is a policy climbing shaped rewards into the true goal.

Where it stops: pay a full biscuit for every pickup and the puppy learns to pick up and drop forever without ever returning. That is reward hacking, and it is exactly what flat shaping pays for. The fix in both worlds is the same. Make crumbs that cancel on loops and keep the jackpot biggest.

How It Actually Works

Start from the naive form and then constrain it. Naive shaping adds any hint you like:

$R^\prime = R + F(s, a, s^\prime)$

Here $R^\prime$ is what the agent optimizes, $R$ is the true sparse reward, and $F$ is the hint for moving from $s$ to $s^\prime$ after $a$ . Any $F$ speeds learning and almost any $F$ can change the optimal path, which is why naive shaping has a bad reputation.

The potential fix

Potential based shaping, from Ng and coauthors in 1999, builds $F$ from a potential $\Phi$ (phi) over states:

$F(s, a, s^\prime) = \gamma \Phi(s^\prime) - \Phi(s)$

Here $\Phi(s)$ scores how promising a state looks, such as negative distance to the goal, and $\gamma$ is the same discount the return uses. Closer means higher potential. Because $F$ is a difference of potentials, extra reward for going forward is taken back on the way back, so detours cannot beat the direct path. That telescoping property is what preserves the optimal policy. Any loop nets near zero shaping, so looping for points stops paying.

Worked example

Grid with goal at $(5, 5)$ . Potential is negative Manhattan distance, so closer is higher. Discount $\gamma$ is 0.99. The agent steps from $s = (0, 0)$ to $s^\prime = (1, 0)$ . True reward $R$ is 0.0 on this step.

Potentials are $\Phi(s) = -(5 + 5) = -10.0$ and $\Phi(s^\prime) = -(4 + 5) = -9.0$ . The shaping term is:

$F = 0.99 \times (-9.0) - (-10.0) = -8.91 + 10.0 = 1.09$

So the shaped reward is $R^\prime = 0.0 + 1.09 = 1.09$ . One step closer turns a silent 0.0 into a 1.09 hint, which is the number the diagram draws. Step back and the hint is $0.99 \times (-10.0) - (-9.0) = -0.90$ , so the round trip keeps only a small discount residue instead of farming points. A flat plus 1.0 for every forward step with no take back would pay forever on a two state loop, which is the failure the right panel of the diagram shows.

Code

Same grid and same 1.09, so the prose, the picture, and the run agree:

1from dataclasses import dataclass2
3
4@dataclass(frozen=True)5class Cell:6    x: int7    y: int8
9
10def potential(cell: Cell, goal: Cell) -> float:11    """Negative Manhattan distance. Closer is higher."""12    return -float(abs(cell.x - goal.x) + abs(cell.y - goal.y))13
14
15def shaped_step(s: Cell, s_next: Cell, base: float, goal: Cell, gamma: float = 0.99) -> float:16    """Shaped reward R prime for one move."""17    return base + (gamma * potential(s_next, goal) - potential(s, goal))18
19
20if __name__ == "__main__":21    goal = Cell(5, 5)22    print(round(shaped_step(Cell(0, 0), Cell(1, 0), base=0.0, goal=goal), 2))23    # -> 1.09

The print is 1.09, the hand computed bonus. Swap in the reverse move to see minus 0.90 come out.

Watch Out For

Flat step pay that funds a loop

The symptom is a policy that paces between two states while its episode return climbs. It never reaches the goal and never needs to. The cause is a hint with no take back, such as plus 1.0 for every forward move and 0.0 for every backward move. The loop harvests plus 1.0 every other step forever.

Fix it by writing hints as potential differences with the same $\gamma$ the learner uses, then test for loops directly. Run the policy from several starts and watch the state trace, not just the return curve. If the trace cycles while return rises, the shaping is paying for motion rather than progress.

Hints that drown the true goal

The symptom is fast early learning that plateaus short of the goal. The agent surfs shaping bonuses and ignores the sparse plus 1 it was meant to find. The cause is scale: shaping terms ten times larger than the goal, or a potential with cliffs that dominate the return.

Fix it by keeping hints small next to the goal and by checking the goal rate separately. Anneal shaping down late in training if needed, and always report the sparse success rate beside the shaped return. When shaped return rises and goal rate stays flat, turn the hints down.

The Quick Version

Sparse goals pay rarely, so random search starves. Shaping adds a hint per step.
Naive hints of the form $R + F$ can change the best path and teach loops.
Potential shaping with $F = \gamma \Phi(s^\prime) - \Phi(s)$ keeps the optimal policy intact.
From $(0, 0)$ to $(1, 0)$ toward $(5, 5)$ at gamma 0.99, the hint is 1.09.
Flat forward pay funds pacing. Potential differences take the loop profit back.
Keep hints small, anneal late, and judge runs by goal rate rather than shaped total.