Hierarchical RL
Hierarchical RL splits a long task in two: a manager picks the next milestone every few steps, and a worker takes the small actions that reach it.
Why Does This Exist?
Flat agents drown in long tasks. When reward arrives only after a thousand correct moves, credit assignment is hopeless: which of the thousand mattered? Exploration is hopeless too, since random thrashing almost never stumbles through the full sequence.
Hierarchy shortens every horizon at once. A manager looks at the state every c steps and names a subgoal, learning from the environment's own sparse rewards. A worker acts every step toward that subgoal, learning from dense progress rewards the manager hands out. A thousand step ordeal becomes ten manager decisions over hundred step worker shifts, and the worker's skills transfer: one manager's trained walker serves the next manager's errand runner. The formal home of these ideas is the options framework, with learned skills in option-critic and stacked control in feudal networks.
The one idea you need first
A semi-Markov step lasts many primitive steps. If the manager commits for c = 3 steps with discount gamma = 0.90, rewards inside the commitment discount normally while the next manager value discounts by gamma^c = 0.729. The manager's Bellman backup is the macro return plus 0.729 times the next manager value.
Think of It Like This
A head chef and a line cook
A head chef never narrates knife work. They call out tickets: three steaks medium rare by half past, then sauces for table nine. The line cook turns each ticket into dozens of small moves: season, sear, rest, plate.
The chef judges plates against real customers, while the cook judges each move against the ticket: closer to done is good, further is bad. The chef plans ten tickets a night instead of a thousand cuts, the cook masters searing once and reuses it under every future chef, and a bad ticket still teaches the cook something about timing.
Where it stops: the cook's hands keep improving while the chef's plans assume fixed hands, so the chef's world shifts under them. Real kitchens retrain together daily; naive hierarchy code does not, which is the first pitfall below.
How It Actually Works
Two policies run at two speeds. The manager policy pi_high(g | s) reads state s and picks a goal g, then waits c steps or until the goal is reached. The worker policy pi_low(a | s, g) reads the state plus the goal and picks a primitive action every step.
The worker learns from intrinsic reward, which pays for progress toward g rather than task success. On a line with goal g0, one honest choice is distance reduced:
The manager learns from extrinsic reward over its commitment with the semi-Markov backup. With macro return R^{macro} across c steps and next state s':
Worked example
Walk a line from s = 0.0 toward 10.0. The manager commits for c = 3 steps with gamma = 0.90 and sets g0 = 3.0. The worker steps +1.0 three times: positions 1.0, 2.0, 3.0. Distances to the goal fall 3.0 to 2.0 to 1.0 to 0.0, so intrinsic rewards are +1.00 each and the worker banks +3.00 total.
The environment pays extrinsic +1.00 only on reaching the milestone at step three. The macro return is 0 + 0.90 * 0 + 0.90^2 * 1.00 = 0.81, and the effective manager discount is 0.90^3 = 0.729. If the best next manager value at the new state is 4.00, the target is 0.81 + 0.729 * 4.00 = 0.81 + 2.916 = 3.726. Against a current estimate of 3.20, the manager error is 3.726 - 3.20 = +0.526, a clean positive push for a subgoal that worked.
Code
This snippet replays the manager backup with plain Python, printing the same 0.81, 0.729, 3.726, and +0.526 derived by hand.
gamma = 0.90steps = 3extrinsic = [0.0, 0.0, 1.0]next_best = 4.00current = 3.20
macro = sum((gamma ** t) * r for t, r in enumerate(extrinsic))manager_discount = gamma ** stepstarget = macro + manager_discount * next_besterror = target - currentprint(f"{macro:.2f} {manager_discount:.3f}") # -> 0.81 0.729print(f"{target:.3f} {error:+.3f}") # -> 3.726 +0.526The prints agree with the hand pass: macro return 0.81, manager target 3.726, error +0.526.
Watch Out For
Training both levels at once without admitting it
While the worker improves, old manager experience goes stale: a goal that failed in week one succeeds in week three with the same manager command. The manager's world breaks its Markov promise, and off policy manager updates wobble or diverge.
Symptom: manager values oscillate as the worker gets competent, and replay from early training actively hurts. Fix: relabel old goals with what the worker can do now, train the manager on fresher data, or freeze the worker while the manager catches up.
Picking a goal space nobody can use
Goals must be easy for the manager to name and possible for the worker to reach. Raw pixels overwhelm the manager, while three hand picked waypoints may skip the one narrow door the task needs.
Symptom: the worker masters its intrinsic game yet task reward never moves, or the manager parks on one goal forever. Fix: start goals as positions or simple features, check that the worker reaches sampled goals most of the time, and only then grow the space toward latent embeddings.
The Quick Version
- Long sparse tasks beat flat agents through the credit assignment problem, since one reward must explain a thousand moves.
- The manager picks subgoals every few steps from extrinsic reward, while the worker acts each step for dense intrinsic progress reward.
- Commitments of length
cdiscount the next manager value bygamma^c, which was 0.729 in the worked example. - The worked walk banked +3.00 intrinsic, a 0.81 macro return, a 3.726 manager target, and a +0.526 learning push.
- Workers transfer across managers, but joint training breaks the manager's stationarity, and bad goal spaces waste all of it.
- Continue with the options framework, then option-critic and feudal networks.