Feudal Networks (FuN)

Why Does This Exist?

In long-horizon reinforcement learning tasks with sparse rewards (such as the infamous Atari game Montezuma's Revenge), standard "flat" RL algorithms fail completely. When an agent must execute thousands of primitive actions before receiving a single scalar reward, credit assignment breaks down: the probability of discovering a rewarding trajectory through random exploratory motor noise is exponentially vanishing $\mathcal{O}(|\mathcal{A}|^{-T})$.

Hierarchical Reinforcement Learning (HRL) aims to solve this by decomposing complex behaviors into multi-level hierarchies. However, early HRL frameworks faced crippling bottlenecks:

• Handcrafted Subgoals: Traditional architectures required engineers to manually define subgoals (e.g., discrete room coordinates or specific inventory flags), destroying end-to-end learning.
• The Options Dilemma: Methods like the Options Framework required learning when to initiate and terminate options, suffering from option collapse where options degraded into single-step primitive actions.
• Information Hiding Bottlenecks: Classical Feudal RL (Dayan & Hinton, 1993) hid state details from higher levels, creating severe communication bandwidth limits.

FeUdal Networks (FuN), introduced by Alexander Vezhnevets et al. at DeepMind in 2017, modernized feudal reinforcement learning for deep neural networks. FuN separates control into two distinct recurrent tiers operating at different temporal resolutions: a Manager operating at a dilated macro-timescale $c$, and a Worker operating at primitive timescale $t=1$.

Crucially, FuN eliminated manual subgoal engineering by defining subgoals as directional unit vectors in a learned continuous latent space. The Worker is motivated purely by cosine-similarity intrinsic rewards, while the Manager is trained using a novel Transition Policy Gradient that eliminates the need to backpropagate gradients through the Worker's motor decisions.

Think of It Like This

How It Actually Works

Architectural Decoupling: Manager and Worker

FuN takes raw environment observations $s_t$ (such as visual image frames) and passes them through a shared convolutional perception backbone to produce a low-dimensional latent state:

$z_t = f_{\text{percept}}(s_t) \in \mathbb{R}^d$

The architecture then splits into two specialized recurrent modules:

1Observation s_t ──> [ Perception Backbone ] ──> Latent Representation z_t2                                                     │3         ┌───────────────────────────────────────────┴───────────────────────────────────────────┐4         ▼                                                                                       ▼5┌─────────────────────────────────────────┐                                     ┌─────────────────────────────────────────┐6│ Dilated Manager RNN (Macro-step c)      │                                     │ Fast Worker RNN (Micro-step t = 1)      │7│ Ticks every c steps (e.g., c = 10)      │                                     │ Ticks every single environment step     │8│ Outputs Directional Goal g_t ∈ ℝ^d      │                                     │ Takes z_t and Goal Embedding            │9└─────────────────────────────────────────┘                                     └─────────────────────────────────────────┘10         │                                                                                       │11         ▼                                                                                       ▼12   Directional Goal g_t                                                            Primitive Motor Action a_t13         │                                                                                       │14         └───────────────────────────┬───────────────────────────────────────────────────────────┘15                                     ▼16                ┌─────────────────────────────────────────┐17                │ Latent Cosine Alignment Module          │18                │ Measures: cos(z_t - z_{t-i}, g_{t-i})   │19                │ Yields: Worker Intrinsic Reward r^I_t   │20                └─────────────────────────────────────────┘

1. The Dilated Manager (Macro-timescale $c$)

The Manager operates at a coarse temporal resolution, ticking once every $c$ steps (typically $c = 10$). To maintain temporal memory across hundreds of steps without vanishing gradients, the Manager uses a Dilated LSTM.

At macro-step $t$, the Manager outputs a continuous latent goal vector $g_t \in \mathbb{R}^d$, normalized to a unit vector:

$\hat{g}_t = \frac{g_t}{\|g_t\|}$

Key Insight: $g_t$ is not an absolute target coordinate in state space. It is a directional displacement vector, commanding: "Over the next $c$ steps, shift the environment's latent state in direction $\hat{g}_t$."

2. The Fast Worker (Primitive timescale $t=1$)

The Worker operates at every single environment step $t$. It receives the perceptual state $z_t$ and an internal goal embedding $w_t = \phi(g_t)$. It produces a probability distribution over primitive motor actions:

$a_t \sim \pi_{\text{worker}}\left(a_t \mid z_t, w_t\right)$

The Worker handles low-level agility, collision avoidance, and motor reflexes.

Worker Intrinsic Reward via Cosine Similarity

The Worker is completely shielded from external environment rewards $R_t$. Instead, it is motivated purely by an internal cosine-similarity intrinsic reward $r^I_t$.

At each step $t$, the intrinsic reward measures how closely the actual state displacement achieved over the past $c$ steps aligns with the Manager's commanded directional goal:

$r^I_t = \frac{1}{c} \sum_{i=1}^c \cos\left(z_t - z_{t-i},\, g_{t-i}\right)$

where the cosine similarity between two vectors $u$ and $v$ is:

$\cos(u, v) = \frac{u \cdot v}{\|u\| \|v\|}$

If the Worker's primitive actions move the agent in the exact direction commanded by the Manager, $\cos(\Delta z, g) = +1.0$, maximizing intrinsic payoff. If the Worker moves perpendicular to the goal, reward is $0.0$; if it moves in the opposite direction, reward is $-1.0$.

The Worker maximizes $r^I_t$ using standard policy gradient algorithms (such as A3C or PPO).

Transition Policy Gradients for the Manager

Training a hierarchical manager usually creates a mathematical dilemma: how can the Manager receive gradients without backpropagating through hundreds of unrolled Worker actions?

Vezhnevets et al. introduced Transition Policy Gradients by exploiting the structure of the goal space. The Worker is actively trained to align state displacements $z_{t+c} - z_t$ with goal $g_t$. Therefore, from the Manager's perspective, the probability of the environment transitioning from $z_t$ to $z_{t+c}$ can be modeled as:

$p\left(z_{t+c} \mid z_t, g_t\right) \propto \exp\left( \cos\left(z_{t+c} - z_t,\, g_t\right) \right)$

Applying the policy gradient theorem to this transition distribution yields the Manager's update rule:

$\nabla_{g_t} J_M = \mathbb{E} \left[ \left(R_t - V_M(s_t)\right) \nabla_{g_t} \cos\left(z_{t+c} - z_t,\, g_t\right) \right]$

where:

• $R_t = \sum_{k=0}^\infty \gamma^k r_{t+k}$ is the cumulative discounted extrinsic reward from the environment.
• $V_M(s_t)$ is the Manager's baseline value function.
• $A_M = R_t - V_M(s_t)$ is the Manager's advantage.

Intuition: If a sequence of actions produced high extrinsic reward ($A_M > 0$), the gradient pushes the Manager's emitted goal $g_t$ to point directly along the realized displacement vector $z_{t+c} - z_t$. The Manager learns to command directions that correlate with winning!

The Non-Hiding Principle

Earlier hierarchical RL architectures enforced "information hiding": the higher level hid sensory details from the lower level. This created artificial communication bottlenecks.

FuN adheres to the Non-Hiding Principle: both the Manager and Worker have full access to the shared perceptual representation $z_t$. Hierarchical abstraction is achieved not by concealing data, but through temporal dilation (the Manager only thinks every $c$ steps) and functional specialization (the Manager plans trajectories, the Worker actuates controls).

Worked numerical example

Let us trace a concrete macro-step of dilation horizon $c = 2$ in a 2-dimensional latent representation space.

Setup

• Initial latent state at $t=0$: $z_0 = [1.0, 0.0]$
• Manager directional goal: $g_0 = [0.0, 1.0]$ (aiming along the positive $y$-axis)
• Dilation horizon: $c = 2$

Step 1: Worker Execution ($t=1$)

The Worker executes primitive action $a_0$, moving the environment to: $z_1 = [1.0, 0.6]$

The displacement vector is: $\Delta z_1 = z_1 - z_0 = [1.0 - 1.0,\, 0.6 - 0.0] = [0.0,\, 0.6]$

Compute cosine similarity with commanded goal $g_0 = [0.0, 1.0]$: $\text{Dot Product} = 0.0 \times 0.0 + 0.6 \times 1.0 = 0.60$ $\|\Delta z_1\| = \sqrt{0.0^2 + 0.6^2} = 0.60, \quad \|g_0\| = 1.00$ $\cos(\Delta z_1, g_0) = \frac{0.60}{0.60 \times 1.00} = +1.0000$

The Worker's intrinsic reward at step 1 (with $c=2$): $r^I_1 = \frac{1}{2} \times \cos(\Delta z_1, g_0) = \frac{1}{2} \times 1.0000 = 0.5000$

Step 2: Worker Execution ($t=2$)

The Worker executes primitive action $a_1$, moving to: $z_2 = [1.2, 1.6]$

The total displacement from the macro-step origin $z_0$ is: $\Delta z_2 = z_2 - z_0 = [1.2 - 1.0,\, 1.6 - 0.0] = [0.2,\, 1.6]$

Compute cosine similarity: $\text{Dot Product} = 0.2 \times 0.0 + 1.6 \times 1.0 = 1.60$ $\|\Delta z_2\| = \sqrt{0.2^2 + 1.6^2} = \sqrt{0.04 + 2.56} = \sqrt{2.60} \approx 1.61245$ $\cos(\Delta z_2, g_0) = \frac{1.60}{1.61245 \times 1.00} \approx 0.992277 \approx 0.9923$

The Worker's intrinsic reward contribution at step 2: $r^I_2 = \frac{1}{2} \times 0.992277 \approx 0.4961$

Step 3: Manager Transition Policy Gradient Update

At step $t=2$, the environment emits an extrinsic reward $R_2 = 5.00$. The Manager's baseline value network estimated: $V_M(z_0) = 3.00$

The Manager advantage is: $A_M = R_2 - V_M(z_0) = 5.00 - 3.00 = +2.00$

Because the advantage is positive ($A_M = +2.00$), the Manager's Transition Policy Gradient updates $g_0$ to align even more strongly with the realized displacement vector $[0.2, 1.6]$. The Manager recognizes that directing the Worker into the upper-right quadrant produced high extrinsic returns!

Code

Below is a self-contained, type-hinted Python implementation of FeUdal Networks' directional goal calculation, Worker cosine-similarity intrinsic reward evaluation across dilation horizon $c$, and Manager Transition Policy Gradient updates:

1from dataclasses import dataclass2import math3from typing import List, Tuple4
5
6@dataclass7class FeUdalStepLog:8    """Stores metrics for a single hierarchical step in FuN."""9
10    timestep: int11    latent_state: List[float]12    displacement: List[float]13    cosine_sim: float14    worker_intrinsic_reward: float15
16
17class FeUdalNetworkSimulator:18    """Simulates the interaction between a dilated Manager and high-frequency Worker in FeUdal Networks."""19
20    def __init__(self, dilation_c: int = 2) -> None:21        self.c = dilation_c22
23    def cosine_similarity(24        self, vec_a: List[float], vec_b: List[float]25    ) -> float:26        """Computes cosine similarity: cos(a, b) = (a . b) / (||a|| * ||b||)."""27        dot = sum(x * y for x, y in zip(vec_a, vec_b))28        norm_a = math.sqrt(sum(x * x for x in vec_a))29        norm_b = math.sqrt(sum(y * y for y in vec_b))30        if norm_a == 0.0 or norm_b == 0.0:31            return 0.032        return dot / (norm_a * norm_b)33
34    def evaluate_worker_step(35        self,36        current_z: List[float],37        start_z: List[float],38        goal_direction: List[float],39    ) -> Tuple[List[float], float, float]:40        """Calculates displacement, cosine alignment, and step intrinsic reward."""41        disp = [curr - start for curr, start in zip(current_z, start_z)]42        cos_sim = self.cosine_similarity(disp, goal_direction)43        intrinsic_reward = (1.0 / self.c) * cos_sim44        return disp, cos_sim, intrinsic_reward45
46    def compute_manager_transition_gradient(47        self,48        extrinsic_return_R: float,49        manager_value_estimate: float,50        total_disp: List[float],51        goal: List[float],52    ) -> Tuple[float, float]:53        """Computes Manager advantage A_M = R - V_M and directional alignment."""54        advantage = extrinsic_return_R - manager_value_estimate55        alignment = self.cosine_similarity(total_disp, goal)56        return advantage, alignment57
58
59if __name__ == "__main__":60    sim = FeUdalNetworkSimulator(dilation_c=2)61
62    # 1. Worked Numerical Example Setup63    # Macro-step origin z_0 and Manager unit goal g_064    z_0 = [1.0, 0.0]65    g_0 = [0.0, 1.0]66
67    print("=== FeUdal Networks Macro-Step Simulation ===")68    print(f"Origin Latent State z_0:      {z_0}")69    print(f"Manager Directional Goal g_0: {g_0} (pointing along +y axis)")70
71    # 2. Worker Step 1: Moves to z_1 = [1.0, 0.6]72    z_1 = [1.0, 0.6]73    disp_1, cos_1, r_I_1 = sim.evaluate_worker_step(z_1, z_0, g_0)74
75    print("\n--- Worker Step 1 (t=1) ---")76    print(f"Arrived State z_1:           {z_1}")77    print(f"Displacement Delta z_1:      {disp_1}")78    print(f"Cosine Similarity:           {cos_1:.4f}")79    print(f"Worker Intrinsic Reward r_1: {r_I_1:.4f}")80
81    assert math.isclose(cos_1, 1.0000)82    assert math.isclose(r_I_1, 0.5000)83
84    # 3. Worker Step 2: Moves to z_2 = [1.2, 1.6]85    z_2 = [1.2, 1.6]86    disp_2, cos_2, r_I_2 = sim.evaluate_worker_step(z_2, z_0, g_0)87    norm_disp_2 = math.sqrt(disp_2[0] ** 2 + disp_2[1] ** 2)88
89    print("\n--- Worker Step 2 (t=2) ---")90    print(f"Arrived State z_2:           {z_2}")91    print(f"Displacement Delta z_2:      {disp_2}")92    print(f"Displacement Norm:           {norm_disp_2:.5f}")93    print(f"Cosine Similarity:           {cos_2:.4f}")94    print(f"Worker Intrinsic Reward r_2: {r_I_2:.4f}")95
96    assert math.isclose(norm_disp_2, 1.61245, rel_tol=1e-4)97    assert math.isclose(cos_2, 0.992277, rel_tol=1e-4)98    assert math.isclose(r_I_2, 0.496138, rel_tol=1e-4)99
100    # 4. Manager Transition Policy Gradient Update101    R_extrinsic = 5.00102    V_manager = 3.00103    advantage_M, final_alignment = (104        sim.compute_manager_transition_gradient(105            extrinsic_return_R=R_extrinsic,106            manager_value_estimate=V_manager,107            total_disp=disp_2,108            goal=g_0,109        )110    )111
112    print("\n=== Manager Transition Policy Gradient Update ===")113    print(f"Extrinsic Environment Return R: {R_extrinsic:.2f}")114    print(f"Manager Value Baseline V_M:     {V_manager:.2f}")115    print(f"Manager Advantage A_M:          {advantage_M:+.2f}")116    print(f"Directional Alignment:          {final_alignment:.4f}")117
118    assert math.isclose(advantage_M, 2.00)119    print("\nAll FeUdal Networks assertions verified successfully!")

Expected output:

1=== FeUdal Networks Macro-Step Simulation ===2Origin Latent State z_0:      [1.0, 0.0]3Manager Directional Goal g_0: [0.0, 1.0] (pointing along +y axis)4
5--- Worker Step 1 (t=1) ---6Arrived State z_1:           [1.0, 0.6]7Displacement Delta z_1:      [0.0, 0.6]8Cosine Similarity:           1.00009Worker Intrinsic Reward r_1: 0.500010
11--- Worker Step 2 (t=2) ---12Arrived State z_2:           [1.2, 1.6]13Displacement Delta z_2:      [0.19999999999999996, 1.6]14Displacement Norm:           1.6124515Cosine Similarity:           0.992316Worker Intrinsic Reward r_2: 0.496117
18=== Manager Transition Policy Gradient Update ===19Extrinsic Environment Return R: 5.0020Manager Value Baseline V_M:     3.0021Manager Advantage A_M:          +2.0022Directional Alignment:          0.992323
24All FeUdal Networks assertions verified successfully!

Watch Out For

The Quick Version

• Two-Tier Temporal Abstraction: FuN decouples hierarchical control into a dilated Manager operating every $c$ steps (e.g., $c=10$) and a high-frequency Worker operating at primitive step $t=1$.
• Directional Latent Subgoals: The Manager commands goals as directional unit vectors $g_t = d_t / \|d_t\|$ in learned representation space $z$, specifying which way to change the state rather than over-constraining the Worker with exact coordinates.
• Cosine Intrinsic Reward: The Worker is motivated purely by internal cosine similarity between achieved state displacement $\Delta z$ and the commanded goal $g_t$, providing dense guidance in sparse-reward environments.
• Transition Policy Gradients: The Manager is trained without backpropagating through unrolled Worker actions, directly reinforcing goals that aligned with state displacements yielding high extrinsic returns.

What to Read Next

• Hierarchical RL
• The Options Framework
• MAXQ Value Function Decomposition