Noisy Nets for Exploration

Why Does This Exist?

In classic deep reinforcement learning, agents rely on heuristic $\epsilon$-greedy exploration: with probability $1 - \epsilon$, the agent selects the greedy action $\arg\max_a Q(s, a)$, and with probability $\epsilon$, it samples a uniformly random action from $\mathcal{A}$.

While simple, $\epsilon$-greedy suffers from three critical deficiencies:

1. Destructive Action Dithering: $\epsilon$-greedy explores in action space by sampling independent random choices at every single time step. This introduces Brownian random-walk dynamics: the agent moves left, then right, then left, canceling its own progress. In environments requiring coherent multi-step navigation (such as navigating a maze or solving sparse-reward games like Montezuma's Revenge), random dithering virtually never reaches distant goals.
2. State-Agnostic Exploration: $\epsilon$-greedy treats every state identically. Even after an agent has completely mastered opening moves, it continues executing random, game-ending blunders with probability $\epsilon$.
3. Brittle Hyperparameter Schedules: Designing an exploration schedule requires manually tuning $\epsilon_{\text{start}}$, $\epsilon_{\text{end}}$, and decay rates across millions of frames, which rarely transfers across different tasks.

Introduced by Meire Fortunato et al. (DeepMind, 2018), Noisy Networks for Exploration (Noisy Nets) replace heuristic action dithering with learnable parametric noise injected directly into the neural network's weights. By perturbing parameters rather than actions, the policy remains temporally consistent across consecutive decisions, enabling structured, deep exploration.

Crucially, because the noise scale parameters $\sigma$ are updated by gradient descent, exploration naturally self-anneals: as value uncertainty declines in mastered states, gradients automatically drive $\sigma \to 0$.

Think of It Like This

How It Actually Works

Mathematical Formulation and Factorised Noise Architecture

In a standard deterministic fully-connected layer, an input vector $x \in \mathbb{R}^p$ is transformed to output $y \in \mathbb{R}^q$ via:

$y = W x + b$

Where $W \in \mathbb{R}^{q \times p}$ and $b \in \mathbb{R}^q$ are learnable parameter tensors.

A NoisyLinear layer replaces fixed weights and biases with random variables parameterized by learnable means ($\mu$) and learnable noise scale deviations ($\sigma$):

$y \doteq \left( \mu^w + \sigma^w \odot \varepsilon^w \right) x + \left( \mu^b + \sigma^b \odot \varepsilon^b \right)$

Where:

• $\mu^w \in \mathbb{R}^{q \times p}$ and $\mu^b \in \mathbb{R}^q$ are learnable mean parameters.
• $\sigma^w \in \mathbb{R}^{q \times p}$ and $\sigma^b \in \mathbb{R}^q$ are learnable noise scale parameters.
• $\varepsilon^w \in \mathbb{R}^{q \times p}$ and $\varepsilon^b \in \mathbb{R}^q$ are zero-mean stochastic noise tensors.
• $\odot$ denotes element-wise (Hadamard) multiplication.

Independent vs. Factorised Gaussian Noise

There are two primary ways to generate the noise tensors $\varepsilon^w$ and $\varepsilon^b$:

1. Independent Gaussian Noise: Every element $\varepsilon^w_{i, j}$ and $\varepsilon^b_i$ is sampled independently from $\mathcal{N}(0, 1)$. This requires generating $p \times q + q$ random variables per forward pass, which becomes computationally prohibitive in deep networks with wide hidden layers.

2. Factorised Gaussian Noise (Standard in Deep RL): Instead of generating $p \times q$ independent variables, factorised noise generates only $p$ unit Gaussian variables for the inputs ($\varepsilon_{\text{in}} \in \mathbb{R}^p$) and $q$ variables for the outputs ($\varepsilon_{\text{out}} \in \mathbb{R}^q$), requiring only $p + q$ random draws.

Each component is transformed using the non-linear real-valued function:

$f(z) \doteq \operatorname{sgn}(z) \sqrt{|z|}$

The weight and bias noise matrices are then constructed via outer product:

$\varepsilon^w_{i, j} \doteq f(\varepsilon_{\text{out}, i}) \cdot f(\varepsilon_{\text{in}, j}), \quad \varepsilon^b_i \doteq f(\varepsilon_{\text{out}, i})$

The Self-Annealing Gradient Mechanism

During backpropagation, gradients flow into both the mean parameters and the scale parameters:

$\frac{\partial \mathcal{L}}{\partial \mu^w_{i, j}} = \frac{\partial \mathcal{L}}{\partial y_i} x_j, \quad \frac{\partial \mathcal{L}}{\partial \sigma^w_{i, j}} = \frac{\partial \mathcal{L}}{\partial y_i} x_j \cdot \varepsilon^w_{i, j}$

$\frac{\partial \mathcal{L}}{\partial \mu^b_i} = \frac{\partial \mathcal{L}}{\partial y_i}, \quad \frac{\partial \mathcal{L}}{\partial \sigma^b_i} = \frac{\partial \mathcal{L}}{\partial y_i} \cdot \varepsilon^b_i$

Why does exploration self-anneal? Consider a state where the agent has already converged to the correct value function. In this regime, weight perturbations $\sigma^w \odot \varepsilon^w$ cause the network's output to deviate from the optimal Bellman target, increasing the loss $\mathcal{L}$. Because the gradient $\frac{\partial \mathcal{L}}{\partial \sigma^w_{i, j}}$ is positively correlated with loss increases, gradient descent pushes $\sigma^w_{i, j}$ downward toward zero.

In contrast, in novel states with high value uncertainty, exploratory perturbations uncover rewarding trajectories that reduce the loss, sustaining non-zero $\sigma$ values. The network autonomously decides where and how much to explore without manual schedules.

Worked Numerical Example

Consider a single NoisyLinear neuron with $p = 2$ inputs and $q = 1$ output.

Step 1: Initial Parameters and Input

• Input vector: $x = [2.0, -1.0]^\top$
• Mean parameters: $\mu^w = [0.5, -0.2]$, $\mu^b = 0.1$
• Scale parameters: $\sigma^w = [0.4, 0.3]$, $\sigma^b = 0.2$
• Learning rate: $\alpha = 0.1$

Step 2: Factorised Noise Generation

• Sample input noise: $\varepsilon_{\text{in}} = [1.0, -4.0]$
• Sample output noise: $\varepsilon_{\text{out}} = [1.0]$

Apply transform $f(z) = \operatorname{sgn}(z)\sqrt{|z|}$:

• $f(\varepsilon_{\text{in}}[0]) = f(1.0) = \operatorname{sgn}(1.0)\sqrt{1.0} = 1.0$
• $f(\varepsilon_{\text{in}}[1]) = f(-4.0) = \operatorname{sgn}(-4.0)\sqrt{4.0} = -2.0$
• $f(\varepsilon_{\text{out}}[0]) = f(1.0) = \operatorname{sgn}(1.0)\sqrt{1.0} = 1.0$

Compute noise matrices:

• $\varepsilon^w_0 = f(\varepsilon_{\text{out}}[0]) \cdot f(\varepsilon_{\text{in}}[0]) = 1.0 \times 1.0 = 1.0$
• $\varepsilon^w_1 = f(\varepsilon_{\text{out}}[0]) \cdot f(\varepsilon_{\text{in}}[1]) = 1.0 \times (-2.0) = -2.0$
• $\varepsilon^b = f(\varepsilon_{\text{out}}[0]) = 1.0$

Step 3: Realized Weights and Forward Pass

Realize perturbed parameters: $w_0 = \mu^w_0 + \sigma^w_0 \varepsilon^w_0 = 0.5 + 0.4 \times (1.0) = 0.90$ $w_1 = \mu^w_1 + \sigma^w_1 \varepsilon^w_1 = -0.2 + 0.3 \times (-2.0) = -0.2 - 0.6 = -0.80$ $b = \mu^b + \sigma^b \varepsilon^b = 0.1 + 0.2 \times (1.0) = 0.30$

Compute perturbed forward output: $y = w_0 x_0 + w_1 x_1 + b = (0.90 \times 2.0) + (-0.80 \times -1.0) + 0.30 = 1.80 + 0.80 + 0.30 = 2.90$

(Note: The deterministic unperturbed mean output is $\mu^w \cdot x + \mu^b = 1.0 + 0.2 + 0.1 = 1.30$. Parametric noise shifted the output by $+1.60$.)

Step 4: Loss and Gradient Backpropagation

Suppose the target Bellman value is $y^* = 1.50$.

• Loss: $\mathcal{L} = \frac{1}{2} (y - y^*)^2 = \frac{1}{2} (2.90 - 1.50)^2 = 0.98$
• Output error: $\delta = y - y^* = 2.90 - 1.50 = 1.40$

Compute parameter gradients:

• $\nabla_{\mu^w} \mathcal{L} = \delta \cdot x = 1.40 \times [2.0, -1.0] = [2.80, -1.40]$
• $\nabla_{\sigma^w} \mathcal{L} = \delta \cdot (x \odot \varepsilon^w) = 1.40 \times [2.0 \times 1.0, -1.0 \times (-2.0)] = [2.80, 2.80]$
• $\nabla_{\mu^b} \mathcal{L} = \delta = 1.40$
• $\nabla_{\sigma^b} \mathcal{L} = \delta \cdot \varepsilon^b = 1.40 \times 1.0 = 1.40$

Step 5: SGD Parameter Update ($\alpha = 0.1$)

$\mu^w \leftarrow [0.5, -0.2] - 0.1 \times [2.80, -1.40] = [0.22, -0.06]$ $\sigma^w \leftarrow [0.4, 0.3] - 0.1 \times [2.80, 2.80] = [0.12, 0.02]$ $\mu^b \leftarrow 0.1 - 0.1 \times 1.40 = -0.04$ $\sigma^b \leftarrow 0.2 - 0.1 \times 1.40 = 0.06$

The Self-Annealing Effect in Action: Because the positive noise perturbation caused the network to significantly overshoot the target ($y = 2.90$ vs. $y^* = 1.50$), gradient descent directly reduced the noise scale parameters:

• $\sigma^w_0$ decreased from $0.40 \to 0.12$
• $\sigma^w_1$ decreased from $0.30 \to 0.02$
• $\sigma^b$ decreased from $0.20 \to 0.06$

The layer autonomously dampened its noise parameters to restore precision.

Code

1import math2import random3from typing import List, Sequence, Tuple4
5
6def f_transform(x: float) -> float:7    """Factorised noise non-linear transform: f(x) = sgn(x) * sqrt(|x|)."""8    return math.copysign(math.sqrt(abs(x)), x)9
10
11class NoisyLinear:12    """Fully-connected layer with factorised Gaussian parametric noise."""13
14    def __init__(self, in_features: int, out_features: int, sigma_zero: float = 0.5) -> None:15        self.in_features = in_features16        self.out_features = out_features17
18        # Initialize mean parameters (mu) and scale parameters (sigma)19        bound = 1.0 / math.sqrt(in_features)20        self.mu_w: List[List[float]] = [21            [random.uniform(-bound, bound) for _ in range(in_features)]22            for _ in range(out_features)23        ]24        self.mu_b: List[float] = [random.uniform(-bound, bound) for _ in range(out_features)]25
26        sigma_w_init = sigma_zero / math.sqrt(in_features)27        self.sigma_w: List[List[float]] = [28            [sigma_w_init for _ in range(in_features)] for _ in range(out_features)29        ]30        self.sigma_b: List[float] = [31            sigma_zero / math.sqrt(out_features) for _ in range(out_features)32        ]33
34        # Internal noise buffers35        self.eps_in: List[float] = [0.0] * in_features36        self.eps_out: List[float] = [0.0] * out_features37
38    def sample_noise(39        self,40        eps_in: Sequence[float] | None = None,41        eps_out: Sequence[float] | None = None,42    ) -> None:43        """Sample or inject factorised Gaussian noise."""44        if eps_in is not None and eps_out is not None:45            self.eps_in = list(eps_in)46            self.eps_out = list(eps_out)47        else:48            self.eps_in = [random.gauss(0.0, 1.0) for _ in range(self.in_features)]49            self.eps_out = [random.gauss(0.0, 1.0) for _ in range(self.out_features)]50
51    def forward(self, x: Sequence[float]) -> List[float]:52        """Compute y = (mu_w + sigma_w * eps_w) x + (mu_b + sigma_b * eps_b)."""53        f_in = [f_transform(v) for v in self.eps_in]54        f_out = [f_transform(v) for v in self.eps_out]55
56        outputs = [0.0] * self.out_features57        for i in range(self.out_features):58            b_realized = self.mu_b[i] + self.sigma_b[i] * f_out[i]59            w_dot_x = 0.060            for j in range(self.in_features):61                weight_noise = f_out[i] * f_in[j]62                w_realized = self.mu_w[i][j] + self.sigma_w[i][j] * weight_noise63                w_dot_x += w_realized * x[j]64            outputs[i] = w_dot_x + b_realized65
66        return outputs67
68
69if __name__ == "__main__":70    # Reproduce the exact worked numerical example (p=2 inputs, q=1 output)71    layer = NoisyLinear(in_features=2, out_features=1)72
73    # Set parameters to exact test values74    layer.mu_w[0] = [0.5, -0.2]75    layer.mu_b[0] = 0.176    layer.sigma_w[0] = [0.4, 0.3]77    layer.sigma_b[0] = 0.278
79    # Inject exact factorised noise vectors80    layer.sample_noise(eps_in=[1.0, -4.0], eps_out=[1.0])81
82    x = [2.0, -1.0]83    y = layer.forward(x)[0]84    print(f"Forward Output y: {y:.4f}")85
86    # Compute loss against target y* = 1.5087    y_star = 1.5088    delta = y - y_star89    loss = 0.5 * (delta**2)90    print(f"Target: {y_star:.2f}, Delta: {delta:.4f}, Loss: {loss:.4f}")91
92    # Compute analytical gradients93    f_in = [f_transform(v) for v in layer.eps_in]94    f_out = [f_transform(v) for v in layer.eps_out]95    eps_w = [f_out[0] * f_in[0], f_out[0] * f_in[1]]96    eps_b = f_out[0]97
98    grad_mu_w = [delta * x[j] for j in range(2)]99    grad_sigma_w = [delta * x[j] * eps_w[j] for j in range(2)]100    grad_mu_b = delta101    grad_sigma_b = delta * eps_b102
103    print(f"Grad mu_w:    {[round(g, 4) for g in grad_mu_w]}")104    print(f"Grad sigma_w: {[round(g, 4) for g in grad_sigma_w]}")105    print(f"Grad mu_b:    {grad_mu_b:.4f}")106    print(f"Grad sigma_b: {grad_sigma_b:.4f}")107
108    # Apply SGD parameter update with learning rate alpha = 0.1109    lr = 0.1110    layer.mu_w[0] = [layer.mu_w[0][j] - lr * grad_mu_w[j] for j in range(2)]111    layer.sigma_w[0] = [layer.sigma_w[0][j] - lr * grad_sigma_w[j] for j in range(2)]112    layer.mu_b[0] -= lr * grad_mu_b113    layer.sigma_b[0] -= lr * grad_sigma_b114
115    print("\nUpdated Parameters after 1 Step:")116    print(f"mu_w:    {[round(v, 4) for v in layer.mu_w[0]]}")117    print(f"sigma_w: {[round(v, 4) for v in layer.sigma_w[0]]}")118    print(f"mu_b:    {layer.mu_b[0]:.4f}")119    print(f"sigma_b: {layer.sigma_b[0]:.4f}")

1# Expected Output:2# Forward Output y: 2.90003# Target: 1.50, Delta: 1.4000, Loss: 0.98004# Grad mu_w:    [2.8, -1.4]5# Grad sigma_w: [2.8, 2.8]6# Grad mu_b:    1.40007# Grad sigma_b: 1.40008# 9# Updated Parameters after 1 Step:10# mu_w:    [0.22, -0.06]11# sigma_w: [0.12, 0.02]12# mu_b:    -0.040013# sigma_b: 0.0600

Watch Out For

The Quick Version

• Weight Space Exploration: Replaces heuristic $\epsilon$-greedy action dithering by injecting stochastic noise directly into neural network weights ($y = (\mu^w + \sigma^w \odot \varepsilon^w)x + (\mu^b + \sigma^b \odot \varepsilon^b)$).
• Temporally Consistent Policies: Because parameters remain stable across decision evaluations, the agent executes coherent multi-step exploratory sequences rather than random Brownian jitter.
• Autonomous Self-Annealing: Gradients naturally penalize noise scale parameters $\sigma$ in mastered states where perturbations increase Bellman loss, driving $\sigma \to 0$ without manual decay schedules.
• Factorised Efficiency: Factorised Gaussian noise generates weight perturbations via outer products of input and output vectors ($f(\varepsilon_{\text{out}}) f(\varepsilon_{\text{in}})^\top$), reducing noise variables from $O(p \cdot q)$ down to $O(p + q)$.

What to Read Next

• Deep Q-Networks
• Rainbow DQN
• Dueling Network Architectures