Soft Actor-Critic (SAC)

Why Does This Exist?

In continuous control reinforcement learning, deterministic off-policy algorithms like Deep Deterministic Policy Gradient (DDPG) often suffer from hyperparameter sensitivity and severe value overestimation. Because deterministic policies converge prematurely to local optima, small errors in action-value estimation compound, driving the policy toward sub-optimal, brittle behaviors.

Conversely, on-policy algorithms such as Proximal Policy Optimization (PPO) provide stability through stochastic policies, but discard historical transitions after every update batch, requiring millions of interactions with physical or simulated environments.

Soft Actor-Critic (SAC), introduced by Haarnoja et al. (2018), bridges this divide. By framing policy optimization under the Maximum Entropy RL framework, SAC instructs the agent to maximize both cumulative task reward and policy entropy. This entropy incentive encourages broad exploration, prevents policy collapse onto near-deterministic narrow trajectories, and yields robust policies capable of adapting to environmental perturbations while retaining full off-policy sample efficiency.

Think of It Like This

How It Actually Works

Maximum Entropy Objective and the Tri-Component Architecture

Standard reinforcement learning maximizes expected return $\sum_t \mathbb{E}[r(s_t, a_t)]$. Soft Actor-Critic generalizes this to the maximum entropy objective:

where $\mathcal{H}(\pi(\cdot | s_t)) = \mathbb{E}_{a \sim \pi}[-\log \pi(a | s_t)]$ is the Shannon entropy of the action distribution, and $\alpha$ is the entropy temperature determining the relative importance of the entropy bonus against reward maximization.

SAC implements this optimization through three interacting mechanisms:

1. Clipped Double Q-Learning with Soft Bellman Backups: To prevent Q-function overestimation bias, SAC maintains two independent critic networks, $Q_{\theta_1}$ and $Q_{\theta_2}$, alongside slowly moving target critics $Q_{\bar{\theta}_1}$ and $Q_{\bar{\theta}_2}$. The soft state-value target takes the minimum between the twin critics and penalizes high certainty (low entropy):

   $$y = r + \gamma \left( \min_{i=1,2} Q_{\bar{\theta}_i}(s', \tilde{a}') - \alpha \log \pi_\phi(\tilde{a}' | s') \right), \quad \tilde{a}' \sim \pi_\phi(\cdot | s')$$

   Both critics are trained by minimizing mean squared Bellman error:

   $$L(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim \mathcal{D}} \left[ \left( Q_{\theta_i}(s, a) - y \right)^2 \right]$$

2. Reparameterized Squashed Gaussian Policy: Continuous actions are bounded to a valid physical range $[-1, 1]$ using the hyperbolic tangent function $\tanh$. To allow backpropagation directly through policy sampling, actions are sampled via the reparameterization trick:

   $$u = \mu_\phi(s) + \sigma_\phi(s) \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$$ $$a = \tanh(u)$$

   The policy parameters $\phi$ are optimized to maximize the expected soft Q-value:

   $$L(\phi) = \mathbb{E}_{s \sim \mathcal{D}, \epsilon \sim \mathcal{N}} \left[ \alpha \log \pi_\phi(a | s) - \min_{i=1,2} Q_{\theta_i}(s, a) \right]$$

Reparameterization Trick, Tanh Squashing, and Jacobian Correction

Because $a = \tanh(u)$ is a nonlinear change of variables from the latent Gaussian variable $u$ to the bounded action $a$, calculating $\log \pi(a|s)$ requires applying the change-of-variables formula:

The derivative of the hyperbolic tangent is $\frac{da_i}{du_i} = 1 - \tanh^2(u_i) = 1 - a_i^2$. Because the transformation is element-wise, the Jacobian matrix is diagonal, and its determinant is the product of diagonal entries. Taking the logarithm gives:

where $p(u | s) = \mathcal{N}(\mu_\phi(s), \sigma_\phi(s)^2)$ and $\delta > 0$ is a tiny stabilizer (e.g., $10^{-6}$) to avoid taking the logarithm of zero when actions reach the boundary $[-1, 1]$.

Automatic Entropy Temperature Tuning

Early iterations of SAC treated $\alpha$ as a fixed hyperparameter. However, the optimal entropy varies significantly across environments and training phases: early exploration requires high entropy, while fine motor control requires lower entropy.

SAC resolves this by framing entropy as an inequality constraint: $\mathbb{E}[-\log \pi(a|s)] \ge \bar{\mathcal{H}}$, where $\bar{\mathcal{H}} = -\dim(\mathcal{A})$ is the heuristic target entropy. This yields the dual optimization problem for $\alpha$:

Optimizing $\log \alpha$ with gradient descent:

If policy entropy $-\log \pi(a|s)$ exceeds the target $\bar{\mathcal{H}}$, $\alpha$ automatically decreases. If the policy becomes overly deterministic, $\alpha$ increases to stimulate exploration.

Worked numerical example

Let us trace a single transition update with concrete numbers:

Suppose an agent in 1D continuous action space ($\dim(\mathcal{A}) = 1$, target entropy $\bar{\mathcal{H}} = -1.0$) transitions to next state $s'$. The policy produces mean $\mu = 1.0$ and standard deviation $\sigma = 0.5$. Suppose sampled noise $\epsilon = 0.0$.

1. Sample Latent and Squashed Action:

   $$u = \mu + \sigma \epsilon = 1.0 + (0.5)(0.0) = 1.0$$ $$a' = \tanh(1.0) \approx 0.7616$$

2. Evaluate Jacobian and Log-Probabilities:

   • Tanh Jacobian factor: $$1 - \tanh^2(1.0) = 1 - (0.7616)^2 = 1 - 0.5800 = 0.4200$$ $$\ln(0.4200) \approx -0.8675$$
   • Latent Gaussian log-probability: $$\log p(u) = -\frac{1}{2} \ln(2\pi \sigma^2) - \frac{(u - \mu)^2}{2\sigma^2} = -\frac{1}{2} \ln(2\pi \cdot 0.25) - 0 = -\frac{1}{2} \ln(1.5708) \approx -0.2258$$
   • Squashed corrected log-probability: $$\log \pi(a' | s') = \log p(u) - \ln(0.4200) = -0.2258 - (-0.8675) = +0.6417$$

3. Compute Soft Bellman Target: Let current temperature $\alpha = 0.2$, reward $r = 1.0$, discount $\gamma = 0.99$. The twin target critics evaluate the pair $(s', a')$ as:

   $$Q_{\bar{\theta}_1}(s', a') = 5.0, \quad Q_{\bar{\theta}_2}(s', a') = 4.2 \implies \min = 4.2$$

   Soft state value with entropy deduction:

   $$V_{\text{soft}}(s') = 4.2 - \alpha \log \pi(a' | s') = 4.2 - (0.2)(0.6417) = 4.2 - 0.1283 = 4.0717$$

   Bellman target $y$:

   $$y = r + \gamma V_{\text{soft}}(s') = 1.0 + 0.99(4.0717) = 1.0 + 4.0309 = 5.0309$$

4. Update Temperature $\alpha$: Compare current negative log-prob with target entropy $\bar{\mathcal{H}} = -1.0$:

   $$\log \pi(a | s) + \bar{\mathcal{H}} = 0.6417 + (-1.0) = -0.3583$$

   Entropy difference: $-\log \pi(a|s) - \bar{\mathcal{H}} = -0.6417 - (-1.0) = +0.3583$. Because $-\log \pi > \bar{\mathcal{H}}$, the policy has more entropy than the target constraint demands. The gradient on $\log \alpha$ is positive, triggering a decrease in $\alpha$ to focus policy execution.

Code

1import numpy as np2
3
4class SoftActorCriticStep:5    """Implements core mathematics of Soft Actor-Critic (SAC) updates:6
7    Reparameterized squashed Gaussian sampling, Jacobian correction,8    clipped double Q soft Bellman targets, and automatic temperature tuning.9    """10
11    def __init__(12        self,13        action_dim: int = 1,14        gamma: float = 0.99,15        initial_alpha: float = 0.2,16        eps_stabilizer: float = 1e-6,17    ) -> None:18        self.action_dim = action_dim19        self.gamma = gamma20        self.log_alpha = float(np.log(initial_alpha))21        self.target_entropy = -float(action_dim)22        self.eps = eps_stabilizer23
24    @property25    def alpha(self) -> float:26        return float(np.exp(self.log_alpha))27
28    def sample_action(29        self,30        mu: np.ndarray,31        sigma: np.ndarray,32        noise: np.ndarray | None = None,33    ) -> tuple[np.ndarray, np.ndarray, float]:34        """Samples action via reparameterization trick: u = mu + sigma * eps, a = tanh(u).35
36        Applies Jacobian change-of-variables correction to calculate log pi(a|s).37        """38        if noise is None:39            noise = np.random.randn(*mu.shape)40
41        # Reparameterization trick: differentiable latent Gaussian sample42        u = mu + sigma * noise43        a = np.tanh(u)44
45        # Gaussian density in unconstrained latent space u46        var = sigma**247        log_p_u = -0.5 * np.sum(np.log(2.0 * np.pi * var) + ((u - mu) ** 2) / var)48
49        # Jacobian change-of-variables correction for tanh squashing50        log_jacobian = float(np.sum(np.log(1.0 - a**2 + self.eps)))51        log_pi = float(log_p_u - log_jacobian)52
53        return u, a, log_pi54
55    def compute_soft_target(56        self,57        reward: float,58        q1_target: float,59        q2_target: float,60        next_log_pi: float,61    ) -> float:62        """Computes soft Bellman value target with clipped double Q and entropy bonus."""63        min_q = min(q1_target, q2_target)64        soft_value = min_q - self.alpha * next_log_pi65        y = reward + self.gamma * soft_value66        return float(y)67
68    def compute_temperature_update(69        self,70        log_pi: float,71        lr_alpha: float = 0.05,72    ) -> tuple[float, float, float]:73        """Computes automatic entropy temperature loss, gradient, and updated alpha."""74        entropy_diff = -log_pi - self.target_entropy75        alpha_loss = -self.log_alpha * (log_pi + self.target_entropy)76        grad_log_alpha = entropy_diff77
78        # Gradient descent step on log(alpha)79        new_log_alpha = self.log_alpha - lr_alpha * grad_log_alpha80        new_alpha = float(np.exp(new_log_alpha))81
82        return alpha_loss, grad_log_alpha, new_alpha83
84
85if __name__ == "__main__":86    np.set_printoptions(precision=4, suppress=True)87
88    # Initialize SAC step for 1D continuous action space89    sac = SoftActorCriticStep(action_dim=1, gamma=0.99, initial_alpha=0.2)90
91    # Transition parameters matching the worked numerical example92    mu = np.array([1.0], dtype=np.float64)93    sigma = np.array([0.5], dtype=np.float64)94    noise = np.array([0.0], dtype=np.float64)95
96    u, a, log_pi = sac.sample_action(mu, sigma, noise=noise)97
98    q1_next, q2_next = 5.0, 4.299    reward = 1.0100    soft_target = sac.compute_soft_target(reward, q1_next, q2_next, log_pi)101
102    alpha_loss, grad_log_alpha, updated_alpha = sac.compute_temperature_update(103        log_pi, lr_alpha=0.05104    )105
106    print(f"Latent sample u: {u[0]:.4f}")107    # -> Latent sample u: 1.0000108
109    print(f"Squashed action a: {a[0]:.4f}")110    # -> Squashed action a: 0.7616111
112    print(f"Corrected log pi(a|s): {log_pi:.4f}")113    # -> Corrected log pi(a|s): 0.6418114
115    print(f"Soft Bellman target y: {soft_target:.4f}")116    # -> Soft Bellman target y: 5.0309117
118    print(f"Temperature alpha: {sac.alpha:.4f}")119    # -> Temperature alpha: 0.2000120
121    print(f"Target entropy H_bar: {sac.target_entropy:.4f}")122    # -> Target entropy H_bar: -1.0000123
124    print(f"Log-alpha gradient: {grad_log_alpha:.4f}")125    # -> Log-alpha gradient: 0.3582126
127    print(f"Updated alpha: {updated_alpha:.4f}")128    # -> Updated alpha: 0.1964129
130    # Assert correctness131    assert np.isclose(a[0], 0.7616, atol=1e-3)132    assert np.isclose(soft_target, 5.0309, atol=1e-3)133    assert updated_alpha < sac.alpha  # Alpha reduced as entropy > target

Watch Out For

The Quick Version

• Maximum Entropy Framework: SAC optimizes for both cumulative rewards and policy entropy, encouraging wide exploration and preventing convergence to brittle, deterministic policies.
• Clipped Double Q-Learning: Uses the minimum prediction of two independent target critics ($\min(Q_1, Q_2)$) in soft Bellman backups to suppress value overestimation.
• Reparameterized Tanh Actor: Enables end-to-end backpropagation through stochastic action choices while bounding continuous actions to $(-1, 1)$ with an analytical Jacobian correction.
• Automatic Temperature Tuning: Continuously modulates the entropy weight $\alpha$ to satisfy an entropy constraint $\bar{\mathcal{H}} = -\dim(\mathcal{A})$, balancing exploration early and exploitation late.

What to Read Next

• Maximum Entropy RL
• Soft Q-Learning
• Twin Delayed DDPG (TD3)