Soft Actor-Critic
SAC maximizes reward plus policy entropy, giving a stable, sample-efficient, exploration-friendly control algorithm.
The maximum-entropy objective
Soft Actor-Critic (SAC) changes the objective: it maximizes expected reward plus the entropy of the policy, weighted by a temperature alpha. The agent is rewarded for succeeding and for staying as random as it can while doing so. This built-in preference for stochasticity improves exploration, robustness, and stability.
Soft value functions
Under this objective the Bellman equations gain an entropy bonus: the target adds alpha times the log-probability of the next action. SAC learns twin Q-critics (taking the minimum, as in TD3, to limit overestimation) and a stochastic actor that outputs a squashed Gaussian. Being off-policy, it reuses a replay buffer for high sample efficiency.
# SAC critic target (entropy-augmented, twin critics)
a2, logp2 = actor.sample(s2)
q_targ = min(Q1_targ(s2,a2), Q2_targ(s2,a2)) - alpha*logp2
y = r + gamma * q_targ
# actor loss: maximize Q(s,a) - alpha*log pi(a|s)Automatic temperature
The temperature alpha trades reward against entropy and is hard to set by hand. SAC can tune it automatically by treating it as a constraint: adjust alpha so the policy's average entropy matches a target level. This removes a fragile hyperparameter and adapts exploration over the course of training, high early, lower as the policy sharpens.
Why SAC is a default
- Sample efficient (off-policy) yet stable, unlike DDPG
- Entropy term gives structured exploration without hand-tuned action noise
- Robust across tasks with little per-task tuning
SAC is a leading choice for continuous control in robotics and simulation. Its maximum-entropy framing also connects RL to probabilistic inference, where acting is cast as sampling from a distribution proportional to exponentiated reward.