Soft Actor-Critic (SAC)
SAC is an off-policy actor-critic that maximizes reward plus policy entropy, yielding robust exploration and strong continuous control.
Maximum-entropy RL
Soft Actor-Critic (SAC) is an off-policy actor-critic algorithm for continuous control built on the maximum-entropy objective: maximize expected reward and the entropy of the policy at the same time. Rewarding randomness encourages the agent to keep multiple good options open, producing robust exploration and stable learning.
The objective
SAC maximizes E[ sum of r + alpha * H(pi(. | s)) ], where H is the policy's entropy and alpha is a temperature that weights exploration against reward. High alpha keeps the policy exploratory; low alpha makes it more greedy. Modern SAC tunes alpha automatically to hold entropy near a target level.
Architecture
- A stochastic actor outputting a distribution over continuous actions (typically a squashed Gaussian).
- Twin critics, as in TD3, taking the minimum to reduce overestimation.
- A replay buffer, since SAC is off-policy and reuses past experience.
- Soft target updates for stable bootstrapping.
Why entropy helps
Building exploration into the objective, rather than bolting on external action noise as DDPG does, gives SAC several advantages: it explores more effectively, it is less sensitive to hyperparameters, and its stochastic policy captures multiple near-optimal behaviors, which improves robustness when conditions shift.
Standing
SAC is a leading choice for continuous-control problems, valued for sample efficiency and stability. Together with TD3 and PPO it forms the modern toolkit for control tasks, including simulated physical systems where robust exploration and reliable convergence matter more than squeezing out the last increment of performance.