SARSA

SARSA is an on-policy temporal-difference control method that learns the value of the policy it actually follows, including its exploration.

On-policy control

SARSA is a temporal-difference control algorithm named for the tuple it uses: State, Action, Reward, next State, next Action (s, a, r, s', a'). Unlike Q-learning, it is on-policy: it evaluates and improves the same policy the agent is following, exploration included.

The update rule

After observing (s, a, r, s', a'), SARSA updates Q(s, a) <- Q(s, a) + alpha [ r + gamma Q(s', a') - Q(s, a) ]. The target uses Q(s', a') for the action a' actually chosen by the current policy, not the maximum over actions as in Q-learning.

On-policy versus off-policy

Because SARSA's target reflects the exploratory action the agent will really take, it learns the value of its actual behavior. This makes it more conservative near danger: if exploration occasionally leads to a bad outcome, SARSA accounts for that risk, whereas Q-learning learns the value of the ideal greedy path and may steer closer to hazards.

The cliff-walking contrast

In the classic cliff-walking task, Q-learning learns the optimal path right along the cliff edge but suffers frequent falls during epsilon-greedy exploration. SARSA learns a safer path a step back from the edge, earning higher reward while still exploring. The example neatly illustrates the on-policy versus off-policy distinction.

Variants and use

Expected SARSA replaces the sampled next action with the expectation over the policy's action distribution, reducing variance. SARSA and its variants extend naturally to eligibility traces (SARSA(lambda)) and to function approximation. Where safety during learning matters, the on-policy caution of SARSA can be an asset.