Reward Shaping

Reward shaping adds guidance signals to speed learning, but only potential-based shaping is guaranteed to be safe.

Guiding the agent

Sparse rewards make learning slow: an agent may wander for a long time before stumbling on any signal. Reward shaping adds intermediate rewards to point the agent toward good behavior, for example rewarding progress toward a goal rather than only reaching it. Done carelessly, though, shaping can change the optimal policy and teach the wrong thing.

The danger of naive shaping

If you reward a soccer agent for touching the ball, it may learn to vibrate against the ball forever, maximizing the shaping reward while never scoring. The added signal created a new, unintended optimum. Any shaping that alters which policy is optimal is unsafe.

Potential-based shaping

Ng, Harada, and Russell proved a clean condition. If the shaping reward has the form F(s, s') = gamma * Phi(s') - Phi(s) for some potential function Phi over states, then the set of optimal policies is unchanged, guaranteed. Intuitively the extra rewards telescope along any trajectory and cancel, so they can accelerate learning without moving the optimum.

# Potential-based shaping preserves optimal policy
# F(s, s') = gamma * Phi(s') - Phi(s)
r_shaped = r + gamma * Phi(s2) - Phi(s)
# a good Phi is an estimate of the value/negative distance-to-goal

Choosing a potential

• A useful Phi resembles the true value function, so a rough heuristic estimate of it works well
• Distance-to-goal (negated) is a common potential for navigation and reaching
• Shaping only speeds convergence; it never changes what the optimal policy is

Reward shaping is the disciplined middle ground between sparse reward and reward hacking. When guidance is needed, express it as a potential difference; when it is not, keep the reward honest and rely on exploration methods instead.