Curiosity and Intrinsic Motivation

Intrinsic rewards drive exploration when the environment's own reward is sparse or absent.

Rewarding the novel

When extrinsic reward is rare, an agent needs another reason to explore. Intrinsic motivation supplies an internal reward for encountering novel or informative states. The total reward becomes r_extrinsic + beta * r_intrinsic, and as the agent masters a region the intrinsic reward there fades, pushing it onward.

Prediction-error curiosity

One influential formulation rewards prediction error: the agent learns a forward model of dynamics and is rewarded when that model is surprised. The Intrinsic Curiosity Module (ICM) predicts the next state in a learned feature space and uses the prediction error as the curiosity bonus. Crucially the features are trained by an inverse model (predicting the action from consecutive states), so the agent is curious only about things it can affect, not about unpredictable but irrelevant noise.

The noisy-TV problem

Naive prediction-error curiosity has a famous failure: a source of pure randomness (a TV showing static) yields permanently high prediction error, trapping the agent in fascination with noise. Feature learning via the inverse model mitigates this; another family avoids it entirely by using random network distillation (RND), rewarding error in predicting the output of a fixed random network, which measures novelty of states rather than dynamics.

# ICM curiosity bonus
phi_s, phi_s2 = encoder(s), encoder(s2)
pred = forward_model(phi_s, a)
r_intrinsic = 0.5 * ((pred - phi_s2)**2).sum()

Count-based cousins

A different tradition rewards visiting rarely seen states directly, approximating a visit count in high-dimensional spaces via density models or hashing (r_intrinsic proportional to 1/sqrt(count)). All these methods share a goal: turn exploration into an optimization target so the agent seeks out the unknown even before any task reward appears.