Exploration Strategies in Deep RL
Beyond random actions, deep RL uses intrinsic curiosity, count bonuses, and posterior sampling to explore hard, sparse-reward tasks.
When random is not enough
Simple epsilon-greedy exploration takes random actions, which suffices for easy tasks but fails when reward is rare and reaching it requires a long, specific sequence of actions. Deep RL has developed richer exploration strategies that direct the agent toward what it does not yet understand.
Intrinsic motivation
Intrinsic motivation adds an internal reward for encountering novelty or surprise, independent of the task reward. Curiosity-driven methods reward the agent when its own predictions about the next state are wrong, pushing it toward states it cannot yet predict. This provides a dense signal to explore even when the extrinsic reward is silent.
Count-based bonuses
Count-based exploration rewards visiting rarely seen states, formalizing optimism under uncertainty. In small problems this means literal visit counts; in large or continuous spaces it uses pseudo-counts from density models or hashing, so that novelty can be estimated where exact counts are impossible.
Posterior and uncertainty methods
- Bootstrapped networks: train an ensemble and act on a sampled member, exploring where members disagree.
- Randomized value functions: inject noise into value estimates to drive deep, consistent exploration.
- Thompson-style sampling: sample plausible models or values and act as if they were true.
Choosing a strategy
Exploration effort should match task difficulty. Dense-reward tasks need little beyond entropy or epsilon-greedy; hard-exploration tasks with long chains of prerequisite actions need intrinsic or count-based bonuses. The cost of exploration also matters: cheap in simulation, potentially damaging on real systems, which is where these methods meet safe RL.