Exploration vs Exploitation
An agent must balance exploiting what it knows to gain reward now against exploring the unknown to find something better.
The central dilemma
The exploration-exploitation trade-off is fundamental to reinforcement learning. To earn reward the agent should exploit the best action it currently knows. But its knowledge is incomplete, so it must also explore less-tried actions that might be better. Too much exploitation locks in a mediocre habit; too much exploration wastes reward chasing dead ends.
Simple strategies
- Epsilon-greedy: act greedily most of the time, but with probability epsilon pick a random action. Simple and surprisingly effective.
- Decaying epsilon: explore heavily early, then anneal toward exploitation as knowledge accumulates.
- Softmax / Boltzmann: choose actions with probability increasing in their estimated value, so near-best actions are tried more than clearly bad ones.
Smarter strategies
More principled methods explore where uncertainty is highest. Upper confidence bound (UCB) adds a bonus for actions tried few times, encouraging optimism under uncertainty. Thompson sampling maintains a distribution over which action is best and samples from it. Count-based and curiosity-driven bonuses reward reaching novel or surprising states, which helps in sparse-reward tasks.
Exploration in deep RL
Entropy regularization, as in SAC and PPO, keeps policies stochastic and exploratory. Parameter-space noise and intrinsic-motivation bonuses tackle hard exploration where reward is rare. The right amount of exploration is problem-dependent and remains one of the hardest practical questions in RL.
The stakes
In simulation, aggressive exploration is cheap. On real or safety-critical systems it is not: a random control action can be damaging. This tension motivates safe RL, where exploration must respect constraints even while the agent is still learning.