Offline Reinforcement Learning
Offline RL learns policies entirely from a fixed dataset of logged experience, without any further interaction with the environment.
Learning without interacting
Offline reinforcement learning (also called batch RL) learns a policy from a fixed dataset of previously collected transitions, with no ability to gather new experience during training. This matters whenever live interaction is expensive, slow, or unsafe — medicine, industrial control, robotics — and a log of past behavior already exists.
The core difficulty
The central problem is distributional shift. A learned policy may prefer actions the dataset rarely or never contains. The value function then has to estimate the worth of those unseen actions, and because it cannot be corrected by real feedback, it tends to overestimate them. The policy chases these overoptimistic estimates and fails when deployed.
How methods cope
- Policy constraints: keep the learned policy close to the behavior that generated the data.
- Conservative value estimation: penalize value for out-of-distribution actions so they are not preferred.
- Uncertainty penalties: downweight actions where the model or value estimate is unsure.
- Model-based offline RL: learn a dynamics model and plan only within its trusted region.
Offline versus off-policy
All offline RL is off-policy, but not all off-policy learning is offline. Off-policy methods like Q-learning can still collect fresh data online; offline RL forbids that, which is exactly why distributional shift cannot be self-corrected and requires the conservative techniques above.
Why it matters
Offline RL promises to turn large archives of logged operational data into improved policies without risky experimentation. For high-value physical systems where each trial is costly and exploratory mistakes are unacceptable, learning a better controller from historical records — validated in simulation before any deployment — is an appealing and active direction.