Partial Observability and POMDPs
When the agent cannot observe the full state, it must act on incomplete information and maintain beliefs about what it cannot see.
Seeing less than the whole
A partially observable Markov decision process (POMDP) generalizes the MDP to cases where the agent does not directly observe the true state. Instead it receives an observation that gives only partial, possibly noisy information. Most real problems are partially observable: sensors are limited, and important variables are often hidden.
The problem with observations
Because the observation does not satisfy the Markov property, a policy that maps the current observation to an action can be badly suboptimal. Two genuinely different situations may look identical, and the agent needs history to tell them apart. Acting on the latest observation alone throws away information the past provides.
Belief states
The principled solution is the belief state: a probability distribution over which true state the agent is in, updated with each observation using Bayes' rule. The belief state is Markov, so a POMDP can in theory be treated as an MDP over beliefs. Exactly maintaining and planning over beliefs is expensive, so this is usually approximated.
Practical approaches
- Frame stacking: include several recent observations so velocity and trend are visible.
- Recurrent networks: an LSTM or GRU summarizes history into a hidden state that stands in for the belief.
- Explicit belief tracking: maintain and update a distribution when a model is available.
Why it matters
Any controller reading a physical system through a handful of diagnostics faces partial observability: the full internal state is never directly measured. A simulated fusion device observed through limited sensor channels is a clear example, so a controller trained on it typically needs memory to integrate readings over time rather than react to a single instantaneous measurement.