Markov Decision Processes

A Markov decision process formalizes sequential decision making as states, actions, transition probabilities, rewards, and a discount factor.

The formal model

A Markov decision process (MDP) is the standard mathematical framework for reinforcement learning. It is a tuple (S, A, P, R, gamma) where S is the set of states, A the set of actions, P the transition dynamics, R the reward function, and gamma the discount factor.

The five components

Kronos motion — safety factor

S, states: the situations the agent can be in.
A, actions: the choices available, possibly depending on state.
P(s' | s, a): probability of moving to s' after taking a in s.
R(s, a) or R(s, a, s'): expected reward for that transition.
gamma in [0,1): how much future reward is discounted relative to immediate reward.

Dynamics and the Markov assumption

The transition function P encodes the environment's dynamics and assumes the Markov property: the next state depends only on the current state and action. This assumption is what lets value functions be written recursively through the Bellman equation.

The objective

The agent seeks a policy that maximizes expected return — the discounted sum of future rewards. When P and R are known the MDP can be solved by dynamic programming. When they are unknown the agent must learn from sampled experience, which is the RL setting proper.

Variants

Extensions include partially observable MDPs (states hidden behind observations), continuous state and action spaces, constrained MDPs that add safety limits, and multi-agent settings. Most practical control problems, including simulated plasma control, are MDPs with continuous states and actions that must be approximated with function approximation.