Value Functions

A value function estimates expected future return, giving the agent a way to compare states and actions without simulating the full future.

Predicting return

A value function estimates how good it is to be in a state, or to take an action in a state, measured as expected return under a given policy. Value functions turn the sprawling problem of long-horizon planning into a local quantity the agent can estimate and improve.

State-value function

The state-value function V^pi(s) is the expected return starting from state s and following policy pi thereafter: V^pi(s) = E[G_t | S_t = s, pi]. It answers: how much total reward can I expect if I start here and keep behaving as I do now?

Action-value function

The action-value function Q^pi(s, a) is the expected return starting from s, taking action a, then following pi: Q^pi(s, a) = E[G_t | S_t = s, A_t = a, pi]. Q is more directly useful for control because comparing Q values across actions tells the agent which action is best without a model of the dynamics.

The relationship

The two are linked: V^pi(s) is the average of Q^pi(s, a) over the actions the policy takes, V^pi(s) = sum over a of pi(a | s) Q^pi(s, a). The advantage function A^pi(s, a) = Q^pi(s, a) - V^pi(s) measures how much better an action is than the policy's average.

Optimal value functions

The optimal value functions V* and Q* are the best achievable over all policies. Given Q*, an optimal policy is simply to pick the action with the largest Q* value in each state. The Bellman equation and its optimality form describe how these functions must be self-consistent.