MuZero

MuZero plans with a learned model of dynamics in a latent space, so it needs no access to the true game rules.

Planning without the rules

AlphaZero requires a perfect simulator to run search. MuZero removes that requirement by learning its own model. It plans in an abstract latent space and never predicts the raw next observation, only the quantities needed for planning: reward, value, and policy.

Three learned functions

• Representation h: observation -> latent state s0
• Dynamics g: (s_k, a_k) -> (s_{k+1}, r_{k+1})
• Prediction f: s_k -> (policy p_k, value v_k)

MCTS runs entirely on these functions. From the encoded root state, the dynamics function unrolls hypothetical action sequences and the prediction function evaluates each latent node, exactly as AlphaZero's network would on real states.

Training signal

The model is unrolled several steps and trained so that its predicted rewards match observed rewards, its values match n-step bootstrapped returns, and its policies match the MCTS visit counts. Crucially the latent state is never asked to reconstruct the observation; it only has to be sufficient for predicting these three targets. This makes the learned model value-equivalent rather than a faithful simulator.

# MuZero unroll (schematic)
s = h(obs)
for a in trajectory_actions:
    p, v = f(s)          # prediction
    s, r = g(s, a)       # dynamics
    # losses on p vs MCTS pi, v vs return, r vs observed reward

Why it matters

MuZero reached AlphaZero-level play in board games while also mastering Atari from pixels, a domain with unknown dynamics and per-step rewards. It unifies model-based planning and model-free value learning: the model exists solely to serve planning, so it can ignore visually complex but decision-irrelevant detail.