Model-Predictive Control and RL

MPC plans a short action sequence with a model, executes one step, then replans, and pairs naturally with learned models.

Plan, act, replan

Model-Predictive Control (MPC) is a receding-horizon method: at each step it optimizes a sequence of actions over a short horizon using a dynamics model, executes only the first action, then discards the plan and repeats from the new state. Replanning at every step corrects for model error and disturbances, giving robustness that open-loop planning lacks.

Optimizing the plan

With a learned, possibly nonlinear neural model, the inner optimization is done by sampling-based planners rather than gradients. Random shooting samples many action sequences and keeps the best; the cross-entropy method (CEM) iteratively refits a distribution to the top-scoring sequences; model-predictive path integral (MPPI) weights sequences by exponentiated return. These handle non-differentiable, multimodal objectives.

# CEM-based MPC step
for _ in range(iters):
    seqs = sample(mean, std, N)                 # action sequences
    scores = [rollout_return(model, s0, seq) for seq in seqs]
    elite = top_k(seqs, scores)
    mean, std = fit_gaussian(elite)
execute(mean[0]); # then observe new state and replan

Relationship to RL

MPC and RL are complementary. MPC needs a model but no long training; RL amortizes planning into a reactive policy but needs many samples. Hybrids are powerful: PETS uses MPC over a learned probabilistic ensemble; MuZero runs tree search (a form of planning) with a learned model; and a learned value function can serve as MPC's terminal cost, extending its effective horizon beyond the planning window.

Trade-offs

• Robust to disturbances and model error thanks to constant replanning
• No policy to train, but planning cost is paid at every control step
• Performance is bounded by model accuracy over the horizon

MPC is the workhorse of classical and learned control alike. In RL it provides a principled way to use a learned model without committing to full-horizon planning or to a fixed reactive policy.