Safe RL and Constrained MDPs

Safe RL maximizes reward subject to constraints, so the agent respects safety limits during and after learning.

Reward is not enough

Many real systems have hard limits: a robot must not exceed torque bounds, a controller must not drive a plant outside safe operating envelopes. Folding such limits into a single reward via penalties is brittle. Safe RL keeps them separate as explicit constraints.

The constrained MDP

A Constrained MDP (CMDP) augments the MDP with one or more cost functions and a budget for each. The goal is to maximize expected return subject to expected cumulative cost staying below its threshold. This is a constrained optimization, not a scalar maximization, and its solution can be a stochastic policy even when the unconstrained optimum is deterministic.

# CMDP: maximize E[return]  subject to  E[cost_j] <= d_j
# Lagrangian relaxation:
# max_pi min_{lambda>=0}  E[return] - sum_j lambda_j (E[cost_j] - d_j)
# lambda is raised when a constraint is violated, lowered otherwise

Solution approaches

• Lagrangian methods add adaptive multipliers that penalize constraint violation, tuning them online
• Trust-region methods (CPO) take policy steps guaranteed to keep constraints satisfied
• Shielding / safety layers filter or project unsafe actions before they execute

Safety during learning

A subtlety separates safe RL from ordinary constrained optimization: constraints must often hold during training, not only at convergence, because exploratory mistakes on real hardware can be catastrophic. This drives interest in conservative exploration, control-theoretic safety filters, and formal guarantees. Where a machine or plant is expensive or safety-critical, learning is typically done first in simulation, with a safety layer retained during any real deployment.