Multi-Objective RL

When several goals conflict, multi-objective RL seeks the Pareto front of policies rather than a single optimum.

More than one reward

Many real decisions trade off competing objectives: speed against energy use, throughput against safety margin. Multi-objective RL (MORL) makes this explicit by using a vector reward, one component per objective, instead of collapsing everything into a single scalar in advance. This preserves the trade-off structure so it can be examined rather than baked in.

Pareto optimality

A policy is Pareto optimal if no other policy is at least as good on every objective and strictly better on one. The set of all such policies is the Pareto front. MORL aims to find this front (or a useful part of it), giving a decision-maker the menu of achievable trade-offs rather than a single answer that hid the choice.

Scalarization

The simplest approach picks weights and optimizes a weighted sum of objectives, recovering ordinary RL. Sweeping the weights traces out part of the Pareto front, but linear scalarization can only find its convex regions. Nonlinear scalarizations (for example maximizing the minimum objective, or Chebyshev) can reach concave portions the weighted sum misses.

# Linear scalarization recovers scalar RL for fixed weights w
scalar_reward = sum(w[i] * reward_vector[i] for i in range(K))
# Sweeping w traces the convex part of the Pareto front

Conditioned policies

• Single-policy: fix a preference (weights), find one optimal trade-off
• Multi-policy: learn a set of policies covering the Pareto front
• Preference-conditioned: one network takes the weight vector as input and outputs the matching policy on demand

Preference-conditioned networks are attractive because they let a user choose the trade-off at deployment without retraining. MORL connects to constrained RL (a constraint is one way to encode a secondary objective) and to reward design, keeping objectives separate so their tensions stay visible.