Offline RL: Implicit Q-Learning

IQL learns entirely within the dataset by never evaluating actions it has not seen, using expectile regression on the value function.

Avoid the query, avoid the shift

Distributional shift in offline RL comes from evaluating Q at out-of-distribution actions. Implicit Q-Learning (IQL) sidesteps this by never querying such actions at all. It approximates the maximum over in-support actions rather than over all actions.

Expectile value learning

IQL fits a separate value function V(s) using expectile regression toward Q(s,a) on dataset actions. A high expectile (for example tau = 0.9) makes V approximate the best returns achievable by actions actually present in the data, without ever proposing new actions. The Q update then bootstraps from this V, keeping the whole target inside the data distribution.

def expectile_loss(diff, tau=0.9):
    # asymmetric squared error
    w = np.where(diff > 0, tau, 1 - tau)
    return (w * diff**2).mean()

# V toward Q on dataset actions; Q toward r + gamma*V(s')
# no max over unseen actions anywhere

Policy extraction

IQL separates value learning from policy learning. After V and Q are fit, the policy is extracted by advantage-weighted regression: behavior-clone the dataset, but weight each action by exp(beta * (Q - V)), so higher-advantage actions dominate. The policy is thus a reweighting of observed behavior and stays on-support by construction.

Trade-offs

• Simple, stable, and does not require sampling from a policy during value learning
• Never evaluates unseen actions, so no explicit conservatism penalty is needed
• Performance is capped by the quality of actions in the dataset and the chosen expectile

IQL and CQL are the two dominant offline-RL recipes. CQL suppresses optimism explicitly; IQL avoids the problematic query. Both aim at the same goal: a policy that improves on the logged behavior without stepping outside what the data can justify.