RMSprop for Optimization

RMSprop adapts per-parameter learning rates using an exponentially decaying average of squared gradients, curing Adagrad's vanishing step size.

Fixing Adagrad's decay

Adagrad divides the learning rate by the square root of the sum of all past squared gradients, which grows without bound, so the effective step size decays to zero and learning stops prematurely. RMSprop, proposed by Geoffrey Hinton in a 2012 lecture, replaces the ever-growing sum with an exponentially weighted moving average of recent squared gradients, so old gradients fade and the accumulator reflects only the recent gradient scale.

The update

RMSprop maintains a running average v of squared gradients: v <- rho * v + (1 - rho) * g^2, with decay rate rho typically 0.9. The parameter update is theta <- theta - (eta / sqrt(v + epsilon)) * g. Because v tracks a moving window rather than the whole history, the effective learning rate can grow again if gradients shrink, so the method keeps adapting throughout training rather than grinding to a halt.

import numpy as np

def rmsprop(grad_fn, theta, eta=0.001, rho=0.9, eps=1e-8, iters=1000):
    v = np.zeros_like(theta)
    for _ in range(iters):
        g = grad_fn(theta)
        v = rho*v + (1-rho)*g*g        # decaying average of g^2
        theta -= eta * g / (np.sqrt(v) + eps)
    return theta

Behavior

By normalizing each step by a recent estimate of the gradient's magnitude, RMSprop keeps effective step sizes roughly comparable across parameters and across time, which stabilizes training on nonstationary and non-convex objectives such as neural-network losses. It handles the ravines and plateaus of deep-learning landscapes far better than plain gradient descent and was a standard optimizer before Adam became dominant.

Relationship to Adam

Adam can be read as RMSprop plus momentum: it keeps RMSprop's decaying average of squared gradients for per-parameter scaling and adds a decaying average of the gradients themselves for momentum, together with bias corrections for the early iterations. RMSprop remains a strong, simpler choice, particularly for recurrent networks and reinforcement learning, where it is still a common default.