RMSprop Optimizer
RMSprop scales each parameter's step by a running average of its squared gradients, adapting the learning rate per parameter to handle varying gradient magnitudes.
Per-parameter step sizes
Different parameters in a network often see gradients of very different magnitudes, so a single global learning rate is either too large for some or too small for others. RMSprop addresses this by keeping, for each parameter, an exponentially decaying running average of its recent squared gradients, then dividing the gradient by the square root of that average. Parameters with consistently large gradients take smaller steps, and parameters with small gradients take larger ones, equalizing progress across the network.
The update rule
At each step RMSprop updates the running mean-square v = rho*v + (1-rho)*g^2, where g is the gradient and rho is a decay factor near 0.9. It then updates the parameter by subtracting the learning rate times g divided by the square root of v plus a small constant for numerical safety. The division normalizes the effective step so it is roughly scale-invariant with respect to the gradient magnitude.
v = rho * v + (1 - rho) * g**2
w = w - lr * g / (v.sqrt() + eps)- Adapts the step size separately for each parameter
- The running average forgets old gradients, tracking the current landscape
- Handles non-stationary objectives well, which suits recurrent networks
- Does not include momentum by default, unlike Adam
Relation to Adam
RMSprop is effectively the second-moment half of Adam. Adam adds a running average of the gradient itself, giving momentum, and bias-corrects both averages in early steps. RMSprop's simplicity makes it a useful baseline and it remains effective for certain recurrent and reinforcement-learning settings, but for large-scale supervised training Adam and AdamW are usually preferred because momentum accelerates convergence in flat directions.
Origins and use
RMSprop grew from the earlier Adagrad, which accumulated squared gradients without forgetting and so let the effective step decay to nearly zero over long training. Replacing the sum with an exponentially decaying average fixed that, keeping the adaptivity alive indefinitely. RMSprop is still a solid, low-memory optimizer where a full Adam state is unnecessary or where the objective shifts over time.