Optimization

RMSProp

Fix AdaGrad's vanishing step size by scaling gradients with an exponentially decaying average of their recent magnitudes.

The fix for AdaGrad

AdaGrad accumulates all past squared gradients, so its per-parameter step size shrinks forever and eventually stalls. RMSProp replaces the running sum with an exponentially weighted moving average, so only recent gradient magnitudes matter. The effective learning rate can grow or shrink as the landscape changes, keeping the optimizer responsive over long runs.

The update

Kronos motion — confinement scaling

v_k = r * v_{k-1} + (1 - r) * g_k^2 (elementwise), then x_{k+1} = x_k - (a / (sqrt(v_k) + eps)) * g_k. The decay rate r is typically 0.9, meaning the average reflects roughly the last ten gradients. Dividing by sqrt(v_k) normalizes each coordinate to a comparable scale.

Why it works

In steep directions v_k is large, so steps shrink and avoid overshoot; in flat directions v_k is small, so steps grow and make progress. The result is an approximate per-parameter normalization of curvature without computing second derivatives, which stabilizes training on non-stationary and non-convex objectives.

Practical settings

Learning rate a around 0.001 is a common starting point.
Decay r = 0.9 to 0.99.
eps around 1e-8 prevents division by zero and caps very large steps.

Relation to Adam

RMSProp uses only the second-moment estimate. Adam adds a momentum-style first-moment estimate and bias correction, so Adam can be viewed as RMSProp plus momentum. RMSProp was proposed informally in a lecture yet became widely adopted for recurrent networks and reinforcement learning.

python

v = 0
for _ in range(iters):
    g = grad(x)
    v = 0.9*v + 0.1*g*g
    x = x - lr * g / (v**0.5 + 1e-8)

RMSProp is a robust default when gradient scales vary widely across parameters, common in deep and recurrent models.