Adadelta

Adadelta extends Adagrad with a decaying gradient average and a matching decaying average of updates, removing the need to choose a global learning rate.

Two problems, one method

Adadelta, from 2012, addresses two shortcomings of Adagrad at once. First, like RMSprop, it replaces the unbounded sum of squared gradients with an exponentially decaying average, so the effective learning rate does not vanish. Second, and distinctively, it removes the global learning rate entirely by keeping the units of the update consistent with the units of the parameter.

Matching the units

Adadelta maintains two decaying averages: one of squared gradients, E[g^2], and one of squared parameter updates, E[dx^2]. Each step scales the gradient by the ratio sqrt(E[dx^2] + epsilon) / sqrt(E[g^2] + epsilon). The numerator, an estimate of the recent update magnitude, replaces the fixed learning rate. This ratio has the correct units of the parameter itself, which is why Adadelta needs no eta to be tuned.

import numpy as np

def adadelta(grad_fn, theta, rho=0.95, eps=1e-6, iters=1000):
    Eg = np.zeros_like(theta); Edx = np.zeros_like(theta)
    for _ in range(iters):
        g = grad_fn(theta)
        Eg = rho*Eg + (1-rho)*g*g
        dx = -(np.sqrt(Edx+eps)/np.sqrt(Eg+eps))*g  # no global lr
        theta += dx
        Edx = rho*Edx + (1-rho)*dx*dx
    return theta

Behavior

Because the step size is derived from the running scale of past updates rather than a fixed constant, Adadelta adapts automatically as training progresses and is relatively insensitive to hyperparameters, needing mainly the decay rate rho. It combines Adagrad's per-parameter adaptivity with a self-scaling step that neither vanishes nor requires tuning, which made it attractive for settings where hyperparameter search is costly.

In context

Adadelta, RMSprop, and Adagrad form a family that share the per-parameter squared-gradient idea and differ in how they use it. In practice Adam, which adds momentum on top of RMSprop-style scaling, became the most widely used, but Adadelta's contribution, that a sensible step size can be derived from the update history rather than set by hand, anticipated the learning-rate-free methods that followed.