Optimization

AdaGrad

Adapt a per-parameter learning rate by dividing by the accumulated history of squared gradients, favoring rarely updated directions.

Per-parameter adaptation

AdaGrad gives each parameter its own learning rate that shrinks in proportion to how much that parameter has been updated. Directions with large, frequent gradients get small effective steps; directions with small, rare gradients keep larger steps. This suits sparse features, where infrequent but informative signals would otherwise be under-weighted.

The update

Kronos motion — parameter scan

Accumulate G_k = G_{k-1} + g_k^2 elementwise, where g_k is the gradient. Then update x_{k+1} = x_k - (a / (sqrt(G_k) + eps)) * g_k. The division is elementwise, so each coordinate scales by the inverse square root of its own accumulated squared gradient.

Strengths

No manual per-parameter tuning; the schedule adapts automatically.
Excellent for sparse, high-dimensional problems such as text and recommendation.
Strong theoretical guarantees for online and convex optimization.

The vanishing step-size problem

Because G_k only accumulates and never decreases, the effective learning rate monotonically shrinks toward zero. On long runs, especially non-convex deep learning, AdaGrad can stop making progress before reaching a good solution. This flaw motivated RMSProp and Adam, which replace the sum with an exponentially decaying average so old gradients fade.

When to use it

AdaGrad remains a good choice for convex problems with sparse gradients and for online learning where the total number of steps is bounded. For deep networks trained over many epochs, prefer its descendants.

python

G = 0
for _ in range(iters):
    g = grad(x)
    G = G + g*g
    x = x - lr * g / (G**0.5 + 1e-8)

AdaGrad established the template of adaptive, per-coordinate step sizes that dominates modern machine-learning optimizers.