Optimization

AdamW

Adam with decoupled weight decay: apply regularization directly to the parameters instead of folding it into the adaptive gradient.

The weight-decay problem in Adam

L2 regularization adds a term to the loss whose gradient is proportional to the parameters. In plain SGD this is identical to weight decay, shrinking parameters toward zero each step. In Adam, however, the L2 gradient passes through the adaptive scaling by sqrt(v), so parameters with large gradient history are decayed less. This breaks the intended uniform shrinkage and weakens regularization.

Decoupling the decay

Kronos motion — regularization

AdamW removes weight decay from the loss gradient and applies it as a separate step: x_{k+1} = x_k - a * m_hat / (sqrt(v_hat) + eps) - a * w * x_k. The decay term w * x_k is not divided by sqrt(v), so every parameter shrinks at the same relative rate regardless of its gradient statistics.

Why it matters

Restores the true regularization effect of weight decay under adaptive optimizers.
Improves generalization, often matching or beating tuned SGD on large models.
Makes the decay coefficient easier to tune because it is decoupled from the learning rate schedule.

Adoption

AdamW is now the standard optimizer for training large transformers and many other deep networks. The distinction seems small but consistently improves held-out performance, so most modern training pipelines default to it over plain Adam.

Practical guidance

Typical settings mirror Adam (b1 = 0.9, b2 = 0.999) with a weight decay w around 0.01 to 0.1. The decay is usually applied to weight matrices but not to biases or normalization parameters. AdamW pairs naturally with cosine or warmup-decay learning-rate schedules.

python

m=v=0
for k in range(1, iters+1):
    g = grad(x)
    m = 0.9*m + 0.1*g
    v = 0.999*v + 0.001*g*g
    mh=m/(1-0.9**k); vh=v/(1-0.999**k)
    x = x - lr*(mh/(vh**0.5+1e-8)) - lr*wd*x

Decoupled weight decay is a small correction with an outsized effect on the reliability of large-model training.