Lion Optimizer
Lion updates each weight by the sign of a momentum-smoothed gradient, using less memory than Adam while remaining competitive on large models.
A sign-based update
Lion (EvoLved Sign Momentum) is an optimizer discovered through automated search over optimizer programs. Its defining feature is that the update direction for every parameter is a sign: plus one or minus one, scaled by the learning rate. This means every weight moves by the same magnitude each step, with only the direction varying. It stands apart from adaptive methods like Adam, which scale each step by an estimate of the gradient's variance.
How it works
Lion maintains a single momentum buffer, an exponential moving average of past gradients. At each step it forms an interpolation between the current gradient and the momentum, takes the sign of that combination, and updates the weight by the negative learning rate times that sign, plus decoupled weight decay. It then updates the momentum buffer with a second, typically slower, decay rate. Because it stores only one buffer per parameter instead of two, it uses less optimizer memory than Adam.
update = torch.sign(beta1*m + (1-beta1)*g)
w = w - lr * (update + weight_decay * w)
m = beta2*m + (1-beta2)*g # momentum for next step- One momentum buffer per parameter, so lower memory than Adam
- Uniform update magnitude from the sign operation
- Typically needs a smaller learning rate and larger weight decay than AdamW
- Competitive on large vision and language models in reported results
Trade-offs and tuning
Because the step size is decoupled from gradient magnitude, Lion behaves differently from adaptive optimizers and needs its own hyperparameter tuning: the effective step is set purely by the learning rate, so that rate is usually roughly an order of magnitude smaller than for AdamW, and weight decay is set larger to compensate. The sign update also acts as a form of regularization by ignoring how confident the gradient is. Lion is attractive when optimizer memory is a constraint at large scale, though AdamW remains the more predictable default.