GELU and Swish

Smooth, self-gated activations that behave like ReLU for large inputs but curve gently near zero, now standard in transformers and modern vision models.

Motivation

ReLU has a sharp corner at the origin and a hard zero for all negative inputs. Smooth activations replace that corner with a gentle curve, which can make the loss surface easier to optimize and lets small negative inputs pass a little signal. GELU and Swish (also called SiLU) are the two most widely used smooth activations in current architectures.

GELU

Kronos motion — q curve

The Gaussian Error Linear Unit multiplies the input by the probability that a standard normal variable is less than that input: GELU(x) = x · Phi(x), where Phi is the standard normal cumulative distribution function. Intuitively, it weights each input by how likely it is to be positive under a Gaussian, a smooth stochastic gate. It is the default activation in BERT, GPT-family feedforward blocks, and most modern transformers.

Swish / SiLU

Swish is x · sigmoid(beta·x). With beta = 1 it is called SiLU, the sigmoid-weighted linear unit. Like GELU it is smooth, non-monotonic (it dips slightly below zero for small negatives before rising), and self-gated: the input gates itself through a sigmoid. Swish appears in EfficientNet and many mobile vision architectures.

python

import numpy as np
def gelu(x):
    return 0.5*x*(1+np.tanh(0.79788456*(x+0.044715*x**3)))
def silu(x):
    return x/(1+np.exp(-x))

Why non-monotonic helps

Both functions are slightly non-monotonic near the origin, dipping below zero for small negative inputs. This lets the network represent a richer set of local shapes than a strictly increasing function and keeps a nonzero gradient on the negative side, avoiding the dead-unit problem of plain ReLU. The smoothness also means the second derivative exists everywhere, which some optimizers exploit.

Practical use

GELU is the safe default inside transformer feedforward layers. Swish/SiLU is common in convolutional networks tuned for efficiency. Both cost slightly more than ReLU because of the exponential or tanh, but the difference is negligible relative to the matrix multiplications that dominate a forward pass. When accuracy matters more than raw speed, they are usually worth the small extra cost.

Smooth, self-gated alternatives to ReLU.
GELU gates by the Gaussian CDF; Swish by a sigmoid.
Slightly non-monotonic, no dead-unit problem.
GELU is standard in transformers; Swish in efficient CNNs.