Activation Function
The nonlinear function applied at each neuron that lets networks represent nonlinear relationships.
Definition
An activation function is the nonlinear transform applied to a neuron's weighted input. Without it, a stack of layers would collapse into a single linear map, so the activation is what gives a neural network its expressive power.
Beyond expressiveness, the choice of activation shapes the geometry of the loss surface the optimizer must navigate. Smooth activations like GELU and SiLU, common in large models, blend the efficiency of ReLU with gentler gradients, illustrating how small architectural choices accumulate into large practical differences.
The choice interacts with initialization and normalization to keep signals well-scaled through depth. A poor pairing can leave a network unable to learn even when its architecture is sound. Because these effects are subtle and empirical, practitioners usually start from established defaults, ReLU or its smooth variants in hidden layers, and change them only with evidence that the specific problem benefits.
Common choices
- ReLU: max(0, x), fast and the default for hidden layers.
- Sigmoid: squashes to (0,1), used for probabilities.
- Tanh: squashes to (-1,1), zero-centered.
- Softmax: normalizes a vector into class probabilities.
Trade-offs
Sigmoid and tanh saturate for large inputs, producing tiny gradients that stall learning, the vanishing gradient problem. ReLU avoids saturation on the positive side and trains faster, though it can leave units permanently inactive; variants such as leaky ReLU address this.
Why it matters
The choice of activation affects trainability, convergence speed, and the range of representable functions. It is one of the first architectural decisions in building a network.
Fusion connection
Kronos surrogate networks typically use ReLU or smooth variants; smoothness matters when the surrogate must be differentiated inside a physics-aware optimizer.