Sigmoid and Tanh

The classic S-shaped activations squash inputs into bounded ranges; still useful at outputs and gates, but prone to vanishing gradients in deep hidden layers.

The sigmoid function

The logistic sigmoid is sigma(x) = 1 / (1 + e^-x). It maps any real number into the open interval (0, 1), which makes its output readable as a probability. Its derivative is sigma(x)·(1 - sigma(x)), which peaks at 0.25 when x = 0 and falls toward zero as the input grows large in magnitude. That small maximum derivative is the root of its weakness in deep stacks.

The hyperbolic tangent

tanh(x) = (e^x - e^-x) / (e^x + e^-x) maps inputs to (-1, 1). Unlike sigmoid it is zero-centered, so its outputs are roughly balanced around zero. Zero-centered activations tend to make optimization better behaved, because gradients passed to the next layer are not all the same sign. tanh is essentially a rescaled, shifted sigmoid: tanh(x) = 2·sigma(2x) - 1.

The saturation problem

Both functions saturate: for large positive or negative inputs the curve flattens and the derivative approaches zero. In backpropagation the activation derivative multiplies the incoming gradient, so saturated units pass almost no signal backward. Stack many such layers and the gradient vanishes, stalling learning in early layers. This is why hidden layers moved to ReLU-family activations.

import numpy as np
def sigmoid(x): return 1/(1+np.exp(-x))
def dsigmoid(x): s = sigmoid(x); return s*(1-s)  # max 0.25 at x=0
def tanh(x): return np.tanh(x)              # zero-centered

Where they still belong

Sigmoid remains the right choice for a single binary output paired with binary cross-entropy loss, and for independent multi-label outputs. Both sigmoid and tanh live inside LSTM and GRU gates, where the bounded range is exactly what a gate needs: sigmoid to produce a 0-to-1 gate value and tanh to produce a bounded candidate state. In these roles their saturation is a feature, not a flaw.

• Sigmoid outputs probabilities in (0,1).
• Tanh is zero-centered, output in (-1,1).
• Both saturate and can vanish gradients in deep hidden stacks.
• Ideal at binary outputs and inside recurrent gates.