Statistical Entropy

Statistical entropy measures the average uncertainty in a random variable, in bits or nats, and grounds information theory.

Definition

The Shannon entropy of a discrete distribution is H(X) = −Σ p(x) log p(x). With a base-2 logarithm the unit is bits; with the natural log it is nats. Entropy is the average surprise per outcome, where surprise for an outcome is −log p(x): rarer outcomes carry more information.

Properties

• Entropy is non-negative and zero only for a certain outcome.
• It is maximized by the uniform distribution over n outcomes, at log n.
• It is additive for independent variables: H(X, Y) = H(X) + H(Y).

Why it equals compressibility

Shannon's source coding theorem states that the average number of bits needed to encode outcomes of X cannot be less than H(X), and codes approaching this bound exist. Entropy is therefore the fundamental limit of lossless compression — a predictable source is cheap to encode, a uniform one expensive.

import math
def entropy(ps):
    return -sum(p*math.log2(p) for p in ps if p > 0)
print(round(entropy([0.5,0.5]),3))    # 1.0 bit
print(round(entropy([0.9,0.1]),3))    # 0.469 bits

Related quantities

Conditional entropy H(X | Y) is the uncertainty remaining in X once Y is known. Mutual information I(X; Y) = H(X) − H(X | Y) measures how much one variable tells you about another. The Kullback-Leibler divergence measures how far one distribution is from another and is minimized when a model matches the data.

Beyond information theory

The maximum-entropy principle chooses the distribution with the largest entropy consistent with known constraints, giving the least-biased model — the uniform on a bounded interval, or the normal for a fixed mean and variance. Statistical entropy also connects directly to the thermodynamic entropy of physics.