Mutual Information

Mutual information quantifies how many bits one variable reveals about another.

Definition

Mutual information I(X;Y) is the reduction in uncertainty about X from learning Y. It equals H(X) − H(X|Y), and equivalently H(X) + H(Y) − H(X,Y).

Symmetry and non-negativity

Kronos motion — many body

Mutual information is symmetric: I(X;Y) = I(Y;X). It is always at least 0, and it is exactly 0 if and only if X and Y are independent. Dependence of any kind produces positive shared information.

As a distance from independence

Formally, mutual information is the Kullback–Leibler divergence between the true joint distribution and the product of the marginals. It measures how far the variables are from being independent.

Where it is used

Feature selection in machine learning
Registration of medical and other images
Measuring the capacity of a communication channel
Detecting statistical dependence beyond linear correlation

Beyond correlation

Correlation captures only linear relationships, but mutual information detects any statistical dependence, including nonlinear ones. Two variables can have zero correlation yet high mutual information.

python

import math
def mutual_info(pxy, px, py):
    mi = 0.0
    for (x,y), p in pxy.items():
        if p > 0:
            mi += p * math.log2(p / (px[x]*py[y]))
    return mi