Covariance

Covariance measures whether two random variables tend to move together, in the same direction or opposite ones.

Definition

The covariance of X and Y is Cov(X, Y) = E[(X − E[X])(Y − E[Y])] = E[XY] − E[X] E[Y]. It is positive when the variables tend to be large together, negative when one is large while the other is small, and zero when there is no linear association.

Properties

• Symmetric: Cov(X, Y) = Cov(Y, X).
• Cov(X, X) = Var(X).
• Bilinear: Cov(aX + b, cY + d) = ac Cov(X, Y).
• Independent variables have zero covariance (the converse is false).

Zero covariance is not independence

Covariance detects only linear association. Variables can be strongly dependent yet have zero covariance — for example Y = X² with X symmetric about zero. So zero covariance rules out linear coupling, not dependence in general.

The covariance matrix

For a vector of variables, all pairwise covariances form a symmetric, positive-semidefinite covariance matrix Σ, with variances on the diagonal. It is the central object of multivariate statistics, principal component analysis, and the multivariate normal distribution.

Why it matters for uncertainty

When propagating uncertainty through a model, ignoring covariance between inputs can badly misstate the output variance. Correlated errors do not cancel the way independent ones do; positively correlated inputs inflate the spread of any quantity computed from them, which is the case that dominates real risk estimates.

Estimating covariance from data uses the sample analogue, averaging the products of paired deviations. As with variance, dividing by n − 1 rather than n removes the small-sample bias. Sample covariances are noisy in high dimensions, where the number of pairs grows as the square of the number of variables, so estimates for many variables from few observations must be regularized before they can be trusted.