Joint Distributions

A joint distribution describes several random variables together, capturing how they co-vary as well as their individual behavior.

Two variables at once

The joint distribution of X and Y is described by a joint PMF p(x, y) = P(X = x, Y = y) in the discrete case, or a joint density f(x, y) in the continuous case with ∫∫ f = 1. It contains everything about the pair, including all dependence.

Marginals

Summing or integrating out one variable recovers the marginal distribution of the other: p_X(x) = Σ_y p(x, y). Marginals describe each variable alone but discard the information about how the two relate — you cannot rebuild the joint from marginals unless they are independent.

Independence as factorization

X and Y are independent exactly when the joint factors into the product of marginals, f(x, y) = f_X(x) f_Y(y), for all values. This is the multivariate meaning of independence and the assumption that makes high-dimensional models tractable.

Expectations over joints

The expectation of a function of both variables is E[g(X, Y)] = Σ Σ g(x, y) p(x, y) or the corresponding double integral. Covariance, correlation, and the distribution of sums all come from such joint expectations.

The curse of dimensionality

A joint distribution over d variables, each with m values, has mᵈ entries — an explosion that makes full joint tables impossible beyond a few variables. Graphical models tame this by factoring the joint into local conditional pieces, exploiting the many independence relations that usually hold.