Hypothesis Testing

A hypothesis test asks whether data are surprising enough under a default assumption to reject it.

The framework

A test contrasts a null hypothesis H₀ — the default, often 'no effect' — with an alternative H₁. You compute a test statistic from the data and ask how probable such a value would be if H₀ were true. If it is too improbable, you reject H₀.

Test statistic and reference distribution

The test statistic is chosen so its distribution under H₀ is known: a z-statistic against the normal, a t-statistic against the Student-t, a chi-squared statistic against χ². Comparing the observed value to this reference distribution yields the evidence.

Significance level

Before looking at the data you fix a threshold α, the tolerated rate of false rejections, commonly 0.05. If the p-value falls below α, the result is called statistically significant. The threshold is a convention, not a law of nature, and should be chosen for the decision at hand.

One- and two-sided tests

A two-sided test asks whether a parameter differs from the null value in either direction; a one-sided test looks in a specified direction only. The choice must be made in advance — switching after seeing the data inflates the false-positive rate.

What a test cannot do

Failing to reject H₀ is not proof it is true; it may only mean the sample was too small to detect an effect. And statistical significance is not practical importance — with enough data, a trivially small effect becomes significant. Report the effect size and its uncertainty alongside any test.