Floating-Point Error Analysis
How finite-precision arithmetic introduces rounding error, and how to reason about its accumulation and worst cases.
Numbers a computer cannot represent exactly
Floating-point numbers store a fixed number of significant digits, so most real numbers are rounded to the nearest representable value. The relative error of a single rounding is bounded by machine epsilon, roughly ten to the minus sixteen for standard double precision. Each arithmetic operation on floating-point numbers can introduce one more such rounding, and understanding how these accumulate is the subject of error analysis.
Catastrophic cancellation
The most notorious source of error is subtracting two nearly equal numbers. The leading digits cancel, leaving a result dominated by the rounding errors in the original operands: the relative error explodes even though each input was accurate. The classic example is the naive quadratic formula, which cancels badly when the two roots differ in magnitude; rewriting the formula to avoid the subtraction restores accuracy. Recognizing and reformulating cancellation-prone expressions is a core skill.
Accumulation of rounding
- Summing many numbers accumulates rounding; compensated (Kahan) summation recovers lost bits
- The order of operations can change the result because floating-point addition is not associative
- Well-scaled computations avoid overflow and underflow at the extremes of the exponent range
Backward error analysis
Rather than tracking the forward error of every operation, the modern approach of Wilkinson asks: of what nearby problem is the computed result the exact answer? If that nearby problem differs from the original only by a tiny amount, the algorithm is backward stable, and the accuracy of the result is then governed by the problem's conditioning. This reframing turns an intractable tangle of individual roundings into a clean statement about the algorithm.
Why it matters in simulation
Large simulations perform enormous numbers of operations, so even tiny per-operation errors can matter, especially in long time integrations and iterative solves near convergence. Sensible scaling, stable formulas, and awareness of cancellation keep computed physics trustworthy; ignoring them can produce plausible-looking but wrong results that are hard to diagnose.