Floating-Point Error and Stability

Rounding errors are individually tiny but can accumulate or cancel; stable algorithms keep the total error close to the problem's inherent limit.

How errors accumulate

Each floating-point operation adds a relative error of at most machine epsilon. In a long computation these errors combine. In the best case they behave like a random walk and grow slowly; in the worst case they align and grow linearly, or subtraction causes cancellation that makes them dominate. Understanding which case applies is the heart of error analysis.

Summing n numbers naively accumulates a worst-case error proportional to n times epsilon. Compensated (Kahan) summation tracks the lost low-order bits in a running correction term and reduces the error to be essentially independent of n, at the cost of a few extra operations per element.

def kahan_sum(xs):
    total = 0.0; c = 0.0
    for x in xs:
        y = x - c
        t = total + y
        c = (t - total) - y
        total = t
    return total

Absolute versus relative error

Absolute error is |computed - true|; relative error divides by |true|. Relative error is usually the meaningful measure because floating point has uniform relative precision. Reporting only absolute error can hide catastrophic loss of significance in small quantities.

Designing for stability

Practical rules keep error under control: avoid subtracting near-equal numbers, add numbers of similar magnitude together, prefer orthogonal transformations (which do not amplify error) over general ones, and use higher precision for critical accumulations. When in doubt, compare results at single and double precision to gauge sensitivity.

Multi-physics codes such as those modeling the breeder Hyperion run millions of coupled updates per time step; small per-step errors are held in check by stable time integrators and by summation schemes chosen so that conserved quantities like energy do not drift.