Testing Scientific Software
Testing code whose correct output is unknown requires standing in for the missing answer with limits, symmetries, and convergence.
The Oracle Problem
Ordinary software testing compares output to a known expected value. Scientific software often has no known answer; that is why the simulation exists. This is the oracle problem, and it forces a different testing toolkit built from properties the true solution must satisfy even when its value is unknown.
Substitutes for a Known Answer
- Analytic limits: cases where the equations have a closed-form solution the code must reproduce.
- Manufactured solutions: insert a chosen function, derive the source term it implies, and check the code recovers it.
- Conservation checks: mass, energy, momentum, or charge that must be conserved to machine precision.
- Symmetry checks: a symmetric problem must produce a symmetric result.
- Convergence: refining the mesh must reduce error at the theoretical order of accuracy.
Layers of Test
Unit tests check individual functions against known values. Integration tests check that components work together. Convergence and conservation tests check the numerical mathematics. Regression tests freeze a trusted output and alarm when a change alters it. A healthy scientific code carries all of these and runs them automatically.
Tolerances, Not Equality
Floating-point arithmetic makes exact equality the wrong test for most numerical code. Tests assert that a result lies within a stated tolerance, and the tolerance is itself part of the specification. A test that passes only by luck of rounding is worse than no test, because it creates false confidence.
At Kronos
Frozen physics results are tied to test suites that check conservation, convergence, and reproduction of analytic limits. This is what lets a design number be defended as the output of a verified solver rather than an unchecked run, and it keeps the numerical ledger separate from the physical one.