Floating-Point Hardware
Floating-point units execute real-number arithmetic; understanding fused operations, throughput, and special values explains real numerical behavior.
The arithmetic engine
A floating-point unit (FPU) performs add, multiply, and related operations on IEEE 754 values. Modern cores contain several FPUs and wide vector units, so peak floating-point rate equals cores times vector width times operations per cycle times clock. Reaching that peak requires code that keeps the units continuously fed.
Fused multiply-add
The fused multiply-add (FMA) computes a*b + c as one instruction with a single rounding step, instead of two. This doubles the arithmetic per instruction for the common multiply-accumulate pattern and is slightly more accurate because it rounds once. Dense linear algebra and convolution are built on FMA throughput.
Special values
- Infinity: results of overflow or division by zero
- NaN (not a number): from invalid operations like 0/0; propagates through arithmetic
- Denormals: very small numbers near zero, sometimes handled slowly in hardware
- Signed zero: distinguishes limits approached from above or below
Non-associativity
Floating-point addition is not associative: (a+b)+c may differ from a+(b+c) because of rounding. This means parallel reductions, which sum in a different order than a serial loop, can give slightly different results. It is not a bug; it is a property of finite precision that reproducibility efforts must account for.
Practical consequences
Understanding FPU behavior explains why precision choices affect both speed and accuracy, why denormals can cause mysterious slowdowns, and why a parallel result may not match a serial one bit for bit. Numerically robust HPC code is written with these realities in mind.