Achieving Numerical Reproducibility in the Parallelized Floating Point Dot Product

The world depends on computers every day to do accurate real-world mathematics. Computers must store real numbers in a finite representation that approximates real numbers, called floating point representation. However, simply by changing the order in which we add a list of floating point numbers can provide a different result that may even be less accurate than another ordering. This is because floating point addition is not associative. That is, (a + b) + c is not necessarily equal to a + (b + c). Parallel computing techniques introduce the ability to reorder computations, thus producing a difference in results between runs. Numerical reproducibility means attaining a bit-wise identical result every time an application is run, and is valuable for debugging and testing purposes. Because scientific applications have come to use parallel computing techniques in order to solve increasingly complex problems, non-reproducibility is often introduced in the form of basic operations these applications rely on, such as the dot product. We use the Kahan and Knuth summation algorithms to reduce rounding error in order to produce more reproducible dot product results