Communication-avoiding Krylov subspace methods

The cost of an algorithm includes both arithmetic and communication.We use "communication" in a general sense to mean data movement,either between levels of a memory hierarchy ("sequential") orbetween processors ("parallel"). Communication costs include bothbandwidth terms, which are proportional to the number of words sent,and latency terms, which are proportional to the number of messages inwhich the data is sent. Communication costs are much higher thanarithmetic costs, and the gap is increasing rapidly for technologicalreasons. This suggests that for best performance, algorithms shouldminimize communication, even if that may require some redundantarithmetic operations. We call such algorithms"communication-avoiding."Krylov subspace methods (KSMs) are iterative algorithms for solvinglarge, sparse linear systems and eigenvalue problems. Current KSMsrely on sparse matrix-vector multiply (SpMV) and vector-vectoroperations (like dot products and vector sums). All of theseoperations are communication-bound. Furthermore, data dependenciesbetween them mean that only a small amount of that communication canbe hidden. Many important scientific and engineering computationsspend much of their time in Krylov methods, so the performance of manycodes could be improved by introducing KSMs that communicate less.Our goal is to take s steps of a KSM for the same communication costas 1 step, which would be optimal. We call the resulting KSMs"communication-avoiding Krylov methods." We can do this under certainassumptions on the matrix, and for certain KSMs, both in theory (formany KSMs) and in practice (for some KSMs). Our algorithms are basedon the so-called "s-step" Krylov methods, which break up the datadependency between the sparse matrix-vector multiply and the dotproducts in standard Krylov methods. This idea has been around for awhile, but in contrast to prior work (discussed in detail in Section1.5), this thesis makes the following contributions:We have fast kernels replacing SpMV, that can compute theresults of s calls to SpMV for the same communication cost as onecall (Section 2.1).We have fast dense kernels as well, such as Tall Skinny QR (TSQR-- Section 2.3) and Block Gram-Schmidt (BGS -- Section 2.4), which can do the work of Modified Gram-Schmidt applied to s vectors for a factor of Theta(s^2) fewer messages in parallel, and a factor of Theta(s/W) fewer words transferred between levels of the memory hierarchy (where W is the fast memory capacity in words).We have new communication-avoiding Block Gram-Schmidt algorithmsfor orthogonalization in more general inner products (Section 2.5).We have new communication-avoiding versions of the followingKrylov subspace methods for solving linear systems: the GeneralizedMinimum Residual method (GMRES -- Section 3.4), bothunpreconditioned and preconditioned, and the Method of ConjugateGradients (CG), both unpreconditioned (Section 5.4) andleft-preconditioned (Section 5.5).We have new communication-avoiding versions of the following Krylovsubspace methods for solving eigenvalue problems, both standard (Ax =&lambda x, for a nonsingular matrix A) and "generalized" (Ax =&lambda Mx, for nonsingular matrices A and M): Arnoldi iteration(Section 3.3), and Lanczos iteration, both for Ax = &lambda x (Section4.2) and Ax = &lambda M x (Section 4.3).We propose techniques for developing communication-avoiding versionsof nonsymmetric Lanczos iteration (for solving nonsymmetric eigenvalueproblems Ax = &lambda x) and the Method of Biconjugate Gradients(BiCG) for solving linear systems. See Chapter 6 for details.We can combine more stable numerical formulations that use differentbases of Krylov subspaces with our techniques for avoidingcommunication. For a discussion of different bases, see Chapter 7.To see an example of how the choice of basis affects the formulationof the Krylov method, see Section 3.2.2.We have faster numerical formulations. For example, in ourcommunication-avoiding version of GMRES, CA-GMRES (see Section 3.4),we can pick the restart length r independently of the s-step basislength s. Experiments in Section 3.5.5 show that this abilityimproves numerical stability. We show in Section 3.6.3 that it alsoimproves performance in practice, resulting in a 2.23 × speedupin the CA-GMRES implementation described below.We combine all of these numerical and performance techniques in ashared-memory parallel implementation of our communication-avoidingversion of GMRES, CA-GMRES. Compared to a similarly highly optimizedversion of standard GMRES, when both are running in parallel on 8cores of an Intel Clovertown (see Appendix A), CA-GMRES achieves 4.3× speedups over standard GMRES on standard sparse test matrices(described in Appendix B.5). When both are running in parallel on 8cores of an Intel Nehalem (see Appendix A), CA-GMRES achieves 4.1× speedups. See Section 3.6 for performance results and Section3.5 for corresponding numerical experiments. We first reportedperformance results for this implementation on the Intel Clovertownplatform in Demmel et al. [78].We have incorporated preconditioning into our methods. Notethat we have not yet developed practical communication-avoidingpreconditioners; this is future work. We have accomplished the following:* We show (in Sections 2.2 and 4.3) what the s-step basis shouldcompute in the preconditioned case for many different types of Krylovmethods and s-step bases. We explain why this is hard in Section 4.3.* We have identified two different structures that a preconditionermay have, in order to achieve the desired optimal reduction ofcommunication by a factor of s. See Section 2.2 for details.We present a detailed survey of related work, including s-step KSMs(Section 1.5, especially Table 1.1) and other techniques for reducingthe amount of communication in iterative methods (Section 1.6)