Optimizing Matrix Computations for Learning (and learning to optimize matrix computations)

Matrix-vector notation is the predominant idiom in which machine learning formulae are expressed; some models, like Gaussian processes (Rasmussen & Williams, 2006), would be extremely difficult to describe without it. Turning a matrix expression into a computer program is not always easy, however. Although good implementations of primitive matrix operations are available (Golub & Van Loan, 1996) as are packages like MĆęđĆć1, which pro-vide a high-level interface to these primitives, two important tasks must still be carried out manually: (i) computing derivatives of matrix functions and (ii) turning a matrix expression into an efficient computer program. Not having tools to do this can and does harm research: even for the relatively simple example of fitting a linear regressionmodel with gradientmeth-ods, the number of types and combinations of basis functions a researcher can experiment with is limited by the need to manually differentiate the objective function and write code for each version. We have addressed these issues by combining a symbolic matrix algebra engine with a superoptimizing compiler: an interesting learning problem in itself. We call our system Coconut. Coconut transforms matrix expressions into fast programs for computing them. For th