Building efficient query engines in a high-level language

In this paper we advocate that it is time for a radical rethinking of database systems design. Developers should be able to leverage high-level programming languages without having to pay a price in efficiency. To realize our vision of abstraction without regret, we present LegoBase, a query engine written in the high-level pro-gramming language Scala. The key technique to regain efficiency is to apply generative programming: the Scala code that constitutes the query engine, despite its high-level appearance, is actually a program generator that emits specialized, low-level C code. We show how the combination of high-level and generative program-ming allows to easily implement a wide spectrum of optimizations that are difficult to achieve with existing low-level query compilers, and how it can continuously optimize the query engine. We evaluate our approach with the TPC-H benchmark and show that: (a) with all optimizations enabled, our architecture signifi-cantly outperforms a commercial in-memory database system as well as an existing query compiler, (b) these performance improve-ments require programming just a few hundred lines of high-level code instead of complicated low-level code that is required by exist-ing query compilers and, finally, that (c) the compilation overhead is low compared to the overall execution time, thus making our ap-proach usable in practice for efficiently compiling query engines. 1