Incremental Elasticity For Array Databases

Relational databases benefit significantly from elasticity, whereby they execute on a set of changing hardware re-sources provisioned to match their storage and processing re-quirements. Such flexibility is especially attractive for scien-tific databases because their users often have a no-overwrite storage model, in which they delete data only when their available space is exhausted. This results in a database that is regularly growing and expanding its hardware proportion-ally. Also, scientific databases frequently store their data as multidimensional arrays optimized for spatial querying. This brings about several novel challenges in clustered, skew-aware data placement on an elastic shared-nothing database. In this work, we design and implement elasticity for an array database. We address this challenge on two fronts: determining when to expand a database cluster and how to partition the data within it. In both steps we propose incremental approaches, affecting a minimum set of data and nodes, while maintaining high performance. We in-troduce an algorithm for gradually augmenting an array database’s hardware using a closed-loop control system. Af-ter the cluster adds nodes, we optimize data placement for n-dimensional arrays. Many of our elastic partitioners incre-mentally reorganize an array, redistributing data only to new nodes. By combining these two tools, the scientific database efficiently and seamlessly manages its monotonically increas-ing hardware resources. 1