Flash-aware Database Management Systems

Flash SSDs are becoming the primary storage technology for single servers and large data centers. In contrast to conventional magnetic disks, which were dominating the storage market for more than 40 years, Flash offers significantly more performance, consumes less energy and has lower cost per IOPS (I/O Operations Per Second). Besides these advantages, an important role in establishment and quick proliferation of Flash storage was played by the black-box design of SSDs, which guaranteed their backwards compatibility with the traditional hard disk drives. This makes the replacement of HDDs seamless as the software stack, including the application, does not require any adjustment. However, such design of SSDs has multiple disadvantages, which become especially critical for database management systems. The backwards compatibility of SSDs is encapsulated in the so-called Flash translation layer (FTL). FTL is a set of Flash management tasks that typically run on device and mask the native behavior of Flash memory. In other words, FTL creates a black-box over Flash memory and emulates the behavior of HDDs. The fact that the database system has no knowledge about FTL, and has no control over the physical data placement on Flash, results in high I/O overhead, which is caused by suboptimal realization of Flash management tasks and functional redundancy along the critical I/O path. Thus, write-amplification of conventional SSDs used in traditional ’cooked’ storage architecture (i.e., with file system indirection) can be as high as 15x, i.e., a single 4KB write request submitted by the DBMS can turn into 60KB being physically written on Flash storage. As a result, the effective I/O throughput and longevity expectations of SSDs are significantly lower than those of Flash memory encapsulated in these SSDs. In this work we describe our approach - the NoFTL storage architecture - that aims to solve the aforementioned disadvantages of modern Flash SSDs. The basic idea behind the NoFTL is to give the full control over the underlying Flash storage to the database management system, which in turn assumes elimination of all intermediate abstraction layers (file system, block device layer and FTL) between the DBMS and physical storage. NoFTL consists of three main elements - (i) native Flash interface; (ii) integration of Flash management into subsystems of the DBMS; and (iii) the concept of configurable Flash storage. The interplay of them allows us to realize the whole performance potential of Flash memory. Native Flash interface allows the DBMS to control physical data placement on Flash storage, and to utilize the computational power of the SSD to perform near-data processing. Integration of typical Flash management tasks (address translation, garbage collection and wear leveling) into different subsystems of the DBMS leads to an optimization of these tasks and of native DBMS algorithms. The concept of configurable Flash storage is a unique approach to organize and manage data on Flash SSDs. With the help of novel storage abstractions, the database system can perform intelligent data placement by clustering objects into different regions. Moreover, for each such region the DBMS can apply a separate set of Flash management algorithms, which would be optimal for data assigned to that region. All this reduces the write-amplification of SSDs to a minimum (up to 15x reduction for OLTP workloads), improves the overall system performance, and significantly increases the lifetime of Flash SSDs (up to 30x improvement). We have realized the NoFTL prototype on an open-source database engine and evaluated it under various scenarios and on different testbeds