EStore: An Effective Optimized Data Placement Structure for Hive

The data warehouse system Hive has emerged as an important facility for supporting data computing and storage. In particular, RCFile is a tailor-made data placement structure implemented in Hive, which is designed for the data processing efficiency. In this paper, we propose several optimized schemes based on RCFile and introduce EStore, which is an optimized data placement structure that is able to improve the query rate and reduce storage space for Hive. Specifically, it adopts both row-store and column-store in blocks, and further classifies the columns by the frequency of each table-column. Moreover, we also employ the classic RDP code to store files of the data table. We conduct experiments on a real cluster, and the results show that EStore has better features in terms of data query rate and storage space compared with RCFile.Natural Science Foundation of China (NSFC) [61671001, 61521003]; National Key Research and Development Program of China [2016YFB0800101]; Guangdong Research Programs [2016B030305005]; Shenzhen Research Programs [ZDSYS201603311739428, JCYJ20150331100723974, 20140509093817684]CPCI-S(ISTP)2996-300