Mining Modern Repositories with Elasticsearch

Organizations are generating, processing, and retaining data at a rate that often exceeds their ability to analyze it effec-tively; at the same time, the insights derived from these large data sets are often key to the success of the organi-zations, allowing them to better understand how to solve hard problems and thus gain competitive advantage. Be-cause this data is so fast-moving and voluminous, it is in-creasingly impractical to analyze using traditional offline, read-only relational databases. Recently, new “big data ” technologies and architectures, including Hadoop and NoSQL databases, have evolved to better support the needs of organizations analyzing such data. In particular, Elasticsearch — a distributed full-text search engine — explicitly addresses issues of scalability, big data search, and performance that relational databases were simply never designed to support. In this paper, we reflect upon our own experience with Elasticsearch and highlight its strengths and weaknesses for performing modern mining software repositories research