Fine-grained Performance Evaluation of Large-scale Graph Processing Systems

In the age of information our society generates data at an increasing and already alarming rate. To keep up with the rapid increase in the amount of available data, the academics have shown strong interests in the emerging research field of Big Data Processing (BDP), which explores technologies aiming at efficient processing of enormous amounts of data. As a result, many different types of BDP systems, for example systems specialized in large-scale graph processing, have been added into the Big Data ecosystem in the past decade. The performance of BDP systems impacts both efficiency and direct operational costs significantly, and may be the primary driving force behind the development of these systems. Performance evaluation aims at providing better understanding of BDP systems, which is proven to be challenging. The coarse-grained "black-box" evaluation is commonly used by performance analysts, which considers a BDP system as a black-box and only observes the overall execution time taken by the jobs of the BDP systems. Such coarse-grained evaluation method fails to provide much insight of the internal operations of the BDP systems. In contrast, fine-grained performance evaluation can be challenging and time-consuming, as it requires in-depth knowledge of the BDP systems on the implementation level, and can involve much repetitive work to process large amounts of performance statistics. Moreover, it is under question how efficiently the users can make use of the results of performance studies, as many studies are often out-dated, have a limited scope, and are difficult to reproduce. This thesis work consists of three main chapters. First, we propose Granula, a fine-grained performance evaluation framework which facilitates performance modeling, performance archiving and performance visualizing of Big Data Processing jobs. This framework is designed to extract fine-grained, comprehensive performance results from BDP systems and to facilitate performance-related knowledge exchanges between system developers, researchers, and end users. Second, using the Granula framework, we build comprehensive performance models for two graph processing systems i.e., Giraph and GraphX. The evaluation method used for the purpose of building these performance models can be reused by others for their custom workload, and these models can be extended by other performance analysts for more in-depth studies. Third, we conduct fine-grained performance evaluation on these two graph processing systems and validate the performance models we built. Among the important achievement of this study, we investigate and quantify the performance overhead and the performance bottleneck of these systems. Overall, we hope to have provided better understanding of the performance of BDP systems by improving on the process of performance evaluation, and by defining a method to integrate performance evaluation into the Big Data ecosystem.Computer ScienceParallel and Distributed Systems GroupElectrical Engineering, Mathematics and Computer Scienc