Benchmarking Graph-Processing Platforms: A Vision

Processing graphs, especially at large scale, is an increasingly use-ful activity in a variety of business, engineering, and scientific do-mains. Already, there are tens of graph-processing platforms, such as Hadoop, Giraph, GraphLab, etc., each with a different design and functionality. For graph-processing to continue to evolve, users have to find it easy to select a graph-processing platform, and de-velopers and system integrators have to find it easy to quantify the performance and other non-functional aspects of interest. How-ever, the state of performance analysis of graph-processing plat-forms is still immature: there are few studies and, for the few that exist, there are few similarities, and relatively little under-standing of the impact of dataset and algorithm diversity on perfor-mance. Our vision is to develop, with the help of the performance-savvy community, a comprehensive benchmarking suite for graph-processing platforms. In this work, we take a step in this direction, by proposing a set of seven challenges, summarizing our previous work on performance evaluation of distributed graph-processing platforms, and introducing our on-going work within the SPEC Re-search Group’s Cloud Working Group