The impact factors on the competence of big data processing

A major challenge of using volunteer computing (VC) for big data problems is the opportunistic environment. In such a dynamic, unreliable, and heterogeneous environment, there exist many uncertain factors that may impair the competence of big data processing. This paper explores these factors and their impact, aiming at exposing the original impaired performance that the undedicated environment of VC can achieve under the impact. Our investigation on this issue is four-fold. First, we define a number of impact factors to represent the opportunistic features of VC environment. Second, we model a distributed hash table-based MapReduce approach to process big data. Third, we proposes an ideal computing environment and then inject impact factors into the running MapReduce to quantitatively evaluate how the performance is impaired. Finally, we analyze the evaluation results to figure out the cause of impact and predict optimization potentials. The developers, who plan to construct MapReduce frameworks by using commodity computers or voluntary cycles on the public internet, will benefit from the evaluation results when considering performance requirements. © 2020, © 2020 Informa UK Limited, trading as Taylor & Francis Group