A Scheduling Strategy to Run Hadoop Jobs on Geodistributed Data

none5noInternet-of-Things scenarios will be typically characterized by huge amounts of data made available. A challenging task is to effi- ciently manage such data, by analyzing, elaborating and extracting use- ful information from them. Distributed computing framework such as Hadoop, based on the MapReduce paradigm, have been used to process such amounts of data by exploiting the computing power of many cluster nodes. However, as long as the computing context is made of clusters of homogeneous nodes interconnected through high speed links, the benefit brought by the such frameworks is clear and tangible. Unfortunately, in many real big data applications the data to be processed reside in many computationally heterogeneous data centers distributed over the planet. In those contexts, Hadoop was proved to perform very poorly. The pro- posal presented in this paper addresses this limitation. We designed a context-aware Hadoop framework that is capable of scheduling and dis- tributing tasks among geographically distant clusters in a way that min- imizes overall jobs’ execution time. The proposed scheduler leverages on the integer partitioning technique and on an a-priori knowledge of big data application patterns to explore the space of all possible task sched- ules and estimate the one expected to perform best. Final experiments conducted on a scheduler prototype prove the benefit of the approach.noneCavallo Marco; Cusmà Lorenzo; Di Modica G; Polito Carmelo; Tomarchio OrazioCavallo Marco; Cusmà Lorenzo; Di Modica G; Polito Carmelo; Tomarchio Orazi