Resource-proportional Job Allocation for MapReduce in Heterogeneous Clouds

[[abstract]]近年來，由於網際網路的快速發展，雲端運算受到大量的關注。藉由MapReduce程式設計模式，使用者可以很簡單的開發能夠有效處理大量資料的應用。根據過去文獻的研究成果，提升MapReduce運算效能的挑戰之一為如何分配適當的Mapper 和 Reducer數量給雲端裡的每個節點來執行。儘管過去已經有許多改善MapReduce運算效能的研究，但是大部分的方法都是假設節點是同質的情況，因此預設採用的工作分配方式是均勻分配，即分配相同數量的Mapper與Reducer給每個節點。然而，均勻分配法並不適用在異質的雲端運算環境，因為每個節點的運算能力與剩餘資源不同，如果分配的工作量相同則會導致節點間工作的不平衡而使得運算效能降低。因此，我們提出一個在異質的雲端運算環境下，可以透過分析工作的特性、資料量大小與節點資源的不同來分配給每個節點不同的Mapper與Reducer數量，使得每個節點能夠有較平均的工作負載，這樣不僅可以提升運算效能，也可以有效的使用節點的資源。[[abstract]]In recent years, the rapid development of the Internet, a new computing paradigm, cloud computing, has received a significant attention. The large-scale data processing is easily obtained with thin clients by utilizing the underlying MapReduce programming model. According to the previous researches, one of the key challenges to the MapReduce performance is determining the appropriate number of mappers and reducers allocated to cluster nodes in a cloud. Although many research results have been proposed to improve the MapReduce performance, most of them assume that cluster nodes are homogeneous. The default job allocates the same numbers of Mapper and Reducer to each node. However, it is not suitable for a heterogeneous cloud, because computing capability of each node and usability of resources are different, if the job allocates the same number of mapper and reducer to each node, will cause the performance degradation of the lower resources in the node. In our study, the resolution is to analysis of the work characteristics, data size and heterogeneous environment to assigned different numbers of mapper and reducer, it will achieve equally workload for each node, this is will improve computing performance and use the resources in the node more efficiency