S-Storm: A Slot-aware Scheduling Strategy for Even Scheduler in Storm

Storm has been a popular distributed real-time computation system for stream data processing, which currently provides an even scheduler to distribute all executors and workers of topology among all worker nodes. In this paper, we find that the even scheduler ignores the allocation and dependence relationship among slots. This would bring the load-unbalancing problem when the topology run failed and is killed by its user, or more new machines are extended in Storm cluster. Aiming at solving them, we design the S-Storm, a slot-aware scheduling strategy for even scheduler in Storm, which achieves a fine-grained EvenScheduler using the slot-aware sorting queue and merger factor. S-Storm has the following desirable features: 1) It evenly allocates slots for multitopologies in load-balancing cluster. 2) When load-unbalancing happen, it distributes workers to slots among light-load worker nodes. 3) It achieves a fine-grained EvenScheduler by accessing one slot to the slot-aware sorting queue, and the slot is on the minimum load worker node. 4) It uses the merger factor to avoid over competition of the same resources among the same Spout/Bolt tasks in machine. We implemented S-Storm based on Storm 0.10.0 and used two different applications for performance evaluation. Extensive experimental results show that compared to Storm, S-Storm can achieve a better performance in the average processing time and throughput, while recovering balance from load-unbalancing.National High Technology Research and Development Program ("863" Program) of China [2015AA016009]; National Natural Science Foundation of China [61232005, 61672062]; Science and Technology Program of Shen Zhen, China [JSGG20140516162852628]CPCI-S(ISTP)623-63