Collectives for multiple resource job scheduling across heterogeneous servers

Abstract. Efficient management of large-scale, distributed data storage and pro-cessing systems is a major challenge for many computational applications. Many of these systems are characterized by multi resource tasks processed across a het-erogeneous network. Conventional approaches, such as load balancing, work well for centralized, single resource problems, but breakdown in the more general case. In addition, most approaches are ofcen based on heuristics which do not directiy attempt to optimize the world utility. In this paper, we propose an agent based control system using the theory of collecti-res. We configure the servers of OUT network with agents who make local job scheduling decisions. These decisions are based on local goals which are constructed to be aligned with the objective of optimizing t k ~ overall efficiency of the system. We demonstrate that multi-agent systems in which all the agents attempt to optimize the same global utility function (team game) only marginally outperform conventional load balancing. On the other hand, agents configured using collectives outperform both team games and load balancing (by up to four times for the latter), despite their distributed nature and their limited access to information