Setting up clusters of computing units to process several data streams efficiently

Abstract. Let us consider an upper bounded number of data streams to be pro-cessed by a Divisible Load application. The total workload is unknown and the available speeds for communicating and computing may be poorly a priori esti-mated. This paper presents a resource selection method that aims at maximizing the throughput of this processing. From a set of processing units linked by a net-work, this method consists in forming an optimal set of master-workers clusters. Results of simulations are presented to assess the efficiency of this method exper-imentally. Before focusing on the proposed resource selection method, the paper comes back on the adaptive scheduling method on which it relies