Scalable real-time monitoring for distributed applications

In order to assess the overall service quality in real time, the performance metrics of a distributed application (such as a streaming session) need to be continuously monitored at some monitoring servers. Much of the previous work has used distributed aggregation tree (DAT) to achieve user scalability in monitoring. However, this approach often leads to high monitoring delay and network stress. In this thesis, we present an efficient monitoring algorithm for large-scale applications. There are multiple distributed monitors keeping track of global performance. The network is of two tiers, where client applications report their performance to a proxy by means of an overlay, and the proxies form another spanning tree among themselves to report to the monitors. Such network provides better isolation against peer churns and hence better monitoring stability. We propose and study the algorithm on how to construct a real-time monitoring network minimizing the monitoring delay. We first formulate the problem and show that it is NP-hard. Then we propose a simple, adaptive and scalable protocol called SMon, which continuously reduces the network diameter in the presence of peer churns. Through simulations and actual measurements through implementation, we show that SMon achieves low monitoring delay and network stress for distributed applications