Feedback Control Theory and Processing System Log Streams

In this project, we focus both on building a general tool that can be used by system operators and machine learning researchers for analyzing data, and on exploring general techniques of applying feedback control theory to distributed computer systems. We have built a scalable and flexible system log processing system using TelegraphCQ [36] as the building blocks. The report is divided into two parts. In the first part, we discuss the needs for processing system log data as data streams and our general design of parallel and multi-tier architectures. In the second part of the report, we address problems regarding to this stream processing system as being a complex distributed system itself. We focus on how random disturbances can have an impact on the performance and reliability. All problems addressed are solved systematically with feedback-control-theory. We discuss three uses of the feedback-control-theory. First, as the workload regulator, then the load balancer and, last, the workload sim-ulator. We illustrate the general techniques for the system identification and the controlle