Considerations on modeling for early detection of abnormalities in locally autonomous distributed systems

The advancements in technology yield complex distributed systems on which we come to depend on as a society as well as for economic welfare. The increasingly higher quality is demanded, however the behavior of complex distributed systems is not always as desired. The prevention of harm and loss depends on early detection of disturbances based on a system model. However the behavior of complex systems is difficult to describe from the systems design and the laws of physics and logic utilized therein. Many assumptions and simplifications are required to develop a theoretical system model when systems are complex. Consequently theoretical systems models do not correspond sufficiently with reality, moreover they are insufficiently capable of revealing undesired behavior. The underlying misconception is the idea that the system and potentially undesirable changes are independent entities. We propose that a simplified description of reality has it’s merits for controlling complex systems, however for detection these simplifications must not cause limitations to the potential of modeling the dependencies in the system behavior as a whole. In short, it is recommended not to mimic the presumed structure in system in the system model. We propose an alternative that includes a flexible abstraction layer in the model. This layer may not be suitable for interpretations, however it does allow for measuring coherency in the change from the residuals. We illustrate the proposed principle by distinguishing superficial from profound system changes from the learning behavior of artificial neural networks. We also sketch potential application of this approach in large scale information processing systems such as the LOFAR radio-telescope. The problem addressed in this thesis is the following. There are harmful failures and production loss in large scale distributed systems. Time related disturbances are symptomatic signs of such failures and losses. It is required to make an early distinction between acceptable variations and potentially harmful trends. This distinction depends on dynamical models of the system behavior as a whole including interactions between self-organizing processes in the system that have not been designed. The existing classical modeling approaches do not allow for this. We have conducted a literature survey for existing approaches to detection and diagnosis. We have also conducted experiments to accommodate time-related disturbances in a batch-oriented production process using neural networks. Simulations illustrate the proposed approach to separate superficial from profound changes in the system. The research provides the essential requirements for modeling a time-variant system for the early detection of abnormalities. We have supported these requirements by revealing the limitations of the existing conventional detection methods and techniques, and also the problematic aspects of locally autonomous distributed systems causing these limitations. The main conclusion is that early detection requires an adaptive so called monolithic model, such a model considers the behavior of the system as a whole even if the system is actually modular and can be extremely partitioned