Continuous evaluation framework for software architectures: an IoT case

Context: Design-time evaluation is essential to build the initial software architecture to be deployed. However, experts’ design-time assumptions are unlikely to remain true indefinitely in systems characterized by scale, heterogeneity, and dynamism (e.g. IoT). Experts’ design-time decisions can be thus challenged at run-time. A continuous architecture evaluation that systematically intertwines design-time and run-time evaluation is necessary. However, the literature lacks examples on how continuous evaluation can be realized and conducted. Objective: This thesis proposes the first continuous architecture evaluation framework. Method: The framework is composed of two phases: design-time and run-time evaluation. The design-time evaluation enables the necessary initial step of system design and deployment. Run-time evaluation assesses to what extent the architecture options adopted at design-time and other potential options, perform well at run-time. For that, the framework leverages techniques inspired by finance, reinforcement learning, multi-objective optimisation, and time series forecasting. The framework can actively track and proactively forecast the performance of architecture decisions and detect any detrimental changes. It can then inform deployment, refinement, and/or phasing-out decisions. We use an IoT case study to show how continuous evaluation can fundamentally guide the architect and influence the outcome of the decisions. A series of experiments is conducted to demonstrate the applicability and effectiveness of the framework. Results: The design-time evaluation was able to evaluate the architecture options under uncertainty and shortlist candidates for further refinement at run-time. The run-time evaluation has shown to be effective. In particular, it enabled a significant improvement in overall quality (about 40-70% better than reactive and state-of-the-art approaches in some scenarios), with enhanced architecture’s stability. It was also shown to be scalable and robust to various noise levels. In addition, it provides the architect with flexibility to set a monitoring interval to profile the quality of candidates and has parameters that enable the architect to manage the trade-off between architecture stability and learning accuracy. Conclusion: The proposed continuous evaluation framework could potentially aid the architect in evaluating complex design decisions in dynamic environments