Systematic scalability assessment for feature oriented multi-tenant services

Recent software engineering paradigms such as software product lines, supporting development techniques like feature modeling, and cloud provisioning models such as platform and infrastructure as a service, allow for great flexibility during both software design and deployment, resulting in potentially large cost savings. However, all this flexibility comes with a catch: as the combinatorial complexity of optional design features and deployment variability increases, the difficulty of assessing system qualities such as scalability and quality of service increases too. And if the software itself is not scalable (for instance, because of a specific set of selected features), deploying additional service instances is a futile endeavor. Clearly there is a need to systematically measure the impact of feature selection on scalability, as the potential cost savings can be completely mitigated by the risk of having a system that is unable to meet service demand. In this work, we document our results on systematic load testing for automated quality of service and scalability analysis. The major contribution of our work is tool support and a methodology to analyze the scalability of these distributed, feature oriented multi-tenant software systems in a continuous integration process. We discuss our approach to select features for load testing such that a representative set of feature combinations is used to elicit valuable information on the performance impact and feature interactions. Additionally, we highlight how our methodology and framework for performance and scalability prediction differs from state-of-practice solutions. We take the viewpoint of both the tenant of the service and the service provider, and report on our experiences applying the approach to an industrial use case in the domain of electronic payments. We conclude that the integration of systematic scalability tests in a continuous integration process offers strong advantages to software developers and service providers, such as the ability to quantify the impact of new features in existing service compositions, and the early detection of hidden feature interactions that may negatively affect the overall performance of multi-tenant services.status: publishe