Modeling end user performance perspective for cloud computing systems using data center logs from big data technology

Information system performance measurement has been a concern for software engineers since the early days of the field’s development. Over time, numerous techniques and methodologies have been developed that help engineers and companies better understand, manage and improve the performance that the end users perceive when using information systems in their daily operations. Some performance measurement techniques employ surveys that investigate which aspects satisfy or do not satisfy end user requirements. Other performance measurement techniques simulate the same operations across different services in order to compare performance given a similar workload. Yet another approach that has been experimented slightly modifies the data exchanged between clients and servers in order to include components that help with tracing the performance of different operation statuses. When we consider surveys or questionnaires as a performance measurement technique, they do not include detailed information about the sources of problems that may be impacted by the time of day, the responder’s mood and many other human factors, thus masking the root cause and they are not sufficiently interactive to allow for a timely reaction when there is a performance problem. Simulation is also proposed as a potential solution where the same operations, over different platforms, correctly report fundamental characteristics of performance. This approach however, removes the user’s perspective. It is difficult to assume that a simulation would be able to, with the current state of technology, accurately reflect the complexity of a user’s reasoning and decisions regarding the use of a specific information system in a particular way. Finally, the manipulation of the transactional data between client and hosts could affect the confidentially and refutability of the data used to determine the performance; if an information system includes the possibility of data being modified, even slightly, the end user could lose trust in it, negatively affecting the human-machine relation. The question that is considered here is how can the end user performance perspective of cloud computing-based applications be modeled in a way so that timely analysis can be enacted upon the information? The best possible solution for understanding performance from the end user’s perspective could emerge from combining the completeness of interactive surveys with the controlled environment of simulations and the traceability of packet manipulation, while minimizing the weaknesses of each of these techniques. As companies continue to rollout cloud computing infrastructures and systems, the difficulty with performance measurement increases due to a number of factors, most noticeably, the increased complexity of these systems in comparison with their previous versions as well as the unreliability of the performance experience as perceived by the end user, which is influenced by socio-technical aspects such as technical knowledge, trust, system performance, availability and efficacy (Armbrust, Fox, & Griffith, 2009) (Gruschka & Jensen, 2010) (Grobauer, Walloschek, & Stocker, 2011). In order to be able to address these particular challenges, one possible solution could be to make better use of the ubiquitous industry standard performance logs. Performance logs are textual representations of different resource consumption and activities performed in the various operational cloud system components. Logs have been extensively deployed in the industry and used for both troubleshooting and punctual investigations of performance problems. In this research, logs are explored more extensively in order to address the need for precision, granularity and responsiveness within the decision time required for the current management/prediction challenges. The amount and granularity of the data harvested could potentially be massive. Each of the analyzed hosts or network components can generate as much as 800 KB of data per minute. This could quickly turn into a very large amount of data that is difficult to process and access using traditional SQL-based technologies. One of the possible alternatives for resolving this issue is employing Big Data technologies such as the Hadoop Distributed File System and Apache Spark in order to interactively collect the data from multiple sources and process the individual files simultaneously, which would prove difficult using classic relational database technology. This research proposes a novel performance measurement model for cloud-based information systems as perceived by end users, with many practical applications in the domain of service level measurement and performance prediction. It identifies meaningful and actionable data center logs of low-level direct and derived measurements to model the end user performance perspective. The cloud computing measurement model and quality characteristics presented by Bautista’s framework (Bautista, Abran, & April, 2012) are implemented and experimented. The model for the end user performance perspective for cloud computing systems using data center logs from Big Data technology expands Bautista’s original work by proposing the utilization of a performance indicator and including end user response in order to forecast possible performance anomalies. A large-scale experimentation is described where the measures are analyzed using a modern Big Data infrastructure in order to model the end user performance perspective as an expression of performance indicators based on the service level agreement for the cloud computing services studied. The experimentation addresses the research question and offers a solution avenue for modelling the end user performance perspective of cloud computing based applications in future service level agreements so that a timely analysis of the data can be expected and a predictive algorithm developed to anticipate upcoming performance issues