Experimental Evaluation of Tools for Mining Test Execution Logs

Data and software analysis tools are considered a very beneficial and advantageous approach that is used in the software industry environments. They are powerful tools that help to generate testing, web browsing and mail server statistics in different formats. These statistics are also known as logs or log files, and they can be generated in different formats, textually or visually, depending on the tool which tests them. Though these tools have been used in software industry for many years, there is still a lack of fully understanding them by software developers and testers. Literature study shows that related work on test execution log analysis is rather limited. Studies on evaluating a subset of features related to test execution logs are missing from existing literature since even those that exist are usually focused only on a one – feature comparison (e.g., fault – localization algorithms). One of the reasons for this issue might be the lack of experience or training. Some practitioners are also not fully involved with the testing tools that their companies use, so lack of time and involvement might be another reason that there are only a few experts on this field, who can understand these tools very well and find any error in a short time. This makes the need for more research on this topic even more important. In this thesis report, we presented a case study focused on the evaluation of tools which are used for analyzing test execution logs. Our work relied on three different studies: - Literature study - Experimental study - Expert - based survey study So, in order to get familiar with the topic, we started with the literature study. It helped us to investigate the current tools and approaches that exist in the software industry. It was a very important, but also difficult step, since it was hard to find research papers that are relevant to our work. Our topic was very specific, while many of research papers had performed just a general investigation on different tools. That is why in our literature search, in order to get relevant papers, we had to use specific digital libraries, terms and keywords, and a criteria for literature selection. In the next step, we experimented with two specific tools, in order to investigate their capabilities and features which they provide to analyze the execution logs. The tools we managed to work with are Splunk and Loggly. They were the only tools available for us which would comform to our thesis demands, containing the features that we needed for our work to be more complete. The last part of the study was a survey, which we sent to different experts. A total of twenty-six practitioners responded and their answers gave us a lot of useful information to enrich our work. The contributions of this thesis will be: 1. The analysis of the findings and results which are derived from the three conducted studies in order to identify the performance of the tools, the fault localization techniques they use, the test failures that occur during the test runs and conclude which one is better in these terms. 2. The proposals on how to improve further our work on log analysis tools. We explain what is needed in addition in order to understand better these tools and to provide correct results during testing.