Formative evaluation of a tool for managing software quality

Context/Background: To achieve high software quality, particularly in the context of agile software development, organizations need tools to continuously analyze software quality. Several quality management (QM) tools have been developed in recent years. However, there is a lack of evidence regarding the quality of QM tools, standardized definitions of such quality, and reliable instruments for measuring it. This, in turn, impedes proper selection and improvement of QM tools. Goals: We aimed at operationalizing the quality of a research QM tool, namely the ProDebt prototype, and evaluating its quality. The goal of the ProDebt prototype is to provide practitioners with support for managing software quality and technical debt. Method: We performed interviews, workshops, and a mapping study to operationalize the quality of the ProDebt prototype and to identify reliable instruments to measure it. We designed a mixed-method study aimed at formative evaluation, i.e., at assessing the quality of the ProDebt prototype and providing guidance for its further development. Eleven practitioners from two German companies evaluated the ProDebt prototype. Results: The participants assessed the information provided by the ProDebt prototype as understandable and relevant. They considered the ProDebt prototype's functionalities as easy to use but of limited usability. They identified improvement needs, e.g., that the analysis results should be linked to other information sources. Conclusions: The evaluation design was of practical value for evaluating the ProDebt prototype considering the limited resources such as the practitioners' time. The evaluation results provided the developers of the ProDebt prototype with guidance for its further development. We conclude that it can be used and tailored for replication or evaluation of other QM tools