Should We Trust Automated Static Analysis Tools? A Comparison on the Agreement of Three Tools

Software developers use Automated Static Analysis Tools (ASATs) to detect issues in their code in the early code development stages. By investigating the code quality, these tools highlight the technical debt (TD) of the code. Currently, developers lack in-depth information regarding the current capabilities of ASATs which makes it harder to determine which tools they should use for their needs. In addition, there is a lack of information regarding what current ASATs identify and how the different ASATs agree among one another with respect to TD detection. This study aims to bring in-depth knowledge concerning ASATs with respect to what TD items they identify as well compare and contrast how the tools differ in TD detection. This would bring more knowledge regarding the TD detection capabilities of current ASATs and in turn allow practitioners to be able to choose more precisely appropriate ASATs for their needs as well as encourage researchers to investigate other ASATs. A portion of this knowledge gap concerning current ASATs and TD detection will be filled by investigating three popular ASATs: SonarQube, Coverity Scan, and Better Code Hub. A total of 53 Java projects from the open-source Java Qualitas Corpus set were analyzed and their results were processed to derive the scope of detectable TD items each of the tools identify as well as examine the agreement among the tools concerning common TD items. The results of the study show that ASATs vary with regards to the type of TD items they identify and hardly agree among one another with respect to TD identification. Further research is needed to examine the accuracy and precision of the tools as well as extending the study towards the TD identification capabilities of other ASATs