A Comparative Study of Industrial Static Analysis Tools (extended version)

Almost all software contains defects. Some defects are found easily while others are never found, typically because they emerge seldom or not at all. Some defects that emerge relatively often even go unnoticed simply because they are not perceived as errors or are not suffciently severe. Software defects may give rise to several types of errors, ranging from logical/functional ones (the program sometimes computes incorrect values) to runtime errors (the program typically crashes), or resource leaks (performance of the program degrades possibly until the program freezes or crashes). Programs may also contain subtle security vulnerabilities that can be exploited by malicious attackers to gain control over computers. Fixing defects that suddenly emerge can be extremely costly in particular if found at the end of the development cycle, or worse: after deployment. Many simple defects in modern programming languages can be found by modern compilers, e.g. in statically typed languages. But the predominating method for finding defects is testing. Testing has the potential of finding most types of defects, however, testing is costly and no amount of testing will find all defects. Testing is also problematic because it can be applied only to executable code, i.e. rather late in the development process. Alternatives to testing, such as data°ow analysis and formal verification, have been known since the 1970s but have not gained widespread acceptance outside academia|that is, until recently; lately several commercial tools for detecting runtime error conditions at compile time have emerged. The tools build on static analysis and can be used to find runtime errors as well as resource leaks and even some security vulnerabilities statically, i.e. without executing the code. This paper is a survey and comparison of three market leading static analysis tools: PolySpace Verifier, Coverity Prevent and Klocwork K7. The list is by no means exhaustive, and the list of competitors is steadily increasing, but the three tools represent state-of-the-art in the field at the moment. The main objective of this study is (1) to identify significant static analysis functionality provided by the tools, but not addressed in a normal compiler, and (2) to survey the underlying supporting technology. The goal is not to provide a ranking of the tools; nor is it to provide a comprehensive survey of all functionality provided by the tools. Providing such a ranking is problematic for at least two reasons: Static analysis is generally only part of the functionality provided by the tool; for instance, Klocwork K7 supports both refactoring and software metrics which are not supported by the two other tools. Even if restricting attention only to static analysis functionality the tools provide largely non-overlapping functionality. Secondly, even when the tools seemingly provide the same functionality (e.g. detection of dereferencing of null pointers) their solutions are often not comparable; each tool typically finds defects which are not found by any of the other tools. Studying the internals of commercial and proprietary tools is not without problems|in particular, it is impossible to get full information about technical solutions. However, some technical information is publicly available in manuals and white papers; some of the tools also originate from academic tools which have been extensively described in research journals and conference proceedings. While technical solutions may have changed since then, we believe that such information is still largely valid. We have also consulted representatives from all three providers with the purpose to validate our descriptions of the tools. Still it must be pointed out that the descriptions of suggested technical solutions is subject to a certain amount of guessing in some respects. The rest of the report is organized as follows: In Section 2 we define what we mean by the term static analysis and survey some elementary concepts and preconditions; in particular, the trade off between precision and analysis time. Section 3 contains a description of basic principles of static analysis. In Sections 4{6 we survey the static analysis functionality provided by PolySpace Verifier/Desktop, Coverity Prevent and Klocwork K7 focusing in particular on the support for the C and C++ programming languages. Section 7 addresses the complementary issue of programming guidelines such as those of The Motor Software Reliability Association (MISRA). Section 8 contains a qualitative comparison of the three tools summing up their relative merits and shortcomings. The section also surveys several industrial evaluations of the tools over time at Ericsson, in particular involving the products from Coverity and Klocwork. Section 9 contains conclusions