Crawling Ajax-based Web Applications through Dynamic Analysis of User Interface State Changes

Using JavaScript and dynamic DOM manipulation on the client-side of web applications is becoming a widespread approach for achieving rich interactivity and responsiveness in modern web applications. At the same time, such techniques, collectively known as Ajax, shatter the metaphor of web ‘pages’ with unique URLs, on which traditional web crawlers are based. This paper describes a novel technique for crawling Ajax-based applications through automatic dynamic analysis of user interface state changes in web browsers. Our algorithm scans the DOM-tree, spots candidate elements that are capable of changing the state, fires events on those candidate elements, and incrementally infers a state machine modelling the various navigational paths and states within an Ajax application. This inferred model can be used, for instance, in program comprehension, analysis and testing of dynamic web states, or for generating a static version of the application. In this paper, we discuss our sequential and concurrent Ajax crawling algorithms. We present our open source tool called Crawljax, which implements the concepts and algorithms discussed in this paper. Additionally, we report a number of empirical studies in which we apply our approach to a number of open-source and industrial web applications and elaborate on the obtained results. This paper is a pre-print of: Ali Mesbah, Arie van Deursen, and Stefan Lenselink, Crawling Ajax-basedWeb Applications through Dynamic Analysis of User Interface State Changes. ACM Transactions on the Web, 2012.Software TechnologyElectrical Engineering, Mathematics and Computer Scienc