Evasive Focused Crawling by Exploiting Human Browsing Behaviour: a Study on Terrorism-Related Content

Terrorists have introduced several new challenges for Law Enforcement Agencies (LEAs), including their extensive use of the Web for communication and diffusion of their knowledge, with particular emphasis on the Dark Web due to the anonymity it provides. Thus, it is necessary for LEAs to be able to discover and collect this terrorism-related content both on the Surface and the Dark Web. An important challenge is the fact that servers hosting such content may identify and block crawlers that attempt to access it. This work proposes a novel crawling framework for the discovery and collection of content relevant to a domain of interest on both the Surface and the Dark Web (in particular its Tor, the I2P and the Freenet darknets) that adopts a hu- manlike browsing behaviour so as evade detection of its bot nature. We evaluated the crawling framework in the context of accessing terrorism-related content. The evaluation experiments indicate the effectiveness of the proposed approach regarding the collection of terrorism-related content and also its efficacy in mimicking human browsing behaviour regarding the number of hyperlinks that are followed and the time interval between requests