Focused Crawlers vs Accelerated Focused Crawlers

Focused crawlers are an efficient method to build a set of Web pages related to a specific topic. In this project, we explored two different methods to navigate the web given a topic: in the first method, we add all the links from a page to the frontier with a score equal to the relevancy of the parent page; in the second method we add each link to the frontier with a score equal to the relevancy of only the local text around the link on the parent page. We used three types of text classifiers—linear SVM, Naive Bayesian, and cosine similarity—to rate relevancy. The user can specify a topic through a set of relevant documents. From them we selected a set of feature words, via document frequency or mutual information. The classifiers only look for the presence of absence of the feature words. Furthermore, we compared performance over a different number of selected features. The evaluation was inconclusive: the SVM performed slightly better than the other two classifiers in the first method of crawling. Also, the results from using the second method for crawling were not significantly better than the result from the first method. Only the performance of cosine similarity and Naive Bayes was affected by the method of feature selection and the number of features. 2 Previous Work Traditional focused crawlers maintain their frontier by adding all the links from a fetched page to the frontie