A NOVEL APPROACH FOR WEB CRAWLER TO CLASSIFY THE WEB DOCUMENTS

Web Crawler is a program used to download documents from the internet. It visits many sites to collect information that can be analyzed and mined in a central location. Focused crawler is designed in such a way that it gathers document on a specific topic. To index a document URL, the Focused crawler should ensure that the document which is under the review is belongs to the specific topic. To identify the relevancy of the particular web page content to coincide with the context specific topic and to avoid the replication of the information, the authors of this paper suggests the application of Cosine Similarity measures after removing the stop words from the web page contents. To implement the above mentioned strategy, we have followed this procedure which will effectively identify the web pages containing relevant web contents of specific topic. First we have to create a context specific dictionary consisting of terms related to the focused topic. Then we consider the two web page document namely A and B. We then remove all the stop words from document A and document B. Then we count the number of words available in the each document. Next, we constructed a matrix for each web page to calculate the frequency of each word appearing in the web document. Then we calculate the Cosine Similarity measure between the two matrices constructed out of the two web documents. This approach not only governs the Frequency of a particular word appearing in the web document but also look at the word belongs to the context specific dictionary which we created at the initial stage. Thus we conclude that this paper will provide an efficient mechanism for the Focused crawler to index a web page which is more relevant to the topic