A Survey of Duplicate And Near Duplicate Techniques

ABSTRACT---- World Wide Web consists of more than 50 billion pages online. The advent of the World Wide Web caused a dramatic increase in the usage of the Internet. The World Wide Web is a broadcast medium where a wide range of information can be obtained at a low cost. A great deal of the Web is replicate or near- replicate content. Documents may be served in different formats: HTML, PDF, and Text for different audiences. Documents may get mirrored to avoid delays or to provide fault tolerance. The problem of finding relevant documents has become much more prominent due to the presence of duplicate data on the WWW. This redundancy in results increases the users ’ seek time to find the desired information within the search results, while in general most users just want to cull through tens of result pages to find new/different results. This survey paper has a fundamental intention to present an review of the existing literature in duplicate and near duplicate detection of general documents and web documents in web crawling