e of Duplicates

Figure: The recursive search strategy uses search result to find more results and then combines them. SimSeer [2] is a similarity search engine that incorporates the recursive similarity search algorithm. It currently supports three similarity functions, though it is possible to add more. They are: •Key phrase similarity •Shingle similarity [3] based on sequences of words in a document. •Simhash similarity [4] based on all of the words in a document. The need to find similar documents arises in many situations: •Plagiarism detection •Near duplicate detection •Research paper recommendation Traditionally, queries are constructed by users and submitted to search engines; however, it may not be obvious to users how they should construct such queries or overly complex to do so. Thus querying with a whole document and allowing a system to automatically construct queries can be useful. This research investigates methods for automatically searching for similar documents using a sample document while dealing with issues such as scalability and multiple notions of similarity. Searching for similar documents with a sample document may not return all similar documents since it’s possible for similar documents to not have any features in common. Swanson’s ABC model [1] provides inspiration for a recursive search algorithm to overcome this shortcoming. The search algorithm based on this model is: 1. Search with initial query document 2. Take the top k results and use them to search for more documents 3. Repeat step 2 a set number of times 4. Combine all results 5. Rank each result either by similarity to the initial query (target ranking) document submitted by the user or by the similarity to the query document that found it (local ranking