Feature Reduction for Information Retrieval

this document frequency). The basic idea is that potentially meaningful words occur with medium document frequency. Hence, we disregard words that occur in only one document or that occur in more than half of the documents. 7 2. 10 7 4. 10 7 6. 10 7 8. 10 50000 100000 150000 200000 250000 300000 350000 Figure 2: Number of different words vs number of words in a growing document collection (fat line, 4 years Financial Times) and the square root in comparison (thin line). The output of the whole preprocessing is one summary file that contains a table of the relevant words together with an id number and the document frequency, respectively. This file also contains a document surrogate of each document, ie, a collection of the relevant words together with the term frequency of this word in the collection. The retrieval process is based on this summary file. Owing to the lack of a retrieval system at the time of the experiments no inverted document list was available. So, in order to retrieve documents, the whole summary file had to be scanned, resulting in an inferior query time of 10 seconds per query for the TREC-7 task. 2.2 Complexity and Resources for the Document Preprocessin