Content extraction for information retrieval in blogs and intranets

As the web keeps growing, identifying and retrieving useful in-formation from this huge amount of data continues to be a prob-lem. One difficulty in web retrieval is that web pages not only contain useful content, but also navigation bars, advertisements, disclaimers, and other boilerplate material. Removing this boiler-plate material, i.e. extracting useful content from web documents, has been shown to improve performance of linguistic applications such as classification and clustering. However, it is not clear how content extraction influences retrieval performance. In this paper we systematically evaluate the impact of a sim-ple content extraction method on retrieval performance on two web collections: (i) a blog collection and (ii) an intranet collection. Our results indicate that blog retrieval substantially benefits from even simple content extraction methods, while applying the same con-tent extraction method to intranet documents does not improve re-trieval performance over the baseline. We analyze the results per-topic to identify collection characteristics that could cause these differences