Autowrapper: automatic wrapper generation for multiple online services

A crucial challenge for information extraction from the WWW is to generate wrappers, which are information extraction patterns or rules, which apply to numerous Web sites with great diversity in both format and content. Generating wrappers manually is tedious, time consuming and errorprone. Recent research has successfully adapted machine learning technology to generate wrappers for semi-structured Web pages. However, these machine learning approaches rely on manually annotated example pages, which create a big overhead. This paper presents a system called AutoWrapper which automatically generates wrappers from HTML source pages based on textual similarity and heuristics. This paper details its two key components, the domain-independent HTML similarity comparison algorithm and the wrapper induction algorithm. Our experiment on 20 semistructured Web sites indexed by an independent search engine achieves a 90% success rate. AutoWrapper can generate one wrapper for each Web site