The VLDB Journal DOI 10.1007/s00778-011-0219-9 SPECIAL ISSUE PAPER The SHARC framework for data quality in Web archiving

Abstract Web archives preserve the history of born-digital content and offer great potential for sociologists, busi-ness analysts, and legal experts on intellectual property and compliance issues. Data quality is crucial for these pur-poses. Ideally, crawlers should gather coherent captures of entire Web sites, but the politeness etiquette and complete-ness requirement mandate very slow, long-duration crawling while Web sites undergo changes. This paper presents the SHARC framework for assessing the data quality in Web archives and for tuning capturing strategies toward better quality with given resources. We define data quality mea-sures, characterize their properties, and develop a suite of quality-conscious scheduling strategies for archive crawl-ing. Our framework includes single-visit and visit–revisit crawls. Single-visit crawls download every page of a site exactly once in an order that aims to minimize the “blur ” in capturing the site. Visit–revisit strategies revisit pages after their initial downloads to check for intermediate changes. The revisiting order aims to maximize the “coherence ” of the site capture(number pages that did not change during the capture). The quality notions of blur and coherence are for-malized in the paper. Blur is a stochastic notion that reflects the expected number of page changes that a time-travel access to a site capture would accidentally see, instead of the ideal view of a instantaneously captured, “sharp ” site. Coherenc