Good Answers from Bad Data: a Data Management Strategy

Data error is an obstacle to effective application, integration, and sharing of data. Error reduction, although desirable, is not always necessary or feasible. Error measurement is a natural alternative. In this paper, we outline an error propagation calculus which models the propagation of an error representation through queries. A closed set of three error types is defined: attribute value inaccuracy (and nulls), object mismembership in a class, and class incompleteness. Error measures are probability distributions over these error types. Given measures of error in query inputs, the calculus both computes and "explains " error in query outputs, so that users and administrators better understand data error. Error propagation is non-trivial as error may be amplified or diminished through query partitions and aggregations. As a theoretical foundation, this work suggests managing error in practice by instituting measurement of persistent tables and extending database output to include a quantitative error term, akin to the confidence interval of a statistical estimate. Two theorems assert the completeness of our error representation