Scalable lineage capture for debugging DISC analysis

A fundamental challenge for big-data analytics is how to efficiently tune and debug multi-step dataflows. This paper presents Newt, a scalable architecture for captur-ing and using record-level data lineage to discover and resolve errors in analytics. Newt’s flexible instrumenta-tion allows system developers to collect this fine-grain lineage from a range of data intensive scalable com-puting (DISC) architectures, actively recording the flow of data through multi-step, user-defined transformations. Newt pairs this API with a scale-out, fault-tolerant lin-eage store and query engine. We find that while active collection can be expensive, real-world analytics often incur modest runtime over-heads (<36%) and it enables novel lineage-based debug-ging techniques. For instance, Newt can efficiently recre-ate errors (crashes or bad outputs) or remove input data from the dataflow to enable data cleaning strategies. Ad-ditionally, Newt’s active lineage collection allows retro-spective analyses of a dataflow’s behavior, such as iden-tifying anomalous stages. As case studies, we instrument two DISC systems, Hadoop and Hyracks, with less than 105 lines of additional code for each. Finally, we use Newt to systematically clean input data to a Hadoop-based de novo genome assembler, significantly improv-ing the quality of the output assembly.