A framework for collecting provenance in data-centric scientific workflows

As wireless networking becomes more robust and sensor technology, cameras, and low-power compute devices reach the consumer market as inexpensive commodity products, the proliferation of relatively inexpensive specialized sensors and networks will have an impact on scientific computing that parallels that of the commercial space. Specifically, the increasing ability to sense the world around us will result in a growing need for scientific data-driven applications that are under the control of data-centric workflows. These workflows are complex, non-deterministic, and event-driven. The focus of our work is on provenance collection for data-centric workflows. The challenge we address is to record uniform and usable provenance metadata that meets the domain needs while minimizing the modification burden on the service authors and the performance overhead on the workflow engine and the services. The framework, which is based on a loosely-coupled publish-subscribe architecture for propagating provenance activities, is argued on the grounds that it reduces programmer overhead and can satisfy the needs of detailed provenance collection with minimal performance overhead (in the range of 1%)