ADAGE: a framework for supporting user-driven ad hoc data analysis processes

Scientists in data-intensive science like to use a variety of software applications, including Grid, Cloud and standalone applications, to support their analysis activities and processes. To effectively support the work of scientists, these applications should satisfy the requirements of interoperability, integration, automation, reproducibility and efficient data handling. To meet all requirements, current application systems adopt software architectural designs based on a horizontal or a vertical design approach. The horizontal approach focuses on a generic system design that can be easily extended through new modules/services; but as new functionalities are added over time, the system's complexity increases and incompatibilities are more likely to arise. The vertical approach focuses on a system design with restricted scope; but the scope itself also limits the system's ability to be extended beyond that scope. Consequently, this thesis proposes the Ad hoc DAta Grid Environment (ADAGE) framework, which satisfies all requirements and overcomes the limitations of the two design approaches.The ADAGE framework consists of an architectural pattern and its associated operational guidelines. The architectural pattern includes a reference data model used to standardise data representation, a set of ADAGE services that analyse and process data, and a service-oriented architecture (SOA) that imposes an overall structure. The operational guidelines offer advice on how to apply the architectural pattern to derive domain-specific software architectures to support the work of scientists. Contributions of the ADAGE framework lie in its flexible architecture, which enables components such as Web portals, workflow systems and business process management systems to be incorporated to satisfy the requirements. Additionally, the reference data model and the ADAGE services are closely coupled to reduce service incompatibilities, improve the composability of services, and allow new analysis processes to be easily supported in the future. Finally, to evaluate the ADAGE framework, a software architecture based on the framework is applied to the financial research domain to support financial time series analysis and event studies using high-frequency stock and news data. Three realistic case studies have also been conducted to show that the performance overheads are reasonable, and that the architecture effectively supports users' data analysis requirements