Requirements Engineering in the Context of Big Data Software Applications

Big Data applications, like traditional applications, serve end-user needs except that underlying the software system is Big Data which the system operates upon to improve or provide different end-user experience with the application. In comparison to traditional software development where the development processes are usually well-established, the development of Big Data applications is - to our knowledge - not explored to any significant degree. With Big Data, characterised by the well-known V attributes, questions arise as to how to elicit, specify, analyse, and document system requirements. While requirements engineering (RE) has long been recognised as critical for downstream development of computer systems, the field is currently passive about how to deal with characteristics of data in the RE process in the development of Big Data software applications. This problem is compounded by the fact that the RE field had no domain model (until recently) for Big Data systems depicting the various artefacts, activities, and relationships amongst them that, in turn, can be used to support RE specifications, product design, project decisions, and maintenance. In this thesis research, we investigated empirically a number of issues in RE involving Big Data applications, leading to the following research contributions: (i) knowledge concerning (a) the state of RE research involving Big Data applications, and (b) RE practices on real-world Big Data applications projects; (ii) a set of RE challenges in creating Big Data applications; (iii) a meta-model depicting the various RE artefacts and their inter-relationships in the context of Big Data software development projects; (iv) a goal-oriented approach (composed of a systematic process, requirements logging templates, checklists, and a requirements language) for modelling quality requirements for Big Data applications; and (v) a prototype tool that implements the proposed Big Data goal-oriented requirements language. These results lay a foundation in RE research involving Big Data applications development with anticipated impact in real-world projects and in RE research