Theory-driven or process-driven prediction? Epistemological challenges of big data analytics

Abstract Most scientists are accustomed to make predictions based on consolidated and accepted theories pertaining to the domain of prediction. However, nowadays big data analytics (BDA) is able to deliver predictions based on executing a sequence of data processing while seemingly abstaining from being theoretically informed about the subject matter. This paper discusses how to deal with the shift from theory-driven to process-driven prediction through analyzing the BDA steps and identifying the epistemological challenges and various needs of theoretically informing BDA throughout data acquisition, preprocessing, analysis, and interpretation. We suggest a theory-driven guidance for the BDA process including acquisition, pre-processing, analytics and interpretation. That is, we propose—in association with these BDA process steps—a lightweight theory-driven approach in order to safeguard the analytics process from epistemological pitfalls. This study may serve as a guideline for researchers and practitioners to consider while conducting future big data analytics. Graphical abstract Epistemological challenges of big data analytics