An Explorative Study on the Perceived Challenges and Remediating Strategies for Big Data among Data Practitioners

Abstract Background: Worldwide, new data are generated exponentially. The emergence of Internet of Things has resulted in products that were designed first to generate data. Big data are valuable, as they have the potential to create business value. Therefore, many organizations are now heavily investing in big data. Despite the incredible interest, big data analytics involves many challenges that need to be overcome. A taxonomy of these challenges is available that was created from the literature. However, this taxonomy fails to represent the view of data practitioners. Little is known about what practitioners do, what problems they have, and how they view the relationship between analysis and organizational innovation. Objective: The purpose of this study was twofold. First, it investigated what data practitioners consider the main challenges of big data and that may prevent creating organizational innovation. Second, it investigated what strategies these data practitioners recommend to remediate these challenges. Methodology: A survey using semi-structured interviews was performed to investigate what data practitioners view as the challenges of big data and what strategies they recommend to remediate those challenges. The study population was heterogeneous and consisted of 10 participants that were selected using purposive sampling. The interviews were conducted between February 27, 2020 and March 24, 2020. Thematic analysis was used to analyze the transcripts. Results: Ninety per cent of the data practitioners experienced working with low quality, unstructured, and incomplete data as a very time-consuming process. Various challenges related to the organizational aspects of analyzing data emerged, such as a lack of experienced human resources, insufficient knowledge of management about the process and value of big data, a lack of understanding about the role of data scientists, and issues related to communication and collaboration between employees and departments. Seventy per cent of the participants experienced insufficient time to learn new technologies and techniques. In addition, twenty per cent of practitioners experienced challenges related to accessing data, but those challenges were primarily reported by consultants. Twenty per cent argued that organizations do not use a proper data-driven approach. However, none of the practitioners experienced difficulties with data policies because this was already been taken care of by the legal department. Nevertheless, uncertainties still exist about what data can and cannot be used for analysis. The findings are only partially consistent with the taxonomy. More specifically, the reported challenges of data policies, industry structure, and access to data differ significantly. Furthermore, the challenge of data quality was not addressed in the taxonomy, but it was perceived as a major challenge to practitioners. Conclusion: The data practitioners only partially agreed with the taxonomy of challenges. The dimensions of access to data, data policies, and industry structure were not considered a challenge to creating organizational innovation. Instead, practitioners emphasized that the 3 dimension of organizational change and talent, and to a lesser extend also the dimension of technology and techniques, involve significant challenges that can severely impact the creation of organizational innovation using big data. In addition, novel and significant challenges such as data quality were identified. Furthermore, for each dimension, the practitioners recommended relevant strategies that may help others to mitigate the challenges of big data analytics and to use big data to create business value