Log Files for Proactive Monitoring of Big Data

This thesis reports on a qualitative study of Big Data and Analytics, with an emphasis on tools that can be used to monitor systems within organizations. An overview of the literature available has been given as to what Big Data means, as well as how this data could be used to provide information to an organization. Big Data is stockpiled into log files by the various systems that have been implemented within an organization. In order to interrogate and find information, as well as deal with issues that may arise, an organization requires evidence to be gathered from log data. This could be used to assist with the decision making process within organizations. Three tools, namely N-Able, AWS CloudWatch and Sumo Logic were implemented in order to gain an understanding of how they could be used to provide information from the data contained in these log files. During the analysis of these tools the focus was on what data these tools provided, and if and how these tools could provide an in-depth analysis by utilizing Big Data and Big Data Analytics. The use of the tools, allows for an organization to monitor and alert on their infrastructure and other environments. In all cases the tools reviewed were able to provide this as a basic foundation. Sumo Logic stood out as the most productive tool, as it had a similar basic foundation as the other two tools, but in addition had the Big Data Analytical capability inbuilt. From the review of tools and literature during this research it came to the fore that there is a requirement for Big Data Analytical Tools. In order for information to be collected and assimilated into something useful – Big Data Analytical tools provide the means and methods to assist in providing information that is both useful and timeous