Design and development of a cognitive data analytics engine for network security, implementing Big Data technologies & machine learning techniques

Thesis submitted in partial fulfilment of the requirements for the degree of Master of Science in Systems EngineeringSummarization: The main objectives of this Master’s thesis may be summarized as follows: a. The presentation of Big Data’s impact on modern applications, with particular emphasis on cybersecurity analytics. The review of the challenging aspects as an aftermath of the world’s transformation towards a data-driven culture and the identification of the arisen opportunities in the field of network security. b. The study of machine learning utilisation in cybersecurity for the extraction of hidden knowledge in accumulated network data and the comparison between cognitive anomaly detection systems and traditional signature-based systems. The categorisation of machine-learning methods to supervised and unsupervised and the presentation of the mathematical background regarding the most common algorithms of each family, along with several cybersecurity implementations. c. The architectural design, development and implementation of a state-of-the-art data analytics engine, in the framework of the SHIELD EU-funded cybersecurity project. The presentation of all the relevant components, placing more focus on the description of the data acquisition and data analytics modules which constitute the core of the platform. d. The deployment, configuration, usage and testing of the Apache Spot platform as an integrated analytics ecosystem for the accomplishment of anomaly detection, using public and captured network traffic datasets. The drawing of conclusions that identify the strong and weak points of the engine and can be generalised for the majority of cognitive analytics systems