Artificial neural networks based techniques for anomaly detection in Apache Spark

Late detection and manual resolutions of performance anomalies in Cloud Computing and Big Data systems may lead to performance violations and financial penalties. Motivated by this issue, we propose an artificial neural network based methodology for anomaly detection especially for the Apache Spark in-memory processing platforms. Apache Spark has become widely adopted by industry because of its speed and generality, however there is still a shortage of comprehensive performance anomaly detection methods applicable to this platform. We propose artificial neural networks driven methodology to quickly sift through Spark logs data and operating system monitoring metrics to accurately detect and classify anomalous behaviors based on the Spark resilient distributed dataset (RDD) characteristics. The proposed method is evaluated against three popular machine learning algorithms, decision trees, nearest neighbor, and support vector machine (SVM), as well as against four variants that consider different monitoring datasets. The results prove that our proposed method outperforms other methods, typically achieving 98%-99% F-scores, and offering much greater accuracy than alternative techniques to detect both the period in which anomalies occurred and their type