An Approach To The Correlation Of Security Events Based On Machine Learning Techniques

Organizations face the ever growing challenge of providing security within their IT infrastructures. Static approaches to security, such as perimetral defense, have proven less than effective - and, therefore, more vulnerable - in a new scenario characterized by increasingly complex systems and by the evolution and automation of cyber attacks. Moreover, dynamic detection of attacks through IDSs (Instrusion Detection Systems) presents too many false positives to be effective. This work presents an approach on how to collect and normalize, as well as how to fuse and classify, security alerts. This approach involves collecting alerts from different sources and normalizes them according to standardized structures - IDMEF (Intrusion Detection Message Exchange Format). The normalized alerts are grouped into meta-alerts (fusion, or clustering), which are later classified using machine learning techniques into attacks or false alarms. We validate and report an implementation of this approach against the DARPA Challenge and the Scan of the Month, using three different classifications - SVMs, Bayesian Networks and Decision Trees - having achieved high levels of attack detection with little false positives. Our results also indicate that our approach outperforms other works when it comes to detecting new kinds of attacks, making it more suitable to a world of evolving attacks. © 2013 Stroeh et al.41116Joosen, W., Lagaisse, B., Truyen, E., Handekyn, K., Towards application driven security dashboards in future middleware (2012) J Internet Serv Appl, 3, pp. 107-115. , 10.1007/s13174-011-0047-6Hale, J., Brusil, P., Secur(e/ity) management: A continuing uphill climb (2007) J Netw Syst Manage, 15 (4), pp. 525-553Ganame, A.K., Bourgeois, J., Bidou, R., Spies, F., A global security architecture for intrusion detection on computer networks (2008) Elsevier Comput Secur, 27, pp. 30-47Giacinto, G., Perdisci, R., Roli, F., (2005) Alarm Clustering For Intrusion Detection Systems In Computer Networks, 19, pp. 429-438. , In: Perner P, Imiya A (eds)Ning, P., Cui, Y., Reeves, D.S., Xu, D., Techniques and tools for analyzing intrusion alerts (2004) ACM Trans Inf Syst Secur (TISSEC), 7, pp. 274-318Boyer, S., Dain, O., Cunningham, R., Stellar: A fusion system for scenario construction and security risk assessment (2005) Proceedings of the Third IEEE International Workshop On Information Assurance, pp. 105-116. , IEEE Computer SocietyJulisch, K., Clustering intrusion detection alarms to support root cause analysis (2003) ACM Trans Inf Syst Security, 6, pp. 443-471Liu, P., Zang, W., Yu, M., Incentive-based modeling and inference of attacker intent, objectives, and strategies (2005) ACM Trans Inf Syst Secur (TISSEC), 8, pp. 78-118Sabata, B., Evidence aggregation in hierarchical evidential reasoning (2005) UAI Applications Workshop, Uncertainty In AI 2005, , Edinburgh, ScotlandChyssler, T., Burschka, S., Semling, M., Lingvall, T., Burbeck, K., Alarm reduction and correlation in intrusion detection systems (2004) Detection of Intrusions and Malware & Vulnerability Assessment Workshop (DIMVA), pp. 9-24. , Dortmund, DeutschlandOhta, S., Kurebayashi, R., Kobayashi, K., Minimizing false positives of a decision tree classifier for intrusion detection on the internet (2008) J Netw Syst Manage, 16, pp. 399-419Haines, J.W., Lippmann, R.P., Fried, D.J., Tran, E., Boswell, S., Zissman, M.A., The 1999 darpa off-line intrusion detection evaluation (2000) Comput Netw Int J Comput Telecommunications Netw, 34, pp. 579-595Project, T.H., (2004) Know Your Enemy: Learning About Security Threats, , (2nd Edition). Addison-Wesley ProfessionalSommer, R., Paxson, V., Outside the closed world: On using machine learning for network intrusion detection (2010) Proceedings of the IEEE Symposium On Security and PrivacyBowen, T., Chee, D., Segal, M., Sekar, R., Shanbhag, T., Uppuluri, P., Building survivable systems: An integrated approach based on intrustion detection and damage containment (2000) DARPA Information Survivability Conference (DISCEX)Vigna, G., Eckmann, S.T., Kemmerer, R.A., The stat tool suite (2000) Proceedings of DISCEX 2000, , Hilton Head, IEEE Computer Society PressLee, W., Stolfo, S.J., Chan, P.K., Eskin, E., Fan, W., Miller, M., Hershkop, S., Zhang, J., Real time data mining-based intrusion detection (2001) Proc. Second DARPA Information Survivability Conference and Exposition, pp. 85-100. , Anaheim, USANeumann, P.G., Porras, P.A., Experience with EMERALD to date (2005) Proceedings 1st USENIX Workshop On Intrusion Detection and Network Monitoring, pp. 73-80. , Santa Clara, CA, USAGrimaila, M., Myers, J., Mills, R., Peterson, G., Design and analysis of a dynamically configured log-based distributed security event detection methodology (2011) J Defense Model Simul: Appl Methodolgy Tech, pp. 1-23Rieke, R., Stoynova, Z., Predictive security analysis for eventdriven processes (2010) MMM-ACNS'10 Proceedings of the 5th International Conference On Mathematical Methods, models and architectures for computer network securityValdes, A., Skinner, K., Probabilistic alert correlation (2001) Proceedings of the 4th International Symposium On Recent Advances In Intrusion Detection (RAID 2001), pp. 54-68. , Davis, CA, USAAsif-Iqbal, H., Udzir, N.I., Mahmod, R., Ghani, A.A.A., Filtering events using clustering in heterogeneous security logs (2011) Inf Technol J, 10, pp. 798-806Corona, I., Giacinto, G., Mazzariello, C., Roli, F., Sansone, C., Information fusion for computer security: State of the art and open issues (2011) Inf Fusion, 10, pp. 274-284Burroughs, D.J., Wilson, L.F., Cybenko, G.V., Analysis of distributed intrusion detection systems using bayesian methods (2002) Proceedings of IEEE International Performance Computing and Communication Conference, pp. 329-334. , Phoenix, AZ, USASabata, B., Ornes, C., Multisource evidence fusion for cyber-situation assessment (2006) Proc. SPIE, 6242. , (Apr. 18, 2006). Orlando, FL, USAEndsley, M.R., Toward a theory of situation awareness in dynamic systems (1995) Human Factors: J Human Factor Ergon Soc, 37, pp. 32-64Debar, H., Curry, D., Feinstein, B., The intrusion detection message exchange format (idmef) (2007) Internet Experimental RFC, p. 4765. , http://tools.ietf.org/html/rfc4765, Available atLan, F., Chunlei, W., Guoqing, M., A framework for network security situation awareness based on knowledge discovery (2010) Computer Engineering and Technology (ICCET)Cox, K., Gerg, C., (2004) Managing Security With Snort and IDS Tools, , O'Reilly Media, SebastopolAlfedaghi, S., Mahdi, F., Events classification in log audit (2010) Int J Netw Secur Appl (IJNSA), 2, pp. 58-73Valdes, A., Skinner, K., International, S., Adaptive, model-based monitoring for cyber attack detection (2000) Recent Advances In Intrusion Detection (RAID 2000), pp. 80-92. , Springer-VerlagMahoney, M.V., Chan, P.K., Learning nonstationary models of normal network traffic for detecting novel attacks (2002) Proceedings of the Eighth ACM SIGKDD International Conference On Knowledge Discovery and Data Mining, pp. 376-385. , ACMMukkamala, S., Sung, A.H., Abraham, A., Intrusion detection using ensemble of soft computing (2003) Paradigms, Advances in Soft Computing, pp. 239-248. , Springer VerlagFaraoun, K.M., Boukelif, A., Securing network traffic using genetically evolved transformations (2006) Malays J Comput Sci, 19 (1), pp. 9-28. , (ISSN 0127-9084)Faraoun, K.M., Boukelif, A., Neural networks learning improvement using the k-means clustering algorithm to detect network intrusions (2006) Int J Comput Intell Appl, 6 (1), pp. 77-99Tandon, G., Chan, P., Learning rules from system call arguments and sequences for anomaly detection (2003) ICDM Workshop On Data Mining For Computer Security (DMSEC), pp. 20-29. , Melbourne, FL, USAMukkamala, S., Sung, A.H., Feature ranking and selection for intrusion detection systems using support vector machines (2002) Proceedings of the Second Digital Forensic Research WorkshopChang, C.C., Lin, C.J., (2001) LIBSVM: A Library For Support Vector Machines, , http://www.csie.ntu.edu.tw/cjlin/libsvm, Available atHsu, W.C., Chang, C.C., Lin, J.C., (2007) A Practical Guide to Support Vector Classification, , http://www.csie.ntu.edu.tw/cjlin, tech. rep., Department of Computer Science, National Taiwan University. Available atWitten, I.H., Frank, E., (2000) Data Mining: Practical Machine Learning Tools and Techniques, , (Second Edition), Morgan KaufmannKayacik, H.G., Zincir-Heywood, A.N., (2003) Using Intrusion Detection Systems With a Firewall: Evaluation On Darpa 99 Dataset, , Tech. rep., NIMS Technical Report 06200