Fixing the foundations : towards generalization for machine learning

An intrusion detection system (IDS) is a security monitoring system capable of detecting potential attacks exceeding the firewall. Traditionally, these systems rely on pattern matching and signatures of known attacks, but in the last decades, security researchers have shifted their interest to machine learning (ML) techniques with varying degrees of success. While the published literature reports outstanding results, we are still waiting for the breakthrough of operational ML-IDS in the real world. To address this issue, we investigate the source of the problem, the datasets themselves, and propose guidelines to work with existing benchmark datasets to improve the generalization power of the ML models. This study evaluates the generalization strength of a ML model using the novel inter-dataset evaluation strategy compared to the traditional intra-dataset strategy. The results show high classification performance on the training dataset but fail to transfer this to a second related but unseen dataset with an average drop in accuracy of 25\%. The proposed inter-dataset evaluation strategy empowers future research to estimate the generalization strength of ML solutions closer to a real-world setting.An intrusion detection system (IDS) is a security monitoring system capable of detecting potential attacks exceeding the firewall. Traditionally, these systems rely on pattern matching and signatures of known attacks, but in the last decades, security researchers have shifted their interest to machine learning (ML) techniques with varying degrees of success. While the published literature reports outstanding results, we are still waiting for the breakthrough of operational ML-IDS in the real world. To address this issue, we investigate the source of the problem, the datasets themselves, and propose guidelines to work with existing benchmark datasets to improve the generalization power of the ML models. This study evaluates the generalization strength of a ML model using the novel inter-dataset evaluation strategy compared to the traditional intra-dataset strategy. The results show high classification performance on the training dataset but fail to transfer this to a second related but unseen dataset with an average drop in accuracy of 25\%. The proposed inter-dataset evaluation strategy empowers future research to estimate the generalization strength of ML solutions closer to a real-world setting.C