A deep‐learning‐ and reinforcement‐learning‐based system for encrypted network malicious traffic detection

Abstract Traditional network intrusion detection methods lack the ability of automatic feature extraction for encrypted network malicious traffic, and thus, the detection rates are low. Moreover, the means of this malicious traffic are concealed, and the key malicious features are usually hidden in many normal data packets, so fewer encrypted malicious traffic samples can be captured. This easily leads to insufficient system training, low detection rate, and high false alarm rate. This letter proposes an encrypted network malicious traffic detection model based on deep learning, in which automatic feature extraction is performed against encrypted network malicious traffic. The proposed model has self‐learning and self‐adaption abilities. Furthermore, a sample generation method of encrypted traffic based on deep Q‐networks and deep convolution generative adversarial networks is proposed, in which new samples are learned from the training samples of encrypted traffic and solves problems, such as insufficient original training samples and unbalanced samples. In a validation experiment, the proposed model could distinguish between normal and abnormal encrypted network traffic, and the accuracy rate reached 99.94%. Experimental results show that the proposed model in this letter can provide a new and better solution for an encrypted network malicious traffic detection system