Large scale malware collection: lessons learned

In order to assure accuracy and realism of resilience as-sessment methods and tools, it is essential to have access to field data that are unbiased and representative. Sev-eral initiatives are taking place that offer access to mal-ware samples for research purposes. Papers are published where techniques have been assessed thanks to these sam-ples. Definition of benchmarking datasets is the next step ahead. In this paper, we report on the lessons learned while collecting and analysing malware samples in a large scale collaborative effort. Three different environments are de-scribed and their integration used to highlight the open is-sues that remain with such data collection. Three main lessons are offered to the reader. First, creation of represen-tative malware samples datasets is probably an impossible task. Second, false negative alerts are not what we think they are. Third, false positive alerts exist where we were not used to see them. These three lessons have to be taken into account by those who want to assess the resilience of techniques with respect to malicious faults.