Toward a standard benchmark for computer security research: the worldwide intelligence network environment (WINE

Unlike benchmarks that focus on performance or reliability evaluations, a benchmark for computer security must neces-sarily include sensitive code and data. Because these artifacts could damage systems or reveal personally identifiable infor-mation about the users affected by cyber attacks, publicly dis-seminating such a benchmark raises several scientific, ethical and legal challenges. We propose the Worldwide Intelligence Network Environment (WINE), a security-benchmarking ap-proach based on rigorous experimental methods. WINE in-cludes representative field data, collected worldwide from 240,000 sensors, for new empirical studies, and it will enable the validation of research on all the phases in the lifecycle of security threats. We tackle the key challenges for security benchmarking by designing a platform for repeatable experi-mentation on the WINE data sets and by collecting the meta-data required for understanding the results. In this paper, we review the unique characteristics of the WINE data, we dis-cuss why rigorous benchmarking will provide fresh insights on the security arms race and we propose a research agenda for this area. 1