Crowdsourcing software vulnerability discovery: expertise indicators, organizations perception and quality control

The complexity of software-based systems is increasing dramatically as development becomes even more distributed across multiple heterogeneous, autonomous, and evolving cloud services. More specifically, the increased reliance on third-party software-based systems (e.g., cloud services, open APIs, external programming libraries and black-box software packages) makes it very difficult for in-house IT experts to deal with the inherent risks of using external software. In order to overcome potential vulnerability issues, several organizations outsource tasks such as vulnerability discovery to third-party providers. More recently, the approach of crowdsourcing vulnerability discovery has emerged. In this research, we examine crowdsourcing vulnerability discovery tasks both analytically and empirically through systematic literature review, interviews, surveys and case studies. We identify models and dimensions of vulnerability discovery tasks. We investigate the pre-adoption fears that preventing organizations from using crowdsourcing for vulnerability discovery, issues faced by organizations who have adopted crowdsourcing for vulnerability discovery, and countermeasures used by organizations to mitigate these fears and issues. We investigate one of the key countermeasures we identified (i.e. people selection) and investigate indicators of the expertise of security professionals involved in the crowdsourced vulnerability discovery tasks. Finally, we derive a quality control model (quality dimensions and attributes). We investigate the methods used for quality assessment in crowdsourced vulnerability discovery tasks, and the assurance strategies used to improve the quality of these tasks. We conduct a case study to evaluate the quality control model. Finally, we provide practical guidelines to support practitioners achieve better quality while deploying a crowdsourced vulnerability discovery task