Trust Me, I’m Confident – Are Confident Members of the Crowd Better at Evaluating Business Model Ideas?

Crowdsourced idea evaluation is often seen as a viable, less expensive alternative to traditional idea selection mechanisms such as expert evaluation. However, previous research found several limitations on the reliability of crowdsourced decision making, especially when simple majority-based aggregation mechanisms are used. In response, researchers suggest new, advanced aggregation mechanisms that rely on additional information (e.g. the confidence of a judge) to overcome these limitations. Therefore, this study investigates whether advanced aggregation methods based on the judge’s confidence can be used to increase the reliability of crowdsourced idea evaluation. To answer this question, we post an idea evaluation task to a crowdworking platform, aggregate the judgements with different mechanisms and compare the predictions from each mechanism with results from expert evaluation. Our findings suggest that confidence-based mechanisms reduce the risk of discarding high quality ideas, at the cost of a slight increase in the misclassification rate on low quality ideas