A Quantitative Study of Software Reviews Using Content Analysis Methods

Online product reviews play a critical role for consumers to make a decision of product purchasing, and become an important data source for vendors to build recommendation systems. Some consumers won’t even buy a product without reading online reviews first, and some vendors invite reviewers to write product reviews before a product is released to the market. However, the review quality could greatly impact the use of the reviews for supporting the purchase decision and recommendation. In order to produce enough high-quality reviews, it is a common practice that vendors provide incentives for writing product reviews. Current research is divided. Some research showed that incentive reviews could be more biased compared to organic reviews, but other scholars showed incentive reviews contain more useful information. Furthermore, other academic publications showed very different results regarding the differences between the incentivized and organic reviews. One of the reasons explaining the observation differences could be due to the quality of the reviewers and reviews. Therefore, it is necessary to control the quality of the reviewers and their reviews for a more objective comparison. In this research, we first discuss an approach for ensuring the quality of the data collection and processing to ensure the quality of the comparison study. Then, we explore the differences between incentivized reviews and organic reviews collected from a website that provides reviews for enterprise software systems. Several parameters of the reviews, such as the overall score, sentiment, and subjectivity of the reviews, were analyzed and compared. Our results could provide a reference for appropriately using reviews and managing the reviewing process