Best-worst scaling: An alternative to ratings data

The purpose of this chapter is to compare best-worst scaling with rating scales in an organizational research application. Category rating scales are commonly used in organizational research, as they are easy for respondents to answer, allow one to ask questions about more items and have reasonable statistical properties (Gupta and Govindarajan, 1984). However, respondents can use different parts of ratings scales (for example, the middle point or end points) or display “response styles” that affect the items’ means and variances (Craig and Douglas, 2000; Baumgartner and Steenkamp, 2001; Diamantopoulos, Reynolds and Simintiras, 2006; De Jong et al., 2008). Consequently, ratings scales may be poor measures of the relationships of interest. Indeed, social desirability bias, acquiescence bias and extreme response bias can affect results obtained from rating scales (Paulhus, 1991). While social desirability bias has often been ignored in organizational research, it is recognized as a potential source of bias that needs to be considered (Banerjee, 2002; Luo, Rindfleisch and Tse, 2007; Tan and Peng, 2003). Similarly, acquiescence bias may be an issue if people rate most items positively (Paulhus, 1991), as this "heightens the correlations among items that are worded similarly, even when they are not conceptually related" (Winkler, Kanouse and Ware, 1982). This has led some researchers to suggest the use of positive and negative items as a way to reduce acquiescence bias (for example, Baumgartner and Steenkamp, 2001; Saxe and Weitz, 1982; Srinivasan, Rangaswamy and Lilien, 2005). However, others have argued that this approach can confound construct unidimensionality (Herche and Engelland, 1996), especially in cross-cultural research (Wong, Rindfleisch and Burroughs, 2003). Consequently, an approach that reduces acquiescence bias without introducing other measurement problems is needed. In particular, perceptions should be measured in ways that do not affect associations between constructs of interest (for example, job satisfaction and performance). Response biases can distort rating scale values, so they do not represent true scores; hence, it seems desirable to measure these perceptions in ways that avoids such distortion. BWS (Finn and Louviere, 1992; Lee, Soutar and Louviere, 2008; Louviere, Swait and Anderson, 1995) can overcome such problems; specifically, researchers select a list of relevant aspects (such as possible organization culture issues), assign them to sets using a suitable experimental design and ask a target sample of people to choose the best (or most appropriate) and the worst (or least appropriate) aspect in each set