Trends in Cognitive Science

A serious methodological weakness affecting much research in syntax and semantics within the field of linguistics is that the data presented as evidence are often not quantitative in nature. In particular, the prevalent method in these fields involves evaluating a single sentence/meaning pair, typically an acceptability judgment performed by just the author of the paper, possibly supplemented by an informal poll of colleagues. Although acceptability judgments are a good dependent measure of linguistic complexity (results from acceptability–judgment experiments are highly systematic across speakers and correlate with other dependent measures, but see Ref., using the researcher's own judgment on a single item/pair of items as data sources does not support effective testing of scientific hypotheses for two critical reasons. First, as several researchers have noted, a difference observed between two sentences could be as a result of lexical properties of the materials rather than syntactic or semantic properties. Multiple instances of the relevant construction are needed to evaluate whether an observed effect generalizes across different sets of lexical items. The focus of this letter, however, is on a second problem with standard linguistic methodology: because of cognitive biases on the part of the researcher, the judgments of the researcher and his/her colleagues cannot be trusted (Box 1). As a consequence of these problems, multiple items and multiple naïve experimental participants should be evaluated in testing research questions in syntax/semantics, which therefore require the use of quantitative analysis methods