Empirical evidence in research on meaning∗

DRAFT – Feedback very welcome! Empirical evidence is at the heart of research on natural language meaning. Surprisingly, however, discussions of what constitutes evidence in research on meaning are almost non-existent. The goal of this paper is to open the discussion by advancing a proposal about the nature of empirical evidence in research on meaning. Our proposal is based primarily on insights we and our colleagues have gained in research on under-studied languages and in quantitative research using offline measures, but we intend the proposal to cover research on natural language meaning more broadly, including research on well-studied languages that the researcher may even control natively. Our proposal has three parts. First, we argue that a complete piece of data in research on meaning consists of a linguistic expression, a context in which the expression is uttered, a response by a native speaker to a task involving the expression in that context, and information about the native speakers that provided the responses. Incomplete pieces of data fail to satisfy our three proposed objectives that data be stable, replicable and transparent. Second, we argue that some response tasks, namely acceptability and implication judgment tasks, are better suited than others (e.g., paraphrase and translation tasks) fo