Measuring Political Deliberation: A Discourse Quality Index

In this paper, we develop a discourse quality index (DQI) that serves as a quantitative measure of discourse in deliberation. The DQI is rooted in Habermas' discourse ethics and provides an accurate representation of the most important principles underlying deliberation. At the same time, the DQI can be shown to be a reliable measurement instrument due to its focus on observable behavior and its detailed coding instructions. We illustrate the DQI for a parliamentary debate in the British House of Commons. We show that the DQI yields reliable data and we discuss how these data could be used in subsequent analysis. We conclude by discussing some limitations of the DQI and by identifying some areas in which it could prove useful. Introduction Over the past decade, deliberative politics has moved to the forefront of political theory. 1 Deliberation implies that political decision-making is or should be 'talk-centric' rather than 'vote-centric Despite the advances in the political theory of deliberation and despite deliberative theorists' claims about the importance of discourse, empirically 2 This is partially a reflection of the continued predominance of social choice theory and other theories that treat preferences as given and view decision-making as a process of preference aggregation. However, a larger issue may be the dearth of measurement instruments that allow researchers to operationalize and quantify the quality of discourse, and that open up deliberation for empirical research. In this article, we develop a measurement instrument of deliberative quality -the discourse quality index (DQI). This measurement instrument has the advantage that it is theoretically grounded, finding its origins in Habermas (1981, 1990, 1991, 1992, 1995, 1996) as well as other theorists. At the same time, the DQI can be applied easily and reliably to a wide range of deliberative contexts. Thus, the DQI opens up deliberation for empirical research, allowing this research to interface with political theory. We should point out at the outset that this is a measurement paper. We lack the space to investigate actually the impact of discourse quality on political outcomes. Although many theorists believe that deliberative politics leads to better outcomes, there is no consensus on this matter, with some arguing vociferously that an automatic connection between deliberation and just outcomes cannot be presumed We organize this article as follows. First, we discuss the criteria that a measure of the nature of discourse should satisfy. Next, we discuss two past efforts at measuring deliberation. Third, we discuss the theoretical foundation of the DQI. Having laid the groundwork, we then provide a detailed discussion of the DQI. This is followed by an empirical illustration that shows the coding procedures as well as the measurement properties of the DQI. We then discuss how the DQI can be employed in empirical research. We conclude by discussing some limitations of the DQI and opportunities for its use. Measurement Criteria A measure of the nature of discourse can serve as a bridge between political theory and empirical scholarship only if it does justice to the former and provides guidance to the latter. We believe, therefore, that such a measure should meet four different criteria: (1) it should be theoretically grounded, (2) it should tap into observable phenomena, (3) it should be general, and (4) it should be reliable. The first criterion is essential because it concerns the validity of the discourse measure. An improperly grounded measure lacks construct validity, would be of little practical use and would fail to convince theorists. The complication here is the lack of agreement among political theorists about what constitute deliberation and discourse. One could attempt to develop a measure that captures all of the different conceptions of these concepts, but this would result in an instrument that is too complex to be of practical value and one that lacks internal consistency (since the different theories are not necessarily compatible). Our approach is different; we have selected a particular theory of deliberation, one that is most closely associated with The second to fourth criteria speak to the empirical power of a discourse measure. Most importantly, the measurement instrument should tap into observable discourse behavior. This is essential if the measurement instrument is to produce reliable data and if it is to convince empirical scholars. In addition, an ideal measure of discourse quality should be general, so that it can be transported from one research domain to the next. Finally, such a measure should be reliable. This requires not only that it is based on observable behavior, but also that its coding instructions are specific, and that its coding categories are sufficiently clear that different coders could agree on the classification of the same discourse. Of course, reliable measurement is never guaranteed, so that empirical reliability assessments should be a standard practice in discourse analysis. Below, we shall argue that the DQI meets these criteria. However, before outlining the logic of this measure, we should review past efforts at measuring discourse. As we shall see, these efforts leave considerable room for improvement