Unveiling Uncertainty: Tools, Concepts, and Philosophy for Defining what is Measured in Educational and Psychological Measurement (and Avoiding a Bewitching)

The quality of a measuring instrument is always to some extent limited by the conceptual clarity and coherence of the target property’s (that which we would like to measure) definition. However, foundational texts about validity theories, psychometrics, and latent variable modeling offer little guidance for transparently defining or analyzing these properties (sometimes called, constructs, attributes, factors, latent variables, or similar). I begin by suggesting, contrary to many methodological approaches, that non-operationally defining properties is the first necessary step in any measurement effort and an ethical imperative in the human sciences. Productive definitional efforts are those that enable what Hasok Chang calls epistemic iteration. This means that definitional efforts serve certain initial scientific aims but should be ever improved and critiqued. I try to present useful recommendations for productive definitional work from the fields of psychometrics, philosophies of language and science, as well as metrology and exploratory statistics. That we know enough about a property to define it and treat it as measurable is presented as an empirical and ethical question. Throughout, I use running examples from definitional debates surrounding the assessment of reading comprehension ability in the National Assessment of Educational Progress (NAEP) and scholarship related to “non-academic” student attributes such as human resilience. In the final section of this dissertation, I present an empirical example from a reading test, repurposing differential item functioning (DIF) and measurement invariance testing from psychometrics as useful inductive tools for demarcating properties we would like to measure. Ideally, these methods for property definition can help facilitate conceptual and linguistic clarity for measurement in education, psychology, and the human sciences more broadly