Advancing Assessment of Quantitative and Scientific Reasoning

Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other four-year institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions’ learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes