The impact of National Science Foundation investments in undergraduate engineering education research: A comparative, mixed methods study

The U.S. invests billions of taxpayers\u27 dollars in research tied to the national priorities that contribute to its competitiveness in a global economy. As the federal funding agency with an explicit focus on engineering education, the National Science Foundation (NSF) contains a portfolio of projects focused on improving the quantity of engineering graduates and the quality of engineering programs. Within the agency, the Division of Undergraduate Education invests approximately $190 million (FY 2012) annually on science, technology, engineering and mathematics (STEM) education projects. Although the DUE portfolio includes a suite a projects with different foci supporting national initiatives and Principal Investigators (PIs) report their results in annual reports and conferences, there is little consistency on how impact is defined, evaluated, and measured. ^ While many agree on the importance of investing in research, the stiff economic climate necessitates that the research that demonstrates impact is what will continue to be supported. However, the dearth of scholarship on impact contributes to the lack of understanding around this topic. This study links the fragmented literature on impact to form a unified starting point for continuing the conversation. While existing literature includes three dimensions of research impact (i.e., scientific, societal, and domain-specific impact), this study focuses on the domain-specific impacts of engineering education research using two guiding frameworks, namely, Toulmin\u27s Model (1958) and the Common Guidelines for Education Research and Development (Earle et al., 2013), and a multiphase mixed methods research design (Creswell & Plano Clark, 2011).^ The qualitative phase of this study explores how researchers on NSF-funded STEM education R&D projects talk about the impact of their work; the findings reveal eight themes that are commonly discussed when PIs articulate the impact of their research, and two themes related to how they support their claims. The findings also indicate that the STEM discipline associated with the study and the project focus have more to do with the types of impact PIs claim than the amount of funding awarded to the project. As a result of identifying the points of alignment between PIs\u27 perspectives on impact and existing literature, a preliminary description of what impact looks like in this context is proposed—using the three dimensions of research impact as an organizing framework. Although this study puts forth a preliminary description of the impact of STEM education research, extensions of this work are necessary before providing practitioners and policymakers with a valid, comprehensive framework characterizing what impact means in this context.^ Ideas supporting the types of claims PIs make when discussing the impact of their work were used to develop a survey that was distributed to a small sample of current and former NSF Program Officers (POs) in the second phase of this study. The survey results provide preliminary evidence on how PIs and NSF PO\u27 perspectives on research impact compare, and affirm that additional studies are needed. Consequently, implications for policy and practice and potential research directions are also discussed