Innovations and student success in online learning: A systematic review of how innovations affect student retention

This study examined disruptive and incremental innovations in online higher education to determine their relative impact on student success—defined as course completion and retention—focusing on two factors associated with success: faculty interaction and student engagement. The premise for this examination is that different types of innovations may have different effects on student success. To complete this investigation, an evidence-based research synthesis was conducted. The process included (a) a systematic selection of studies that examined innovations and student success for online learners, (b) a quality assessment of the selected studies, (c) thematic coding of the selected studies, and (d) thematic analysis to understand the relationships amongst variables. In addition, a panel of stakeholders was interviewed to review the study and provide feedback on the design and the value of this study. Results indicate that incremental innovations showed positive results towards student success in 74% of the cases. There were too few studies on disruptive innovations (18%) to draw conclusions, and these studies showed mixed results at best. Disruptive innovations may require a greater investment of time and resources in order to be successful. Many institutions used a comprehensive approach to student success saw an increase in course success or retention. It was determined that incremental innovations have a positive effect on student success. Further, innovations that utilized multiple strategies and targeted more than one factor associated with success are more likely to be successful.Running Head: INNOVATION AND SUCCESS IN ONLINE EDUCATION Innovations and Student Success in Online Learning: A Systematic Review of How Innovations Affect Student Retention Denise Nadasen University of Maryland University College Dissertation submitted to the Faculty of the Graduate School of the University of Maryland University College in partial fulfillment of the requirements for the degree of Doctor of Management Dissertation Advisors: Leslie Dinauer, Ph.D. Sharon Hadary, Ph.D. University of Maryland University College December 2016 INNOVATION AND SUCCESS IN ONLINE EDUCATION 2 © 2016 Denise Nadasen ALL RIGHTS RESERVED INNOVATION AND SUCCESS IN ONLINE EDUCATION 3 Abstract This study examined disruptive and incremental innovations in online higher education to determine their relative impact on student success—defined as course completion and retention—focusing on two factors associated with success: faculty interaction and student engagement. The premise for this examination is that different types of innovations may have different effects on student success. To complete this investigation, an evidence-based research synthesis was conducted. The process included (a) a systematic selection of studies that examined innovations and student success for online learners, (b) a quality assessment of the selected studies, (c) thematic coding of the selected studies, and (d) thematic analysis to understand the relationships amongst variables. In addition, a panel of stakeholders was interviewed to review the study and provide feedback on the design and the value of this study. Results indicate that incremental innovations showed positive results towards student success in 74% of the cases. There were too few studies on disruptive innovations (18%) to draw conclusions, and these studies showed mixed results at best. Disruptive innovations may require a greater investment of time and resources in order to be successful. Many institutions used a comprehensive approach to student success saw an increase in course success or retention. It was determined that incremental innovations have a positive effect on student success. Further, innovations that utilized multiple strategies and targeted more than one factor associated with success are more likely to be successful. Keywords: higher education, online learning, innovation, retention, course success INNOVATION AND SUCCESS IN ONLINE EDUCATION 4 Acknowledgements First, I wish to thank my family for their patience and support: my husband, Tom Reynolds, who has read numerous drafts and took on additional responsibilities while I retreated to my dissertation cave; my two sons, Clay Nadasen-Reynolds and Jeremy Nadasen-Reynolds, who have been incredibly patient and understanding; my father, Dr. Aruna Nadasen, who encouraged me to finish; and my sister, Dr. Premilla Nadasen, whose support was both operational and emotional. Second, I owe a great deal of gratitude to an amazing dissertation chair, Dr. Leslie Dinauer. Without her support, this dissertation would not have been completed. The confidence she had in me carried me through difficult times. She motivated and challenged me to do more than I thought I could. I would also like to thank the doctor of management faculty members at the University of Maryland University College who have advised, guided, and encouraged me over the years: Dr. Sharon Hadary, Dr. Tom Mierzwa, Dr. Mike Evanchek, Dr. Kay Edwards, Dr. John Sherlock, and Dr. James Gillette. I am incredibly fortunate to have several colleagues who have provided tremendous support through the dissertation process: Dr. Karen Vignare, Dr. Marcia Watson, Dr. Peter Shea, Dr. Karen Swan, Mr. William Pelz, and fellow doctoral students with whom I enjoyed a community of support and academic discourse. A special thank you goes to Dr. Karen Vignare who has been a friend, a mentor, and an advisor. INNOVATION AND SUCCESS IN ONLINE EDUCATION 5 Table of Contents Abstract ............................................................................................................................... 3 Acknowledgements ............................................................................................................. 4 Table of Tables ................................................................................................................... 9 Table of Figures ................................................................................................................ 10 Chapter 1: Introduction ..................................................................................................... 11 Background ................................................................................................................... 13 Problem Statement ........................................................................................................ 15 Innovation ..................................................................................................................... 17 Definitions .................................................................................................................... 20 Purpose .......................................................................................................................... 22 Significance .................................................................................................................. 23 Summary ....................................................................................................................... 24 Chapter 2: Literature Review ............................................................................................ 25 The Adult Learner ......................................................................................................... 26 Success in Online Learning .......................................................................................... 27 Theoretical Framework ................................................................................................. 29 Factors Influencing Success in Online Learning .......................................................... 36 Innovation in Higher Education .................................................................................... 38 Incremental innovation. ............................................................................................ 39 Disruptive innovation. ............................................................................................... 40 Conceptual Framework ................................................................................................. 45 Summary ....................................................................................................................... 48 INNOVATION AND SUCCESS IN ONLINE EDUCATION 6 Chapter 3: Methodology ................................................................................................... 50 Research Questions and Variables ................................................................................ 52 Keyword Development ................................................................................................. 54 Search and Selection Process ........................................................................................ 55 Inclusion/Exclusion Criteria ......................................................................................... 56 Quality Assessment Process ......................................................................................... 57 Publications. .............................................................................................................. 58 Citations. ................................................................................................................... 58 Research questions. ................................................................................................... 58 Variables. .................................................................................................................. 60 Assumptions. ............................................................................................................. 60 Sample size. .............................................................................................................. 60 Internal validity. ........................................................................................................ 61 External validity. ....................................................................................................... 61 Research design. ....................................................................................................... 61 Results and conclusions. ........................................................................................... 62 Fit for purpose. .......................................................................................................... 62 Expert Panel Process ..................................................................................................... 62 Analysis and Synthesis Process .................................................................................... 64 Summary ....................................................................................................................... 67 Chapter 4: Results ............................................................................................................. 69 Quality Assessment Results .......................................................................................... 69 Stakeholder Review Results ......................................................................................... 71 INNOVATION AND SUCCESS IN ONLINE EDUCATION 7 Results of Thematic Analysis ....................................................................................... 79 Coding. ...................................................................................................................... 79 Analytic memos. ....................................................................................................... 85 Themes .......................................................................................................................... 86 Institutional innovations. ........................................................................................... 87 Faculty interaction. ................................................................................................... 90 Student engagement. ................................................................................................. 91 Course design. ........................................................................................................... 92 Student support. ........................................................................................................ 93 Disruptive and incremental innovations. .................................................................. 94 Concept Diagram .......................................................................................................... 97 Answering the Research Questions ............................................................................ 102 Summary ..................................................................................................................... 104 Chapter 5: Discussion, Conclusions, and Implications ................................................... 106 Innovations .................................................................................................................. 108 Incremental Innovations. ......................................................................................... 108 Disruptive Innovations. ........................................................................................... 108 Factors ......................................................................................................................... 109 Themes ........................................................................................................................ 111 Conclusions ................................................................................................................. 113 Recommendations ....................................................................................................... 114 Implications to Management ...................................................................................... 115 Limitations .................................................................................................................. 116 INNOVATION AND SUCCESS IN ONLINE EDUCATION 8 Areas of Future Research ............................................................................................ 116 References ....................................................................................................................... 118 Appendix A: Keyword Search Criteria ........................................................................... 137 Appendix B: Databases Used in Systematic Search ....................................................... 138 Appendix C: List of Selected Articles ............................................................................ 139 Appendix D: Stakeholders .............................................................................................. 142 Appendix E: Summary of Stakeholder Interviews ......................................................... 143 Appendix F: Codes Used for Thematic Analysis ........................................................... 144 INNOVATION AND SUCCESS IN ONLINE EDUCATION 9 Table of Tables Table 1. Quality Assessment Tool (QAT) ....................................................................... 59 Table 2. Raw Assessment Scores by Author and Criterion ............................................. 70 Table 3. Quality Assessment Total Weighted Scores by Author..................................... 71 Table 4. Frequency of Unique Codes by Category .......................................................... 80 Table 5. Disruptive and Incremental Strategies by Outcome .......................................... 94 INNOVATION AND SUCCESS IN ONLINE EDUCATION 10 Table of Figures Figure 1. Tinto’s student integration model (SIM). .......................................................... 30 Figure 2. Bean & Metzner’s student attrition model (SAM). .......................................... 33 Figure 3. Rovai’s composite persistence model .............................................................. 34 Figure 4. Conceptual framework: Factors of online learning student success. ............... 46 Figure 5. Evidence-based research synthesis process. ..................................................... 52 Figure 6. Search and selection for evidence-based research synthesis. ........................... 57 Figure 7. Emergent conceptual diagram. ......................................................................... 97 INNOVATION AND SUCCESS IN ONLINE EDUCATION 11 Chapter 1: Introduction Higher education has changed dramatically with the advent of online education and a market that has expanded to online learners. Technologically savvy young people and working adult students are drawn to this new way of learning. It is particularly appealing because it allows students to learn anywhere, anytime. As a result, online education has become an important part of student education. Institutions of higher education are embracing online learning by incorporating online courses into their academic programs. Success rates for students using online education, however, are lower than success rate for students in traditional, face-to-face education (Frydenberg, 2007; Lee, Choi, & Kim, 2013; Rovai & Downey, 2010). Dropout rates for students enrolled in online programs in the United States have been reported between 25% and 45% (Willging & Johnson, 2009). As a result, leaders in higher education are seeking to find out what factors lead to attrition in order to develop innovative solutions to improve persistence (Lee & Choi, 2011). Innovations are strategies that change the standard business practice to improve organizational goals. In higher education, institutional goals are broad but typically include improving student success. Student success can be defined as course completion, student retention, or graduation. Within online learning, innovations that address student success may include strategies such as adaptive learning, online learning communities, and even virtual faculty or students. Innovations are not always as effective as expected (Tinto, 2006). Institutions employ innovations for a variety of reasons (Whitworth, 2012). Some innovations are INNOVATION AND SUCCESS IN ONLINE EDUCATION 12 expected to increase enrollments and improve efficiencies (Anderson, Maier, & Shepard, 2010), while other innovations may face challenges related to an institution’s social and political culture (Whitworth, 2012). In particular, incremental innovations may be more prevalent at large universities where the political and social context favors innovations that adapt to the conventional practices (Whitworth, 2012). The complex environment in higher education, particularly online learning, makes evaluating innovations related to student success challenging. For example, Heyman (2010) examined student retention in online programs and found that student support, instructor-related interactions, and course design were important factors contributing to student success. However, other practices and policies such as admissions, financial aid, and advising may confound the effects of these factors on student success. Despite the complexity of evaluating innovations, institutions employ numerous innovations to address student success. For example, the massive open online course (MOOC) is perceived as an innovation that could disrupt the standard classroom experience for students. While MOOCs have become increasingly popular, they have shown poor course completion rates (de Frietas, Morgan, & Gibson, 2015). Innovations related to online student orientation, on the other hand, have shown positive results with regard to online student success (Hall, 2011). Hall (2011) examined a community college’s voluntary orientation and found that it led to social and academic integration, which led to increased course completion for online students. Given these mixed results, the need to examine the effectiveness of innovations in higher education is more prevalent for online education (Casanovas, 2010). INNOVATION AND SUCCESS IN ONLINE EDUCATION 13 While there is an abundance of literature documenting the differences between online and face-to-face learning, there is little research on how different types of innovations influence student success in online learning. There are innovations that are incremental, focusing on small improvements and sustaining the business model; or innovations that are disruptive, more radical in nature, creating a substantial change to the education business model. Incremental innovations are popular in higher education as a means to sustain and slightly improve existing practices. Disruptive innovations, by nature, are less appealing to institutions where change is difficult. Using a systematic review and an evidence-based research synthesis (EBRS), this dissertation explores both incremental and disruptive innovations to determine their influence on student success in online learning. This introductory chapter provides the background, the problem statement, the research questions, the theoretical framework, and definitions of key terms related to student success. Chapter 2 provides a literature review, followed by Chapter 3, which describes the methodology. Chapter 4 presents the results and analysis and Chapter 5 discusses conclusions and implications. Background An educated population stimulates the economy, reduces poverty, and experiences lower crime rates (Haskins, 2012; McKiernan, 2012; Nolan, 1999). In 2008, the United States reported that 39% of its population had received a college degree while Canada reported the highest rate of college graduates at 55% of its population (National Center for P