Reconciling the disconnect between information technology and information systems using an organizational epistemology: A framework to improve success with technology

Abstract There is a disconnect between information technology (IT) and information systems (IS) that lies at the foundation of frequent failure in cost, schedule, and/or performance of IT/IS. This disconnect can perhaps be reconciled through a focus on the socially constructed and emergent nature of IT as it enters and is used by an organization. The argument is that IT investment is not working with the right knowledge for evaluation and justification and does not reflect human nature in decision-making. The hypothesis is that a structured dialog technique, by focusing on properly perceiving human felt needs rather than exclusively technological aims, improves the process of IT project evaluation and justification. This breaks the requirements development process into multiple, time-efficient steps in order to properly acquire and convert tacit knowledge into explicit and then embedded knowledge. The research questions include: What are the foundations of IT/IS failure, and how do they relate to the absence or presence of structured dialog? What are the dimensions of structured dialog that help fulfill the gap between users’ felt needs and IT specifications? Finally, what are the aspects of tools and methods applied in early investment lifecycle activities that can, or seem to, fulfill these dimensions? This study posits a conceptual decision framework using structured dialog to perhaps prevent IT failure. The research starts with an analysis of IT/IS failure factors using a metaevaluation approach tailored for qualitative, interpretive analysis as a metainterpretation with respect to IT case studies and a synthesis of those factors. Then, a pilot of a survey approach is performed to profile and test elements of structured dialog and associated early lifecycle tools with the Dissertation - POWELL ii business/mission process community and technologists. This is then made practical by creating a conceptual decision framework for management to use in framing complex investment decisions such as IT investment. The framework includes obtaining early clarification of the purpose of the new IT/IS system, improving team dynamics to enable better collaboration, making a specific, shared decision as the basis of collaboration, and focusing on structured dialog. The elements of structured dialog found to underlie this decision framework are the production and use of shared meaning, creation and use of common language, repetition and continuation of dialog, and facilitation of shared experience. These elements achieve an organizational epistemology, or knowledge framework, that can potentially facilitate more accurate acquisition and development of IT (that is, the system-as-created), and perhaps lay the foundation for subsequent transition into IS (that is, the system-as-used) that an organization should use in the manner needed and intended. In addition, this epistemology underlies the process, and products, of successful IT/IS architecture.Dissertation POWELL i Title: Reconciling the Disconnect between Information Technology and Information Systems using an Organizational Epistemology: A Framework to Improve Success with Technology Author: Chris R. Powell Abstract There is a disconnect between information technology (IT) and information systems (IS) that lies at the foundation of frequent failure in cost, schedule, and/or performance of IT/IS. This disconnect can perhaps be reconciled through a focus on the socially constructed and emergent nature of IT as it enters and is used by an organization. The argument is that IT investment is not working with the right knowledge for evaluation and justification and does not reflect human nature in decision-making. The hypothesis is that a structured dialog technique, by focusing on properly perceiving human felt needs rather than exclusively technological aims, improves the process of IT project evaluation and justification. This breaks the requirements development process into multiple, time-efficient steps in order to properly acquire and convert tacit knowledge into explicit and then embedded knowledge. The research questions include: What are the foundations of IT/IS failure, and how do they relate to the absence or presence of structured dialog? What are the dimensions of structured dialog that help fulfill the gap between users’ felt needs and IT specifications? Finally, what are the aspects of tools and methods applied in early investment lifecycle activities that can, or seem to, fulfill these dimensions? This study posits a conceptual decision framework using structured dialog to perhaps prevent IT failure. The research starts with an analysis of IT/IS failure factors using a metaevaluation approach tailored for qualitative, interpretive analysis as a metainterpretation with respect to IT case studies and a synthesis of those factors. Then, a pilot of a survey approach is performed to profile and test elements of structured dialog and associated early lifecycle tools with the Dissertation - POWELL ii business/mission process community and technologists. This is then made practical by creating a conceptual decision framework for management to use in framing complex investment decisions such as IT investment. The framework includes obtaining early clarification of the purpose of the new IT/IS system, improving team dynamics to enable better collaboration, making a specific, shared decision as the basis of collaboration, and focusing on structured dialog. The elements of structured dialog found to underlie this decision framework are the production and use of shared meaning, creation and use of common language, repetition and continuation of dialog, and facilitation of shared experience. These elements achieve an organizational epistemology, or knowledge framework, that can potentially facilitate more accurate acquisition and development of IT (that is, the system-as-created), and perhaps lay the foundation for subsequent transition into IS (that is, the system-as-used) that an organization should use in the manner needed and intended. In addition, this epistemology underlies the process, and products, of successful IT/IS architecture. Dissertation - POWELL iii Reconciling the Disconnect between Information Technology and Information Systems using an Organizational Epistemology: A Framework to Improve Success with Technology Doctor of Management Dissertation Christopher R. Powell UMUC November 15, 2009 Committee Dr. Hasan Sayani, Chair Dr. Husein Abdul-Hamid, Member Dr. Claudine SchWeber, Member Dissertation - POWELL iv Dissertation - POWELL v Dedication and Acknowledgements This is dedicated to my wife, Bonnie Jean, and my daughter, Emilie Alyson Grace, who provided ongoing support and encouragement throughout my pursuit of a Doctorate while working full time. Without their love and strength, this effort would not have been possible. I acknowledge the following people, who provided key support throughout this endeavor: Tim Keenan – President & CEO, High Performance Technologies, Inc. Kristen Baldwin – Deputy Director, OUSD(AT&L) DDRE/SE/System Analysis Dissertation POWELL vi Dissertation - POWELL vii Table of Contents Abstract ........................................................................................................................................... i! Dedication and Acknowledgements ............................................................................................... v! List of Tables & Figures ................................................................................................................ ix! List of Abbreviations ..................................................................................................................... xi! CHAPTER 1: BACKGROUND, PURPOSE, RESEARCH QUESTIONS, SIGNIFICANCE ...... 1! Introduction and Relevance ..................................................................................................... 1! Purpose and Research Questions ............................................................................................ 6! Purpose .................................................................................................................................. 6! Research Questions ............................................................................................................. 10! Theoretical Background ......................................................................................................... 11! Thesis, Focus, and Argument ............................................................................................. 17! Significance .......................................................................................................................... 22! Limitations and Constraints .............................................................................................. 23! CHAPTER 2: LITERATURE REVIEW ...................................................................................... 29! Concept Map ........................................................................................................................... 29! Literature Review and Analysis ............................................................................................ 32! IT Research – A Review ......................................................................................................... 63! Knowledge Framework – A Foundation ............................................................................... 67! Conclusion ............................................................................................................................... 71! CHAPTER 3: RESEARCH APPROACH AND METHOD ........................................................ 74! Research Approach ................................................................................................................. 76! Phase I: Case Study Metaevaluation ................................................................................. 78! Phase II: Reviewing Customer Experiences Using a Survey .......................................... 84! Scope .................................................................................................................................. 95! Validity ................................................................................................................................ 96! Data sources ....................................................................................................................... 101! Detailed Research Plan ..................................................................................................... 102! Conclusion ............................................................................................................................. 115! CHAPTER 4: RESEARCH DATA AND DISCUSSION .......................................................... 118! Phase I – Data from the Case Study Metaevaluation ........................................................ 119! Phase I Conclusion ................................................................................................................ 139! Phase II – Data from the Survey Pilots ............................................................................... 140! Step 1: Survey Development and Pre-Pilot (Eight BPA Professionals) ........................... 140! Step 2: Survey Pilot (24 BPA Users) ................................................................................... 140! Phase II Conclusion .............................................................................................................. 145! CHAPTER 5: ANALYSIS, FUTURE APPLICATIONS, AND EXTENSIONS ...................... 148! Analysis ................................................................................................................................. 148! Research Activity Summary ............................................................................................ 154! Summary of Phase I - Metaevaluation ............................................................................ 155! Summary of Phase II – Survey ........................................................................................ 157! Definition of Decision Framework .................................................................................. 160! Future Applications .............................................................................................................. 172! Potential Future Research ................................................................................................ 178! Extensions .............................................................................................................................. 179! Dissertation - POWELL viii Conclusion ............................................................................................................................. 183! Glossary ..................................................................................................................................... 188! References .................................................................................................................................. 190! Appendix A. Institutional Review Board Form .......................................................................... 212! Appendix B. Survey Instrument ................................................................................................ 218! Appendix C. Case Study Metaevaluation Data – Research Question 1 .................................... 240! Appendix D. Linkage of Case Study Metaevaluation to the Survey Instrument and Research Questions .................................................................................................................................... 256! Appendix E. Survey Data – Research Question 2 ..................................................................... 258! Appendix F. Survey Data – Research Question 3 ...................................................................... 282! Index .......................................................................................................................................... 292! Dissertation - POWELL ix List of Tables & Figures Figure 1 - Concept Map ................................................................................................................ 30! Figure 2 - Summary of appraisal techniques, approaches, and references (Irani, et al., 1997) .... 42! Figure 3 - The matrix of benefits and their impact (Farbey et al., 1993) ...................................... 47! Figure 4 - Synthesizing Requirements Development ................................................................... 61! Figure 5 - Process of Metainterpretation (Weed, 2005) ............................................................... 80! Table 1 - IT Failure Factors from Jones (1996) .......................................................................... 120! Table 2 - Root Causes for Failure Factors from Jones (1996) .................................................... 124! Table 3 - IT Failure and Risk Factors from Ewusi-Mensah (2003, p.43) ................................... 132! Figure 6 - Crosstab of Survey Questions .................................................................................... 142! Figure 7 - Hierarchy of IT Failure Factors .................................................................................. 175! Figure 8 - Concept Map of IT Success Factors ........................................................................... 181! Dissertation - POWELL x Dissertation - POWELL xi List of Abbreviations AHP Analytic Hierarchy Process CBA Cost-Benefit Analysis CEO Chief Executive Officer CLIN Contract Line Item CST Critical Social Theory DCF Discounted Cash Flow DoD U.S. Department of Defense DoDAF Department of Defense Architecture Framework FAP Financial Appraisal Profile FMS Flexible Manufacturing System(s) GAO General Accountability Office GB Gigabyte (1 billion bytes) HPC High Performance Computing ICT Information and Communication Technologies ID Identification IEC International Electrotechnical Commission IEEE Institute of Electrical and Electronics Engineers IPT Integrated Program Team IRB Institutional Review Board IRR Internal Rate of Return ISO not an acronym, but commonly used for International Organization for Standardization IS Information System IT Information Technology Km Kilometer (1000 meters) KM Knowledge Management MB Megabyte (1 million bytes) MIS Management Information System(s) NPV Net Present Value OE Organizational Epistemology OUSD Office of the Undersecretary of Defense PADR Prospector-Analyzer-Defender-Reactor PB Petabyte (1 quadrillion bytes) PC Personal Computer PMO Program Management Office PPBS Programming-Planning-Budgeting System R&D Research & Development ROI Return on Investment SOS System of Systems TB Terabyte (1 trillion bytes) U.S. United States Dissertation POWELL 1 CHAPTER 1: BACKGROUND, PURPOSE, RESEARCH QUESTIONS, SIGNIFICANCE Introduction and Relevance Evaluation and justification of IT are both complex and difficult (Serafeimidis, 2001, p. 99; Wang, 2007; Mondragon et al., 2001), particularly given the high rate of failure in IT projects (upwards of 40-95%, depending on methodology and definition of failure [Dalcher & Drevin, 2003; Charette, 2005; Reel, 1999; Verner et al., 2007; Whiting, 1998; Ewusi-Mensah, 2003; Flowers, 1996; Jones, 1996]). The financial measures typically used by firms to evaluate any project (such as ROI [Return on Investment], NPV [Net Present Value], IRR [Internal Rate of Return, etc.]), when applied to technology projects, attempt to be quantitative, precise, and certain. However, as an innovation project applied to a social group (organization), an IT project has benefits that are frequently intangible and uncertain (Lefley, 1996). This clash between the need for discrete performance measurement and the intangibility of innovation in social groups caused a long period of research (from about 1976 through 2001; see Ch. 2) that attempted to bring these financial and intangible poles together. Many techniques were brought to the table, but few could be empirically tested because of many factors including political, cultural, and accounting tradition. The need for insightful evaluation has, if anything, become greater as IT becomes more complex and touches more parts of an organization’s value proposition, foundation, and partners, particularly leading to the “death of geography”; as the enterprise model becomes extended, strategic alliances become more important, and the organization begins its shift to more of a “virtual” model (Irani et al., 1997). So, there is a need for a new look and new tools to be applied to this area, to arrive at something practical for management to apply and for technologists to support. A key part of this problem is defined as a lack of balance Dissertation - POWELL 2 between episteme (knowledge or science, of the organization and of IT) and techne (craft or art, of implementing IT and other business/mission systems) in the organization—based on a lack of shared meaning and a lack of a basis for that shared meaning—which reinforces the existence of only tangential touch points between these frequently opposing forces and which concomitantly and continually reduces IT project success. An enterprise-wide approach that enables IT to take a strategic view of the organization is required (Irani et al., 1997, p. 697). We know from management literature (particularly Simon, 1997, and related, such as Horkheimer, 1954; Farbey, Land, & Targett, 1992; and Dörner, 1996) that we humans, making microeconomic and many other decisions, are subject to our own bounded rationality. As a result, we satisfice decisions, using information we believe is salient to the decision, almost regardless of whether that information is the most microeconomically optimal. This is particularly applicable in executive decision-making and in decision-making related to IT, which Farbey et al. (1992) found is subject to four main heuristics: act of faith, obeying orders, step-by-step, and got-to-do all more deeply explained in Chapter 2). This research will show that many IT evaluation and justification methods attempt to get around human satisficing through many objective and subjective decision methods, including financial (objective) and heuristic (subjective) methods, among others. In executive teams, financial metrics have particular weight, and so we may expect to find that the heuristic methods at that level may favor financial metrics. In fact, Farbey et al. (1992) found that while executives seem to favor such metrics, that does not mean that they are truly making their decisions based on the objective financial analysis at hand. This leaves us with a question about whether management information used in satisficing decisions can be “set” to some degree, submerging less relevant information and maximizing the Dissertation - POWELL 3 prominence of information with the most utility to the decision at hand. It could be said that many aids to management decision-making attempt to do just that, as they “load the deck” with report after report that contain much interesting and relevant information. Yet, we know that many decisions fail to attain their objectives, particularly in IT, where project failure can occur as high as 40-95% (Dalcher & Drevin, 2003), depending on the definition of failure. The types of dysfunctional executive behavior that drive this failure include acting without prior analysis of the situation and failing to anticipate side effects and long-term repercussions, among others (Dörner, 1996). So, the present “deck-loading” does