Measuring progress and defining productivity in model-based engineering

Have you ever walked into a model review session or looked into a model and wondered about the level of maturity towards the deliverables? Over the past decade, the adoption of a model-driven approach to engineering has dramatically changed the delivery and coordination methods in capital projects. Many of the firms that are already engaged in model-driven engineering processes have a positive perception of the value they receive for the time, money, and efforts they have expended on their modeling programs. Although the industry is adjusting to a data-rich model-driven approach to engineering, there are no established and standardized processes for measuring progress and productivity of a model-driven engineering process; particularly, when the deliverable type, schedule reporting, and their impacts have changed. The key underlying challenge is how progress and level of effort in a model-driven engineering process can be measured and reported. Any process or metric for measurement, should provide accurate and timely information about both modeling progression and productivity to the project stakeholders. It should also measure factors that provide insight, manage, and control opportunities, and it must account for the unavailability of vendor data, design changes, re-work and design development, model review comments, operability reviews, hazard and operability studies, among other factors. These processes and metrics should be easy-to-use, flexible, and extendable for implementation across various project types and lifecycle phases of today’s capital projects. To create a solution for these challenges and needs, this research aims to address the following essential question: How can we accurately measure progress and productivity towards the deliverables in a model-driven approach to engineering without imposing unnecessary work or taking away from actual productivity? With the previous question as the central catalyst, this research offers a new guideline for measuring progress and defines productivity metrics in a model-driven approach to engineering. As part of this guideline, a series of standardized definitions were created to measure the maturity of various modeling disciplines. These definitions can be used to analyze the maturity of the design components and the quality of the information used in the modeling process. The definitions -categorized into a set of discrete Model Maturity Index (MMI) levels ranging from 100 to 600 levels- provide owners and engineering firms with a clear set of modeling requirements that must be fulfilled during each engineering phase. To successfully implement the MMI definitions and measure progress across both green and brownfield projects, this research also offers a Model Maturity Risk Index (Model MRI) toolkit, together with an addendum to existing model execution plans (ModelXP). Based on the MMI definitions and by accounting for the inter-disciplinary relationships between modeling disciplines, the Model MRI toolkit easily and quickly determines the MMI levels for each model discipline, and by Work Breakdown Structure (WBS) location. The toolkit also offers a detailed analysis of the risk associated with the remaining modeling work needed to achieve a certain MMI level, within the same discipline and across other related modeling disciplines. The Model MRI toolkit can be used, externally with clients, as part of any model review session to communicate modeling progress. Or, internally within the engineering team, to assess the actual modeling progress against the client's expectations. The ModelXP Addendum was created to bring transparency to the modeling process by outlining the modeling process and assuring all project stakeholders are aware of their responsibilities for achieving certain maturity levels for each modeling discipline, per model review session or other project milestones. By connecting the modeling expectations to the MMI levels, the project team can compare their actual progress against the expected progress. The expected progress per project specific project milestone should be outlined in the ModelXP Addendum. The guideline provided, empowers engineering firms to systematically track productivity in the form of engineering hours between MMI levels and per modeling discipline. Companies with previous productivity data can use the Model MRI toolkit to evaluate their team’s schedule conformance, measure risk at each stage of the project, and devise control strategies to keep their projects on schedule and within budget. This can be accomplished by benchmarking project specific progress and productivity data to historic performance productivity data. An analysis with such granularity helps to track the modeling effort needed to achieve each MMI level per model discipline; as well as, providing enough data to benchmark previous engineering productivity rates in future projects. To gain better insight on current best practices for measuring progress and productivity in model-driven and traditional engineering processes, a thorough literature review and an industry survey were conducted on the state-of-the-art research and practice. The existing definitions for Level of Development (LoD) and model breakdown structure defined by AIA, ASTM, and other groups were also reviewed and synthesized. The MMI definitions, the Model MRI toolkit, and the model execution (ModelXP) addendum were all validated internally through a beta test conducted within the research team. Externally, they were validated via three separate day-long charrettes. The charrettes participants were comprised of industry experts ranging from engineering modeling to project controls experts. In these charrettes, real-world project models were used to simulate nine different working scenarios to validate and improve the adoptability and adaptability of the developed MMI definitions, the Model MRI toolkit, and the ModelXP addendum. The charrettes validated three objectives: The MMI definitions are capable of outlining model-based requirements per engineering phase and discipline. The Model MRI Toolkit can be used to measure progress in model-based engineering. The model execution plan, established around MMI levels, provides project teams a clear set of engineering milestones with quantitative and qualitative requirements that must be completed to claim engineering progress. The MMI definitions, the Model MRI toolkit, and the Model Execution Plan Addendum establish procedures and define metrics by which project stakeholders can reliably measure progress and productivity in a model-driven engineering process. While the resources provided in this research are adoptable and adaptable for different types of projects and applications across the process industrial sector, they can be extended for use by other construction industries such as commercial or industrial buildings. The framework created as part of this research has great potential to automate the progress measurement in the modeling environment. This great potential for automation is demonstrated via a real-world project example using AVEVA modeling solutions and is also being explored as part of ongoing research