A methodology for the evaluation of training effectiveness during early phase defense acquisition

Today's economic environment requires for a greater emphasis to be placed on the development of cost-effective solutions to meet military capability based requirements. The Joint Capabilities Integration and Development System (JCIDS) process is designed to identify materiel and non-materiel solutions to fill defense department capability requirements and gaps. Non-materiel solutions include: Doctrine, Organization, Training, Materiel, Leadership and Education, Personnel, Facilities, and Policy (DOTMLPF-P) changes. JCIDS specifies that all non-materiel solutions be analyzed and recommendations be made accordingly following a capability-based assessment (CBA). Guidance for performing CBA analysis provides minimal information on how to predict training effectiveness and as a result training investments are not properly assessed and considered as a viable alternative. Investigations into the ability to predict versus evaluate training performance and to quantify uncertainty in training system design are two identified gaps in the capability of existing training evaluation methods. To address these issues, a Methodology to Predict and Evaluate the Effectiveness of Training (MPEET) has been developed. To address the gap in predictive capability MPEET uses primary elements of learning theory and instructional design to predict the cost-effectiveness of a training program, and recommends training alternatives based on decision-maker preferences for each of the cost and effectiveness criteria. The use of educational and instructional theory involves developing and ensuring human performance requirements will be met after training. Utility theory is used to derive an overall criterion consisting of both cost and effectiveness attributes. MPEET uses this criterion as a key variable in determining how to properly allocate resources to gain maximum training effectiveness. To address the gap in quantifying uncertainty in training performance, probability theory is used within a modeling and simulation environment to create and evaluate previously deterministic variables. Effectiveness and cost variables are assigned probability distributions that reflect the applicable range of uncertainty. MPEET is a systems engineering based decision-making tool. It enhances the instructional design process, which is rooted in the fields of education and psychology, by adding an objective verification step to determine how well instructional strategies are used in the design of a training program to meet the required learning objectives. A C-130J pilot case study is used to demonstrate the application of MPEET and to show the plausibility of the approach. For the case study, metrics are derived to quantify the requirement for knowledge, skills, and attitudes in the C-130J pilot training system design. Instructional strategies were defined specifically for the C-130J training program. Feasible training alternatives were generated and evaluated for cost and effectiveness. Using information collected from decision-maker preferences for cost and effectiveness variables, a new training program is created and comparisons are made to the original. The case study allows tradeoffs to be performed quantitatively between the variable importance weightings and mean value of the probabilistic variables. Overall, it is demonstrated that MPEET provides the capability to assess the cost and effectiveness of training system design and is an enabler to the inclusion of training as an independent non-materiel alternative solution during the CBA process. Although capability gaps in the defense acquisition process motivated the development of MPEET its applicability extends to any training program following the instructional design process where the assumed constraints are not prohibitive.Ph.D