Assessing the Accuracy of Task Time Prediction of an Emerging Human Performance Modeling Software - CogTool

There is a need for a human performance modeling tool which not only has the ability to accurately estimate skilled user task time for any interface design, but can be used by modelers with little or no programming knowledge and at a minimal cost. To fulfill this need, this research investigated the accuracy of task time prediction of a modeling tool – CogTool - on two versions of an interface design used extensively in the petrochemical industry – DeltaV. CogTool uses the KeyStroke Level Model (KLM) to calculate and generate time predictions based on specified operators. The data collected from a previous study (Koffskey, Ikuma, & Harvey, 2013) that investigated how human participants (24 students and 4 operators) performed on these interfaces (in terms of mean speed in seconds) were compared to CogTool’s numeric time estimate. Three tasks (pump I, pump II and cascade system failures) on each interface for both participant groups were tested on both interfaces (improved and poor), on the general hypothesis that CogTool will make task time predictions for each of the modeled tasks, within a certain range of what actual human participants had demonstrated. The 95% confidence interval (CI) tests of the means were used to determine if the predictions fall within the intervals. The estimated task time from CogTool did not fall within the 95% CI in 9 of 12 cases. Of the 3 that were contained in the acceptable interval, two belonged to the experienced operator group for tasks performed on the improved interface, implying that CogTool was better in predicting the operators’ performance than the students’. A control room monitoring task, by its nature, places great demand on an operator’s mental capacity. This also includes the fact that operators work on multiple screens and/or consoles, sometimes requiring them to commit information to memory that they have to revisit a screen to check on some vital information. In this regard, it is suggested that the one user mental operator for “think time” (estimated as 1.2sec), should be revised in CogTool to accommodate the demand on the operator. For this reason, the present CogTool prediction did not meet expectations in estimating control room operator task time, but it however succeeded in showing where the poor interface could be improved by comparing the detailed steps to the improved interface