Are Experts Difficulty Guessing And Statistical Results Comparable

HEZINET, our Learning Management System (LMS), is prepared to use calibrated items according to the Item Response Theory (IRT). Until now, pedagogues, based on their experience, used to judge the difficulty level of each item and the results of the system have been acceptable enough. But, could the results be improved? The IRT provides methods to detect and remove bad items from item bank plus methods to find the difficulty level of each item and other parameters. However, these methods are very expensive, since they require many resources. This paper shows an evaluation plan for LMSs to decide whether it is more adequate to use statistical results or experts guessing. Moreover, it describes the issues to be considered in order to perform such an evaluation and it gives some clues to choose one of them depending on the specific conditions of the concrete LMS. 1 Introduction. Ite