LEARNING OBJECT EVALUATION: COMPUTER-MEDIATED COLLABORATION AND INTER-RATER RELIABILITY

Learning objects offer increased ability to share learning resources so that system-wide production costs can be reduced. But how can users select from a set of similar learning objects in a repository and be assured of quality? This article reviews recent developments in the establishment of learning object repositories and metadata standards, and presents a formative reliability analysis of an online, collaborative method for evaluating quality of learning objects. The method uses a 10-item Learning Object Review Instrument (LORI) within a Convergent Participation evaluation model that brings together instructional designers, media developers, and instructors. The inter-rater reliability analysis of 12 raters evaluating eight learning objects identified specific items in LORI that require fur-ther development. Overall, the collaborative process substantially increased the reliability and validity of aggregate learning object rat-ings. The study concludes with specific recommendations including changes to LORI items, a rater training process, and requirements for selecting an evaluation team