Evolving Expert Knowledge Bases: Applications of Crowdsourcing and Serious Gaming to Advance Knowledge Development for Intelligent Tutoring Systems

This dissertation presents a novel effort to develop ITS technologies that adapt by observing student behavior. In particular, we define an evolving expert knowledge base (EEKB) that structures a domain\u27s information as a set of nodes and the relationships that exist between those nodes. The structure of this model is not the particularly novel aspect of this work, but rather the model\u27s evolving behavior. Past efforts have shown that this model, once created, is useful for providing students with expert feedback as they work within our ITS called Rashi. We present an algorithm that observes groups of students as they work within Rashi, and collects student contributions to form an accurate domain level EEKB. We then present experimentation that simulates more than 15,000 data points of real student interaction and analyzes the quality of the EEKB models that are produced. We discover that EEKB models can be constructed accurately, and with significant efficiency compared to human constructed models of the same form. We are able to make this judgment by comparing our automatically constructed models with similar models that were hand crafted by a small team of domain experts. We also explore several tertiary effects. We focus on the impact that gaming and game mechanics have on various aspects of this model acquisition process. We discuss explicit game mechanics that were implemented in the source ITS from which our data was collected. Students who are given our system with game mechanics contribute higher amounts of data, while also performing higher quality work. Additionally, we define a novel type of game called a knowledge-refinement game (KRG), which motivates subject matter experts (SMEs) to contribute to an already constructed EEKB, but for the purpose of refining the model in areas in which confidence is low. Experimental work with the KRG provides strong evidence that: 1) the quality of the original EEKB was indeed strong, as validated by KRG players, and 2) both the quality and breadth of knowledge within the EEKB are increased when players use the KRG