User Modeling via Machine Learning and Rule-Based Reasoning to Understand and Predict Errors in Survey Systems

User modeling is traditionally applied to systems were users have a large degree of control over their goals, the content they view, and the manner in which they navigate through the system. These systems aim to both recommend useful goals to users and to assist them in achieving perceived goals. Systems such as online or telephone surveys are different in that users have only a singular goal of survey completion, extremely limited control over navigation, and content is restricted to prescribed set of survey tasks; changing the user modeling problem to one in which the best means of assisting users is to identify rare-actions hazardous to their singular goal, by observing their interactions with common contexts. With this goal in mind, predictive mechanisms based on a combination of Machine Learning classifiers and survey domain knowledge encapsulated in sets of rules are developed that utilize user behavioral, demographic, and survey state data in order to predict when user actions leading to irreparable harm to the user\u27s singular goal of successful survey completion will occur. We show that despite a large class imbalance problem associated with detecting these actions and their associated users, we are able to predict such actions at a rate better than random guessing and that the application of domain knowledge via rule-sets improves performance further. We also identify traits of surveys and users that are associated with rare-action incidence. For future work, it is recommended that existence of potential sub-concepts related to users who perform these rare-actions be explored, as well as exploring alternative means of identifying such users, and that system adaptations be developed that can prevent users from performing these rare and harmful actions. Advisor: LeenKiat So