The Validity of the Stimulated Retrospective Think-Aloud Method as Measured by Eye Tracking

Retrospective Think aloud (RTA) is a usability method that collects the verbalization of a user’s performance after the performance is over. There has been little work done to investigate the validity and reliability of RTA. This paper reports on an experiment investigating these issues using the method called stimulated RTA. By comparing subjects’ verbalizations with their eye movements, we found stimulated RTA to be valid and reliable: the method provides a valid account of what people attended to in completing tasks, it has a low risk of introducing fabrications, and its validity is unaffected by task complexity. More detailed analysis of RTA shows that it also provides additional information about user’s inferences and strategies in completing tasks. The findings of this study provide valuable support for usability practitioners to use RTA and to trust the users ’ performance information collected by this method in a usability study. Author Keywords Retrospective think aloud, validity, reliability