Qualitative Data Differences between a Focus Group and Online Forum Hosting a Usability Design Review: A Case Study

Traditional usability assessment test methods often do not accommodate conventional resource constraints which typically result in testing renouncement; thus, new methods are needed. This case study reviews a new test method, the internet forum, and compares data outcomes related to participant participation between this forum and the traditional focus group. The same design review was hosted in all groups: two forum groups (n=28 and 7) and two focus group sessions (n=14 and 10). Data was coded to determine what percentage of the users responded to each posed question, how many user conversations occurred and how many design suggestions were provided. Thus, more participants responded to questions in the forum rather than the focus group but more forum conversations occurred comparatively. However, these conversations did not inspire more designs suggestions comparatively. Also, forum participants most often directly answered the questions and provided more design suggestions directly related to the questions posed while focus group participants offered more off-topic responses that provided important rich contextual information (e.g., explanation behind user requirements). Research implications, threats to validity and future work are also discussed