Manual Versus Online Card Sorts: A Cautionary Tale for IS Researchers

Card sorting is an important and frequently used survey validation technique for information systems (IS) research. A card sort can be time consuming and thus, with the advent of technology, card sorting has been automated with a number of online card sorting programs available for use by researchers. Since the early 1990s, there have been numerous studies that compared online to manual card sorts with most reporting that manual and online card sorts yield similar results. This study found that observation may impact the efficacy between online and manual card sorts thereby casting doubt into the interchangeability of manual and online card sorts. Specifically we find a significant difference between the results of a manual observed card sort and an online unobserved card sort. With the proliferation of card sorts as a construct validation technique, this paper is a caution for IS researchers to be mindful of the design of their cards sort and to be consistent in the delivery of them