The Effect of Sample Size on Cognitive Interview Findings

An often-mentioned strength of cognitive interview pretesting is its ability to identify most question problems using a few interviews. However, this is not based on empirical research. We report a study investigating how the number of cognitive interviews affects the number of problems identified by conducting a 90 interviews, drawing samples of size 5 through size 50 from the pool of 90 interviews, and comparing the number and impact of problems identified at each samples size. It is clear that small numbers of cognitive interviews, typical of most pretests, fail to detect many problems including some that are quite serious. Even in samples of size 50, some problems are not uncovered. We conclude that conducting more cognitive interviews than are typically carried out is probably a good investment