Examination of the 'web mode effect'

Declining response rates is one of the most significant challenges for survey based research today. Seen in isolation, traditional interviewer based data collection methods are still the most effective but also the most expensive, especially the greater difficulty in gaining responses taken into account. As a solution, mixed-mode designs have been employed as a way to achieve higher response rates, while keeping the overall costs low. In particular, the use of web based surveys has expanded considerably during the last few years, both as a single data collection method and as a component in mixed mode designs. But web surveys are subject to points of criticism. In addition to obvious errors in relation to coverage, sampling and non-response, the underlying construct is that web surveys produce measurement errors per se due to more superficial cognitive processing by respondents. Thus, including web based surveys in mixed-mode designs may improve response rates at a low cost but serious mode effects on measurement presumably occur and lead to poorer survey quality. However, the objections are still open to argument. A large part of literature on the subject matter compares web surveys with telephone surveys, not enabling determination of a “web mode effect”. In this case, differences might as well be due to differences between self-administered and interviewer-administered collection methods. Other parts of literature on mixed-mode design including a web option are using stratified sampling for different modes, and mode differences then are influenced by stratification differences. In both cases the real mode differences are nearly impossible to determine and remains rather speculative. The purpose of this contribution is to examine potential “web mode effects” in mixed-mode surveys. Compared to self-administered surveys by pen and paper, do web based surveys then produce greater measurement errors? Are there indications of greater tendency to ‘mental coin flipping’ when responding via web and are the answers less consistent? The questions asked will be answered on the basis of a survey experiment among political science students divided into to groups, one asked to fill out a questionnaire on attitudes towards environmental issues on paper and the other asked to fill out a web based version.