The effects of mixed mode survey designs on simple and complex analyses

Survey data can be collected in a number of ways. The survey organisation may use face-toface interviewing or telephone interviewing, or may ask respondents to complete questionnaires themselves, either online or on paper. It is becoming increasingly common for surveys to use a combination of these methods, a so-called “mixed mode” design. However, the choice of mode, or mix of modes, can affect the data that are collected and consequently also the estimates that are made based on those data. This can happen because different kinds of people are more likely to participate in different modes, or because people will give slightly different answers depending on the mode of interviewing. In this paper we compare estimates from surveys that used three different designs. One survey was a single mode survey, carried out entirely through face-to-face interviewing. The other two surveys were mixed mode surveys, involving face-to-face and telephone interviewing as well as online (web) questionnaires. The difference between the two mixed mode surveys was that in one case respondents were offered an explicit choice between the three interview modes, while in the other case respondents were first asked to complete a web questionnaire and only if they were unable or unwilling to do so were they then asked instead for a telephone interview. If the respondent was also unable or unwilling to carry out the telephone interview they were then asked for a face-to-face interview. Our main interest lies in comparing each of the mixed mode designs with the single mode design. An identical questionnaire was administered to each of the three samples, namely the round 4 questionnaire of the European Social Survey. We compare the distributions of answers to each survey question, and we also compare some estimates of regression coefficients from statistical models of the kind often used by political scientists. While many estimates show no significant difference between the survey designs, we conclude that some estimates are affected by the survey design. We find some suggestive evidence that this is likely to be, at least partly, due to differences in how people answer questions in different modes rather than due to different types of people taking part in the survey. We consequently urge caution in comparing estimates based on data collected using different (mixes of) modes and in the use of mixed mode survey designs