Evaluating the extent of non-response and non-coverage bias in the Swiss European Social Survey

Surveys increasingly require expensive response enhancement methods to achieve target response rates. In the European Social Survey, the specifications for participating countries stipulate a response rate target of 70%, intended originally as a way to improve cross-national equivalence, guide sample design, as well as to encourage the pursuit of high rates of participation, generally accepted as the best way to minimize bias from nonresponse. Yet the specification of such a target on the ESS has long been the focus of controversy, mainly due to mounting evidence that the relation between response rates and nonresponse bias is not as clear cut as it was originally assumed to be, and that the efforts needed to increase participation – as well as becoming increasingly unaffordable – may in fact aggravate the problem of bias on certain survey variables, and threaten comparability across countries. This literature has not only called into question the value of setting response rate targets, but has highlighted the need for alternative indicators of survey quality that take into account the extent to which data are affected by different forms of survey error. One of the proposed measures is the ‘R-indicator’ or Representativeness-indicator (Schouten and Cobben, 2007), which relies on estimates of individual response propensities based on available auxiliary data, to assess the representativeness of the responding sample. In this paper, we draw on this literature to assess the effectiveness of different response enhancement strategies used on the European Social Survey in Switzerland, both in terms of costs, as well as in terms of their impact on errors due to non-response and non-coverage. In Switzerland, after a comparatively low response rate in the first wave of the ESS, significant improvements have been made, but at considerable financial cost. Until recently, insufficient data relating to the non-responding sample were available to enable a detailed cost-benefit analysis of the effectiveness of methods used to increase participation. In the most recent wave of the survey, however, the availability of new data means that such an analysis is now possible and desirable. Since 2010, the Swiss Centre of Expertise in the Social Sciences (FORS) has been allowed (under strict conditions) to draw their survey samples from the Federal Office of Statistics population register, which covers all (registered) people resident in Switzerland. This new frame offers several distinct advantages: it decreases error due to non-coverage substantially compared to past social surveys, which generally relied on telephone lists for sampling purposes; it also permits the sampling of individuals, thereby reducing the likelihood of refusals by proxy; and importantly, it means that for the first time, auxiliary data are available for the analysis and possible correction of coverage and nonresponse error. The Swiss ESS 2010 was one of the first studies to profit from this new sampling frame, and is somewhat unusual in terms of the richness of the data available for this kind of analysis. For each sampled person, we possess information about basic socio-demographic variables like age, gender, marital status, nationality, and address, and we also know if this person has a registered phone number or not. Additionally, paradata are available relating to fieldwork effort (the number, timing and outcome of contact attempts, and of course, final response status), and interviewer observations (e.g. about the condition of housing, neighbourhood characteristics and an evaluation of wealth). Finally, about 50% of the non-respondents accepted to fill a non-response follow-up paper questionnaire that was sent to them a few months after the end of the main data collection, providing information about the extent to which key survey variables (including attitudinal measures) may be affected by bias. To evaluate the effectiveness of different fieldwork strategies used on the ESS 2010, we use R-indicators to measure the representativeness of different sub-groups of respondents, participating after different amounts of fieldwork effort (based on the number of contacts needed to obtain participation, the need for refusal conversions, etc.). We also assess the representativeness of the participants to the non-response follow-up survey. Moreover, we are interested in the cross-section of these subgroups and the subgroups of sampled persons having a registered telephone number. This information is interesting for at least two reasons. First, it may provide insight into the extent of coverage error in previous survey waves, when samples were drawn from the Swiss telephone-register, which is known, to suffer increasingly from under-coverage. Second, telephone contacts are permitted for sample members with available telephone numbers after five unsuccessful face-to-face contact attempts. This difference in contact mode between people with and without a registered phone number could also introduce bias that could be detected using R-indicators. Finally, partial R-indicators will be calculated for different subsets of participants (e.g. after five face-to-face contact attempts) to distinguish under-represented groups in different steps of the fieldwork process. Preliminary analyses suggest that the ‘representativeness’ of the respondents compared to the sample, based on the auxiliary variables available in the frame (e.g., gender, age and urbanization) does not necessarily improve with the number of contacts attempts and hence with increasing response rates. On the other hand, including the respondents to the non-response survey significantly increases the value of the R-indicator. These first results will be extended to other auxiliary variables available, for instance, from the interviewer observations, and crossed with the subset of people in the sample who have a registered number. We will also compare the final group of respondents to the main survey with the set that includes participants to the non-response survey, to allow us to compare R-indicators based on socio-demographic auxiliary variables with indicators incorporating attitudinal variables collected in the non-response survey. Ultimately, our aim is to gain insight into how to develop targeted contact or refusal conversion strategies for different type of persons based on knowledge of the representativeness of the responding sample, with a view to reducing costs and survey error. Such a targeted fieldwork strategy could help to improve data quality compared to blind efforts to increase the response rate to meet internationally required thresholds.status: publishe