Measuring sexual behaviours and attitudes in hard-to-reach groups. A comparison of a non-probability web survey with a national probability sample survey

Introduction: Hard-to-reach and minority groups are often at higher risk for adverse sexual health outcomes. While such groups are therefore of interest to sexual health researchers, it can be difficult to locate and recruit sufficient sample sizes using probability sampling methods. This study aims to establish whether web-panel surveys can provide a viable less resource intensive means of boosting sample sizes of two hard-to-reach groups (people of Black African ethnicity, and gay men) for a sexual health survey, and the extent of any bias. Methods: Results from a national probability sample survey (Natsal-3, administered using a computer-assisted personal interview (CAPI) and self-interview (CASI) with 15,162 participants), which included 211 black African participants and 83 gay men, were compared with results from a web-panel survey (using identical questions) of 529 black Africans and 592 gay men. Web-panel survey results for socio-demographics were compared with external benchmarks, and for sexual behaviours and attitudes reported in Natsal-3. Odds ratios (ORs) were used to examine differences between variables and the average absolute OR, along with the number of estimates for which the web-panel survey differed significantly from the benchmarks, were used to summarise survey performance. Results: At least 18% of estimates differed significantly between surveys for gay and black African men, and 28% for black African women. For black African women average absolute ORs were: 1.6 for attitudinal questions asked in CAPI, 1.5 for attitudinal CASI questions, 3.2 for behaviour questions asked in CAPI and for 1.7 for behaviour CASI. For black African men average absolute ORs were: 1.5 for attitudinal questions asked in CAPI, 1.8 for attitudinal CASI questions, 2.5 for behavioural questions asked in CAPI and 1.6 for behavioural questions asked in CASI=1.6. For gay men, average absolute ORs were: 2.2 for attitudinal questions asked in CAPI, 2.8 for attitudinal CASI questions, 1.8 for behavioural questions asked in CAPI to 1.6 for behavioural questions asked in CASI. Discussion: Web-panel surveys may be able to sample hard-to-reach groups but may not be able to replace probability-sample surveys where accurate population-level estimates of sensitive sexual behaviours are required. Differences between web and CASI responses, where mode effects may be similar, suggest web-panel survey selection bias