Rapid responses Topic collections Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go?

Study objective: There is little guidance on how to select the best available evidence of health effects of social interventions. The aim of this paper was to assess the implications of setting particular inclusion criteria for evidence synthesis. Design: Analysis of all relevant studies for one systematic review, followed by sensitivity analysis of the effects of selecting studies based on a two dimensional hierarchy of study design and study population. Setting: Case study of a systematic review of the effectiveness of interventions in promoting a population shift from using cars towards walking and cycling. Main results: The distribution of available evidence was skewed. Population level interventions were less likely than individual level interventions to have been studied using the most rigorous study designs; nearly all of the population level evidence would have been missed if only randomised controlled trials had been included. Examining the studies that were excluded did not change the overall conclusions about effectiveness, but did identify additional categories of intervention such as health walks and parking charges that merit further research, and provided evidence to challenge assumptions about the actual effects of progressive urban transport policies. Conclusions: Unthinking adherence to a hierarchy of study design as a means of selecting studies may reduce the value of evidence synthesis and reinforce an ''inverse evidence law'' whereby the least is known about the effects of interventions most likely to influence whole populations. Producing generalisable estimates of effect sizes is only one possible objective of evidence synthesis. Mapping the available evidence and uncertainty about effects may also be important. D espite increasing calls for systematic reviews of health effects of social interventions, there is little methodological research or even guidance on how such reviews should be done. We have lifted the lid on the ''private life'' of the input side of one such systematic review to expose some of our methodological processes and decisions to critical analysis. 1 In a companion paper, we set the scene and examined one phase of the review, the search for evidence. 2 In this paper, we examine another phase of the review: the selection of evidence for inclusion. We investigate the effect of varying our inclusion criteria on the findings and overall value of the review. SELECTING EVIDENCE FOR INCLUSION Researchers designing systematic reviews of intervention studies are advised to specify their research questions in terms of four facets: the intervention, the population receiving the intervention, the outcome of interest, and the study designs deemed worthy of inclusion. 3 This approach is undoubtedly helpful for structuring research questions and protocols, but we aimed to synthesise population level evidence in a cross disciplinary field where comparatively little empirical intervention research has been done. A broad understanding of population health and its wider determinants implied a need to frame our primary research question rather differently. We were not asking, for example, ''What is the evidence that traffic calming leads to a change in travel behaviour?'', but rather ''What interventions, of any kind, lead to such a change?'' In other words, we focused on the outcome of interest and were open to the possibility that any kind of intervention might contribute towards achieving it. This is an example of addressing a ''broad'' review questionacknowledged as a valid, but often difficult, type of review to carry out. 5 Many published systematic reviews have only considered evidence from randomised controlled trials (RCTs). The motivation is to minimise bias, but the Cochrane handbook recognises that this can compromise the relevance of a review, and asks (but does not answer) the question ''How far is it possible to achieve a higher level of relevance by including evidence other than that derived from RCTs without violating the central principle: minimising bias? '' 4 There are already precedents for varying the inclusion criteria for study design according to the nature of the available evidence. For example, although some reviews published by the Cochrane Tobacco Addiction Group are restricted to RCTs, those on community or population level interventions include other study designs including, in some cases, uncontrolled before and after studies. It is increasingly recognised that the usual approach to selecting studies based on a ''hierarchy of evidence'' may rely too heavily on study design as a marker of validity or utility. 8 This type of bias has been described as ''methodological imperialism'' that could distort, rather than strengthen, the evidence base. 11 The relative lack of methodological research on how to deal with evidence from studies other than RCTs may make researchers feel vulnerable at key decision points in the process of synthesising evidence. METHODS We have reported details of our methods previously. We carried out full data extraction and critical appraisal on all of these relevant studies, and therefore formed an overview of the full range of study populations, interventions, and study designs available in the field as well as the range of outcome metrics used and effect sizes identified. It became clear that both the types of study design and the nature of the study populations varied widely. Some studies had used comparatively robust methods to measure, for example, changes in vehicle flows along certain roads, but these studies could tell us nothing about the people using those vehicles or about their non-vehicular (walking) trips. Similarly, we found studies showing how the distribution of transport mode choice had changed among weekend shoppers interviewed in a city centre street, but these studies could tell us nothing about where the shoppers had come from or whether their overall travel behaviour had changed. We also found particular difficulty in deciding what to do with articles-typically book chapters-about ''successful'' towns or cities in which trends in travel patterns were linked post hoc to a variety of interventions, often part of a complex integrated urban policy that included land use planning, public transport improvements, widespread traffic restraint, cycle routes, pedestrianisation, and related measures. These articles did not seem to report the results of specific studies of specific interventions as such, so we characterised them as ''case studies'' in which authors had reported trends of interest to us, but had not presented data in a way that enabled us to assess the strength of the causal assertions being made. These findings led us to devise a simple matrix, or two dimensional hierarchy, of study utility 13 When our review was complete, we also conducted a sensitivity analysis to examine what the content and findings of the review would have been if we had taken one of two extreme approaches to inclusion-either (a) by restricting the review to randomised controlled trials, or (b) by including all relevant studies. This sensitivity analysis was intended to answer two questions: were the conclusions of our review sensitive to the inclusion criteria, and could we have reached our conclusions more efficiently? RESULTS Two dimensional hierarchy We examined the distribution of studies in the matrix We first excluded studies whose design was neither prospective nor controlled (n = 29). We then excluded studies whose populations did not represent a local population or subset thereof (n = 9). This left 31 studies (represented by the dark columns in the Sensitivity analysis Effect of including only RCTs We found only three RCTs. If we had included only these studies, we would have benefited from reviewing a small set of studies that were well written and comparatively easy to appraise. These were also the only studies that contained robust data on direct health outcomes. However, we would only have been able to include evidence about two small categories of intervention: targeted behaviour change programmes for commuters, and school travel coordinators. We would not have identified any evidence about, or perhaps even the existence of, any population wide health promotion activities, ''environmental'' engineering or transport service developments, or financial incentives, and we would not have Criteria for relevance Studies were selected as relevant studies if