What doesn't work and how to show it

What doesn’t work and how to show it Ineffectiveness is hard to prove and accept Archie Cochrane, the British epidemiologist, posed three key questions to ask about a healthcare intervention: “Can it work?” “Does it work in practice?” and “Is it worth it?”1 It would be great if the answers to these were always positive, but life isn’t like that. The possible answers might be classed as “yes,” “not sure,” and “no.” The rules for deciding “yes” are relatively clear and well known, but less has been written about deciding that something doesn’t work. This theme issue looks at examples of interventions that don’t appear to work, the dilemmas of trying to decide between an answer of “not sure” and “no,” and what to do when we are not sure. Doust and Del Mar admit (p 474)2 that “Don’t just do something, stand there!” sounds ludicrous. But this is sometimes good advice, as their round up of commonly used but apparently ineffective interventions shows. This issue also includes new evidence of varying kinds and strengths on other interventions that didn’t seem to have a marked effect: a systematic review of mentoring for children with behavioural problems (p 512);3 randomised controlled trials of adenoidectomy for children with recurrent otitis media (p 487);4 rocking exercises in late pregnancy to turn babies from a posterior lie (p 490);5 low dose ramipril to reduce cardiovascular and renal complications among diabetic people with microalbuminuria (p 495);6 and an n-of-1 trial of vitamin B-6 for nausea and vomiting in pregnancy (p 503).7 In each case the type of evidence, its quality, and the potential benefits and harms might lead us to be more or less convinced that an intervention does more good than harm. This is very much a judgment, and if people want to believe there is an effect it can be very hard to persuade them that any effect is too small to be important. As a school textbook puts it when discussing the dilemma of industries or official bodies trying to convince the public that something poses no risk: “It is very hard to persuade people that a factor has no effect. This involves ‘proving a negative’.”8 When it is difficult to judge effectiveness and benefit from new research, we need all the help we can get from existing evidence. Two articles this week develop this point. McPherson and Hemminki argue that if trials done by pharmaceutical companies to obtain drug licences had to include adequate data on harms and ineffectiveness we could learn much more quickly what we need to know about new drugs (p 518).9 Pound et al propose systematic reviews and meta-analyses of the animal studies that supposedly underpin much human clinical research.10 Such reviews could assess the validity and generalisability to humans of animal research and could prevent unnecessary treatment trials. Even with all the evidence we can muster, we are often left being uncertain about the right treatment choice.11 As US defense secretary Donald Rumsfeld said at a press briefing on Iraq: “Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know.We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know.”12 This much ridiculed statement won the Plain English Campaign’s Foot in Mouth award for 2003, but it does make sense when read carefully. Perhaps what we don’t know we don’t know would be a good topic for a BMJ theme issue: let us know if you agree