Do Consensus Conferences Work? A Process Evaluation of the NIH Consensus Development Program

Abstract. The purpose of this evaluation study is to identify problems and suggest modifications in the NIH Consensus Development Program. The current program consists of three-day conferences in which experts assess medical technologies for issues of efficacy, safety, conditions of use, and other related topics (e.g., costs and social impact). Eight consensus conferences held between 1980 and 1982 were stud-ied in depth using a variety of methods; five of the conferences were investigated concurrently. In addition, archival material was examined for all but one of the 33 conferences held up to that time, and four planning meetings for future conferences were observed. The delay in publishing our findings provided an opportunity to ex-amine the changes introduced by NM; it also allowed us to avoid the criticism of numerous prior evaluations for finding fault with programs that are still developing. NIH adopted many of the recommendations in our evaluation report and has inves-tigated others. Based on our evaluation and more recent evidence, however, we con-clude that the major problem that was uncovered-selection bias, particularly with respect to the choice of questions and panelists-remains a significant threat to the credibility of the consensus process. More specifically, the results indicate that con-troversial issues cannot be properly addressed within the present conference format, although that was one of its major purposes. Recommendations for improving the consensus process are presented, as are their implications for a larger set of consensus activities that are currently being conducted. During the 1970s there was increasing public pressure for more accountability in the U.S. medical care system. Part of this concern focused on the introduction and use of innovative medical technologies, including devices, procedures, and drugs. Members of Congress and others voiced concern about the ways research This paper is based on a report submitted to and accepted by the Office of Medical Applications o