Peer review in the funding of research in higher education : the Australian experience

In this article we evaluate the peer review process used to fund Australian university research across all disciplines. Peer reviews of research proposals (2,989 proposals, 6,233 external reviewers) submitted to the Australian Research Council (ARC) are related to characteristics of the researchers and of external reviewers. The reliability of the peer reviews was disappointingly low (interrater agreement of .53 for researcher ratings based on an average of 4.3 external reviewers per proposal). The gender and age of a researcher and the number of researchers on a research team did not affect the probability that funding would be granted, but professors were more likely to be funded than nonprofessors. Australian external reviewers gave lower ratings than did non-Australian reviewers, particularly those from North America. The number of external reviewers for each proposal and the number of proposals assessed by each external reviewer had small negative effects on ratings. Researcher-nominated external reviewers (those chosen by the authors of a research proposal) gave higher, less-reliable ratings than did panel-nominated external reviewers chosen by the ARC. To improve the reliability of peer reviews, we offer the following recommendations: (a) Researcher-nominated reviewers should not be used; (b) there should be more reviews per proposal; and (c) a smaller number of more highly selected reviewers should perform most of the reviews within each subdiscipline, thereby providing greater control over error associated with individual reviewers