Peer review in the assessment and funding of research by the Australian Research Council

In higher education settings the peer review process is highly valued and used for evaluating the academic merits of grant proposals, journal submissions, academic promotions, monographs, text books, PhD thesis and a variety of other academic products. The purpose of this thesis was to evaluate the peer review process for awarding research grants used by the Australian Research Council (ARC) Large Grants Program and to propose strategies to address potential shortcomings of the system. This study also evaluated psychometric properties such as the reliabilities of various ratings that are part of the assessment process of the ARC Large grants Program. Data for the all grant applications submitted for the 1996 round of the Large Grants Program were provided by the ARC. In a variation to the typical peer review process, applicants were given an opportunity to nominate assessors to review their proposals. The results indicated that global ratings given by the researcher-nominated assessors were systematically higher and less reliable than those by panel-nominated external reviewers chosen by the ARC. The reliability of peer reviews is not adequate by most standards. A critical direction for future research is considering what strategies need to be put in place to improve the quality of the reviews. To improve the reliability it is recommended that researcher-nominated reviewers should not be used; that there should be more reviews per proposal and a smaller more highly selected core of reviewers should perform most of the reviews within each sub-discipline providing a greater control over error associated with individual reviewer