The comparison of normalization procedures based on different classification systems

In this paper, we develop a new methodology for comparing normalization procedures based on different classification systems. Firstly, a pair of normalization procedures should be compared using their own classification systems for evaluation purposes. Secondly, when the two procedures are noncomparable according to the above test, then evaluation using a third (or more) classification systems may be forthcoming. In the empirical part of the paper we use: (i) the IDCP method for the evaluation of normalization procedures; (ii) two nested classification systems consisting of 219 sub-fields and 19 fields, together with a systematic and a random assignment of articles to sub-fields (or fields) with the aim of maximizing or minimizing differences across sub-fields (or fields); (iii) six normalization procedures using mean citations in each of the classification systems as normalization factors, and (iv) a large dataset, indexed by Thomson Reuters, in which 4.4 million articles published in 1998-2003 with a five-year citation window are assigned to Web of Science subject-categories, or sub-fields using a fractional approach. The results obtained indicate that this methodology may lead to useful conclusions in specific instances.The authors acknowledge financial support by Santander Universities Global Division of Banco Santander. Ruiz-Castillo also acknowledges financial help from the Spanish MEC through grant ECO2010-1959