RESEARCH ARTICLEPublications as predictors of racial and ethnicdifferences in NIH research awardsDonna K. GintherID1,2☯*, Jodi Basner3‡, Unni Jensen3‡, Joshua SchnellID3‡,Raynard Kington4☯, Walter T. SchafferID5☯1 Department of Economics and Center for Science, Technology & Economic Policy, Institute for Policy &Social Research, University of Kansas, Lawrence, Kansas, United States of America, 2 National Bureau ofEconomic Research, Cambridge, Massachusetts, United States of America, 3 Discovery Logic/ClarivateAnalytics, Rockville, Maryland, United States of America, 4 Grinnell College, Grinnell, Iowa, United States ofAmerica, 5 National Institutes of Health, Bethesda, Maryland, United States of America☯ These authors contributed equally to this work.‡ These authors also contributed equally to this work.* dginther@ku.eduAbstractThis research expands efforts to understand differences in NIH funding associated with theself-identified race and ethnicity of applicants. We collected data from 2,397 NIH Biographi-cal Sketches submitted between FY 2003 and 2006 as part of new NIH R01 Type 1 applica-tions to obtain detailed information on the applicants’ training and scholarly activities,including publications. Using these data, we examined the association between an NIH R01applicant’s race or ethnicity and the probability of receiving an R01 award. The applicant’spublication history as reported in the NIH biographical sketch and the associated biblio-metrics narrowed the black/white funding gap for new and experienced investigators inexplanatory models. We found that black applicants reported fewer papers on their Bios-ketches, had fewer citations, and those that were reported appeared in journals with lowerimpact factors. Incorporating these measures in our models explained a substantial portionof the black/white funding gap. Although these predictors influence the funding gap, they donot fully address race/ethnicity differences in receiving a priority score.IntroductionThe National Institute of Health (NIH) has worked to characterize reasons for the underrepre-sentation of certain racial and ethnic groups within the population of Principal Investigatorswith NIH research grants. Earlier studies documented attrition rates for different groups tran-sitioning through important career stages from high school to faculty attainment [1]. Studiesalso examined race/ethnicity and gender differences in NIH funding [2–4] identifying largedifferences for PhDs and smaller differences for MDs at medical schools as well as no addi-tional gender gap. The results of these earlier studies sparked an immediate response from thebiomedical research community and the NIH [5, 6]. A recent study has shown that the black/white funding gap at the NIH has persisted using data from 2010–2013: 19% of applicationsfrom white investigators were funded during this time period compared to only 11.8% ofPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 1 / 24a1111111111a1111111111a1111111111a1111111111a1111111111OPEN ACCESSCitation: Ginther DK, Basner J, Jensen U, SchnellJ, Kington R, Schaffer WT (2018) Publications aspredictors of racial and ethnic differences in NIHresearch awards. PLoS ONE 13(11): e0205929.https://doi.org/10.1371/journal.pone.0205929Editor: Leonidas G. Koniaris, Indiana University,UNITED STATESReceived: May 15, 2018Accepted: October 2, 2018Published: November 14, 2018Copyright: This is an open access article, free of allcopyright, and may be freely reproduced,distributed, transmitted, modified, built upon, orotherwise used by anyone for any lawful purpose.The work is made available under the CreativeCommons CC0 public domain dedication.Data Availability Statement: Pre-decisional grantapplication and personally identifiable informationdata covered in this manuscript cannot be madepublic by law. The federal law, the Privacy Act,precludes us from releasing personally identifiableinformation on grant applicants. Under theFreedom of Information Act ("FOIA"), 5 U.S.C. 552,individuals may submit a formal request to obtaininformation on funded biomedical research grantsnot publicly available. The government maywithhold information pursuant to the nineexemptions and three exclusions contained in theAct. Inquiries may be directed to FOIA Coordinator,applications from black investigators [7]. Thus, a more-complete understanding of the black/white funding gap is warranted. This study adds to our evolving understanding of the complexproblem of differences related to race/ethnicity in NIH funding by extracting additional detailsabout applicants’ prior education, training, and publications from their biographical sketchessubmitted at the time of application. Previous studies did not include the full range of informa-tion available to reviewers from the biographical sketch and could have contributed to unex-plained differences in NIH funding. In the new analysis, we examined whether improvedinformation on undergraduate and postdoctoral training, academic rank, scholarly awards,prior grant activity, publications, and associated bibliometrics available at the time of applica-tion help to explain the funding gap for applicants of different races and ethnicities.To receive NIH funding, grant applications are peer-reviewed to evaluate the significance,innovation, and approach of the proposed research along with the investigator(s) and theresearch environment. The NIH application consists of several parts: the research plan, thebiographical sketches of key personnel, and the budget. Assigned reviewers provide a prelimi-nary score for each application. The most meritorious applications (approximately half of thetotal) are discussed in detail and provided a final priority (more recently, impact) score. NIHInstitutes and Centers determine which highly meritorious applications are ultimately fundedbased on budgets and scientific priorities [8]. In our previous analysis, we hypothesized thatscientists of different races and ethnicities with similar academic records would have similarlikelihoods of being awarded research grants, but that was not the case. In response, the NIHDirector established a high-level Working Group on Diversity in the Biomedical ResearchWorkforce (WGDBRW), and their report [9] pointed out that potentially important explana-tory variables were missing from the previous analysis [2]. The report argued that the ability todistinguish between the competing explanations of the black/white NIH funding gap—appli-cation merit, investigator characteristics, or bias in the peer review process—was insufficientlyexplained by variables included in the analysis, prompting a the need for a more detailedevaluation.Data and methodsThe current study fills in the missing data by using information from biographical sketches(Biosketches) that accompany each NIH grant application. As before, we approached the datawith the null hypothesis that information contained in the application determined the proba-bility of an application receiving a priority score and receiving funding. If this is an accuratedepiction of the peer review process, then incorporating previously unexplored measuresabout the investigator from the Biosketch, including his or her academic record, has the poten-tial to explain more of the observed black/white funding gap. In the event that the additionaldata do not explain the observed gap, and in the absence of other factors, bias in the peerreview process remains an open possibility.We randomly sampled 600 applications for each of the self-identified race/ethnicity catego-ries (Asian, black, Hispanic and non-Hispanic white) from grant proposals submitted betweenFY 2003 and FY 2006 that had been included in the original sample of 83,188 first-time (Type1) R01 grant applications from PhD investigators at US institutions [2]. We were limited tothat period of time because of the availability of Biosketch information in the NIH IMPAC IIgrants data system [10, 11]. Our sample included 2,397 applications from 2,025 unique investi-gators. Three observations were dropped from the original random sample because theseapplications had incomplete Biosketches. (Fig A in S1 File) indicated no significant differencesin R01 Award probabilities by race between the full population previously described and thesubsamples used in this study [2].Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 2 / 24Office of Extramural Research at email:OERFOIA@mail.nih.gov. For complete informationon the FOIA submission process, visit the NIH FOIAwebsite at http://www.nih.gov/icd/od/foia/index.htm.Funding: The NIH provided support in the form ofa contract [HHSN276200700235U] awarded toDiscovery Logic/Clarivate Analytics [DKG, JS, UJ,JB] for data license and labor fees required for datapreparation and analysis, but did not have anyadditional role in the study design, data collectionand analysis, decision to publish, or preparation ofthe manuscript. This work was also supported bythe National Institute on Aging, 1R01AG36820-0 toDKG. WTS was an employee of NIH at the time ofthis study. NIH reviewed and provided feedback onthe manuscript prior to publication. The specificroles of these authors are articulated in the ‘authorcontributions’ section.Competing interests: DKG served as a paidexternal consultant to Discovery Logic/ClarivateAnalytics in support of contractHHSN276200700235U. JDS, UJ and JB areemployees of Discovery Logic/Clarivate Analytics.Discovery Logic/Clarivate Analytics does not alterour adherence to PLOS ONE policies on sharingdata and materials.As part of the application process, an NIH principal investigator included a Biosketch thatlists information, in up to four pages, on education and training, positions and honors,selected peer-reviewed publications, and research support. Information from these 2,397 Bios-ketches was manually entered into a data collection instrument that coded the following infor-mation: academic rank of the applicant at the time of application, undergraduate degrees, PhDdegrees, postdoctoral positions, publications, scholarly awards and advisory panel experience,and prior grants. Details of data collection and classification are described in the supplementalmaterial, but we provide an overview of items collected here. Altogether, over one million dataitems were entered or created from the 2,397 Biosketches.Education and trainingInformation on bachelor’s and doctoral degree institution was previously available for US-trained PhDs, but missing for foreign-trained students [2]. Information on postdoctoral insti-tutions was missing for the entire sample in the earlier study. By coding Biosketches we wereable to collect undergraduate, PhD, and postdoctoral institutions for all applicants in the sub-sample and link the data to the Department of Education Integrated Postsecondary EducationSystem Database (IPEDS) [12]; Carnegie Classification of Institutions of Higher Education[13]; and the Oberlin Group of 50 liberal arts colleges [14] in order to examine factors thatmight be related to the prestige and selectivity of the applicant’s institutional affiliations. Wealso assigned an NIH funding rank to the undergraduate, PhD, and postdoctoral institutionsbased in the US using aggregate NIH funding data for the institutions for the same timeperiod.We identified 1,302 predoctoral and postdoctoral fellowships on applicant Biosketches aswell as 170 traineeships and 220 diversity-related fellowships and scholarships. In addition, weincorporated data from IMPAC II on T32 predoctoral traineeships, T32 postdoctoral trainee-ships, F31 fellowships, and F32 fellowships, including diversity fellowships [15], for the entiresample used previously [2].We attempted to identify PhD and postdoctoral advisors from multiple sources, but foundthat few Biosketches identified PhD or postdoctoral advisors. We had limited success in identi-fying PhD or postdoctoral advisors by using the last author’s name from the applicant’s publi-cations during the time they were doctoral students or postdoctoral researchers. For the subsetof applicants for whom we could identify advisors’ names from other sources, only 56% of thelast authors listed on their doctoral and postdoctoral publications were the applicant’s advi-sors. Thus, our analysis could not control for doctoral and postdoctoral advisors.Positions and honorsBiosketch data allowed us to identify the academic rank, affiliation, and professional service ofthe applicant at the time the application was submitted. Applicants were associated with 3,107advisory panels, 2,005 professional society memberships, and 121 leadership positions.Selected peer-reviewed publicationsEach peer-reviewed publication listed on the Biosketch was assigned to the application andused to build the bibliographic record for the application. Nearly 54,000 publications wereidentified on the Biosketches, and the majority of these publications were matched to biblio-graphic information from the Clarivate Analytics Web of Science and Journal Citation Reportsby article title, journal title, volume, and author names. Publications were assigned to under-graduate, graduate, postdoctoral, and independent researcher career stages using the dates ofdegree receipt and postdoctoral appointments. For each publication, we collected bibliometricPublications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 3 / 24data such as the number of times cited, the number of self-citations, the narrow and broadjournal subject category, the impact factor of the journal, the number of coauthors, and theauthor position of the applicant.Not all scientific fields have the same publication and citation patterns, so we used field-normalized measures based on the journal subject category of the publication. Citations to agiven publication over the first two years were compared to citations to papers in journals in asimilar Web of Science journal subject category during the same two years. The publicationwas then assigned a field-normalized quartile rank in the publication distribution. We identi-fied the percentage of the applicant’s publications that were published in each quartile of thefield. In addition, we identified the percentage of papers that were uncited and the percentageof papers where the applicant was either first or last author. We also identified the three mostfrequent last authors and the three most frequent coauthors, regardless of the order of appear-ance in the list of authors, on the papers published by the applicant and listed on the Biosketch.We then created bibliometric measures similar to those used for the author’s publicationsbased on the publications of the last authors and coauthors. These bibliometric measures serveas proxies for the professional networks of the applicants.Research supportAn indicator of prior NIH Grants was an important explanatory variable in previous studies[2–4]. We enhanced these data by the counts of prior non-R01 NIH grants by funding mecha-nism as well as prior NIH R01 Type 2 (competing renewal) awards from the IMPAC II datasystem [10]. Applicant Biosketches provided additional information on non-NIH federalgrants, foundation grants, private sources, and international grants that were not available inthe data used in the earlier papers. The Biosketch data was then combined with the variablesfrom the original studies [2, 16]. Table A in S1 File lists the variables used in the analysesdescribed below.Modeling grant success outcomesWe analyzed the probability of receiving an R01 award using probit models to evaluatewhether different combinations of Biosketch data explained the funding gap. Our analysis wasbuilt on a main model that included controls for age, gender, foreign PhD, and race/ethnicity,and progressed through eight models that added sets of explanatory variables for academicrank, grants, scholarly honors, publications and associated bibliometrics, coauthor publica-tions, journal field, and a full model (Table A in S1 File).In place of reporting probit coefficients, we report the marginal effect of race/ethnicity rela-tive to whites on the award probability, which is the change in the award probability due torace/ethnicity with other variables evaluated at their mean values. These regression estimatesare correlations between race/ethnicity and the probability of receiving an R01 award andshould not be interpreted as having a causal impact. Our goal is to determine whether addi-tional covariates from the Biosketch explain (reduce the size) of the NIH funding gap. Manyvariables included in the analysis are endogenous to the process of obtaining additional NIHfunding. Thus, we begin our analysis by only including the exogenous variables in the mainmodel. Education variables will also be exogenous to the NIH funding process, whereas train-ing, positions and honors, publications, and prior research support are all endogenous to theNIH funding process. As a result we estimate multiple specifications in order to examine howthe race/ethnicity coefficients change as these endogenous variables are added to the model.Since we are examining differences in NIH funding for multiple race/ethnicity categories, allp-values are adjusted for multiple comparisons using the Bonferroni correction.Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 4 / 24The Biosketch data features improved measures of publications and associated biblio-metrics. Other than our previous works [2–4] that modeled R01 funding as a function of bib-liometric measures, we are not aware of other papers that model funding as a function ofbibliometrics. Although one study did calculate a variety of bibliometric measures for a smallnumber of NIH awardees [17]. In this study, we observed more publications using Biosketchdata (an average of 22.5 publications per investigator) than in the earlier study that used name-matched data (an average of 17.6 publications per investigator) in the earlier study [2]. After alengthy evaluation of a variety of bibliometric measures detailed in the supplemental material,we determined that the following bibliometric measures provide the optimal fit and explana-tory power: the logarithm of the sum of the journal impact factors for all the applicant’s publi-cations, the percentage of publications that were first-authored, the percentage of papers thatwere last-authored, the percentage of papers in the top quartile of the field (measured by two-year citations to the paper compared with two-year citations to papers published in the journalsubject category), the percentage of uncited papers, and the percentage of the top three coau-thors’ papers in the top quartile of the field (measured by two-year citations to the paper com-pared with two-year citations to papers published in the journal subject category).Use of the journal impact factor to evaluate individual publications is controversial [18, 19].However, no measure of research impact is perfect, and the journal impact factor comparesfavorably to other measures [20]. In addition, the sum of the journal impact factors of publica-tions has been used previously to examine academic research productivity [21, 22]. Essentially,the sum of the impact factors weights the publication by the impact factor of the journal. Thus,a publication in a journal like Science (impact factor 34.66 in 2015) is given more weight than apublication in Human Biology (impact factor .88 in 2015). Journal impact factors have beencriticized because citation distributions of papers in journals are highly skewed [19], and wefound that citation distributions of papers listed on an applicant’s Biosketch are also highlyskewed. Consistent with this observation, we used the natural log of these bibliometric mea-sures to improve the fit of the model.After establishing the main results that include publications and bibliometrics, we conducta number of robustness tests to gain a better understanding of the factors contributing to therace/ethnicity gap in NIH funding. Our analysis examines whether race/ethnicity gaps differbetween New and Experienced Investigators, those with MD/PhD and PhD degrees, and thefunding rank of the institution. NIH has long recognized that new investigators are less-likelyto receive NIH research awards than experienced investigators [23]. NIH policy during ourstudy time frame [24] directed NIH Institutes and Centers (ICs) to maintain the number offunded new investigators at the average rate of the previous five years as a way of promotingindependent research careers. A recent study showed that mid-career scientists were facingdifficulties in receiving additional NIH funding [25]. Given NIH’s focus on differences infunding outcomes for new and experienced investigators [26], we investigated outcomes usingthese career milestones.Our previous analysis examined whether the black/white funding gap differed for MDs andMD/PhDs [3]. We found that the black/white funding gap for MDs and MD/PhDs was smallerthan for PhD scientists, and could be explained by whether grants included human subjects.Unlike PhD scientists, MDs often have clinical duties in addition to teaching and research thatmay affect their ability to publish papers.In addition, our previous analysis also showed that the NIH funding rank of an institutionexplained a significant share of the race/ethnicity funding gap [2]. The NIH funding rank is aproxy for the research intensity of the institution, where organizations with higher fundingconduct more research. We partitioned institutions into the top 100 and the 101+ NIH fundedinstitutions and included our Biosketch measures of publications to determine whether thePublications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 5 / 24funding gap differed according to institution type. Next we compared estimates of the race/ethnicity gap when using name-matched and Biosketch measures of publications andbibliometrics.We then used our model to examine the effect of a counterfactual policy experimentdesigned to increase publications. We considered a mentoring intervention that provides feed-back and professional development for black investigators related to the publication process.This type of intervention increased the number and quality of publications in a randomizedcontrolled trial for female economists [27]. We assumed that this intervention increases boththe quantity and quality of publications for black investigators. To understand the impact ofthis counterfactual policy experiment, we used our models to predict what would happen ifblack investigators had the average publication and bibliometric profiles as well as averageType 2 awards that matched R01 awardees.We also used the timing of publications to identify when black and white researchers’careers diverged. Finally, the black/white funding gap consists of two parts: the probability ofreceiving a priority score, and then conditional on the score, the probability of receiving fund-ing. We examined whether our measures of publications narrowed the race/ethnicity gap inreceiving a priority score.ResultsProbability of receiving NIH fundingTable 1 shows the marginal effects for Asian, black, and Hispanic investigators relative towhite investigators from the probit models of receiving an R01 grant. Consistent with our pre-vious analysis we find that the Asian and Hispanic funding gaps could be explained by covari-ates in our model. Accordingly, we focus on the black/white funding gap in our discussion.The main model includes only covariates for race/ethnicity, gender, age, and foreign PhD andindicates that applications from black investigators are -13.4 percentage points (ppt) (p<.001) less likely to receive an R01 award. Sets of covariates from the Biosketches includinginformation on academic rank, prior non-NIH grants, and scholarly activities are added to themain model (in subsequent columns of Table 1), but the inclusion of these variables does notchange the significant black/white funding gap. In results not reported, being a full professorincreased the probability of NIH funding by 11.9 ppt (p< .001), but the funding gap remained-12.8 ppt (p< .001). Only prior NIH grants had a significant impact on R01 awards—non-NIH grants had no significant effect. Participating on non-NIH review panels (-3.6 ppt (p<.01)) had negative and significant associations with NIH R01 award probabilities.When we added the logarithm of the sum of the journal impact factors and controls for firstand last authorship, the funding gap falls from -13.3 ppt (p< .001) to -9.5 ppt (p< .001). Aone percent increase in the sum of the journal impact factors increases the probability of NIHfunding by 6.7 ppt (p< .001). Likewise a one percent increase in the percentage of firstauthored publications increases the probability by 9.1 ppt (p< .05) (Table 1). Field normalizedpublications, measured by the percentage of papers in the top quartile of the field, and the per-centage of uncited papers reduced the gap to -8.1 ppt (p< .01) (Table 1). Although coauthorswho publish in the top quartile of the field are positively associated with NIH research awards,this variable does not change the black/white funding gap. Controlling for field of publicationreduces the gap to -7.7 ppt. (p< .001). Next, we added important explanatory variables fromour previous analysis such as NIH funding rank of the employer and the number of prior NIHgrants, and the black/white funding gap fell to -6.9 ppt (p< .01).We separately analyzed the effect of training variables on the probability of receiving anNIH R01 award. Building on the main model, we progressed through nine models that addedPublications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 6 / 24Table 1. Probit Estimates of NIH R01 award.Academic Scholarly Publications Add Field Add Add FullVARIABLES Main Rank Grants Activities Normalized Coauthor Pub Field ModelAsian -0.044 -0.038 -0.038 -0.047� -0.058�� -0.051 -0.050 -0.038 -0.021[0.022] [0.022] [0.022] [0.022] [0.022] [0.022] [0.022] [0.022] [0.023]Black -0.134�� -0.128�� -0.126�� -0.131�� -0.095�� -0.081�� -0.082�� -0.077�� -0.069��[0.020] [0.020] [0.020] [0.020] [0.022] [0.023] [0.022] [0.022] [0.022]Hispanic -0.039 -0.033 -0.030 -0.038 -0.030 -0.029 -0.026 -0.021 -0.004[0.022] [0.023] [0.023] [0.022] [0.023] [0.022] [0.022] [0.023] [0.023]Log Sum of Impact Factors 0.067�� 0.056�� 0.056�� 0.083�� 0.044��[0.010] [0.010] [0.010] [0.013] [0.011]Percentage First Authored 0.091� 0.093� 0.091� 0.080 0.093�Publications [0.045] [0.044] [0.044] [0.045] [0.045]Percentage Last Authored 0.036 0.027 0.027 0.039 -0.035Publications [0.043] [0.042] [0.042] [0.043] [0.045]Percentage of Uncited -0.003� -0.003� -0.004� -0.003�Publications [0.001] [0.001] [0.001] [0.001]Percent Publications Top Quartile of Field 0.002�� 0.001 0.000 0.001[0.001] [0.001] [0.001] [0.001]Co-Authors have Publications 0.002�� 0.002�� 0.002��Top Quartile of Field [0.001] [0.001] [0.001]NIH Fund Rank 1–30 0.108��[0.032]NIH Fund Rank 31–100 0.043[0.030]NIH Fund Rank 101–200 0.081�[0.036]Total Prior NIH Grants 0.008�[0.003]NIH Review Committee 0.094��[0.020]Grant includes Human -0.048��Subjects [0.019]Additional Controls:Age X X X X X X X X XNativity X X X X X X X X XGender X X X X X X X X XApplication Year X X X X X X X X XAcademic Rank (1) XPrior Grants (2) XScholarly Activities (3) XPublication Field Missing X X X X XPublication Field XNIH Institute (4) X(Continued)Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 7 / 24sets of explanatory variables for each stage of training: undergraduate, predoctoral, receipt ofPhD, postdoctoral, and pre and postdoctoral fellowships (specifications Table B, estimatesTable C in S1 File).Adding undergraduate training, predoctoral fellowships, and variables describing the PhD-granting institution to the main model does not diminish the funding gap for black investiga-tors. Adding postdoctoral characteristics in the last four models reduces the black/white fund-ing gap to about -12.7 ppt (p< .001), and when all training variables are entered into themodel, the black/white funding gap is only slightly smaller at -12.3 ppt (p< .001). The finalspecification limits the analysis to US citizens in order to evaluate the impact of NIH training,and the black/white gap falls to -11.9 ppt (p< .001). These results indicated that bibliometricmeasures better explain the black/white funding gap when compared to training measures.Given the importance of publication and bibliometric measures for explaining the funding gap,we used a method developed by Gelbach [28] to estimate the contribution of covariates in ourmodel to the reduction in the black/white funding gap. To implement this method we estimated alinear probability model using the main specification and the Biosketch model that includes all thecovariates from the main, rank, grants, scholarly activities, field-normalized publications, andcoauthor publications models (see Table A in S1 File for the specification). We omitted the trainingcovariates because they do not appreciably change the race/ethnicity coefficients. Results arereported in the top panel of Table D in S1 File. Linear probability estimates in the main model indi-cate that black investigators are 14.1 ppt (p< .001) less likely to receive funding than white investi-gators. Once the covariates from the Biosketch model are included, the gap falls by 40% to 8.4 ppt(p< .001). Almost all of the reduction in the gap is due to the inclusion of bibliometric measuresderived from the publications reported on the application at the time of review.Next we compare the main model to the full model in the bottom panel of Table D in S1 File.The full model uses Biosketch publications and improved information on previous NIH fund-ing (see Table A in S1 File for a complete list of covariates). The full model explains 52% of theblack/white funding gap. Half of the explained gap is due to Biosketch publications (p< .001)and an additional 38% is due to improved measures of prior NIH funding and Type 2 awards.New and experienced investigators, MD/PhDs and funding rank of theinstitutionFig 1 shows the R01 award probability, and Table E in S1 File shows the counts of applicationsand awards by race for new and experienced investigators. We found that 31% of experiencedTable 1. (Continued)Academic Scholarly Publications Add Field Add Add FullVARIABLES Main Rank Grants Activities Normalized Coauthor Pub Field ModelObservations 2,397 2,396 2,397 2,397 2,397 2,397 2,397 2,397 2,397Robust standard errors in brackets.�� p<0.001�� p<0.01� p<0.05. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of Medical Colleges faculty roster, select NIH Biosketches,Web of Science1.1) Includes controls for Assistant, Associate, Full and Research Faculty. Only Full Professor significant .119 (p < .001).2) Includes controls for prior grants by funder, role on grants as principal investigator, co-investigator, director. Only NIH grants significant .022 (p < .001).3) Includes controls for scholarly activities. Only Non-NIH Panel member -0.036 (p < .01) significant).4) Includes eight of the 27 institutes and centers that have a significant impact on probability of R01 Award.https://doi.org/10.1371/journal.pone.0205929.t001Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 8 / 24white investigators were funded compared to only 21% of new white investigators. With theexception of Asians, experienced investigators have an approximately 10 percentage pointadvantage in NIH research funding. However, the funding disadvantage for black investigatorsremains significant. Table 2 examined whether the black/white funding gap differed for newand experienced investigators. Using the main model that only controls for demographic char-acteristics, we found that the black/white funding gap for new investigators was smaller(-10.2 ppt (p< .001)) than for experienced investigators (-12 ppt, (p< .001)). Experiencedinvestigators can receive renewals (Type 2 R01 awards) on their previously funded Type 1 R01awards. We incorporated this variable in the full model and found that each additional Type 2award increased the probability of receiving a new Type 1 R01 award by 2 ppt (p < .01), but itdid not change the overall black/white funding gap. We tested whether the factors that pre-dicted NIH research awards differ significantly between new and experienced investigators inour probit models, and rejected the null hypothesis that the coefficients were the same (p<.04). The black/white funding gap falls to 4.6 ppt. (p = .33) for new investigators. However, theblack/white funding gap for experienced investigators is almost double that at 7.8 ppt. (p = .13)(Table 2, Fig 1).The new and experienced investigator estimates must be interpreted with care. It is impor-tant to note that our sample contains almost all of the black investigators who submittedFig 1. R01 Awards by race and investigator experience. A comparison of R01 award probability by race/ethnicity, by investigator experiencewith 95% confidence intervals. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of MedicalColleges faculty roster, select NIH Biosketches, Web of Science. Full Sample N = 53,454. Subsample N = 2,397.https://doi.org/10.1371/journal.pone.0205929.g001Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 9 / 24applications between 2003 and 2006, and this is the maximum possible sample size. When wesplit the sample into new and experienced investigators, the standard errors became larger,making it more likely to commit a Type 2 error (failing to reject the null hypothesis of noblack/white funding difference when it is false and there is a difference). There are 150 moreblack new investigators than experienced investigators, but the p-value for new investigators isTable 2. Probit Estimates of NIH R01 Award by investigator experience.Main Main Main Full Full + New ExperiencedVARIABLES New Investigator Experienced Investigator Type 2 Investigator InvestigatorAsian -0.044� -0.010 -0.063 -0.021 -0.018 -0.001 -0.027[0.022] [0.031] [0.033] [0.023] [0.023] [0.029] [0.035]Black -0.134�� -0.102�� -0.120�� -0.069�� -0.068� -0.046 -0.078[0.020] [0.026] [0.034] [0.022] [0.022] [0.027] [0.036]Hispanic -0.039 -0.024 -0.023 -0.004 -0.004 -0.005 0.001[0.022] [0.030] [0.035] [0.023] [0.023] [0.028] [0.036]Log Sum of Impact Factors 0.044�� 0.043�� 0.043�� 0.042�[0.011] [0.011] [0.012] [0.020]Percentage First Authored 0.093� 0.098� 0.049 0.140Publications [0.045] [0.044] [0.049] [0.079]Percentage Last Authored -0.035 -0.044 -0.005 -0.034Publications [0.045] [0.045] [0.062] [0.066]Percentage of Uncited -0.003� -0.003� -0.004�� -0.001Publications [0.001] [0.001] [0.001] [0.003]Percent Publications Top Quartile of Field 0.001 0.001 0.000 0.001[0.001] [0.001] [0.001] [0.001]Co-Authors have Publications 0.002�� 0.002�� 0.002�� 0.002��Top Quartile of Field [0.001] [0.001] [0.001] [0.001]NIH Fund Rank 1–30 0.108�� 0.107�� 0.128�� 0.052[0.032] [0.032] [0.041] [0.051]NIH Fund Rank 31–100 0.043 0.041 0.067 -0.016[0.030] [0.030] [0.036] [0.049]NIH Fund Rank 101–200 0.081� 0.078� 0.077 0.049[0.036] [0.036] [0.042] [0.059]Total Prior NIH Grants 0.008� -0.001 0.004 -0.006[0.003] [0.005] [0.011] [0.006]NIH Review Committee 0.094�� 0.099�� 0.073�� 0.105��[0.020] [0.020] [0.028] [0.030]Grant includes Human -0.048�� -0.042� -0.026 -0.057Subjects [0.019] [0.019] [0.023] [0.031]Type 2 Award 0.020�� 0.027��[0.007] [0.009]Observations 2,397 1,232 1,165 2,397 2,397 1,232 1,165Robust standard errors in brackets.�� p<0.001�� p<0.01� p<0.05.Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of Medical Colleges faculty roster, select NIH Biosketches, Web ofScience.https://doi.org/10.1371/journal.pone.0205929.t002Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 10 / 24much larger than for experienced investigators, suggesting that the remaining variation is sam-ple noise. Since there are a smaller number of black experienced investigators, we would expectto have less precise estimates resulting in larger standard errors and p-values for experiencedcompared to new investigators. The smaller p-value for experienced investigators contradictsthis intuition.We observed considerable evidence that the factors associated with new investigator fund-ing differ from those associated with experienced investigator funding (Table 2). In particular,the percentage of uncited papers had a larger negative effect on new investigators. Likewise,the NIH funding rank of the employer and prior NIH grants, increased the probability ofresearch funding for new, but not experienced investigators. The coefficient on NIH reviewcommittee experience is positive and significant for both new (7.3 ppt, p< .01) and experi-enced (10.5 ppt, p < .001) investigators but it is much larger for the experienced PIs. Investiga-tors eligible to obtain renewals on their previous R01 awards (R01 Type 2 awards) are bydefinition, experienced investigators. Each additional Type 2 award increases the probabilitythat experienced investigators receive funding for a new R01 project by 2.7 ppt (p< .001).We partitioned our data into MD/PhDs and MDs as well as Top 100, and 101+ NIH fundedinstitutions to see whether Biosketch publications narrowed the race/ethnicity funding gap inthese subsamples. Table 3 shows the race/ethnicity coefficients for the main model that onlyincludes exogenous variables and the full model. In the main model, the estimate of the black/white funding gap is remarkably similar in magnitude and statistically significant for the fullsample, MD/PhDs, PhDs, 101+ NIH Funded, and Top 100 institutions. The sample size ofMD/PhDs is small (N = 476) as is the sample of 101+ NIH Funded institutions (N = 763).When we included the covariates in the full model, the black/white funding gap for MD/PhDsdrops by half compared to the full sample (-3.8 ppt compared with -6.8 ppt) and is no longerstatistically significant. However, the black/white funding gap for PhDs is larger than the coef-ficient for the full sample (-7.6 ppt, p< .01). The black/white funding gap for the NIH Fundingrank subsamples are not statistically significant, but the magnitude suggests economic signifi-cance. The black/white funding gap for the 101+ NIH funded institutions is -7.4 ppt while thegap is -5.5 ppt for the top 100 NIH funded institutions.Table 3. Probit Estimates of NIH R01 Award by degree and NIH funding rank.Main Model Full ModelVARIABLES FullSampleMD/PhDsPhDs 101+ NIHFundingTop 100 NIHFundingFullSampleMD/PhDsPhDs 101+ NIHFundingTop 100 NIHFundingAsian -0.044 -0.140� -0.024 -0.020 -0.051 -0.018 -0.116� 0.005 -0.012 -0.018[0.022] [0.048] [0.025] [0.037] [0.028] [0.023] [0.048] [0.026] [0.036] [0.029]Black -0.134�� -0.132� -0.139�� -0.127�� -0.124�� -0.068� -0.038 -0.076�� -0.074 -0.055[0.020] [0.047] [0.021] [0.031] [0.026] [0.022] [0.055] [0.024] [0.033] [0.030]Hispanic -0.039 -0.049 -0.043 0.020 -0.063 -0.004 -0.026 0.000 0.035 -0.022[0.022] [0.053] [0.025] [0.040] [0.027] [0.023] [0.051] [0.026] [0.040] [0.029]Observations 2,397 476 1,921 763 1,634 2,397 476 1,921 763 1,634Robust standard errors in brackets.�� p<0.001�� p<0.01� p<0.05.Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of Medical Colleges faculty roster, select NIH Biosketches, Web ofSciencehttps://doi.org/10.1371/journal.pone.0205929.t003Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 11 / 24A comparison of name-matched and Biosketch publicationsIn our previous studies we included measures of publications and citations, but these variablesdid not fully explain the black/white funding gap. As discussed above, publications were iden-tified in the previous study using a name-matching algorithm, a standard approach for assign-ing publications to applicants. However, this resulted in an undercount of publications.Estimates in Table 4 demonstrated that 2 percentage points of the -13.4 ppt (p< .001) black/white funding gap could be explained without including measures of publications. Our name-matched publication measures from the previous study did not include the log sum of journalimpact factors. Instead, we included the log of publications, the log of citations, the maximumjournal impact factor on all publications, the median journal impact factor on all publications,and the percentage of first, last, and single-authored publications from the name-matchedmeasures. The name-matched publication variables explain 0.8 ppt of the black/white fundinggap. We can replicate these measures with our Biosketch publication variables, and the black/white funding gap reduces from -13.4 to -8.8 ppt (p< .05). We replaced these measures withour new measures that included the log sum of the impact factors of publications, and theblack/white funding gap drops from -13.4 to -8.6 ppt (p< .001).Why do bibliometric measures based on the journal impact factor have better explanatorypower than citation counts of an individual’s publication record? Journals are observed on theBiosketch, and citations are not. Furthermore, review panel members are likely to be familiarwith the impact factors of journals in their fields. Reviewers appear to be using their under-standing of the journal impact factor as a proxy for the quality of the publication, despite warn-ings to avoid this attribution [18].Race/Ethnicity differences in publications and bibliometricsUnlike our previous studies, publications and the associated bibliometrics explained a signifi-cant portion of the black/white funding gap, suggesting the need to explore factors that mayhelp us understand the differences in publication-related metrics. Figs 2 and 3 (Tables F and Gin S1 File) examined differences in average publications and associated bibliometrics by race,R01 award status and investigator experience. Fig 2 (Tables F and G in S1 File) showed thatnew and experienced black investigators report between two to three fewer papers than whites,and those papers are cited at less than half the frequency of whites for non-awardees. Blackinvestigators also have fewer coauthors (Tables F and G in S1 File), and their reported publica-tions have lower average sums of impact factors. These gaps are larger for non-awarded appli-cants than for those who receive an R01 award, and black non-awardees have significantlylower sum of impact factors than white non-awardees (p< .001). Fig 3 indicates that a higherpercentage of reported papers from black new and experienced applicants were uncited; alower percentage of reported papers from new and experienced black applicants were pub-lished in the top quartile and the top 10% of the field (measured by citations relative to cita-tions to papers in the journal subject category). Fig B in S1 File shows that black experiencedinvestigators have had one Type 2 award compared to two Type 2 awards for white experi-enced investigators.Counterfactual impact of publications on R01 awardsOur analysis has shown that publications explain about half of the black/white funding gap.Figs 2 and 3 (Fig C, Tables F and G in S1 File) show there are large and statistically significantdifferences in publications and associated bibliometrics. We now use our model to examinethe effect of a counterfactual policy experiment designed to simulate an increase in publicationquantity and quality. We used our models to predict what would happen if black investigatorsPublications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 12 / 24Table 4. Probit Estimates of NIH R01 Award comparing name-matched and Biosketch publications.Main Main + Name-matched Biosketch Biosketch Pubs &VARIABLES Covariates Publications Publications New MeasuresAsian -0.044� -0.019 -0.027 -0.016 -0.028[0.022] [0.023] [0.023] [0.041] [0.023]Black -0.134�� -0.110�� -0.103�� -0.088� -0.086��[0.020] [0.020] [0.021] [0.043] [0.021]Hispanic -0.039 -0.012 -0.010 -0.024 -0.008[0.022] [0.023] [0.023] [0.040] [0.023]NIH Fund Rank 1–30 0.143�� 0.128�� 0.142�� 0.119��[0.032] [0.032] [0.051] [0.032]NIH Fund Rank 31–100 0.071� 0.060� 0.094 0.053[0.030] [0.030] [0.051] [0.030]NIH Fund Rank 101–200 0.093� 0.087� 0.123� 0.082�[0.037] [0.037] [0.058] [0.036]Prior NIH Grants 0.064�� 0.052� 0.059 0.051�[0.021] [0.022] [0.042] [0.021]NIH Review Committee 0.102�� 0.098�� 0.082� 0.095��[0.019] [0.020] [0.035] [0.020]Grant includes Human -0.069�� -0.056�� -0.103�� -0.050��Subjects [0.018] [0.019] [0.034] [0.019]Log Publications -0.003 0.002[0.013] [0.024]Log Citations 0.006 0.029�[0.009] [0.013]Maximum Impact Factor 0.002 -0.000[0.001] [0.001]Median Impact Factor 0.007� 0.012��[0.003] [0.004]Percentage First Authored -0.041 0.086 0.098�Publications [0.035] [0.045] [0.045]Percentage Last Authored -0.030 -0.009 -0.030Publications [0.037] [0.044] [0.044]Percentage of Single-Authored -0.046 -0.077Publications [0.052] [0.093]Log Sum of Impact Factors 0.042��[0.011]Percentage of Uncited -0.003�Publications [0.001]Percent Publications Top Quartile of Field 0.001[0.001]Co-Authors have Publications 0.002��Top Quartile of Field [0.001]Additional Controls:Age X X X X XNativity X X X X XGender X X X X XApplication Year X X X X X(Continued)Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 13 / 24had the publication and bibliometric profiles as well as average Type 2 awards that matchedR01 awardees. Fig 4 shows the predicted probability of receiving an NIH R01 award condi-tional on having the average characteristics of the sample for all other variables except forbeing white (blue bar) and black (red bar). The difference between these two bars is the esti-mated effect of being black on the probability of receiving NIH funding in Table 1. The greenTable 4. (Continued)Main Main + Name-matched Biosketch Biosketch Pubs &VARIABLES Covariates Publications Publications New MeasuresPublication Field Missing X XNIH Institute (1) X X X XObservations 2,397 2,397 2,397 2,397 2,397Robust standard errors in brackets.�� p<0.001�� p<0.01� p<0.05.Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of Medical Colleges faculty roster, select NIH Biosketches, Web ofScience.1) Includes eight of the 27 institutes and centers that have a significant impact on probability of R01 Award.https://doi.org/10.1371/journal.pone.0205929.t004Fig 2. Average productivity by race, experience. Average publications, citations, coauthors, and sum of impact factors by race and investigatorexperience with 95% confidence intervals. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association ofMedical Colleges faculty roster, select NIH Biosketches, Web of Science. N = 2,397.https://doi.org/10.1371/journal.pone.0205929.g002Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 14 / 24bar is a counterfactual analysis for black investigators where the model is estimated at themeans of the sample for all variables except publications, bibliometrics, and in the case of expe-rienced investigators, Type 2 awards—these variables are assigned the means of R01 awardees(significantly higher for most measures according to Tables F and G and Figs B and C in S1File). In the full sample, an additional 57% (3 ppt) of the predicted black/white funding gap isexplained when the model is evaluated at the means of publications, bibliometrics and Type 2awards for R01 awardees (Fig 4). We tested whether the difference between the white awardprobability and the black counterfactual award probability was statistically significant andfailed to reject this hypothesis (p = .25). Providing the same average publications, biblio-metrics, and Type 2 awards as the average R01 awardee essentially closes the black/white fund-ing gap.We conducted the same thought experiment for new investigators, and the gap nearlycloses: 98% (4.2 ppt) of the predicted funding gap is explained by race/ethnicity differences inpublications and associated bibliometrics, and that gap was not significantly different fromzero (p = .94). For the experienced investigator sample, 36% (3 ppt) of the predicted black/white funding gap is explained by race/ethnicity differences in publications, bibliometrics, andType 2 awards, but the gap was not significant (p = .22). In all cases, the difference between thewhite and black investigators’ probability of R01 awards evaluated at the means of R01 award-ees are not significantly different. It is important to note that in the case of new andFig 3. Relative productivity by race, experience. Measures of uncited publications, field normalized publications, and coauthor publications ofNIH R01 applications by race/ethnicity, 2003–2006 with 95% confidence intervals. Source: NIH IMPAC II, National Science FoundationDoctoral Record File, American Association of Medical Colleges faculty roster, select NIH Biosketches, Web of Science. N = 2,397.https://doi.org/10.1371/journal.pone.0205929.g003Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 15 / 24experienced investigators, the confidence intervals are wide given the smaller sample size.However, our sample contains almost all of the black investigators that applied for fundingfrom 2003 to 2006.Our previous study found that proposals that were resubmitted were more likely to befunded and that black investigators were less likely to resubmit proposals [2]. We estimatedmodels that included controls for resubmitting proposals and evaluated the counterfactual pre-dicted probability for black investigators at the mean characteristics of R01 awardees for publi-cations and resubmissions (Fig C in S1 File). In the full sample and for new investigators, thepredicted probability of black investigator R01 awards evaluated at the mean characteristics ofR01 awardees exceeds that of white investigators. The predicted probabilities for white andblack counterfactual experienced investigators are virtually identical.Identifying career divergenceWe examined whether an applicant attended a top 100 NIH funded bachelor’s, doctoral andpostdoctoral institution as well as a top 100 NIH ranked employer. Fig 5 (Tables F and G in S1File) show that black investigators were less likely than white investigators to attend top bache-lor’s institutions but equally likely to attend top doctoral institutions. Black investigators wereFig 4. Predicted probability of R01 award. Predicted probability of NIH R01 awards by race and investigator experience with 95% confidenceintervals. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of Medical Colleges faculty roster,select NIH Biosketches, Web of Science. N = 2,397.https://doi.org/10.1371/journal.pone.0205929.g004Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 16 / 24significantly less likely (p< .001) to be employed at top 100 NIH funded institutions. Not hav-ing access to top bachelor’s programs does not seem to have disadvantaged black researchersat the PhD and postdoctoral stage. However, black investigators were less likely to be at topNIH funded employers.We examined differences in publications and citations by career stage as well in Fig 6 andTables F and G in S1 File. Although black and white investigators published the same numbersof papers (approximately 2.5) during their PhD and postdoc years, papers from black investi-gators are cited less than half as much (p< .001) as papers by other race/ethnicity groups.Tables F and G in S1 File indicate that only publications during the postdoc are significantlylower for black new investigator awardees (p< .05) compared to white awardees. However,average black publications and citations at all career stages are significantly lower for newinvestigator non-awardees (p< .001), and black publications at the principal investigator stageare lower for non-awardee experienced investigators (p< .001) as are citations during thepostdoc and principal investigator stage (p< .05, p< .001 respectively). This suggests thatcareers begin to diverge at the doctoral stage for new investigators and at the principal investi-gator stage for experienced investigators.Probability of receiving a priority scoreOur previous analysis indicated that the probability of receiving a priority score on grant appli-cations differed significantly by race/ethnicity. This is seen for the full sample, as well as fornew and experienced investigators (Table B in S1 File). In Table 5, we demonstrated the effectof dropping unscored applications for full sample and new and experienced investigators. Theblack/white funding gap falls by half for both new and experienced investigators and is neverstatistically significant. We considered whether our new bibliometric measures and the fullmodel could explain the black/white difference in receiving a priority score on a grant applica-tion in the last three columns of Table 4. Black investigators are -12.3 ppt (p< .001) less likelythan whites to receive a priority score in the full sample, a much larger difference than theresulting funding gap of -6.9 ppt (Table 1). This gap was largely being driven by new investiga-tors who were -11.7 ppt (p< .05) less likely to receive a score. Black experienced investigatorswere 10 ppt less likely to receive a score than whites, but this difference is not statistically sig-nificant at conventional levels (p = .11).DiscussionThis analysis has re-examined the black/white funding gap for NIH R01 awards identified inGinther et al. [2] using detailed data on academic rank, scholarly awards, prior grant activity,training, and reported publications collected from Biosketches combined with bibliometricinformation from Web of Science. We found that academic rank, scholarly awards, prior non-NIH grants, and training had little explanatory power in our models. However our improvedpublication and bibliometric measures explained half of the black/white funding gap, and oncethe analysis was separated by investigator experience, the gap was no longer statistically signifi-cant for new investigators, and the gap narrowed and became less significant for experiencedinvestigators. We also found that publications cut the black/white funding gap for MD/PhDs.We conducted a counterfactual analysis designed to simulate improved publication outcomesfor black investigators. Assuming this intervention increased black publication outcomes,results from the full sample indicated that assigning the average publications, bibliometrics,and Type 2 awards of R01 awardees to black investigators closed the black/white funding gapby nearly 60%, and it was no longer statistically significant. The same thought experimentshowed that practically all of the new investigator black/white funding gap could be closed ifPublications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 17 / 24black investigators were assigned the average productivity of R01 awardees, and over one-third of the experienced investigator gap could be closed. Adding resubmissions to the modelclosed the funding gap for the full sample.Why did black investigators report fewer papers that were cited less frequently on averagethan whites? And why did this gap widen with investigator experience? Our data suggestedthat cumulative advantage plays a role [29]. Although black investigators published the samenumber of papers during their PhD and postdoctoral training, these papers received fewercitations. This implies that black investigators may not receive the same advice from mentorsrelated to research topics and publication strategies as whites during training and at the begin-ning of their careers. These disadvantages appear to accumulate as the black/white publicationand citation gaps widen when black researchers become principal investigators (especially forinvestigators who were not funded). Furthermore, black researchers had fewer coauthors andfewer coauthors publishing in the top quartile of their research fields, a result identified else-where in the literature [30]. This indicates that black investigators had smaller professionalnetworks that may constrain their publication counts, citations, and research impact as well astheir prospects for promotion [30]. Finally, black publication records may be affected by insti-tutional pressure to provide more service and mentor more students in order to promote thediversity initiatives of their institutions overburdening black researchers with a form of “cul-tural taxation” [31, 32].Fig 5. Trained/Employed at top 100 NIH funded institutions by race. Race/ethnicity differences in training and employment at the top 100 NIHfunded institutions with 95% confidence intervals. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, AmericanAssociation of Medical Colleges faculty roster, select NIH Biosketches, Web of Science. N = 2,397.https://doi.org/10.1371/journal.pone.0205929.g005Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 18 / 24The black/white funding gaps for new and experienced investigators are intriguing andinstructive. By including more information from the Biosketch, the potential role of bias in thereview process appears to be diminished. When black new investigators have publication rec-ords similar to all new investigator R01 awardees, the model predicts that they are equallylikely to be funded as whites, and the gap for experienced investigators and the full samplecloses considerably (Fig 4, Fig C in S1 File). The new investigator model also indicates thatpublications, bibliometrics, and the funding rank of the employer act as positive signals of thenew investigator’s future research potential. Once a researcher becomes an experienced inves-tigator, these factors have less explanatory power as evidenced by the lack of statistical signifi-cance of these variables in the experienced investigator model. For experienced investigators,receiving a Type 2 R01 award significantly increases the probability of a new Type 1 R01award being funded. However, black experienced investigators receive one fewer Type 2awards (p< .01) than whites.These results have refined our understanding of the complex factors contributing to race/ethnicity differences in NIH R01 awards. We set out to distinguish between competing expla-nations: application merit, investigator characteristics, and unidentified factors that operateduring the peer review process, including bias. Our approach leaned heavily on controlling forrefined and expanded characteristics of the applicants in order to control for potential omittedvariable bias. We succeeded in showing that publications and associated bibliometricsFig 6. Average productivity by career stage, race, experienced investigators. Average publications and citations by career stage and race forexperienced investigators with 95% confidence intervals. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, AmericanAssociation of Medical Colleges faculty roster, select NIH Biosketches, Web of Science. N = 1,165.https://doi.org/10.1371/journal.pone.0205929.g006Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 19 / 24explained a substantial portion of race/ethnicity differences in funding. However, just becausewe can “explain” the gap does not mean that the problem is solved. Bias is notoriously difficultto identify in observational studies, with direct evidence of bias being found in audit studiesthat have come to different conclusions with respect to the role of gender in science careers[33, 34]. A recent study found that reviewers were not biased against NIH proposals fromresearchers with “Black names” [35]. However, this is a very restrictive test of bias since aname is only one factor associated with the ways that race is embedded in a researcher’s career.In fact, our previous study found that there was no funding gap for US citizen Asian investiga-tors, and the four percentage point Asian gap resulted from Asian, foreign-born investigators[2]. This was despite the fact that Asian investigators publish the same or more papers andhave a larger sum of impact factors than white investigators (Fig 2, Tables F and G in S1 File).It is possible that the peer review process may reflect bias against training from outside of theUS. In addition, we found that new investigators and experienced Asian investigators were sig-nificantly less likely to receive a score on their proposals after controlling for all of the covari-ates in the full model (Table 5). Thus, bias could still be operating in the review and fundingprocess given these remaining unexplained differences. Finally, bias could operate throughother pathways outside of peer review and could be a factor in the quality of postdoctoral andearly-career mentoring, or it could limit the size of professional networks that we observed.Using the publication process to identify that careers diverge by race/ethnicity starting asearly as the postdoc and then widen over time also is an important finding. These findings sug-gest opportunities for intervention. NIH announced pilot modifications to the Biosketch thatallow a narrative account of “up to five of their most significant contributions to science” [36].These changes may “level the Biosketch playing field,” perhaps reducing the weight that anindividual’s publications might play in the assessment of merit. In addition, efforts recentlyannounced by NIH to enhance the diversity of the scientific workforce, including the NationalResearch Mentoring Network, may bolster professional networks for researchers from under-represented groups [37]. A randomized trial of mentoring in the economics profession showedthat the mentoring treatment improved the number of publications, the quality of publica-tions, and the number of federal research grants for junior female economists [28]. MentoringTable 5. Estimates of race/ethnicity differences in NIH funding and the probability of receiving a score.Drop Unscored Probability of ScoreFull New Experienced Full New ExperiencedVARIABLES Model Investigators Investigators Model Investigators InvestigatorsAsian -0.007 -0.027 0.029 -0.034 0.051 -0.107�[0.041] [0.064] [0.054] [0.031] [0.047] [0.042]Black -0.047 -0.017 -0.052 -0.123�� -0.117� -0.104[0.045] [0.069] [0.064] [0.032] [0.043] [0.050]Hispanic -0.014 -0.022 -0.010 -0.004 0.005 0.010[0.040] [0.065] [0.051] [0.032] [0.045] [0.043]Observations 2,397 1,232 1,162 2,397 1,232 1,162Robust standard errors in brackets.�� p<0.001�� p<0.01� p<0.05.Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Association of Medical Colleges faculty roster, select NIH Biosketches, Web ofScience. All estimates use the full model described in Table A in S1 File.https://doi.org/10.1371/journal.pone.0205929.t005Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 20 / 24interventions should focus on conveying tacit knowledge related to the choice of research top-ics, grantsmanship, and the publication process for new investigators. In addition, institutionsmight consider lightening the service load for experienced black investigators so that the diver-sity needs of institutions do not derail their scientific research.Given the importance of submitting a successful R01 Type 2 proposal for subsequent R01Type 1 funding for experienced investigators, an investigation of the determinants of Type 2award funding as a function of publications and bibliometrics is warranted. Previous studieshave found mid-career researchers are disadvantaged [26] and that women are significantlyless likely to receive Type 2 awards [38, 39]. Our data indicated blacks, Asians, and Hispanicshad fewer Type 2 awards than whites. However, our data did not have the requisite informa-tion on Type 2 applications needed to probe these differences further. Although a recent studyfound no race/ethnicity differences in R01 awards after controlling for criterion and impactscores, their analysis did not model how race affected the decision to provide an overall impactscore [7]. We could not fully probe this decision because the application review processemployed during our study period is no longer in use at NIH. Under the current NIH reviewprocess [40], all applications are given an initial set of criterion scores ranging from 1 to 9 (bestto worst) on the approach, significance, investigator, innovation and environment. The appli-cations with the best criterion scores are “fully discussed” and then given an “overall impact”score. Although the “overall impact” score is not an aggregation of the criterion scores, there issignificant correlation [41]. The WGDBRW report examined race/ethnicity differences inoverall impact scores (for grants that received scores) and showed that blacks received slightlyworse impact scores (1.2 points higher) than whites where scores are measured on the 10–90point scale [9]. Considering that significant unexplained differences involve the score-no scoredecision, such analyses should be extended to preliminary criterion scores for applications thatdo not receive impact scores.Our results also have emphasized the importance of using more robust measures of publica-tions from the Biosketch to examine the determinants of R01 funding. Efforts such as ORCID[42] and Clarivate Analytics ResearcherID [43], which link researchers to their publications,and SciENCV [44], which creates structured Biosketch data, will facilitate any future analysisof the relationship between academic productivity and the probability of an application receiv-ing a score. This suggests that combining the current NIH peer review system with covariatesthat explain the black/white funding gap identified in this and our previous analysis will likelyyield insights about the factors associated with black/white differences in the probability ofreceiving an overall impact score and more explicitly inform opportunities for intervention.Supporting informationS1 File. Materials and methods appendix. This appendix describes the processes used toselect the sample used in the study, develop the data collection instrument, conduct data entryusing IMPAC II application images for the sample, and ensure the accuracy of the datathrough standardized data cleaning and quality assurances procedures. Fig A. Probability ofR01 Award 2003–2006. A comparison of R01 award probability by race/ethnicity, full samplefrom Ginther et al. [1] and subsample used in this analysis with 95% confidence intervals.Source: NIH IMPAC II, National Science Foundation Doctoral Record File, American Associ-ation of Medical Colleges faculty roster, select NIH Biosketches, Web of Science. Full SampleN = 53,454. Subsample N = 2,397. Table A. Variables Used in Academic Rank, Prior Grants,Scholarly Awards, Publications Models. Table A lists the variables used in models thatinclude academic rank, prior grants, scholarly awards, publications and associated biblio-metrics. Fig B. Average Number of Type 2 Awards by Race. Average number of Type 2 R01Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 21 / 24Awards for Experienced Investigators by race/ethnicity, 2003–2006 with 95% confidence inter-vals. Source: NIH IMPAC II, National Science Foundation Doctoral Record File, AmericanAssociation of Medical Colleges faculty roster, select NIH Biosketches, Web of Science.N = 2,397. Fig C. Predicted Probability of R01 Award—Resubmission Model. Predictedprobability of NIH R01 awards by race and investigator experience including controls for pro-posal resubmissions with 95% confidence intervals. Source: NIH IMPAC II, National ScienceFoundation Doctoral Record File, American Association of Medical Colleges faculty roster,select NIH Biosketches, Web of Science. N = 2,397. Table B. Variables Used in TrainingModels. Table B lists the variables used in models that include characteristics related to under-graduate institutions, predoctoral activities, PhD institution, postdoctoral appointments andfellowships. Table C. Probit Estimates of NIH R01 Award Controlling for Training Charac-teristics. Table C shows probit estimates of the effect of training variables on race/ethnicitydifferences in NIH awards.Table D. Contribution of Covariates to Explained Portion of theBlack NIH Funding Gap. Table D shows linear probability estimates of the percentage ofrace/ethnicity gap in NIH funding explained by covariates. Table E. Counts of Applicationsand Awards by Race and Investigator Status. Table E shows the counts of applications,scored applications and awards by race/ethnicity and investigator experience. Experiencedinvestigators are significantly more likely to receive scores and funding. Table F. Average ofProductivity Measures by Race/Ethnicity for New Investigators, NIH R01 Awardees andNon-awardees. Table F lists average productivity (publication and bibliometric) measures byrace/ethnicity, R01 award status and investigator experience. Regardless of award status, blackinvestigators publish fewer papers, are cited less often, and publish in lower impact journalsthan whites. Table G. Average of Productivity Measures by Race/Ethnicity for ExperiencedInvestigators, NIH R01 Awardees and Non-awardees. Table G lists average productivity(publication and bibliometric) measures by race/ethnicity, R01 award status and investigatorexperience. Regardless of award status, black investigators publish fewer papers, are cited lessoften, and publish in lower impact journals than whites.(DOCX)AcknowledgmentsWe thank Bernadette Gray-Little, Laurel Haak, Clarence Lang, Michael Lauer, Sally Rockey,Della Hann, Lawrence Tabak, Richard Nakamura, David Slusky, David Ribar, and HannahValentine for insightful comments. Pre-decisional grant application and personally identifiableinformation data covered in this manuscript cannot be made public by law. The federal law,the Privacy Act, precludes us from releasing personally identifiable information on grant appli-cants. Under the Freedom of Information Act ("FOIA"), 5 U.S.C. 552, individuals may submita formal request to obtain information on funded biomedical research grants not publiclyavailable. The government may withhold information pursuant to the nine exemptions andthree exclusions contained in the Act. Inquires may be directed to FOIA Coordinator, Officeof Extramural Research at email: OERFOIA@mail.nih.gov. For complete information on theFOIA submission process, visit the NIH FOIA website at http://www.nih.gov/icd/od/foia/index.htm.Author ContributionsConceptualization: Donna K. Ginther, Joshua Schnell, Raynard Kington, Walter T. Schaffer.Data curation: Jodi Basner, Unni Jensen, Joshua Schnell.Formal analysis: Donna K. Ginther.Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 22 / 24Funding acquisition: Walter T. Schaffer.Investigation: Donna K. Ginther, Jodi Basner, Walter T. Schaffer.Methodology: Donna K. Ginther.Project administration: Unni Jensen, Joshua Schnell.Resources: Raynard Kington, Walter T. Schaffer.Supervision: Unni Jensen.Writing – original draft: Donna K. Ginther, Jodi Basner, Raynard Kington, Walter T.Schaffer.Writing – review & editing: Donna K. Ginther, Unni Jensen, Joshua Schnell, Raynard King-ton, Walter T. Schaffer.References1. Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL, Kington R. Diversity in academic bio-medicine: An evaluation of education and career outcomes with implications for policy. 2009. Availablefrom: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=16779932. Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL, Kington R. Race, ethnicity, and NIHresearch awards. Science. 2011; 333:1015–19. https://doi.org/10.1126/science.1196783 PMID: 218524983. Ginther DK, Haak LL, Schaffer WT, Kington R. Race, ethnicity, and medical school affiliation associatedwith NIH R01 type 1 award probability for physician investigators? Acad Med. 2012; 87:1516–24.https://doi.org/10.1097/ACM.0b013e31826d726b PMID: 230183344. Ginther DK, Kahn S, Schaffer WT. Gender, race, ethnicity and NIH R01 research awards: Is there evi-dence of a double bind? Acad Med. 2016; 91:1098–107. https://doi.org/10.1097/ACM.0000000000001278 PMID: 273069695. Tabak L, Collins FS. Weaving a richer tapestry in biomedical science. Science. 2011; 333: 940–1.https://doi.org/10.1126/science.1211704 PMID: 218524766. NIH established the Working Group on Diversity in Biomedical Research Workforce, reporting to theAdvisory Committee to the Director (ACD), and chaired by Reed Tuckson, John Ruffin, and LawrenceTabak. Available from: http://acd.od.nih.gov/dbr.htm; accessed 2011 Feb 127. Eblen MK, Wagner RM, RoyChowdhury D, Patel KC, Pearson K. How criterion scores predict the over-all impact score and funding outcomes for National Institutes of Health peer-reviewed applications.PLoS ONE. 2016; 11(6):e0155060. https://doi.org/10.1371/journal.pone.0155060 PMID: 272490588. Scarpa T. Peer review at NIH. Science. 2006; 311:41. Available from: http://www.sciencemag.org/content/311/5757/41.full?ijkey=uRKtvtaIXUGN.&keytype=ref&siteid=sci https://doi.org/10.1126/science.1122796 PMID: 164001359. National Institutes of Health. Draft report of the Advisory Committee to the Director, Working Group onDiversity in the Biomedical Research Workforce. 2012 Jun 13. Available from: http://acd.od.nih.gov/Diversity%20in%20the%20Biomedical%20Research%20Workforce%20Report.pdf10. National Institutes of Health, IMPAC II (Information for Management, Planning, Analysis, and Coordina-tion), http://era.nih.gov/.11. Data sources and methods are described more fully in the supporting material.12. National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS),http://nces.ed.gov/ipeds/.13. The Carnegie Classification of Institutions of Higher Education, http://carnegieclassifications.iu.edu/.14. The Oberlin Group of Libraries, Oberlin Group Institution Members, http://www.oberlingroup.org/group-members.15. National Institutes of Health, Office of Extramural Research, https://researchtraining.nih.gov/.16. A complete description of the data for Ginther et al. [2] is available online at: http://www.sciencemag.org/content/333/6045/1015/suppl/DC1.17. Yang J, Vannier MW, Wang F, Deng Y, Ou F, Bennett J, et al. A bibliometric analysis of academic publi-cation and NIH funding. J. Informetr. 2013; 7:318–24.Publications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 23 / 2418. San Francisco Declaration on Research Assessment: Putting science into the assessment of research.2012. Available from: http://www.ascb.org/files/SFDeclarationFINAL.pdf19. Using bibliometrics in evaluating research. 2008. Available from: http://wokinfo.com/media/mtrp/UsingBibliometricsinEval_WP.pdf20. Eyre-Walker A, Stoletzki N. The assessment of science: The relative merits of post-publication review,the impact factor, and number of citations. PLoS Biol. 2013; 11(10):e1001675. Available from: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001675 PMID: 2411590821. Levin SG, Stephan PE. Age and research productivity of academic scientists. Res High Educ. 1989; 30(5):531–49.22. Stephan PE, Levin SG. Inequality in scientific performance: Adjustment for attribution and journalimpact. Soc Stud Sci. 1991; 21(2):351–68.23. Kaiser J. Zerhouni’s parting message: Make room for young scientists. Science. 2008; 322:834–35.https://doi.org/10.1126/science.322.5903.834 PMID: 1898881324. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-07-030.html25. Charette MF, Oh YS, Maric-Bilkan C, Scott LL, Wu CC, Eblen M, et al. Shifting demographics amongresearch project grant awardees at the National Heart, Lung, and Blood Institute (NHLBI). PLoS ONE.2016; 11(12):e0168511. Available from: https://doi.org/10.1371/journal.pone.0168511 PMID: 2797854426. We define new investigators as those that have never received an R01 or R01-equivalent award in oursample. https://grants.nih.gov/policy/early-investigators/index.htm.27. Blau FD, Currie JM, Croson RTA, Ginther DK. Can mentoring help female assistant professors? Interimresults from a randomized trial. Am Econ Rev., Papers and Proceedings. 2010; 100:348–52.28. Gelbach JB. When do covariates matter? And which ones, and how much? J. of Labor Economics.2016; 34(2 Pt 1):509–43.29. DiPrete TA, Eirich GM. Cumulative advantage as a mechanism for inequality: A review of theoreticaland empirical developments. Annu. Rev. Sociol. 2006; 32:271–97.30. Warner ET, Carapinha R, Weber GM, Hill EV, Reede JY. Faculty promotion and attrition: The impor-tance of coauthor network reach at an academic medical center. Journal of General Internal Medicine.2016; 31(1):60–67. https://doi.org/10.1007/s11606-015-3463-7 PMID: 2617354031. Mahoney MR, Wilson E, Odom KL, Flowers L, Adler SR. Minority faculty voices on diversity in academicmedicine: Perspectives from one school. Acad Med. 2008; 83(3): 781–86.32. Joseph TD, Hirshfield L. Why don’t you get somebody new to do it?: Race and cultural taxation in theacademy. Journal of Ethnic and Racial Studies. 2010; 34:121–4.33. Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J. Science faculty’s subtle gen-der biases favor male students. PNAS. 2012; 109:16474–79. https://doi.org/10.1073/pnas.1211286109PMID: 2298812634. Williams WM, Ceci SJ. National hiring experiments reveal 2:1 faculty preference for women on STEMtenure track. PNAS. 2015; 112:5360–65. https://doi.org/10.1073/pnas.1418878112 PMID: 2587027235. Forscher, P. S., Cox, W. T. L., Brauer, M., & Devine, P. G. (2018, May 28). No race or gender bias in arandomized experiment of NIH R01 grant reviews. https://doi.org/10.31234/osf.io/r2xvb.36. National Institutes for Health. Piloting Modified NIH Biosketches. Notice Number: NOT-OD-14-091.2014. Available from: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14-091.html37. National Institutes of Health. NIH awards $31 million to enhance diversity in the biomedical researchworkforce. 2014. Available from: http://www.nih.gov/news/health/oct2014/od-22.htm38. Ley TJ, Hamilton BH. The gender gap in NIH grant applications. Science. 2008; 322(5907):1472–74.https://doi.org/10.1126/science.1165878 PMID: 1905696139. Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, Pinn VW. Sex differences in application, success,and funding rates for NIH extramural programs. Acad Med. 2011: 86:759–67. https://doi.org/10.1097/ACM.0b013e31821836ff PMID: 2151235840. https://grants.nih.gov/grants/peer-review.htm41. Rockey S. National Institutes of Health, Office of Extramural Research. Correlation between overallimpact scores and criterion scores. 2011. Available from: http://nexus.od.nih.gov/all/2011/03/08/overall-impact-and-criterion-scores/42. http://orcid.org/43. http://www.researcherid.com/44. http://rbm.nih.gov/profile_project.htmPublications as predictors of racial and ethnic differences in NIH research awardsPLOS ONE | https://doi.org/10.1371/journal.pone.0205929 November 14, 2018 24 / 24

J Adolesc Health

Abstract

Extracted data

J Adolesc Health

Abstract

Extracted data

Related items

Related items