Publication counting methods for a national research evaluation exercise

ContentslistsavailableatScienceDirect

Journal

of

Informetrics

jou rn al h om ep a ge : w w w . e l s e v i e r . c o m / l o c a t e / j o i

Regular

article

Publication

counting

methods

for

a

national

research

evaluation

exercise

Przemyslaw

Korytkowski

_,

_Emanuel

_Kulczycki

a_{WestPomeranianUniversityofTechnology,Szczecin,Poland}

b_{ScholarlyCommunicationResearchGroup,AdamMickiewiczUniversityinPozna´n,Poland}

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received22December2018 Receivedinrevisedform31May2019 Accepted4July2019

Availableonline30July2019 Keywords: Evaluation Countingmethod Wholecounting Fractionalcounting Completecounting

a

b

s

t

r

a

c

t

Inthispaper,weinvestigatetheeffectsofusingfourmethodsofpublicationcounting (com-plete,whole,fractional,squarerootfractional)andlimitingthenumberofpublications(at researcherandinstitutionlevels)ontheresultsofanationalresearchevaluationexercise acrossfieldsusingPolishdata.Weusebibliographicinformationon0.58million publica-tionsfromthe2013–2016period.Ouranalysisrevealsthatthelargesteffectsareinthose fieldswithinwhichavarietypublicationandcooperationpatternscanbeobserved(e.g. inPhysicalsciencesorHistoryandarcheology).Wearguethatselectingthepublication countingmethodfornationalevaluationpurposesneedstotakeintoaccountthecurrent situationinthegivencountryintermsoftheexcellenceofresearchoutcomes,levelof internal,externalandinternationalcollaboration,andpublicationpatternsinthevarious fieldsofsciences.Ourfindingsshowthatthesocialsciencesandhumanitiesarenot signif-icantlyinfluencedbythedifferentpublicationcountingmethodsandlimitingthenumber ofpublicationsincludedintheevaluation,aspublicationpatternsinthesefieldsarequite differentfromthoseobservedintheso-calledhardsciences.Whendiscussingthegoals ofanynationalresearchevaluationsystem,weshouldbeawarethatthewaysof achiev-ingthesegoalsarecloselyrelatedtothepublicationcountingmethod,whichcanserveas incentivesforcertainpublicationpractices.

©2019ElsevierLtd.Allrightsreserved.

1. Introduction

Multi-authoredpublicationsarethekeyoutputwithinvariousassessmentsofnationalresearchproductivityandimpact (Huang,Lin,&Chen,2011;Zacharewicz,Lepori,Reale,&Jonkers,2018).Inongoingdiscussionsonpublicationcounting,the effectsofthedifferentmethodsareusuallyconsideredinrelationtocountryanduniversityrankingsbasedonpublications andcitations(Gauffriau,Larsen,Maye,Roulin-Perriard,&VonIns,2008;Hagen,2014;Waltmanetal.,2012).

Thetwo mostoftenusedpublicationcountingmethodsarewhole(full)countingandvariousvariantsoffractional counting(Larsen,2008;Waltman&vanEck,2015).Intheformermethod,eachentity(country/institution/author)getsfull creditforco-authoredpapers.Usingthelattermethodallowstoproportionallyfractionalizecreditacrossallthecontributing entities.VanHooydonk(1997)arguethatfractionalcountingcanberefinedintoproportionalcountingtocalculatearelative creditdependingontheauthor’srankonamultiauthoredpublication.However,thismethodisapplicableonlyinfieldsin whichtheorderofauthorlistisnotalphabetical.

∗ Correspondingauthor.

E-mailaddress:emek@amu.edu.pl(E.Kulczycki).

https://doi.org/10.1016/j.joi.2019.07.001

Gauffriauetal.(2008)showthatwholecountingisfavorabletocertaincountrieswithahighlevelofinternationalization. WaltmanandvanEck(2015)presentanempiricalanalysiswhichillustratesthatthebestchoiceistousefractionalcounting insteadoffullcountingbecauseonlythismethodallowsustogeneratethefield-normalizedresults.Aksnes,Schneider, andGunnarsson(2012)followthisconclusionandshowthatthedifferencebetweenwholeandfractionalizedcountsin rankingsbycitationindicatorsis greatestforthecountrieswiththehighestproportionofinternationallyco-authored articles.Gauffriau(2017)providesanoverviewofargumentsforcountingmethodsandshowsthatthereisnotoftenan explicitmotivationforchoosingaspecificmethod.

Moststudiesonpublicationcountingmethodsarebasedontheinternationalindexesofpublications,suchastheWeb ofScience(WoS)orScopus.However,publicationcountingisalsousedinperformance-basedresearchfundingsystems (PRFSs)whichcalculateavarietyofbibliometricindicatorstoproducerankingsacrossfieldsorinstitutionsinonecountry (Aagaard&Schneider,2015).InPRFSsystems,theyalsousepublicationscollectedinnationaldatabasesthatcovernotonly WoSorScopuslistingsbutalsoarticlesfromlocaljournalsandalltypesofscholarlybookpublications(S¯ıleetal.,2018).This iscrucialforthesocialsciencesandhumanities,especiallywheretheWoS/Scopuscoveragedegreeisverylow(Ossenblok, Engels,&Sivertsen,2012;Prins,Costas,VanLeeuwen,&Wouters,2016;Sivertsen&Larsen,2012)andwherescholarlybook publicationsplaythemajorrole(Kulczyckietal.,2018).OnthebasisoftheNorwegiandatabase,Piro,Aksnes,andRørstad (2013)showhowpublicationcountingmethodscanchangethepictureofaresearcher’sproductivityacrossdifferentfields. Whenwholecountingisused,researchersfromtheso-calledhardsciencesarefoundtobemoreproductivethatfromthe softsciences.Changingthemethodintofractionalcountingcompletelyreversesthispicture.

In thispaperweusetheterm‘publicationcountingmethodsfora nationalresearchevaluation exercise’(inshort: publicationcountingmethods)inabroadsensetocoverallaspectsofbibliographicdataintoascoreforafield(withinan institution)usedinaperformance-basedresearchfundingsystem.

Weidentifiedfivedimensionsofpublicationcountingmethodsforanationalresearchevaluationexercise:(1)unitof assessment,(2)countingmethod,(3)institutionlimit,(4)researcherlimit,and(5)pointscale.Thepublicationcounting methodisjustoneofmanyelementsofthebroaderchallengefacinganationalresearchevaluationsystem,whicharenot focusedonjustpublications.Nonetheless,publicationsarethemostimportantcriterion.

Thefirstdimensionreflectstheunitofassessment.Forinstance,inNorway,scientificinstitutionsareassessed,while Polandhaschangedtheirunitofassessmentfromwholeinstitutions,e.g.afacultycomprisingresearchersfromafewfields (the2013and2017evaluations)toseparatefieldswithininstitutions(the2021evaluation).Inthisway,thenewPolishunit ofassessmentissimilartothesolutionusedintheResearchExcellentFramework(REF)intheUnitedKingdom.

Theseconddimension,countingmethod,isrelatedtoafairapproachtowardsmulti-authoredpublications(Sivertsen, 2018).InNorway,squarerootfractionalcountingisused,whilePolanduseacombinationofthreemethods,i.e.whole counting,fractionalcounting,squarerootcounting.Sivertsen(2016),inadiscussiononredesigningpublicationcounting methodsinPRFSs,highlightsthatthereisaneedforamethodwhichwillallowforbalancingacrossdifferentfield-dependent co-authorshippractices.AccordingtoSivertsen,suchamethodissquarerootfractionalcounting.Havingasimilarpoint ofviewwithSivertsen,webelievealsothatsquare-rootcountingcanbeusefulnotonlyforbalancingresultsacrossfields butalsoforanalysescarriedoutforseparatefields.Allfieldshavesubfields,whichhavediversepublicationpractices,for instance,theoreticalphysicsandhighenergyphysicswithinthefieldofphysics.

Thethirddimensionconcernsaninstitutionlimit,i.e.limitingthenumberofsubmittedpublicationsfromasingleunitof assessment(institutionorfield,dependingontheadoptedevaluationmodel).ThenumberofpublicationsperFTEmostoften expressesthislimit.Forinstance,inNorway,allpublicationsofanevaluatedinstitutionareused,whereasinPolandinthe 2013and2017evaluations,onlyalimitednumberofpublicationswereincludedinthecalculation.Thislimitisexpressed bythe3N–2N0formula,whereNisthearithmeticmeanofthefull-timeequivalent(FTE)ofacademicstaffmemberswho

workinagivenscientificunitduringtheevaluatedfour-yearperiod,whileN0isthenumberofacademicstaffmembers

whowerenotauthorsofanypublicationduringtheperiodinquestion(Kulczycki,Korze ´n,&Korytkowski,2017).IntheUK ResearchExcellenceFramework2021,theaveragenumberofpublicationsrequiredperFTEintheunitofassessmentis2.5 (ResearchExcellenceFramework,2018).

Thefourthdimensionisrelatedtoaresearcherlimit,i.e.limitingthenumberofsubmittedpublicationsbyoneresearcher. Forinstance,thereisnoresearcherlimitinNorway.InPoland,accordingtoregulationsforthePolishevaluationin2021, themaximumnumberofpublicationsrequiredperFTEis4.InREF2014,oneresearchercouldsubmituptofourresearch outputswhereasinREF2021atleastoneandamaximumoffiveoutputs.

Thefifthdimensionconcernsthepointsassignedtothepublicationchannelsandthustopublications.Thepointscale canbelinearlikeinthePolishevaluationexercisein2017(institutionscouldobtainfrom1to50pointsforanarticle),or non-linearasinFinlandandNorway(1or3pointsperarticledependingonthelevelofthescientificpublicationchannel). TheNorwegianPublicationIndicatorusedwithintheNorwegianPRFSisapointsystemwhichcategorizesallpublications intotwolevelsandassignsdifferentnumbersofpointstothem(Aagaard,Bloch,&Schneider,2015).Thispointsystemis weightedintermsofbothlevelandpublicationtype(journalarticles,articlesinanthologies,andmonographs).Thepoint systemsarealsoimplementedamongothersinDenmark(Aagaard,2018),Finland(Pölönen,2018),Flanders(Engels&Guns, 2018),andPoland(Kulczyckietal.,2017).Ineachofthesecounties,variousmethodsofdistributingpointstoinstitutions whichhadcontributedtothepublicationmightbefound.

Inthisstudy,weconsiderpublicationcountingmethodsratherassciencepolicytoolsinresearchevaluation,thanas toolswhichservetodescribetheexistingcharacteristicsofresearchers’productivityandtheirpublicationpatterns.We

comparetheeffectsofusingfourmethodsofpublicationcountingandoflimitingthenumberofpublications,ontheresults ofaresearchevaluationexerciseusingPolishdata.

Weusedatafromthenationalevaluationexercisein2017inwhichresearchoutputsfromthe2013–2016periodofall universitieswereassessed.Weanalyzehowthechangeofpublicationcountingmethodinfluencestheresultsofrankings acrossthefields.Moreover,weinvestigatetheeffectsoflimitingthenumberofconsideredpublicationsintheevaluation exercise.

InPoland,theresultsofevaluationarenotdirectlytranslatedintofundingforinstitutions.Apartfrompublications,data concerningseveralotherparametersaregatheredforthepurposesoftheevaluationexercise(Kulczyckietal.,2017).These parametersareaggregatedintofourmaincriteria,whicharelaterweightedandsummed.Asaresult,thepositionofa scientificunitisdeterminedamongsimilarunitsintermsoffield.Basedonthepositionoftheunitintheranking,ascientific category(A+,A,BorC)isassignedbytheMinistry.Ultimately,thescientificcategorytranslatesintothesizeablockgrant fromtheMinistry.Theblockgrantsincaseofunitsfromuniversitiesisabout10%oftheirannualbudget,whileforbasicand appliedresearchinstitutesitisupto30%oftheirannualbudget.

Thisarticleaddstotheongoingdiscussionbyshowingtheeffectsofthedifferentpublicationcountingmethodsonthe resultsofanationalresearchevaluationsystemcalculatedattheleveloffields.Weareinterestedinexaminingwhich meth-odsofpublicationcountingfavortheso-calledhardandsoftsciences,andtheeffectsoflimitingthenumberofpublications requiredperresearcher.

Ourmainresearchquestionistwofold:howdodifferentpublicationcountingmethodsinfluencethefieldrankings,and howdoesaresearcherlimitchangetherankings?Thispaperpresentsanoriginalstudybasedonbibliographicaldatafrom thePolishnationalresearchevaluationsystem.Thankstothis,thestudyislimitedneithertodatafromtheWebofScience norfromScopus,fromwhichthecoveragedegreeisinsufficienttoevaluatethesocialsciencesandhumanities,especially fromanon-Englishspeakingcountry.Datafromanationalevaluationismorecomprehensiveandbalanced,i.e.dataincludes scholarlybookpublicationsandarticlesfromlocalscholarlyjournals.

Theusefulnessofthisarticleisthatweusethepointscale(from1to50)–withlittlemodification–thatisknowntoall Polishresearcherssince2008.Thisscalewasusedintheevaluationexercisesin2013andin2017(detailsaboutthepoint scalearepresentedinDatasetsection).Itisalsoquiteoftenusedinpromotionproceduresandintheperiodicassessmentof employees.Therefore,researchersarefamiliarwiththeframeworkandthedetailsofthepointscaleandresearchershave adapted–tosomeextent–theirownpublishingpracticestothisscale.Publicationchannelswithahighernumberofpoints arewidelyrecognizedasmoreprestigiousandthereforeareperceivedaschannelsinwhichitisworthtopublish.Thus, usingtheoneandwell-assimilatedpointscaleinallanalysesallowustocoherentlyassesstheconsequencesofthevarious countingmethodsformulti-authorpublicationsinanationalresearchevaluationsystem.

Therestofthispaperisorganizedasfollows.InSection2,wepresentthedata,methodsfocusingonhowthedatahave beenpreparedforanalysis,andwhatvariantsofpublicationcountingmethodwereused.InSection3,wepresenttheresults focusingontheeffectsofvariantsforfields.Then,inSection4,wediscussthemainfindings.InthefinalSection5wepresent conclusions.

2. Dataandmethods

2.1. Dataset

Inouranalysis,weuseadatasetfromthelastcycleofresearchevaluation inPolandconductedin2017.Scientific unitssubmittedbibliographicalrecordsof581,106publications,withtheFTEofacademicstaffmembersat86,461.84. Foreachevaluatedpublication,thegivenscientificunitobtainsaspecifiednumberofpoints,dependingonavarietyof factors,includingthetypeofpublicationchannelandthenumberofauthors.ArticlesfromjournalsindexedintheJournal CitationReportscouldobtainfrom15to50points(onthebasisofthefive-yearimpactfactornormalizedusingtheWeb ofSciencesubjectcategories).Articlesfromlocaljournalscanobtainfrom1to15points.Articlesfromjournalsindexedin theEuropeanReferenceIndexforHumanitiesfrom10to25points.Monographscouldobtain25pointsandachapterin monographsobtains5points.DetailedinformationonassessingpublicationsinthePolishsystemhasbeenpresentedin previouspublications(Kulczycki,2017;Kulczycki&Rozkosz,2017).

2.2. MappingpublicationstotheOECDfields

ThePolishevaluationwasconductedatthelevelofscientificinstitutions,andoriginallypublicationsareclassifiedtofields accordingtotheorganizationalclassification(Daraio&Glänzel,2016).Forthepurposeofthisanalysis,wehaveorganized allacademicstaffmembersintofieldsaccordingtodisciplines(fromthePolishclassification)declaredbythemforthe purposeofevaluation.WemappedthesedisciplinestothefieldsofscienceandtechnologyintheOrganisationforEconomic Co-operationandDevelopment(OrganisationforEconomicCo-OperationandDevelopment(2007)–tomakeouranalyses clearer.Usingoriginalclassificationwouldbevagueforreadersbecausetheorganizationalunitsweregroupedintothe JointEvaluationGroups(Kulczyckietal.,2017)builtacrossthefields.Moreover,Polandhaschangedfieldanddiscipline classificationtosomevariantofOECDFOSclassificationandreducedthenumberofdisciplinesto44.Allthesenuancesare notrelevantfortheanalysesresultsbuttheyshowrationalesforourdecisionofusingthemapping.

Table1

Characteristicsofeightvariants.

Variant Countingmethod Researcherlimit Slot Points

1 Wholecounting No 1 x

2 Completecounting No k x·k

3 Fractionalcounting No k/(k+m) x





4 Squarerootfractionalcounting No





5 Wholecounting 3 1 x

6 Completecounting 3 k x·k

7 Fractionalcounting 3 k/(k+m) x





8 Squarerootfractionalcounting 3





Finally,inthisanalysis,allpublicationspublishedbyresearchersclassifiedassociologistsarecountedaspublicationsfrom sociology.Allscholarsrepresentingartisticproductionwereexcludedfromtheanalysis,asintheoverwhelmingmajority ofcasestheysubmittedartworktotheevaluation.

PerformingthemappingrequiredexpertdecisionsduetothefactthatsomedisciplinesfromthePolishclassification canbeattributedtoseveralfieldsfromtheOECDclassification.Forexample,inthePolishclassification,computerscience isinmathematicalsciencesandtechnicalsciences,whereasintheOECDclassificationitisinnaturalsciences(thevast majorityofresearchers)andpartlyinelectricalengineering,electronicengineering,informationengineering,andmedia& communications.Forthisreason,inthecaseofseveraldisciplines,theresearchersrepresentingthesehavebeenentirely attributedtothedominantOECDfield.Itwasourexpertdecisionbasedonthedataofresearchfieldsandpublicationsof researchersfromagivendiscipline.

2.3. Variantsofpublicationcountingmethods

Inthisanalysis,weuseeightvariantsofpublicationcountingmethodsforthenationalresearchevaluationexercise. IntermsofthefivedimensionspresentedinSection1,eachvariantunitofassessment,institutionlimitandpointscale (Dimensions1,3,and5)oftheresearchevaluationsystemisinvariable,soweinvestigatethecountingmethod(Dimension 2)andresearcherlimit(Dimension4).

Werestricttheanalysistocaseswithaninstitutionlimitsetatthelevelof3FTE(3NinPolishterminology)duetothe availabledata.ThedataatourdisposaldoesnotcontainfullinformationaboutallscientificpublicationsofPolishresearchers. Duringtheanalyzedperiod,i.e.2013–2016,accordingtothenationalcurrentresearchinformationsystem(PolishScholarly Bibliography),Polishresearcherssubmittedaround1.03millionscientificoutputs,whiletotheevaluationexercise0.58 millionpublicationsweresubmittedforthesametimeperiod.InthePolishresearchevaluationonly3N–2N0publications

weretakenintoaccountwhendeterminingthefinalresult.Forthisreason,someofthescientificunitsforevaluationpurposes wereonlyapartofthetotalnumberoftheiremployees’publications,uploadedtofilltherequiredlimitbyacertainmargin. Intheanalysis,wecouldnotusedatafromthePolishScholarlyBibliographyduetoalackofinformationaboutassigned pointstopublications.Itshouldalsobehighlightedthatexperts,examiningwhetherpublicationswithintheinstitution limitmetthevariousformalrequirements,wereincontrolofthequalityofevaluationdata.

Intheanalysis,wetookintoaccountfourauthor-levelcountingmethods:whole(full)counting,completecounting, fractionalcounting,andfractionalcountingwithsquareroot.Weseparatedthesefourvariantsintocaseseitherwitha researcherpublicationlimit(3slotsperresearcher)orwithoutaresearcherpublicationlimit.Onepublicationoccupiesone slotinthecaseofwholecounting.Incaseofcompletecounting,onepublicationoccupieskslots,wherekisthenumberof authorsaffiliatedtotheinstitutionrepresentingthefield.Anattributiontokisbasedontheaffiliationinthepublication andthefieldselectedbythescholar.Inthecaseoffractionalcounting,onepublicationoccupiesk/(k+m)ofaslot,where misthenumberofotherauthorsrepresentingotherfield(s)fromthesameinstitutionorotherinstitution(s).Inthecaseof squarerootfractionalcounting,onepublicationoccupies



Thenumberofassignedpointsxisequaltothewholenumberofpointsattributedtoasingle-authorpublicationmultiplied bytheoccupiedslot(Table1).

Aswasdescribedabove,allscientificunitssubmittedatleast3N–2N0 publications.However,onlyafewinstitutions

providedinformationabouttheirallpublicationsfromthewholeperiod.

Intheanalysis,therefore,werestrictedthedatasettoonlythosefieldsinwhichwewereabletocollect3FTEpublications inVariant1afterchangingtheunitofanalysisfromorganizationunitstofields.Variantswithfractionalcountinganda researcherlimitrequiredmorepublicationstofilltheavailableslots,andinthecaseofsomefieldswithininstitutions,the publicationpoolwasnotlargeenough.

Inordertorankinstitutionswithinafield,weonlyusedpublicationcriterionC ∈ [0,50].Cshouldbeinterpretedasthe scientificpowerofafieldfromagiveninstitution,wherealowvaluemeansaweakscientificpowerandahighvaluemeans strongscientificpower.Avalueof50isachievableifall3FTEpublicationsarepublishedinthebestchannelsforwhich50

pointsareawardedaccordingtothePolishlist.CiscalculatedusingoneoftwoEqs.(1or2).Iftherewereenoughpublications tofillallavailableslots,i.e.



i si=3FTEthen C=



Iftherewerenotenoughpublicationstofillallavailableslots,i.e.



i si<3FTEthen C=









where_xiisthenumberofpointsassignedtoasingle-authorpublicationaccordingtorulesofthePolishresearchevaluation system2017forpublicationi,siistheslotoccupiedbypublicationi

si=

⎧

⎪

⎪

⎨

⎪

⎪

⎩



i xiistheminimumnumberofpointsassignedtoasingle-authorpublicationfromafieldfromaninstitution.

Thenumberofavailableslotsforafieldfromaninstitutionislimitedinallanalyzedvariants(theinstitutionlimit)to threetimesthenumberoffulltimeequivalents



i

si≤3FTE (3)

ForVariants5through8,weintroducedanadditionallimitofthreeslotsperresearcher(researcherlimit)



whereristheindexfortheresearcher.

Thepublicationswereorderedonthebasisofthenumberofpoints,andthentheslotswerefilleduntilthepublication listwasexhausted,e.g.becauseoftheresearcherlimit.Ifafterexhaustingthepoolofallpublicationstheslotswerenotfilled withpublications,weadoptedtheprinciplethatunoccupiedslotsarefilledwithvirtualpublicationswiththelowestscores fromthoseallocated.Asithasalreadybeenshown,Polishscientistspublishedtwiceasmanyworksaswerereportedfor researchevaluation.Thus,ourdecisionisjustifiedbytheobservationsfromthepreviousPolishresearchevaluationexercises in2013and2017,whereinstitutionshadintheirpoolsalargenumberofpublicationsnotincludedinthe3Nlimit,witha scoresimilartothatfromthecut-offpoint.

2.4. Fieldasaunitofanalysis

Inthisstudy,weanalyzedthedataattheleveloffields.Weassignedacodetoeachfieldwherethefirstdigitrepresented oneofthesixmajorOECDFields(1:Naturalsciences,2:EngineeringandTechnology,3:MedicalandHealthsciences,4: Agriculturalsciences,5:Socialsciences,6:humanities).

Table2presentseachfieldintermsofthetotalnumberofinstitutionsclassifiedinagivenfield(e.g.universities,research institutes)andtheFTEofacademicstaffmembersclassifiedinagivenfield.Afieldintheinstitutionissubjecttoresearch evaluationwhentheFTEisgreaterthan12,whichisinlinewithPolishlawonscienceandhighereducationfrom20th_July

2018.Thisanalysisisrestrictedtofieldswithatleast10institutionstoensureenoughunitsofanalysisforrankings.These twoprovisionsreducedtheanalyzeddatavolumedowntoalmost70,000FTE.

Finally,theresearcherswereclassifiedinto29fieldsrepresentingallsixmajorOECDfieldstoproduce875unitsof assessmentacross245institutions,andthenanalyzed.Aunitofassessmentisoneresearchfieldinoneinstitution.The largestfieldintermsofthenumberofinstitutionsisEconomicsandbusinesswith77institutions.Thelargestfieldinterms ofFTEisClinicalmedicinewithalmost8000FTE.ThesmallestfieldisEnvironmentalbiotechnologywithonly11institutions andabout280FTE.

Fig.1showsboxplotsoftheFTEofacademicstaffmembersacrossthefields.Themeansizeofaunitofassessment(a fieldwithinaninstitution)is78.96FTE,rangingfrom28FTEinEnvironmentalbiotechnologyto248FTEinClinicalmedicine. Themedianforallfieldsismuchlowerthantheaverageatonly46.76FTE.Thesinglebiggestunitofassessmentis971FTE. Thereare170unitsofassessmentwithlessthan20FTE,and48unitsofassessmentwithFTEhigherthan250.

Table2

Characteristicsoftheanalyzedfields.

Code Field Numberofinstitutions SumofFTE

11 Mathematics 29 1,167.56

12 Computerandinformationsciences 38 1,816.28

13 Physicalsciences 43 3,052.82

14 Chemicalsciences 44 3,024.92

15 Earthandrelatedenvironmentalsciences 23 1,956.02

16 Biologicalsciences 50 3,281.82

21 Civilengineering 33 2,898.06

22 Electricalengineering,electronicengineering,informationengineering 33 3,168.99

23 Mechanicalengineering 36 4,143.81 24 Chemicalengineering 15 824.6 25 Materialsengineering 33 1,744.26 27 Environmentalengineering 26 1,816.88 28 Environmentalbiotechnology 11 279.4 31 Basicmedicine 22 3,169.54 32 Clinicalmedicine 32 7,922.47 33 Healthsciences 40 2,684.24

41 Agriculture,forestry,andfisheries 27 3,194.44

42 Animalanddairyscience 11 742.31

51 Psychology 19 906.97

52 Economicsandbusiness 77 6,397.29

53 Educationalsciences 27 1,596.85

54 Sociology 29 1,439.47

55 Law 34 2431.44

56 PoliticalScience 36 1,553.48

58 Mediaandcommunications 12 339.5

61 Historyandarchaeology 29 1,904.36

62 Languagesandliterature 29 4,048.62

63 Philosophy,ethicsandreligion 25 1,388.99

64 Art(arts,historyofarts,performingarts,music) 12 543.81

Total 875 69,439.20

Note:FTE–full-timeequivalentofacademicstaffmembers.

Fig.1.Boxplotsoffull-timeequivalentofacademicstaffmembersacrossthefields.

2.5. Limitationsofthestudy

Fortheanalysisweincludedonlythosescientificunitsinwhichatleast12FTEacademicstaffmemberswereassignedto agivenfield.Ourdecisionwasinspiredbyanewregulationforthe2021Polishevaluation,accordingtowhichonlyscientific unitswith12FTEinagivenfieldwillbeevaluated.Moreover,weanalyzedonlythosefieldstowhichthereareatleast10 scientificunitsassigned.

Fig.2. Thecompletenessofpublicationdatadependingonthevariantofpublicationcountingmethods.

Fig.3.TheaveragenumberofpublicationsperresearcheracrossVariantsofpublicationcountingmethodsandOECDfieldsofscienceandtechnology.

InFig.2,wepresentthedatacompletenessforalltheanalyzedfieldsandvariants.Itshowshowmanyvirtualpublications hadtobeaddedinagivenvariantforagivenfield.Thelowestdatacompletenessisfortheso-calledhardsciencesinVariant 7(becauseoffractionalcountingandtheresearcherlimitofthreeslots),andalittlebettercompletenessforvariant8due tosquarerootfractionalcounting.

Thisincompletenessofthedatalimitsthevalidityoftheresults,however,aswehavewrittenabove,experiencefromthe twolastevaluationexercisesshowsthatinstitutionshavealargepoolofpublicationsnotreportedforevaluation.Thus,using virtualpublicationsisagoodproxy,whichshouldnotsignificantlyaffecttheresults.Duetoalackofnecessaryinformation tocarryouttheseanalyses(e.g.pointsassignedtothepublications),wecouldnotusedatafromanationallevelcurrent researchinformationsystem,i.e.thePolishScholarlyBibliography.

Duetolowdataqualityaboutthenumberofauthors,wehaveexcludedfromanalysisalleditedvolumesandmonographs withsuspiciouslylargenumbersofauthors(e.g.100editorsofaneditedvolume).Inmanycases,editorsweremixedwith chapterauthors.Thiswasaresultofimpreciseprovisionsofimplementingtheactandexplanationsinthedatacollection softwareforevaluationpurposes.

Weanalyzedhowmanypublications(sharesinpublicationsincaseofVariants5–8)havetobeprovidedforevaluation byaresearcherdependingontheselectedcountingmethod.Fig.3presentstheresultsofsuchananalysisbrokendownby fields.ForVariant1,itisalwaysthreepublications,asherewholecountingwith3FTElimitisused.Inthehardsciences,the numberofprovidedpublicationsinVariant2fallsbelowthreewhileinthesocialsciencesandhumanitiesitstayscloseto three.Thisshowsthatinsocialsciencesandhumanities,researchersworkmoreoftenworkaloneandpublishthehighest

Fig.4.FieldrankingsofinstitutionsthroughtheeightanalyzedvariantsforMaterialsengineering.

Fig.5.FieldrankingsofinstitutionsthroughtheeightanalyzedvariantsforPhilosophy,ethicsandreligion.

shareofsingle-authorpublicationsthanresearchersfromthehardsciences.Thevariantrequiringthegreatestnumberof requiredpublicationsisVariant7,justbeforeVariant8.ThisisespeciallyvisibleinPhysicalsciencesandinComputerand informationsciences.

3. Results

AccordingtothemethodologypresentedinSection2.3,foreachunitofassessmentwecalculatedthevalueofparameter C(Eqs.(1)and(2)),whichdeterminesthescientificpowerofthefield.Next,webuiltarankingofinstitutionsforeach fieldbasedonthevalueofparameterC.Werepeatedthisprocedureforalleightvariantsofpublicationcountingmethods presentedinTable1.

Figs.4–7presentschartswithfieldrankingsofinstitutionsthroughtheeightanalyzedvariants.Thesizeofthesign representstheunitsizeexpressedinFTE.Analogouschartsforall29analyzedfieldsareinAppendix1.Weuseranksinstead ofthetotalnumberofpointsbecauserankingsareeasiertoreadasscientificinstitutionsarespreadalloverthescale.In Appendix2weshowtwoplotswhichallowonetocomparethesetwotypesofresultspresentationontheexampleof Physicalsciences.

Fig.6. FieldrankingsofinstitutionsthroughtheeightanalyzedvariantsforPhysicalsciences.

Fig.7.FieldrankingsofinstitutionsthroughtheeightanalyzedvariantsforBasicmedicine.

Inthearticle,weincludedchartsforonlyfourfields:twoofthefields(i.e.Materialsengineering,andPhilosophy,ethics andreligion)inwhichthepositionsintherankingofinstitutionsonlyslightlychangebecauseofthecountingmethods.The nexttwochartspresenttheresultsforPhysicalsciencesandBasicmedicine.Inthesefieldsthepositionsintherankingof theinstitutionsstronglydependonthecountingmethod.

InFigs.4–7,itcanbenoticedthatthelargestinstitutionsfromthepointofviewofagivenfieldusuallyoccupythemiddle positionsintheranking.Atthetopoftherankingaremostoftenmedium-sizedinstitutions.Thebestinstitutionsfromthe fieldquiteoftenremainoneofthebestregardlessoftheanalyzedcountingvariant.Similarly,institutionsfromtheendofthe rankingremainthereregardlessofthecountingmethod.Thebiggestchangescanbeobservedinthemiddleoftheranking. Fig.8presentstheSpearman’srankcorrelationcoefficientforeachfield.Itcanbeseenthatforsomefieldstherankings areresistanttochangesinthecountingmethod.Alowcorrelationbetweentherankingsindicatesthattherearevarious publishingpracticeswithinthefield,inparticularthelengthofthelistofauthors.InPhysicalsciences,differentpractices arefoundinhigh-energyphysicswheregreatcollaborationsdominate(kilo-authorpublications)ratherthaninTheoretical physics(infourinstitutiontheaveragenumberofauthorsisabove1000perpublicationwhileinsixinstitutionsit’sless thanfourperpublication).Similarly,inonefield,historyandarcheologyareincluded,butthesetwosubfieldshavedifferent researchmethodsandcooperationpatterns(anaveragenumberofauthorsis1.28and2.22,respectively).Evenin mathe-maticsthechangeofthecountingmethodhasalargeimpactontheranking(anaveragenumberofauthorsperinstitution rangesfrom1.52to7.23).

Fig.9presentstheSpearman’srankcorrelationcoefficientsperapairofvariantsaggregatedforallfields.Onecanobserve thatingeneralatransitionfromwholecountingtocompletecounting(1–2and5–6)doesnotcausesubstantialchangesin

Fig.8.Spearman’srankcorrelationcoefficientforeachOECDfieldsofscienceandtechnology.

Fig.9. Spearman’srankcorrelationcoefficientsperapairofvariantsaggregatedforOECDfieldsofscienceandtechnology.

therankings.Thesamesituationcanbeobservedforatransitionfromfractionalcountingtosquarerootfractionalcounting (3–4and7–8).Moresignificantchangesoccurwhenaresearcherlimitisincluded(variants5–8).ThetransitionfromVariant 1–4hasahigherSpearmancorrelationthantothosewherearesearcherlimitispresent.Thechangeisevenbiggerwhen thecountingmethodisthesamebuttheresearcherlimitispresentorabsent(1–5).Variability,duetothetransitionamong variantswitharesearcherlimitimposed(Variants5–8),issignificantlylower(theSpearmancorrelationishigher)when comparingvariantswithoutaresearcherlimit(Variants1–4).

4. Discussion

Inthispaper,wediscussedhowpublicationcountingmethodsforanationalresearchevaluationexerciseinfluencethe rankingsacrossthefieldsofscience.Westudiedover0.5millionpublicationssubmittedbyPolishscientificinstitutionsand analyzedeightvariants.

Ourstudyrevealsthatthelargestdifferencesareinthosefieldswithinwhichvariouspublicationandcooperationpatterns (e.g.thenumberofauthors)canbeobserved.Forinstance,thesubstantialeffectsobservedinHistoryandarcheologyand inPhysicsshowthatselectingthepublicationcountingmethodshouldbebasedonthepropergranularityofthefields. Determiningtheproperdetailedgranularityistosomeextentamatterofmerit,astheresearchevaluationsystemisa

sciencepolicyinstrument,whichservesnotonlyforassessinghighereducationinstitutionsorfieldswithinthem,butalso functionsasasetofincentivestoinfluenceresearchers’publicationpractices.

Wepositionourresearchalsowithinresearchevaluationstudiesbecauseineveryevaluationexercisesomecounting methods(explicitlyorimplicitly)isused.Inthispaper,weanalyzecountingmethodstogetherwithfourotherdimensions, thatistheunitofassessment,theinstitutionlimit,theresearcherlimit,andthepointscale.Therefore,webelievethatour resultsmighthaveimplicationsalsooutsideofevaluationandfundingregimes,forexample,universityrankings.

Thesubstantialeffectsofthedifferentcountingmethodscanbeclearlyobservedinsomefields(e.g.Clinicalmedicine) whileotherfieldsarenotsosensitive.Ourfindingsshowthatsocialsciencesandhumanitiesarenotsignificantlyinfluenced bychangesinpublicationcountingmethodbecausepublicationpatternsin thosefieldsarequitedifferentfromthose observedinhardsciences.

Ourobservationsanddiscussionswiththeacademiccommunityonthistopicleadustoaconclusionthatsocialscientists andhumanitiesperceivewholecountingandcompletecountingasanunfairwayofassessingpublicationwithinthenational system.Onecansaythatevaluationisconductedattheleveloffields,andresearchersfromonefieldarecomparedonlywith researchersfromthesamefield.However,researchersfromdifferentfieldscomparethemselveswitheachotherbecause theyworkinthesamehighereducationandsciencesystem.Moreover,publicopinionandsocietytreatallresearchersas representativesofthesamegroup.

Belowwediscusseachofthefivedimensionsoftheresearchevaluationsystemrelatedtopublicationcountingmethods.

Dimension1:Unitofassessment

Anorganizationalunitwithinaninstitution(e.g.faculty,researchinstitute,otherhighereducationinstitution)orafield (discipline)canbeaunitofassessment withinthenationalresearchevaluationsystem.Inevaluatinginstitutions,itis importanttoassesshomogeneousunits,i.e.toassessandcompare,forinstance,afacultyofhistorywithanotherfaculty ofhistory.Whenorganizationalunitsareheterogenous(researchersrepresentvariousfields),thenonefield–favoredby thepublicationcountingmethod–candominate(intermsofobtainedpoints)withinthisinstitution.Actually,thesame situationexistswithevaluatingthefield.Whenweconstituteaunitofassessmentasafieldandatthesametimeaggregatethe differentfields(intermsofdifferentpublicationpatterns)intoasinglefield,thenwehaveheterogenousunitsofassessment. SuchasituationcanbeobservedinourresultsinPhysicalsciencesandinHistoryandarcheology.

Dimension2:Countingmethod

Fourresearcher-levelcountingmethodsfavorvariouspublicationpatternsandbehaviors.Belowwediscusseachmethod andarguewhatisfavoredandwhatisunderestimatedbyagivenmethod.

Wholecounting:thismethodfavorsanytypeofinternalorexternalcooperationregardlessofthecontributionfroma givenunitofassessment.Wholecountingunderestimatesthegivenunitofassessmentwhenitplaysakeyroleinthe publications.

Completecounting:thismethodfavorsaunitofassessmentfromwhichtherearemanyauthorsofagivenpublication, whichcanreflectthecontributionofthisunit.Completecountingunderestimatessingle-authoredpublications.Thereis asubstantialdifferencetobetheonlyauthorofamonographandtobeoneoffourauthorsofamonograph.Therefore, completecountingisnotabalancedmethodforassessingdifferentfieldswithinonenationalsystemfromtheperspective offieldsinwhichsingle-authoredpublicationsconstitutethemajorityoftotalvolume.Completecountingrequiresreporting fewerpublicationsfortheevaluationthanwholecountingwhenaninstitutionlimitisused(seeFig.9).

Fractionalcounting:thismethodfavorssingle-authoredpublicationsandahighshareofcontribution(intermsofthe numberofauthors)inpublications.However,thisbonusisnotassignificantasincompletecounting.Fractionalcounting underestimateswidecooperation-networkseventhoughparticipatinginmanyjointstudiesandprojectsrequiresa signif-icantworkloadandusuallyistheresultofeffectivenetworking.Fractionalcountingrequiresreporting1.5–2timesmore publicationsfortheevaluationthanwholecountingwhenaninstitutionlimitisused.Moreover,researchersfromsocial sciencesandhumanitiesperceivefractionalcountingasamorebalancedandfairwayofpublicationcountingbecausea single-authoredmonographisnotequaltoonemulti-authoredarticle.

Squarerootfractionalcounting:thisvariantoffractionalcountingisusedtomitigatetheconsequencesoffractional countinginordertogivemorecredittounitsofassessmentwithawidecooperation-network.Atthesametime,squareroot fractionalcountingcanbeperceivedasabalancedandfairwayofpublicationcountingbyresearchersfromsocialsciences andhumanities.Thismethodrequiresafewlesspublicationsfortheevaluationthanfractionalcountingwhenaninstitution limitisused.

Dimension3:Institutionlimit

Thislimitisapragmaticwayofshowingthatthequalityofpublicationsismoreimportantthequantityofresearchoutputs. ThiscommonsenseintuitionmightbeconnectedwithBradford’slawofscatteringoraParetodistribution,whichshowthat onlysomepartofresearchoutputisimportant.Inotherwords:onlyarticlespublishedincorejournalsormonographs publishedbythemostimportantpublishersshouldbereportedforevaluation.Fromtheoperationalpointofview,imposing aninstitutionlimitisadvantageousbecauseitlimitstheburdenrelatedtotheacquisitionandverificationofmetadataabout publications.

Dimension4:Researcherlimit

Whenaresearcherlimitisnotused,top-performingresearchersarefavoredintheevaluation.Thankstothis,oneunit ofassessment(e.g.afacultyorafieldwithinaninstitution)canbeassessedveryhighlywhilethisresultisproducedbya

fewtop-performingresearchersandotheracademicstaff-membersprovideaverysmallshareofevaluatedpublications. Moreover,suchtop-performingresearchersmaynotevenhaveco-workersintheirinstitutions.

Usingtheresearcherlimitcausesall(oralmostall)academic-staffmembertoneedtoprovidesomepublicationsfor theevaluation.Top-performingresearchersmightperceivethislimitasaninstrument,whichdepreciatestheirvaluefor institutions.Atthesametime,forsuchresearchers,thislimitcanencourageatop-performingresearchertoenlargethe researchgroupsintheirinstitutions.

Imposingaresearcherlimithasanimpactonthecountingmethod,makingitlessimportantwiththeresearcherlimit.In Fig.8,SpearmancorrelationsacrossVariants5–8arehigh,whichsignifiesthattheorderingoftheunitsofanalysisissimilar. Aresearcherlimitdoesnothaveastrongimpactonthenumberofreportedpublications.

Dimension5:Pointscale

Pointsattachedtocertainpublicationchannelsinformresearcherswhatchannelsarepreferredfromasciencepolicypoint ofview.Overtime,thereshouldbemorepublicationsinchannelswithahigherrating.Researcherspaymoreattentionto thresholdsandtheirrelativedifferencemeasuredinpointsthantothewidthofthescale,from0to1pointorfrom0to 100points.Forinstance,inthePolishsystem,aconcomitanceofthelinearpointscalewithaninstitutionlimitcauseda differencebetweenthresholdsintermsofpoints(e.g.adifferencebetween10-pointpublications11-pointpublications) whichcouldbeverysubstantial.InNorway,fromtheperspectiveofagiveninstitution,three1-pointpublicationscouldbe equivalenttoone3-pointpublication,whereasinPolandtherewasastrictcut-pointexpressedbytheinstitutionlimitof publicationthatcouldbeassessed.

5. Conclusion

Ourpapershowsthatselectingthepublicationcountingmethodfornationalevaluationpurposesneedstotakeinto accountthecurrentsituationinthegivencountryintermsoftheexcellenceofresearchoutcomes,levelofinternal,external andinternationalcollaboration,andpublicationpatternsinthevariousfieldsofscience.Wehaveshownhowdifferent variantsofpublicationcountingmethodsinfluencetherankings.Wecouldconstructothervariants,butitwillnotmakeour task,i.e.selectingtheproperwayofcounting,anyeasier,becausethereisnoexternalandobjectivereferencepoint.

Indiscussing thegoalsofanynationalresearchevaluationsystem,weshouldbeawarethatthewaysof achieving thesegoalsarecloselyrelatedtopublicationcountingmethods.Forinstance,ifourgoalistoappreciatetop-performing researchersintheevaluation,weshouldnotimplementaresearcherlimit.Ifourgoalistoincreasethelevelofinternational collaboration,weshoulduseratherthesquarerootfractionalcountingthancompleteorwholecounting.Therefore,onecan assesswhetherapublicationcountingmethodwasproperlyselected,notbylookingintothefieldrankingsintheevaluation results,butratherbylookingintotheindicators(showinghowincentivesactuallywork),whichreflectthegoalsofthe researchevaluationsystem.Publicationcountingmethodshaveahigherimpactonhardsciencesthanonsocialsciences andhumanities.

Funding

ThisworkwassupportedbytheDIALOGProgram(Grantname“ResearchintoExcellencePatternsinScienceandArt”) financedbytheMinistryofScienceandHigherEducationinPoland.

Authorcontributions

PrzemysławKorytkowski:Conceivedanddesignedtheanalysis,Performedtheanalysis,Wrotethepaper.

EmanuelKulczycki:Conceivedanddesignedtheanalysis,Wrotethepaper.

AppendixA. Supplementarydata

Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10. 1016/j.joi.2019.07.001.

References

Aagaard,K.(2018).Performance-basedresearchfundinginDenmark:Theadoptionandtranslationofthenorwegianmodel.JournalofDataand InformationScience,3(4),19–29.http://dx.doi.org/10.2478/jdis-2018-0018

Aagaard,K.,Bloch,C.,&Schneider,J.W.(2015).Impactsofperformance-basedresearchfundingsystems:ThecaseoftheNorwegianPublicationIndicator. ResearchEvaluation,24(2),106–117.http://dx.doi.org/10.1093/reseval/rvv003

Aagaard,K.,&Schneider,J.W.(2015).Researchfundingandnationalacademicperformance:ExaminationofaDanishsuccessstory.Science&Public Policy,1–14.http://dx.doi.org/10.1093/scipol/scv058

Aksnes,D.W.,Schneider,J.W.,&Gunnarsson,M.(2012).Rankingnationalresearchsystemsbycitationindicators:Acomparativeanalysisusingwhole andfractionalisedcountingmethods.JournalofInformetrics,6(1),36–43.http://dx.doi.org/10.1016/j.joi.2011.08.002

Daraio,C.,&Glänzel,W.(2016).Grandchallengesindataintegration—Stateoftheartandfutureperspectives:Anintroduction.Scientometrics,108(1), 391–400.http://dx.doi.org/10.1007/s11192-016-1914-5

Engels,T.C.E.,&Guns,R.(2018).Theflemishperformance-basedresearchfundingsystem:Auniquevariantofthenorwegianmodel.JournalofDataand InformationScience,3(4),45–60.http://dx.doi.org/10.2478/jdis-2018-0020

Gauffriau,M.(2017).Acategorizationofargumentsforcountingmethodsforpublicationandcitationindicators.JournalofInformetrics,11(3),672–684.

http://dx.doi.org/10.1016/j.joi.2017.05.009

Gauffriau,M.,Larsen,P.O.,Maye,I.,Roulin-Perriard,A.,&VonIns,M.(2008).Comparisonsofresultsofpublicationcountingusingdifferentmethods. Scientometrics,77(1),147–176.http://dx.doi.org/10.1007/s11192-007-1934-2

Hagen,N.T.(2014).Countingandcomparingpublicationoutputwithandwithoutequalizingandinflationarybias.JournalofInformetrics,8(2),310–317.

http://dx.doi.org/10.1016/j.joi.2014.01.003

VanHooydonk,G.(1997).Fractionalcountingofmultiauthoredpublications:Consequencesfortheimpactofauthors.JournaloftheAmericanSocietyfor InformationScience,48(10),944–945.http://dx.doi.org/10.1163/000000086790082109

Huang,M.-H.,Lin,C.-S.,&Chen,D.-Z.(2011).Countingmethods,countryrankchanges,andcountinginflationintheassessmentofnationalresearch productivityandimpact.JournaloftheAmericanSocietyforInformationScienceandTechnology,62(12),2427–2436.http://dx.doi.org/10.1002/asi.21625

Kulczycki,E.(2017).Assessingpublicationsthroughabibliometricindicator:ThecaseofcomprehensiveevaluationofscientificunitsinPoland.Research Evaluation,26(1),41–52.http://dx.doi.org/10.1093/reseval/rvw023

Kulczycki,E.,Engels,T.C.E.,Pölönen,J.,Bruun,K.,Duˇsková,M.,Guns,R.,...&Zuccala,A.(2018).Publicationpatternsinthesocialsciencesand humanities:EvidencefromeightEuropeancountries.Scientometrics,http://dx.doi.org/10.1007/s11192-018-2711-0

Kulczycki,E.,Korze ´n,M.,&Korytkowski,P.(2017).Towardanexcellence-basedresearchfundingsystem:EvidencefromPoland.JournalofInformetrics, 11(1),282–298.http://dx.doi.org/10.1016/j.joi.2017.01.001

Kulczycki,E.,&Rozkosz,E.A.(2017).Doesanexpert-basedevaluationallowustogobeyondtheImpactFactor?Experiencesfrombuildingarankingof nationaljournalsinPoland.Scientometrics,111(1),417–442.http://dx.doi.org/10.1007/s11192-017-2261-x

Larsen,P.O.(2008).Thestateoftheartinpublicationcounting.Scientometrics,77(2),235–251.http://dx.doi.org/10.1007/s11192-007-1991-6

OrganisationforEconomicCo-OperationandDevelopment.(2007).Revisedfieldofscienceandtechnology(FOS)classificationinthefrascatimanual DSTI/EAS/STP/NESTI(2006)19/FINALRetrievedfrom.http://www.oecd.org/science/inno/38235147.pdf

Ossenblok,T.L.B.,Engels,T.C.E.,&Sivertsen,G.(2012).TherepresentationofthesocialsciencesandhumanitiesintheWebofScience:Acomparisonof publicationpatternsandincentivestructuresinFlandersandNorway(2005-9).ResearchEvaluation,21(4),280–290.

http://dx.doi.org/10.1093/reseval/rvs019

Piro,F.N.,Aksnes,D.W.,&Rørstad,K.(2013).Amacroanalysisofproductivitydifferencesacrossfields:Challengesinthemeasurementofscientific publishing.JournaloftheAmericanSocietyforInformationScienceandTechnology,64(2),307–320.http://dx.doi.org/10.1002/asi.22746

Pölönen,J.(2018).Applicationsof,andexperienceswith,theNorwegianmodelinFinland.JournalofDataandInformationScience,3(4),31–44.

http://dx.doi.org/10.2478/jdis-2018-0019

Prins,A.A.M.,Costas,R.,VanLeeuwen,T.N.,&Wouters,P.F.(2016).Usinggooglescholarinresearchevaluationofhumanitiesandsocialscience programs:Acomparisonwithwebofsciencedata.ResearchEvaluation,25(3),264–270.http://dx.doi.org/10.1093/reseval/rvv049

ResearchExcellenceFramework.(2018).REF2021Decisionsonstaffandoutputs(April).

S¯ıle,L.,Pölönen,J.,Sivertsen,G.,Guns,R.,Engels,T.C.E.E.,Arefiev,P.,...&Teitelbaum,R.(2018).Comprehensivenessofnationalbibliographicdatabases forsocialsciencesandhumanities:FindingsfromaEuropeansurvey.ResearchEvaluation,1–13.http://dx.doi.org/10.1093/reseval/rvy016(June) Sivertsen,G.(2016).Abibliometricindicatorwithabalancedrepresentationofallfields.InInternationalConferenceonScienceandTechnologyIndicators.

pp.910–914.(September).

Sivertsen,G.(2018).TheNorwegianmodelinNorway.JournalofDataandInformationScience,3(4),3–19.http://dx.doi.org/10.2478/jdis-2018-0017

Sivertsen,G.,&Larsen,B.(2012).Comprehensivebibliographiccoverageofthesocialsciencesandhumanitiesinacitationindex:Anempiricalanalysisof thepotential.Scientometrics,91(2),567–575.http://dx.doi.org/10.1007/s11192-011-0615-3

Waltman,L.,Calero-Medina,C.,Kosten,J.,Noyons,E.C.M.,Tijssen,R.J.W.,vanEck,N.J.,...&Wouters,P.(2012).TheLeidenranking2011/2012:Data collection,indicators,andinterpretation.JournaloftheAmericanSocietyforInformationScienceandTechnology,63(12),2419–2432.

http://dx.doi.org/10.1002/asi.22708

Waltman,L.,&vanEck,N.J.(2015).Field-normalizedcitationimpactindicatorsandthechoiceofanappropriatecountingmethod.JournalofInformetrics, 9(4),872–894.http://dx.doi.org/10.1016/j.joi.2015.08.001

Zacharewicz,T.,Lepori,B.,Reale,E.,&Jonkers,K.(2018).Performance-basedresearchfundinginEUmemberstates:Acomparativeassessment.Science& PublicPolicy,1–11.http://dx.doi.org/10.1093/scipol/scy041(June)