https://openaccess.leidenuniv.nl
License: Article 25fa pilot End User Agreement
This publication is distributed under the terms of Article 25fa of the Dutch Copyright Act (Auteurswet) with explicit consent by the author. Dutch law entitles the maker of a short scientific work funded either wholly or partially by Dutch public funds to make that work publicly available for no consideration following a reasonable period of time after the work was first published, provided that clear reference is made to the source of the first publication of the work.
This publication is distributed under The Association of Universities in the Netherlands (VSNU) ‘Article 25fa implementation’ pilot project. In this pilot research outputs of researchers employed by Dutch Universities that comply with the legal requirements of Article 25fa of the Dutch Copyright Act are distributed online and free of cost or other barriers in institutional repositories. Research outputs are distributed six months after their first online publication in the original published version and with proper attribution to the source of the original publication.
You are permitted to download and use the publication for personal purposes. All rights remain with the author(s) and/or copyrights owner(s) of this work. Any use of the publication other than authorised under this licence or copyright law is prohibited.
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please contact the Library through email:
OpenAccess@library.leidenuniv.nl
Article details
Wouters P. (2018), The failure of a paradigm Henk F. Moed. Applied Evaluative
Informetrics, Springer International Publishing (2017) Bespreking van: Moed H.F. (2017) Applied Evaluative Informetrics, Qualitative and Quantitative Analysis of Scientific and Scholarly Communication, Journal of Informetrics 12(2): 534-540.
Doi: 10.1016/j.joi.2018.03.002
ContentslistsavailableatScienceDirect
Journal of Informetrics
jou rn al h om ep a ge : w w w . e l s e v i e r . c o m / l o c a t e / j o i
Book Review
Thefailureofaparadigm
HenkF.Moed.AppliedEvaluativeInformetrics,SpringerInternationalPublishing(2017) 1. Introduction
AppliedEvaluativeInformetricsisanambitiousbook.ThelatestworkofHenkMoed,aprize-winningbibliometricianwho workedbetween1986and2010attheCentreforScienceandTechnologyStudies,aimstogiveasweepingoverviewof allaspectsoftheapplicationofinformetricsintheevaluationofscientificandscholarlyresearch.Thebookisintended fornon-specialistscholarsfromalldomainsofresearchaswellasforallwhoareinterestedinresearchevaluation:senior researchers,students,researchmanagers,fundersandsciencepolicyactors.
Thebookconsistsofsixparts.Thefirstfourpartscoverpast,presentandfutureofevaluativeinformetrics,whilethe lasttwopartsconsistoflecturesandarticlesbytheauthor.Inthefirstfourpartsthechapterspresentoriginalarguments incombinationwithexcerptsandpresentationsofresultsandinsightsfromolderarticles.Thebookismoreasynthesis ofthestateoftheartinevaluativeinformetricsthananovelanalysis.Butitismoreambitiousthanjustareviewinbook format.Itisalsoframedasaninterventioninthecurrentdebateaboutresearchevaluationandtheroleofindicators.Moed triestomakethecaseforapoliticallyneutralformofinformetrics,therebyhopingtoclarifythecurrentdiscoursewhichhe apparentlyfindsoftenconfusedandsometimesoverlypolitical.
2. Broadoverview
Letmestartwithpraisingtheauthorforhisendeavor.Notmanybibliometricianshavebothsuchabroadoverviewof thefieldandsuchafirmgroundinginthepracticeofresearchevaluation.Mostpractitionersinourfieldlimitthemselvesto theirspecificareaanddonotstartsuchanambitiousproject.So,theideatowritethisbook,whichcanalsobeseenasan extensionofhisearlierbookCitationAnalysisinResearchEvaluationfrom2005(Moed,2005),iscommendable.
Readerswhoarenotthemselvesscientometriciansorinformetricianswillfindvaluablereferencestheymaywishto followup.PolicyofficialsandresearchmanagerspressedfortimewillfindPartIespeciallyuseful.Itgivesashortintroduction tothefieldofinformetricsandbibliometricswhilstalsopresentinganextensivesynopsis(inChapter2)ofallfollowing chapters.Indeed,formanyusersofevaluativeinformetricsthesefirst41pageswillbesufficienttounderstandthekey argumentsandreferences.Itisalsoagoodstartingpointforthoseinneedofinformationaboutparticulartopicswithout wishingtocoverthefieldasawhole.
PartsII,IIIandIVcontainthekeyargumentsofthebook.PartIIdiscussesindicatorsofresearchperformance,PartIII zoomsinontheapplicationcontext,andPartIVpresentsMoed’sideasaboutthewayforward.Eachpartconsistsofseveral relativelyshortchapters(mostarebetween5and20pages).Thebookhasawidescope;Icouldnotthinkoftopicsthatwere notmentionedatleastonceinthebook.Perhapsasaconsequenceofthissweepingexcursion,thebookdoesnotdigdeep intotheissues.Readerswithatasteformorein-depthanalysisordiscussionmaybedisappointediftheydonotfindmore thanwhattheyseeasobviousorcommonknowledge.Thisraisesthequestionwhethertheauthorwouldnothaveserved hisaudiencebetterbyfocusingthebookmoreonthemostburningquestions.Ofcourse,thismayalsobeamatteroftaste.
3. Performanceindicators
PartIIaboutindicatorsofresearchperformancehasthreechapters.Thefirst(Chapter3)introducesthedifferentdimen- sionsofresearchperformance.Itdistinguishescomponentsoftheresearchprocess(input,output,process,andimpact)and presentsatableof28importantinformetricindicatorsandliststheirdefinitions,theirstrongpointsaswellastheirlimits.
Theindicatorslistedvaryfromverygeneralindicators(suchas“citations”or“measuresofeconomicvalue”)toveryspecific https://doi.org/10.1016/j.joi.2018.03.002
1751-1577/
BookReview 535
ones(suchasthe“IntegratedImpactIndicator”or“Glanzel’s(sic)negative-binomialmodel”).Chapter4thendiscussesa numberoftheseindicatorsinmoredetailundertheheadingofinformetrictools.Thecurrentlyexistingtypesofindicators arediscussedinshortsections:publicationandcitationindicators,journalmetrics,patentandusageindicators,altmet- ricsandwebometricindicators,economicindicators,reputationandesteembasedindicators,collaborationindicatorsand indicatorsofresearchinfrastructure.Eachtypeofindicatorshasaboutonepage,soreaderswhowishtoworkwiththese indicatorsorunderstandtheminsomedepthwillalsohavetoreadthelaterchaptersinwhichtheseindicatorscomeback orconsulttheliteraturereferences.Thechapterconcludeswithashortdiscussionofthefactthatscientometricdatahave becomebigdataandoftheapplicationofsciencemaps.
Chapter5discusseswhatMoedseesasthreeimportantproblemsinappliedinformetrics:arejournalimpactfactors goodpredictorsofcitationsofindividualarticles?;towhatextentdoerrorscanceloutinlargedatasets?;andwhatisthe correctinterpretationofrankcorrelationcoefficients?Theinterpretationofthejournalimpactfactoriscurrentlyahottopic andanimportantproblemforallusersofinformetrictoolsandindicators.Moedisoneofthebestinformedexpertsonthis matter(Moed,2002).Hence,IhadexpectedarigorousupdatetoguideusersintheapplicationoftheJournalImpactFactor.
Instead,thechapterisashortpresentationofthedistributionofcitationsoverarticles(whichisknowntobehighlyskewed) comparedtothelengthofsoccerplayers(whichisanormalone).Moedshowsthatthismeansthatifonecomparestwo journalswithdifferentimpactfactors,theprobabilitythatarandomlyselectedpaperfromthejournalwiththelowerimpact factorhasacitationrateatleastashighasthatofarandomlyselectedpaperfromthejournalwiththehigherimpactfactor cannotbeignored.SoMoedbuildshisargumentthatthejournalimpactfactorisnotagoodpredictorofindividualarticles’
citationratesontheskeweddistributionofcitationsoverarticleswithinjournals.Andyet,thesectionofthischapterends inconclusively.Moedmentionsintheconcludingsentencesthatwhilepreparingthefinalversionofthebook,anarticleby WaltmanandTraag(2017)waspublishedwhichengageswithpreciselythetypeofstatisticalargumentsagainstthejournal impactfactorthatMoedpresentsinthissection.TheseauthorsagreewithMoed’sstatisticalcritiqueofthejournalimpact factorbutarguethatthisdoesnotmeanthatthejournalimpactfactorcannotbeusedatthelevelofindividualpublications.
Basedoncomputersimulations,theyevenconcludethatthejournalimpactfactormaybeamoreaccurateindicatorofthe
“value”ofanarticlethanthenumberofcitationstothatarticle.Moedsummarizestheirargumentbutdoesnotengagewith thiscounter-argument.Thereaderisleftonherown.
Unfortunately,thisisarecurringphenomenoninthisbook:themostinterestingpointsofdiscussionarebrushedover, ignored,orleftforthereadertosortout.Imustsay,Iamabitpuzzledaboutthisnoncommittalattitudeofanauthorwho hasspenthisprofessionallifeinthefieldofscientometricsandinformetrics.ThethirdsectionofChapter5isonepageabout errorswiththeconclusionthatrandomerrorsdocanceloutinlargedatasetsbutsystematicerrorsdonot.Thatmightbe conspicuousevenforreaderswhoarenotfamiliarwithinformetricsbutdohavesomebasicstatisticalknowledge.Thelast sectionofthischapterisatwo-pagediscussionofcorrelationcoefficients,basicallyshowingthatcorrelationcoefficients shouldneverbetakenatfacevalueandthattheunderlyingformofthedistributionaswellasthedatarangeshouldbetaken intoaccount.Well,indeed.
4. Thecontextofapplication
PartIIIisabouttheapplicationcontextandconsistsofthreechaptersonrespectivelyresearchassessmentasanevaluation science,non-informetricfactorsinfluencingindicators,andthepolicycontext.Thechapteronresearchassessmentasan evaluationscience(Chapter6ofthebook)aimstoexplore therelationshipbetweenthefieldofquantitativeresearch assessmentandevaluationscience.Iseethisindeedasaveryimportanttopic,anditwasthereasontoinviteakeynote speakerfromevaluationscience,PeterDahler-Larsen,totheENID/STI2014conferenceinLeiden(Moedalsogaveakeynote atthisconference).However,thischapterisquitedisappointing.Itobserves,correctly,thatevaluationscienceisamulti- disciplinaryfieldwithconceptsandanalyticaldistinctionsthatcanbequiteusefulinresearchassessment,butthenlimits itselftoashortpresentationbasedonaverylimitedsliceoftheliteratureinevaluationscience.Moedhasmissedthe,inmy view,mostinterestingareasinthatfieldandseemstohavethoughtthatselectingoneapproach(Hunter&Nielsen,2013) wouldbesufficient.
Perhapsmoreimportantly,Moedshowsin thischaptertobesurprisingly uninformedabouttheresearchinwhich scholarsfromsociologyofscienceandscience&technologystudieshavealreadymadetheconnectionbetweenresearch assessmentandevaluationscience(Fochler&DeRijcke,2017;Gläser,Lange,Laudel&Schimank,2010;Gläser&Laudel, 2016;Hammarfelt&DeRijcke,2015;Hammarfelt&Rushforth,2017;Kaltenbrunner&DeRijcke,2016;Laudel&Gläser, 2006;Sauder&Espeland2009;Whitley,2010;Woutersetal.,2015).Forexample,MoedcitesanarticlebytheDanish evaluationresearcherPeterDahler-LarsenbutdoesnotmentionhisseminalbookTheEvaluationSociety(Dahler-Larsen, 2012).Asaresult,heisregularlystatingtheobviousinthischapterbutpresentsthisasanewapproach.Unfortunately,this decreasesnotonlytheextenttowhichthechaptercanbecalledapresentationofthestateoftheart,butalsothequality oftheargument.Theresultisamundanesummingupofratherpracticaldistinctions,suchasthedifferencebetweena summativeandaformativeevaluationandbetweennormativeandcriterionbasedreferenceframeworks,inadditionto somehairsplittingaboutthesupposeddifferencebetweentheconceptsofevaluationandassessment.
Chapter7discussesnon-informetricfactorsthatinfluencethedevelopmentofindicators.Thischaptershowshowper- formanceindicatorsinevitablyarebasedonviewsonwhatconstitutesresearchperformancethatcannotthemselvesbe basedoninformetricdata.Thisisclearlyacrucialchapterfortheargumentofthebook.Moedshowstheroleofassumptions
aboutperformanceintheuseofsize-dependentversussize-independentindicators,inthedecisiontofocusonthetoppart oftheperformancedistribution,inthewayindicatorsarenormalized,inthedecisiontofocusontheshort-termversusthe long-term,andinthedefinitionofaproperreferenceframework.Thechapterarguesconvincinglythatthechoiceforapar- ticulartypeofindicatorinevitablybringswithitaparticulardefinitionofwhatshouldcountasperformance.Forexample, ifonethinksthatproductionofarticlesshouldberewarded,ratherthanpunished,onetypicallychoosesasize-dependent indicator.Andincludingparticular“corrections”inanormalizationprocedureis,accordingtoMoed,“notatheoretically neutralact,asthedeveloperexpressesthatthesefactorsdistorttheindicator,sothatitdoesnotproperlymeasurewhat itissupposedtomeasure.¨(p.109).InthissenseMoedrediscoverstheargumentsthathavebeenmadeinthesociologyof scienceandscienceandtechnologystudiesregardingthepoliticalnatureofmeasuresandmeasuring.
Thechapteralsocontainstwoshortersectionsontheneedtobeawareofthewidercontext(forexamplewhetherone seestheresearchgrouportheindividualresearcherasthebasicunitofanalysisofperformance)andontherelationbetween indicatordevelopmentandbusinessinterests.IwascurioustoseewhetherinthissectionMoedwouldsharewithushis experiencesatElsevier(theinformationcompanytowhichhemovedafterhisdeparturefromCWTS)butthisisnotthetopic ofthissection.Itisaratherdrysummaryofthevariousrelationsinthefieldbetweencompaniesandacademicunits.Moed concludesthatindicatorsareincreasinglybecomingatoolinbusinessstrategies,butdoesnotspelloutwhatthismeansfor thefieldofresearchevaluation,informetricsorforthepracticalapplicationofevaluativeinformetrics.
Chapter8delvesintotheimportanttopicofthepolicycontext.Inninepagesthechapterpresentsthemulti-dimensional researchmatrixbasedonanearlierEuropeanexpertgroupreportofwhichMoedwasoneofthemembers.Indicators constructedtomeasureoneparticularaspectmaynotbesuitabletomeasureotheraspects.“Diverseinstitutionalmissions, anddifferentpolicyenvironmentsandobjectivesrequiredifferentassessmentprocessesandindicators”(p.119).Usersof thematrixaresupposedtomakeexplicitchoiceswithregardstotheunitofanalysis,therelevantdimensionofresearch, thepurposeoftheassessment,andtherelevantsystemiccharacteristicsoftheunitofanalysis.Thisthenshouldleadtothe choiceoftheproperindicatorstobeused.Readerswhowishtoseewhetherthematrixleadstoindicatorsthatarepractical intheirsituationwillhavetogototheAUBRreportsincethechapterdoesnotfurtherspecifythis(EuropeanCommission ExpertGrouponAssessmentofUniversity-BasedResearch,2010).Theremainderofthechapterisdevotedto“systemic characteristics”inanattempttoshowhowpolicyobjectivesmayinfluenceindicatorchoice.Forexample,accordingto Moedtheuseofthejournalimpactfactortomeasureinternationalorientationisinformetricallydefensibleinasituation whereinternationalpublicationisnotthenormandpolicywantstostimulatethelocalscientificcommunitytobecome moreinternationalinitsambitions.
5. Thewayforward
PartIVsketchesMoed’sideasaboutthewayforward.Itconsistsoffourchaptersdealingwiththeuseofinformetric indicators,researchperformancemeasurement,altmetrics,andindicatordevelopment.Chapter11discussesaltmetrics, usingabookbytheconsultantandessayistMichaelNielsen(Nielsen2011).ThechapterisapracticalexposeofwhatMoed calls“computerizationoftheresearchprocess”.IttreatstheroleofICTinacademicresearchataverybasiclevel,disregarding theexcitinginsightsthathavebeengeneratedinthefieldsofdigitalhumanities,computerscience,networkedscienceand science&technologystudies(Boonstra,Breure,&Doorn,2004;Borgman,2007;Bulgeretal.,2011;Hey,Tansley,&Tolle, 2009;Kaltenbrunner2014;Wouters,Beaulieu,Scharnhorst,&Wyatt,2013).Ifeltsorrytoseethatitdoesnotgenerate interestingnewinsightswithrespecttothepotentialofaltmetricsforresearchevaluation(Bornmann&Haunschild,2017;
Costas,Zahedi,&Wouters.,2015;Konkiel,2016;Thelwall,2005;Thelwall,2008;Thelwall,Haustein,Larivière,&Sugimoto, 2013;Woutersetal.,2015).
Chapter9aboutindicatoruselistswhatMoedseesasthemainproblemsthatneedtobetackledtomakeprogress:
assessingindividualresearchers,thelimitedtimehorizonofcitationanalysis,assessingsocietalimpact,effectsofindicator useonauthorsandeditors,constitutiveeffectsofindicators,andtheneedforwhatMoedcalls“evaluativeframeworks”.
WithrespecttoassessmentofindividualscholarsMoedsupportstheconclusionoftheACUMENprojectthatresearcher- providednarrativesshouldbethecoreofthisassessment(Woutersetal.,2014).Inthesectiononthelimitedtimehorizon ofbothaltmetricandcitationindicators,Moeddefendsthepositionthatthevalueofcontributionstothescientificrecord willintheendonlybevisibleinthelongterm.“Indicatorsdonotmeasurecontributiontoscientific-scholarlyprogress inthissense,buttendtoindicateattention,visibility,orshorttermimpact”(p.134).AlthoughMoedhasalwaysmade thedistinctionbetweenintellectualinfluenceandcitationimpact(seeeg.Moed,2005:222),heseemstoemphasizethe limitationsofcitationanalysisnowevenmorethanhedidinthepast(seeeg.Moed,2002:731).
ThesectionontheeffectsofindicatoruseonauthorsandeditorslimitsitselftostudiesinwhichMoedwasinvolved:a studyonmechanismstoraisethejournalimpactfactorandananalysisoftheeffectsoftheUKresearchassessmentexercises onpublicationpatterns.Moststudiesinthefieldsofevaluationscience,managementstudiesorscience&technologystudies onthistopicareignored(Fochler&DeRijcke,2017;Gläseretal.,2010;Gläser&Laudel,2016;Hammarfelt&DeRijcke, 2015;Hammarfelt&Rushforth,2017;Kaltenbrunner&DeRijcke,2016;Laudel&Gläser,2006;Sauder&Espeland2009;
Whitley,2010;Woutersetal.,2015).Sincethefieldofevaluationstudiesisrecognizedasanimportantfieldearlierinthe book,thissystematicomissionisratherpuzzlinganddoesdecreasethevalueofthebookasanoverviewofthestateof theart.Acomprehensiveinclusionoftherelevantliteraturewouldhavebeenparticularlyvaluablesincethebooktriesto defendaspecificpositionwithrespecttoeffectsofindicatoruse.AccordingtoMoed,theissueisnotwhetherscholarly
BookReview 537
practiceischangedbyindicators,butwhetherornottheapplicationofindicatorsenhancesresearchperformance(p.137).
Ofcourse,thisraisesthequestionhowresearchperformanceisdefined.Howdoweknowwhetherornotperformancehas increased?Moeddoesnotprovidetheanswerbutshowshowdifficultitistosaysomethingmeaningfulaboutitinthecase studiesinwhichhewasinvolved.Astudyofincreasesofthejournalimpactfactorleadshimtoconcludethat“itisdifficult tomaintainthattheobservedchangesindicateanincreaseinjournalperformance”(p.137).Acasestudyoftheeffectsof theUKResearchAssessmentExercisewasmorecomplextointerpret.Moeddoesnotwanttoexcludethepossibilitythatit stimulateduniversitiestoincreasethevolumeandthesignificanceoftheirresearch.“Ontheotherhand,onecannotclaim apriorithatthisuseindeeddidhaveapositiveeffectuponperformance.Withoutadditionalqualitativeandquantitative evidence,itisimpossibletodrawanysolidconclusionsaboutthismatter.¨(p.137)Basically,Moedwrestleswiththeproblem ofconstitutiveeffectsofindicatorsbecausetheymakeitmoredifficulttointerprettheeffortstomeasureperformance.This isnotaminorissuebuttheveryheartoftheproblemofperformancemanagementandresearchevaluationinacademia.
Butbeforegoingintothisinmoredepth,thequestionmustbeaddressedwhetherMoedisrightinbracketingoffpossible effectsonacademicpractice.Ithinkheiswrong.Evaluationpracticesandtheirinstruments,bothquantitativemetricsand indicatorsandqualitativeprocedureshaveeffects.Theseeffectsdonotonlyrelateto“enhancement”ofresearchperformance butalsoinclude,amongothers:theresearchagendaofdisciplines;theprioritiesthatthemostinfluentialgroupssetintheir practices;thediversityintermsofgenderandethnicityoftheworkforceinacademia;thenatureofthePhDtrainingboth atthegrouplevel(whatkindofinteractionsarestimulated)andattheindividuallevel(whichpsychologicaltraitsaremost conducivetoanacademiccareer);thetechnicalandsocialinfrastructureofscientificandscholarlyresearch;andlastbut notleastthepowerrelationshipsbetweendifferentfieldswithrespecttofundingandcareerpossibilities.Inotherwords, indicatorsmaydirectlyandindirectlyinfluencetheverycharacterofwhatdoingresearchmeansinourcurrentsocieties, bothforbetterandforworse.Inmyview,thesemattersareactuallyfarmoreimportant,alsoforinformetricians,thanthe narrow-mindedfocusonperformanceenhancement.
Thisbringsustotheproblemofconstitutiveeffects,thetopicofthenexttwo-pagesectioninthischapter(Section 9.5).Moeddefinesconstitutiveeffectsasfollows:“Thenotionofconstitutiveeffectsofindicatorsquestionsthevalidityof thedistinctionbetweenintendedandunintendedeffects,andclaimsthatthereisatendencythattheuseofindicatorsofa theoreticalconceptsuchasresearchqualitycreatesarealityinwhichtheconceptandtheindicatortendtocoincide”.Hence,
“researchqualitywouldbemoreandmoredefinedastowhatcitationsmeasure.¨(p.137)Moedseesthisas“acritique”that shouldbetaken“veryseriously”andproceedstomakethreeargumentsrespondingtothiscritique.First,helabelsthe tendency“toreplacerealitywithsymbols”as“magicalthinking”(italicsintheoriginal)whichheclearlyrejects.“Ifsuch abeliefisdominant,asituationwouldhaveemergedthat,whilemodernsciencesurpassedamagicalattitudetowards nature,suchanattitudehasemergedagain,butnowtowardsscience,namelyinresearchassessmentandsciencepolicy”
(p.138,italicsintheoriginal).Second,hedisputestheempiricalbasisofthehypothesisoftheconstitutiveroleofresearch performanceindicators.“Thisbasisisintheperceptionofthecurrentauthorstillratherweak,andnotseldombasedon personalconsiderationsandinformalevidences.¨(p.138).Thethirdcommentisthatconstitutiveeffectsshouldbeseenin anhistoricalcontext.Ifthereareconstitutiveeffectsofindicators,“oneshouldnotpointthecritiqueoncurrentassessment practicesmerelytowardsinformetricindicatorsassuch,butrathertowardsanyclaimforanabsolutestatusofaparticular waytoassessquality”(p.138).InMoed’sview,itdoesnotfollowthatanintelligentcombinationofpeerjudgementand indicatorsisfundamentallyflawedaslongasonedoesnotattributeanabsolutestatustoeitheroneofthemainassessment methods.
Ofcourse,Moed’slastargumentiscorrect,constitutiveeffectsarenotlimitedtoinformetricindicators.Butwhowould disputethis?Idonotmeetmanypeoplewhowishtolaytheblameonlyonindicatorsandscientometricians.Andmost scientistsaswellasevaluatorsprefertocombineindicatorswithhumanjudgement.Thedebateaboutabuseofindicators isessentiallyadebateaboutaresearchassessmentsystemasawholethathasgoneawry.Butscientometriciansincluding Moeddidplayanimportantroleintheshapingofthisassessmentsystemandtheyshouldthereforetakeresponsibilityfor thatrole(DeRijcke&Rushforth,2015).ItisdifficulttoreadMoed’sthirdargumentasawayofbeingaccountable,itreads moreasanattempttodeflectthecritique.Hisfirstargumentisnotmuchstronger.Clearly,Moedseemstothinkthatitis possibletohaveaworldwithoutconstitutiveeffectsofevaluationmechanisms,aratherfundamentalmisunderstanding.
Apparentlyhedoesnotrecognizethatthephenomenonofconstitutiveeffectisonlyaspecificinstantiationofthemore generalphenomenonthatsocialindicatorschangetherealitytheyarepurportedtomeasure.Everysocialscientistwhohas beenengagedinsomehowmeasuringorstudyingpartsofsocialrealityhastodealwiththis(Berger&Luckmann,1967;
Bijker,Hughes,&Pinch,1989;Hacking,1999;Porter,1995;Schinkel,2013).Ithasnothingwhatsoevertodowithmagical thinking.
AdistortedandnarrowviewofsocialscienceisalsobuiltintoMoed’ssecondcommentontheconceptofconstitutive effects.Hedoubtstheempiricalevidencefortheoccurrenceofconstitutiveeffects.Thatisaratherboldclaimtomakegiven thefactthatMoeddoesnotgiveanyindicationthatheattemptedtoprocesstherelevantliterature.Thesuggestionthat theevidenceis“informal”and“basedonpersonalconsiderations”isitselfmadewithoutanyempiricalevidence.Ofcourse, thestudyoftheeffectsof,forexample,citationindicators,cannotitselfbepurelyinformetricandahostofqualitative methodologiesandtheoriesareneeded,rangingfromethnographicobservations,surveys,focusgroups,tointerviews,often incombinationwithformalizedqualitativeandquantitativeapproaches(Fochler&DeRijcke,2017;Gläseretal.,2010;
Gläser&Laudel,2016;Hammarfelt&DeRijcke,2015;Hammarfelt&Rushforth,2017;Kaltenbrunner&DeRijcke,2016;
Laudel&Gläser,2006;Sauder&Espeland,2009;Whitley,2010;Woutersetal.,2015).Informetriciansandbibliometricians
couldlearnalotfromthistypeofstudies.Forexample,thejournalimpactfactorplaysdifferentsymbolicrolesindifferent contextswhichexplainsagoodpartofitspopularity(Rushforth&DeRijcke,2015).Thesuggestionthattheseapproaches canbesummarizedundertheheadingof“informalevidence”isapoorpresentationindeed.Iwillcomebacktothisinthe conclusionofthisreview.
ThelastsectionofChapter9statesthatad-hocargumentsofevaluatorsorunreflectiveassumptionsaboutindicatorscan playaroleinresearchassessmentbecauseofthelackofanevaluativeframeworkandanassessmentmodel.Thesolution totheseproblemsinresearchassessmentsisthereforethesystematicdevelopmentandapplicationofsuchframeworks.In thefollowingchaptersofPartIVMoedsetsoutanumberofproposalsformovingforwardwhichshouldclarifyhowsuch evaluativeframeworksshouldlooklike.Inarecentblogpost,Moedhastriedtofurtherclarifytheconcept(Moed,2018).He doesnotargueinfavorofoneorotherready-madeframeworks.Rather,heproposesthatforeachnewevaluationexercise, theorganizersoftheassessmentcreatesuchaframeworkbydecidingon,inthisorder:theunitofassessment,thedimension ofresearchthatshouldbeassessed,theobjectiveoftheassessment,andtherelevantcharacteristicsoftheunitofassessment.
Usingthemulti-dimensionalindicatormatrix,onecanthendecidewhichindicatorsshouldbeused.Hence,theconclusion thatanevaluativeframeworkshouldbecentralisamethodologicalguideline,notasubstantiveone.Thestrengthofthis proposalisthattheparticipantsinaresearchassessmentareforcedtothinkhardaboutwhattheyactuallywanttoreach.
Thisisagoodrecipeagainsttheunreflectiveuseofstandardizedindicators.Atthesametime,Moed’sproposalforevaluative frameworkssuffersfromamajorflaw.Itdisregardsthewayprocessesofattributingvalue,ofwhichresearchevaluation isanexemplar,aredeeplyembeddedinsociallifeandassumesitispossibletoneatlyseparateresearchevaluationfrom theotherworkprocessesinacademicresearch.Forexample,researchevaluationdoesnotonlytakeplacewhenanexpert panelvisitsaresearchgroupinanacademicmedicalhospitalintheNetherlandsintheframeworkoftheDutchStandard EvaluationProtocolrequirements.Evaluationtakesplaceallthetime:intheprocessofhiringaPhDcandidate,inthedecision tosubmittojournalAinsteadoftojournalB,inthedecisiontosubmitaproposaltoonecallratherthananother,andin careerchoicesofseniorresearchers.Andtheseevaluativemomentsinfluenceeachother.Knowledgecreationandresearch evaluationhavebecomesoentangledthattheideatoneatlyseparatethemisalmostlaughable.Intheend,suchanendeavor remainsatechnocraticsolutiontoadeeplysocialproblem.
InChapter10,Moedlaunchesanumberofproposalsthatexemplifyhowhisideaofevaluativeframeworkscanbeapplied.
Amongthemareindicatorsofcommunicationeffectiveness,indicatorsforthevariousfunctionsofjournals,andindicators ofresearchtrainingandthecontributioninstitutionsmaketoresearchindevelopingcountries.Inaddition,Moedsuggests todefineminimumperformancestandardsasacriteriontodecidewhichstaffmemberscanbeconsideredresearchactive.
AccordingtoMoed,thiswouldentailashiftfrommeasuringperformancetomeasuringthepreconditionsofperformance.
Thechapterincludesasectiononuniversityrankingswithproposalsforuniversitiesinwhich,tobehonest,Ididnotfind muchnew.Andlast,thechapterprovidesanalternativetoperformancebasedfundinginwhichemerginggroupsareput central,ratherthanestablishedresearchgroupswhichiscurrentlycommonlythecase.Theseareallinterestingproposals thatmayfitinresearchassessmentsdependingonthecontextandgoalsoftheevaluation.Idonotsee,however,inwhat waystheywouldcontributetoasystemicchangeinthewayweevaluateresearchandresearchers.
IhadthesamefeelingabouttheproposalsinthelastchapterofPartIVofthebook,whichpresentsfourideasthathavealso beenpublishedasseparatearticlesbytheauthor:indicatorsofthepeerreviewprocessofjournalsandbooks;anontology baseddatamanagementsystem;informetricself-assessmenttools;andinformetricmodelsofscientificdevelopment.All ofthemaresensibleideas,someofthemarealreadyindevelopmentatpublishersanduniversitiesinavarietyofprojects (e.g.theACUMENportfolioisasuiteofself-assessmenttoolsinthemaking)andsomeofthemaremoreresearchprojects ratherthantoolsforassessment(suchasinformetricmodelsofscientificdevelopment).Inotherwords,Chapters10and12 ofthebookpresentvaluableideasbutitisnotclearwhatbindsthemtogetherandwhytheseproposals,ratherthanothers, areselectedforthisoverview.IguesswhatImisshereisanoverarchingvision.Insteadwegetasetoftechnicalideasthat maywork,ormaynotwork.
6. Caninformetricsbevaluefree?
Letmebeclear,Idonotthinkthislackofoverarchingvisioniscoincidental.Ratheritistheconsequenceoftheperspective thatMoeddefendsinhisbook.AsIindicated,thebookisnotonlyastateoftheartperspectiveasperceivedbytheauthor,but alsoaninterventioninthecurrentdebateaboutresearchassessments.Ittriestomakethecaseforafoundationalparadigm ininformetricswhichhasguidedmanybibliometriciansandscientometriciansinthepast(VanRaan,1988).Hence,Moed doesnotonlydefendhisownposition,hisbookisessentiallytheexpressionofthisparadigm.Moedstatesitsuccinctly intheintroductiontothebook:“Thecurrentauthorconceivesinformetricsasavaluefree,empiricalscience.Beingvalue freeisconceivedasamethodologicalrequirement.Thisbookshowshowstatisticaldefinitionsofindicatorsofresearch performancearebasedontheoreticalassumptionsonwhatconstitutesperformance.Theinformetriccomponentandthe domainofevaluativeorpoliticalvaluesinanassessmentaredisentangledbydistinguishingbetweenquantitative-empirical, informetricevidencesontheonehand,andanevaluativeframeworkbasedonnormativeviewsonwhatconstitutesresearch performanceandwhichpolicyobjectivesshouldbeachievedontheother.¨(p.12)Thismeansthatinformetriciansshould maintainaneutralpositionintheirinformetricwork.
ForMoed,valuesplayaroleinthreedifferentways.First,theyareimportantinthechoiceofanevaluativeframework.
Informetriciansshouldnotbeinvolvedinthissincethesechoicescannotbebasedoninformetricevidence.Second,infor-
BookReview 539
metriciansarealsoresearcherssubjecttoassessmentandassuchtheywillalsohaveopinionsbuttheyshouldseparate thesefromtheirroleasinformetrictechnician.Thirdly,everyindicatorhassomenormativeviewsbuilt-inandinthissense noindicatoristrulyvalue-free.Butifindicatorsareconstructedonvalues,howcanoneimagineavalue-freeinformetrics?
Moedanswersthisconundrumbypretendingthat,althoughallindicatorsareinfestedwithvalues,itisstillpossibleto analyticallyseparatevaluefromfact.
Moed’sevaluativeframeworkworkslikeacookbookandtheinformetricianistheneutralcookwhopreparesthefood butabstainsfromjudgementaboutitshealth.Inacontrivedworldthismightwork.Butsuchworldsdonotexistnorwill theyevercometoexist.Toframeitinmoretheoreticalterms,theparadigmthatMoedadherestoisbasedonatriple purificationofthecomplexrealityofthelifeworldinwhichweexist(Latour,2005).First,ittriestoseparatethetechnical characteristicsofindicatorsfromtheirvalue-ladendimensions.Second,ittriestoinsulatetheprocessofresearchassessment fromalltheotherworkprocessesinwhichtheseassessmentsareembedded.Ialreadyarguedwhythisisfallacious.For example,supposeahospitalwouldfollowMoed’sadviceanddevelopanealuativeframeworkforitsformalizedresearch assessmentexercises.Itwouldstillbeconfrontedwiththehundredsofevaluationdecisionsthathavetobetakeneveryday inthecourseofthedailyworkofitsresearchers.Third,Moed’sparadigmtriestoignoretheformativeeffectsofevaluations ingeneralandinformetrictoolsinparticularonthecharacteroftheprocessofknowledgecreationbynarrowlyfocusing on“performanceenhancement”andignoringthehugevarietyofsocialeffects(notatallnecessarilydetrimentalbythe way)ontheresearchsystemandonindividualresearchgroups.AsIalreadyindicated,theprocessesofresearchassessment areintimatelyintertwinedwiththeprocessesofagendasetting,funding,recruitment,projectmanagement,infrastructure developmentandmaintenance,andhumanresourcemanagement.Aninformetricsthatisoblivioustotheseconnections andthefeedbackprocessesthatresultfromthem,andwhichdoesnotdigestthestateoftheartknowledgeofthesocial sciencesandhumanities,willalwaysbeblindtoitsownroleinsocietyandtheintendedandunintendedeffectsitwill generate.
Moed’sbook is testimonytothis. Itis a bookthat showsnothingless thanthefailure ofa paradigm.And thisis, paradoxically,animportantcontributiontothefieldofinformetrics.
Acknowledgements
IwouldliketothankThomasFranssen,ThedvanLeeuwen,SarahdeRijcke,andLudoWaltmanforcommentsonan earlierversionofthisreview.
References
Berger,P.L.,&Luckmann,T.(1967).Thesocialconstructionofreality:Atreatiseinthesociologyofknowledge.Doubleday.
Bijker,W.E.,Hughes,T.P.,&Pinch,T.(1989).Thesocialconstructionoftechnologicalsystems.Cambridge,Mass:TheMITPress.
Boonstra,O.,Breure,L.,&Doorn,P.(2004).Past,presentandfutureofhistoricalinformationscience.
Borgman,C.(2007).Scholarshipinthedigitalage.Cambridge:MITPress.
Bornmann,L.,&Haunschild,R.(2017).Dobibliometricsandaltmetricscorrelatewiththequalityofpapers–Alarge-scaleempiricalstudybasedonF1000 prime,altmetrics,andcitationdata.InSTI2017.openindicators:innovation,participationandactor-basedSTIindicators.[Paris].
Bulger,M.,Meyer,E.T.,Flor,G.dela,Terras,M.,Wyatt,S.,Jirotka,M.,etal.(2011).Reinventingresearch?Informationpracticesinthehumanities..Retrieved (https://www.rin.ac.uk/our-work/using-and-accessing-information-resources/information-use-case-studies-humanities)
Costas,R.,Zahedi,Z.,&Wouters,P.(2015).Doaltmetricscorrelatewithcitations?extensivecomparisonofaltmetricindicatorswithcitationsfroma multidisciplinaryperspective.JournaloftheAssociationforInformationScienceandTechnology,66(10),2003–2019.
Dahler-Larsen,P.(2012).Theevaluationsociety.StanfordUniversityPress.RetrievedAugust9,2013.
http://www.amazon.com/The-Evaluation-Society-Peter-Dahler-Larsen/dp/080477692X
DeRijcke,S.,&Rushforth,A.(2015).Tointerveneornottointervene;isthatthequestion?Ontheroleofscientometricsinresearchevaluation.Journalof theAssociationforInformationScienceandTechnology,66(9),1954–1958.http://dx.doi.org/10.1002/asi.23382.RetrievedFebruary20,2018
EuropeanCommissionExpertGrouponAssessmentofUniversity-BasedResearch.(2010).Assessingeurope’suniversity-basedresearch–Expertgroupon assessmentofuniversity-basedresearch.EuropeanCommission.
Fochler,M.,&DeRijcke,S.(2017).Implicatedintheindicatorgame?Anexperimentaldebate.EngagingScience,Technology,andSociety,3,21–40.
Gläser,J.,&Laudel,G.(2016).Governingscience.EuropeanJournalofSociology,57(1),117–168.Retrieved (http://www.journals.cambridge.org/abstractS0003975616000047)
Gläser,S.,Lange,G.,&Laudel,U.(2010).Informedauthority?thelimiteduseofresearchevaluationsystemsformanagerialcontrolinuniversities.InR.
Whitley,J.Gläser,&L.Engwall(Eds.),Reconfiguringknowledgeproduction:
Changingauthorityrelationshipsinthesciencesandtheirconsequencesforintellectualinnovation(pp.149–183).Oxford:OxfordUniversityPress.Retrieved (http://www.worldcat.org/title/reconfiguring-knowledge-production-changing-authority-relationships-in-the-sciences-and-their-consequences-for- intellectual-innovation/oclc/548626398&referer=briefresults)
Hacking,I.(1999).Thesocialconstructionofwhat.HarvardUniversityPress.
Hammarfelt,B.,&DeRijcke,S.(2015).Accountabilityincontext:effectsofresearchevaluationsystemsonpublicationpractices,disciplinarynorms,and individualworkingroutinesinthefacultyofartsatuppsalauniversity.ResearchEvaluation,24(1),63–77.
Hammarfelt,B.,&Rushforth,A.D.(2017).Indicatorsasjudgmentdevices:Citizenbibliometricsinbiomedicineandeconomics.ResearchEvaluation,1–25.
ResEvalrvx018.
Hey,T.,Tansley,S.,&Tolle,K.(2009).Thefourthparadigm:Data-Intensivescientificdiscovery.InT.Hey,S.Tansley,&K.Tolle(Eds.),Microsoftresearch.
Retrieved(http://www.amazon.com/dp/0982544200)
Hunter,D.E.K.,&Nielsen,S.(2013).Performancemanagementandevaluation:Exploringcomplementarities.NewDirectionsforEvaluation,2013(137), 7–17.RetrievedFebruary8,2018(http://doi.wiley.com/10.1002/ev.20042)
Kaltenbrunner,W.,&DeRijcke,S.(2016).Themicropoliticsofquantifyingresearchoutputforevaluationindutchlawschools.pp.1–7inSTI2016:peripheries, frontiersandbeyond.Valencia,Spain.
Kaltenbrunner,W.(2014).Reflexiveinertia.Reinventingscholarshipthroughdigitalpractices.LeidenUniversity.
Konkiel,S.(2016).Altmetrics:Diversifyingtheunderstandingofinfluentialscholarship.PalgraveCommunications,2,16057.Retrieved http://www.palgrave-journals.com/articles/palcomms201657
Latour,B.(2005).Reassemblingthesocial:Anintroductiontoactor-network-theory.Oxford:OxfordUniversityPress.
Laudel,G.,&Gläser,J.(2006).Tensionsbetweenevaluationsandcommunicationpractices.JournalofHigherEducationPolicyandManagement,28(3), 289–295.
Moed,H.(2002).Theimpact-factorsdebate:TheISI’susesandlimits.Nature,415(6873),731–732.http://dx.doi.org/10.1038/415731a.Retrieved Moed,H.F.(2005).Citationanalysisinresearchevaluation.Dordrecht:Springer.
Moed,H.(2018).Responsibleuseofmetrics:Anevaluativeframeworkisessential,butcannotbegroundedininformetricresearch.TheBibliomagician.
RetrievedFebruary20,2018(https://thebibliomagician.wordpress.com/2018/02/06/in-responsible-use-of-metrics-an-evaluative- framework-is-essential-but-cannot-be-grounded-in-informetric-research/)
Nielsen,M.(2011).Reinventingdiscovery:Theneweraofnetworkedscience[Hardcover].PrincetonUniversityPress.RetrievedJanuary10,2012 (http://www.amazon.com/gp/product/0691148902/ref=aslitfil?ie=UTF8&tag=michaniels-20&linkCode=as2&camp=217145&creative=
399373&creativeASIN=0691148902)
Porter,T.M.(1995).Trustinnumbers:Thepursuitofobjectivityinscienceandpubliclife.Princeton,NJ:PrincetonUniversityPress.Retrieved (http://books.google.nl/books?id=oK0QpgVfIN0C)
Rushforth,A.,&DeRijcke,S.(2015).Accountingforimpact?ThejournalimpactfactorandthemakingofbiomedicalresearchintheNetherlands.Minerva, 53(2),117–139.Retrieved(http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=4469321&tool=pmcentrez&rendertype=abstract)
Sauder,M.,&Espeland,W.N.(2009).Thedisciplineofrankings:Tightcouplingandorganizationalchange.AmericanSociologicalReview,74(1),63–82.
Schinkel,W.(2013).Theimaginationof‘society’inmeasurementsofimmigrantintegration.EthnicandRacialStudies,36(7),1142–1161.Retrieved (http://www.tandfonline.com/doi/abs/10.1080/01419870.2013.783709)
Thelwall,M.,Haustein,S.,Larivière,V.,&Sugimoto,C.R.(2013).Doaltmetricswork?Twitterandtenothercandidates.PublicLibraryofScience,8(5), e64841.
Thelwall,M.(2005).Linkanalysis:Aninformationscienceapproach.SanDiego:AcademicPress.
Thelwall,M.(2008).Bibliometricstowebometrics.JournalofInformationScience,34,605–621[RetrievedOctober24,2011 (http://jis.sagepub.com/cgi/doi/10.1177/0165551507087238)
VanRaan,A.(Ed.).(1988).Handbookofquantitativestudiesofscienceandtechnology.Amsterdam:ElsevierSciencePublishers.
Waltman,L.,&Traag,V.A.(2017).Useofthejournalimpactfactorforassessingindividualarticlesneednotbewrong..Retrieved (http://arxiv.org/abs/1703.02334)
Whitley,R.(2010).Reconfiguringthepublicsciencestheimpactofgovernancechangesonauthorityandinnovationinpublicsciencesystems.InR.
Whitley,J.Gläser,&L.Engwall(Eds.),Reconfiguringknowledgeproduction:Changingauthorityrelationshipsinthesciencesandtheirconsequencesfor intellectualinnovation(pp.3–47).Oxford:OxfordUniversityPress.Retrieved
(http://www.worldcat.org/title/reconfiguring-knowledge-production-changing-authority-relationships-in-the-sciences-and-their-consequences-for- intellectual-innovation/oclc/548626398&referer=brief results)
Wouters,P.,Beaulieu,A.,Scharnhorst,A.,&Wyatt,S.(2013).Virtualknowledge.InP.Wouters,A.Beaulieu,A.Scharnhorst,&S.Wyatt(Eds.),Experimenting inthehumanitiesandthesocialsciences.CambridgeUSA:MITMassPress.Retrieved(http://mitpress.mit.edu/books/virtual-knowledge-0)
Wouters,P.,Bar-Ilan,J.,Thelwall,M.,Aguillo,I.F.,Must,Ü.,Havemann,F.,etal.(2014).ACUMENfinalreport..Brussels.Retrieved (http://cordis.europa.eu/result/rcn/157423en.html)
Wouters,P.,Thelwall,M.,Kousha,K.,Waltman,L.,DeRijcke,S.,Rushforth,A.,etal.(2015).Themetrictide:Literaturereview(SupplementaryreportItothe independentreviewoftheroleofmetricsinresearchassessmentandmanagement).pp.188.Retrieved
(http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/)
PaulWouters CentreforScienceandTechnologyStudies(CWTS),LeidenUniversity,TheNetherlands
E-mailaddress:p.f.wouters@cwts.leidenuniv.nl Availableonline12April2018
