STI 2018 Conference Proceedings
Proceedings of the 23rd International Conference on Science and Technology Indicators
All papers published in this conference proceedings have been peer reviewed through a peer review process administered by the proceedings Editors. Reviews were conducted by expert referees to the professional and scientific standards expected of a conference proceedings.
Chair of the Conference Paul Wouters
Scientific Editors Rodrigo Costas Thomas Franssen Alfredo Yegros-Yegros
Layout
Andrea Reyes Elizondo Suze van der Luijt-Jansen
The articles of this collection can be accessed at https://hdl.handle.net/1887/64521 ISBN: 978-90-9031204-0
© of the text: the authors
© 2018 Centre for Science and Technology Studies (CWTS), Leiden University, The Netherlands
This ARTICLE is licensed under a Creative Commons Atribution-NonCommercial-NonDetivates 4.0 International Licensed
experimental physics in Italy
Marco de Santis Puzzonia**, Fabio Beltram*, Tindaro Cicero** and Marco Malgarini**
*fabio.beltram@anvur.it,
Board of Directors, ANVUR, Via Ippolito Nievo 35, Rome, 00153 (Italy)
** marco.desantispuzzonia@anvur.it; marco.malgarini@anvur.it
Research Evaluation Dept., ANVUR, Via Ippolito Nievo 35, Rome, 00153 (Italy)
Introduction
Co-authorship practices are known to differ among distinct scientific fields, but may do so even within research communities belonging to the same field. As a consequence, standard normalisation procedures may fail and not allow a correct comparison of individual scientific performance within a given field. For instance, in recent years groups of scientists in experimental physics, particularly in the high-energy sector, have started to work within projects involving very large research groups, sometimes comprising thousands of researchers, while other scientists operating in the same general field have continued to work within more traditional, smaller scientific collaborations: as a consequence, evaluating the scientific contribution of researchers in experimental physics has become particularly difficult, even if traditional field normalisation is taken into account. The aim of this paper is to investigate the relevance of this phenomenon and discuss a method to take into account co-authorship when evaluating scientist considering the case of the Italian National scientific Habilitation program as a case study. The paper is organized as follows: the next session briefly introduces the Italian Habilitation programme; we then analyse differences in publication practices in the field of experimental physics in Italy and show the relevance of these on current scholar performance evaluation. We finally discuss a method that can provide a first improved evaluation scheme.
The Italian Habilitation program
Since 2012, the Italian university recruitment system requires an initial ‘‘scientific habilitation’’
awarded by sectorial committees of national experts, as a pre-requirement to access the ranks of associate or full professor. Evaluations are conducted separately for each scientific sector;
moreover, in order to be admitted to the final evaluation candidates must satisfy pre-determined minimal criteria concerning their publication record. More specifically, in STEM areas –as in the case here of interest- candidates must meet or exceed at least 2 out of 3 pre-determined thresholds, defined in terms of number of publications, number of citations and h-index value in a given time interval (for a more thorough description of the procedure, see for instance Bagues, Sylos Labini and Zynovieva, 2015). Thresholds are separately provided for candidates to Associate Professor positions, for candidates to Full Professorships, and for Full Professors aiming to access the role of Evaluator in the Ministerial Commissions granting the Habilitation.
According to Italian Law, the Agency for the Evaluation of the University and Research System (ANVUR) is in charge of calculating these thresholds for each academic rank and for each field, and of proposing their official adoption to the Ministry of Education (MIUR). The definition of scientific fields is provided by the MIUR; the level of aggregation is the so-called Settore concorsuale (SC), specifically designed to allow recruitment in homogenous fields: overall, in
STI Conference 2018 · Leiden
Italy MIUR identified 190 SC’s. If publication practices are found to be different within one such SC, the calculation of the thresholds can be moved to a lower level of aggregation: the so- called Settore Scientifico Disciplinare (SSD); overall, MIUR identified 370 SSD. For the most recent National Habilitation programme started in the fall of 2016, MIUR officially determined the thresholds on the basis of the ANVUR proposal with the Ministerial Decree n. 602/2016 (http://attiministeriali.miur.it/anno-2016/luglio/dm-29072016.aspx).
The case for SSD FIS/01 “Experimental physics”
In Italy, the SSD FIS/01 - “Experimental physics”, is one of the two sub-sectors composing the SC 02/A1 – “Experimental physics of fundamental interactions”1. Currently it comprises 431 scientists, 97 of which are Full Professors, 190 Associate Professors and 144 Assistant Professors (Ricercatore). In the period 2002-2017 these scholars appeared as co-authors in 13450 articles indexed in the Web of Science Database; on average, these articles received 10,6 citations each. Looking at this dataset and its time evolution, it is possible to observe a remarkable change in co-authorship practices in the period under consideration (Fig. 1): the average number of co-authors has grown rapidly since 2010, passing from around 500 to almost 2000 per paper in the period 2011-20172. In parallel to this, a significant change in the variance occurred that indicates an increased divergence in publication practices within this scholarly population.
Figure 1: Average number of co-authors per paper and standard deviation, experimental physics
1 Another group of 367 researchers is affiliated to the SSD Fis/01, but they are comprised in SC 02/B1,
”Experimental physics of Matter”: they are not included in this analysis.
2 The remarkable increase in the average number of co-authors observed after 2008 may be linked to the start up of the Large Hadron Collider (LHC), the world’s largest particle accelerator. Operation of this infrastructure requires rather large collaboration involving many scientists.
So far threshold calculations for the National Habilitation programme did not take into account the number of co-authors: in the presence of such significant changes, however, this may lead to a bias in the selection procedure. In order to evaluate this aspect, let us consider the quartiles of the scholar distribution on the basis of their average number of authors per paper for each academic rank, as reported in Tab. 1.
Table 1. Quartiles of the distribution of academics accordingly to the average number of paper authors
Quartile Full
Professors
Associate Professors
Assistant Professors
I Quartile 1-136 1-123 1-148
II Quartile 137-536 124-544 149-532
III Quartile 537-1769 545-1953 533-2134
IV Quartile >=1770 >=1954 >=2134
By considering the threshold values adopted in the last Habilitation round, we can now calculate the fraction of Professors in each quartile that meets or exceeds the relevant thresholds (i.e.
Evaluator for Full Professors, Full Professor for Associate Professors and Associate Professor for Assistant Professors). Table 2 shows that the rate of success in the National Habilitation programme is much higher for those publishing with a larger average number of co-authors.
Those in the first quartile in the distribution according to the number of authors (i.e., in the case of Assistant Professors, those publishing on average with a number of co-authors comprised between 0 and 148) have a success rate equal to zero for Full professors and Associate Professors (e.g. no Full or Associate Professor with these publication characteristic is able to access the higher rank), while Assistant Professors have a success rate of only 8,6%.
Differently, all those belonging to the third and fourth quartile have a success rate above 90%.
Table 2. Success rates by number of co-authors in experimental physics
Quartile Full
professors
Associate Professors
Assistant Professors
I Quartile 0,0 0,0 8,6
II Quartile 58,3 58,3 64,7
III Quartile 91,7 91,7 94,3
IV Quartile 100,0 100,0 97,1
Total 62,5 62,5 65,9
Figure 2 better illustrates this by presenting the distribution of the number of publications and citations, and of the h-index value with respect to the average number of co-authors, separately considering Full, Associate and Assistant Professors. Each dot in the graph represents an individual researcher: a strong correlation between scientific performance (according to any of the indicators) and the average number of co-authors does emerge. In other words, those publishing with a larger number of co-authors publish more papers, receive more citations and show a higher h-index in the period considered for the analysis, regardless of the academic rank.
Hence, those working with a fewer number of co-authors are penalised, ceteris paribus, in the
STI Conference 2018 · Leiden
access to higher ranks in the academic career. Since all this happens within the same SSD there is a clear need to re-define performance indicators in order to correct for this bias.
Figure 2: Scientific output and average number of co-authors for scientists in experimental physics. The thresholds to access the higher academic ranks are shown as red lines
Accounting for different co-authorship practices in the National Habilitation program In order to overcome this problem, literature usually suggests fractionalised counting, i.e.
counting for each author only a fraction of a paper, depending on the number of authors. The simplest way to do so is to divide by n, where n is equal to the number of co-authors (see for instance Schreiber, 2008). This approach was criticised, however, since it can discourage collaborations even when it is desirable to associate more than a single author to a paper owing to the complexity and multi-dimensionality of the problems it addresses (for a review of this literature, see Waltman, 2016). These cases are common in modern science, therefore a balance must be found between the need to provide the correct incentive to researchers and that of taking into account the possible bias arising from multi-authorships.
For this purpose, we analyse the relationship between the average number of co-authors per scholar in the group under consideration (i.e. the SSD) and the indicator of choice (i.e. number of publications, number of citations, the h-index value). More specifically, we consider for each individual i his/her average number of co-authors (Avgi) and three exponential models3:
3 The exponential function provided the best fit with respect to an inverse logarithmic or a quadratic polynomial function; results for the latter are available with the authors upon request.
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑎𝑝𝑒𝑟𝑠𝑖= 𝛼 ∗ 𝐴𝑣𝑔𝑖𝛽 (1) 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠𝑖= 𝛾 ∗ 𝐴𝑣𝑔𝑖𝛿 (2)
ℎ𝑖= 𝜇 ∗ 𝐴𝑣𝑔𝑖𝜈 (3)
Results of non-linear least-squares fits of (1), (2) to the whole publication set are presented in Tab. 3: the fit is generally good (see also Fig. 3). The fitting parameters can be used to weight individual author ownership to each paper and the corresponding number of citations. More specifically, each paper with n authors and the associated number of citations are fractionally counted by using the following weighting factors4:
𝜂 = 100 𝛼 ∗ 𝑛𝛽 𝜃 = 100
𝛾 ∗ 𝑛𝛿
For what concerns the h-factor, since it applies to individuals, (3) is separately fit to the three author subpopulations. We obtain three sets of pairs that yield rank-specific weighting factors5:
𝜅 = 100
𝜇 ∗ 𝐴𝑣𝑔𝑖𝜈
Table 3. Number of publications, number of citations and average number of co-authors per researcher
Coefficients Whole population
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑝𝑎𝑝𝑒𝑟𝑠𝑖= 𝛼 ∗ 𝐴𝑣𝑔𝑖𝛽
5.92 (1.47)
0.57 (0.03)
Observations 420
R2 0.82
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑖𝑡𝑎𝑡𝑖𝑜𝑛𝑠𝑖= 𝛾 ∗ 𝐴𝑣𝑔𝑖𝛿
675.27 (180.82)
0.44 (0.03)
Observations 420
R2 0.87
Full professors
Associate professors
Assistant professors ℎ𝑖= 𝜇 ∗ 𝐴𝑣𝑔𝑖𝜈
9.35 (1.96) 10.73 (1.61) 7.65 (1.83)
0.25 (0.03) 0.23 (0.02) 0.25 (0.03)
Observations 96 189 138
R2 0.90 0.89 0.82
4 As a result of the proposed weighting system, single-author papers and their citations have a weight different from one. If the value one is preferred for this case, one can re-scale all weights by dividing each by the n=1 value.
5 It is important to point out that k operates a non linear transformation on the original h index, and as such the resulting indicator must not be interpreted with reference to the original concept of the h index.
STI Conference 2018 · Leiden
Figure 3: Scientific output and average number of co-authors for scientists in experimental physics. The fitted curves are shown in red dotted lines.
We can now redefine the threshold values for the three academic ranks in order to reach an average success rate for the three professor types (Full, Associate, Assistant) analogous to that seen in Tab. 2 and verify the difference among the different quartiles. Results are reported in Tab. 4: scholars with a higher number of co-authors are still obtaining higher success rates, but now 16,7% of those in the first quartile are able to access the higher rank for Full professors, the success rate being as high as 29,8% for Associate and 26.5% for Assistant Professors in the first quartile.
Table 4. Success rates by number of co-authors after correcting for the number of co-authors using the exponential model
Quartile Full professors Associate Professors
Assistant professor
I Quartile 16,7 29,8 26,5
II Quartile 62,5 57,4 57,1
III Quartile 70,8 68,1 88,2
IV Quartile 79,2 87,5 91,4
Total 57,3 60,8 65,9
Figure 4 also reports the distribution of the number of publication and citations and of the h index with respect to the average number of co-authors.
Figure 4: Average number of co-authors and scientific output corrected for the number of co- authors using the exponential model. The thresholds to access the higher academic ranks are
shown as red lines
Discussion
Co-autorship practices can be remarkably different also within sectors that are homogenous in terms of scientific content; this is the case of experimental physics, where in the last few years some groups of researchers operated within large collaborative projects that lead to publications with a very high number of co-authors. This drastically affects the results of selection procedures based on indicators measuring scientific productivity and impact, such as the Italian National Habilitation program. In the paper, considering this latter case, we have shown the severity of this bias effect and the need to address this problem. As a first example we have shown that a correction based on the empirical evaluation of the average-performance of individuals belonging to collaborations of a given size can mitigate this bias and partially balance success chances of individual scientists. Further studies are however needed to satisfactorily address this issue and thus avoid undermining the selection process itself.
References
STI Conference 2018 · Leiden
Bagues M., Sylos Labini M., and Zynovieva N. (2015), Connections in Scientific Committees and Applicants’ Self-Selection: Evidence from a Natural Randomized Experiment, IZA Dicussion Paper n. 9594
Buckland, M. & Gey, F. (1994). The relationship between recall and precision. Journal of the American Society for Information Science, 45, 12-19.
Borgman, C.L. (Ed.). (1990). Scholarly Communication and Bibliometrics. London: Sage.
Bauin, S. & Rothman, H. (1992). ”Impact“ of journals as proxies for citation counts. In P.
Weingart, R. Sehringer & M. Winterhager (Eds.), Representations of Science and Technology (pp. 225-239). Leiden: DSWO Press.
Hoppe, K., Ammersbach, K., Lutes-Schaab, B. & Zinssmeister, G. (1990). EXPRESS: An experimental interface for factual information retrieval. In J.-L. Vidick (Ed.), Proceedings of the 13th International Conference on Research and Development in Information Retrieval (ACM SIGIR ’91) (pp. 63-81). Brussels: ACM.
Kling, R. & Elliott, M. (1994). Digital library design for usability. Retrieved December 7, 2001 from: http://www.csdl.tamu.edu/DL94/paper/kling.html.
Schreiber M (2008) A modification of the h-index: The hm-index accounts for multi-authored manuscripts. Journal of Informetrics Volume 2, Issue 3, July 2008, Pages 211-216
Waltman L. (2016), A review of the literature on citation impact indicators, Journal of Informetrics, 10, 365-391, https://doi.org/10.1016/j.joi.2016.02.007
