Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings

Severe language effect in university rankings:

particularly Germany and France are wronged in citation-based rankings

Raan, A.F.J. van; Leeuwen, T.N. van; Visser, M.S.

Citation

Raan, A. F. J. van, Leeuwen, T. N. van, & Visser, M. S. (2011).

Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings. Scientometrics,

Version: Publisher's Version

License:

Downloaded from:

Note: To cite this publication please use the final published version

(if applicable).

Severe language effect in university rankings:

particularly Germany and France are wronged in citation-based rankings

Anthony F. J. van Raan^•Thed N. van Leeuwen^•Martijn S. Visser

Received: 25 December 2010 / Published online: 13 April 2011

Ó The Author(s) 2011. This article is published with open access at Springerlink.com

Abstract We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurements of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.

Keywords Bibliometric analysis
University rankings
Language of publication

The WoS data system covers a number of journals in non-English languages, particularly in German and in French. Publications in these non-English language journals are counted as part of the output of countries, but they generally have a very low impact because only a few scientists outside Germany, Austria and Switzerland are able to read German; we have a similar situation for French. Thus, these non-English publications will ‘dilute’

considerably the average impact of countries such as Germany, Austria and France. This is particularly the case for the more application-oriented fields such as clinical medicine and engineering, and also for the social sciences and humanities. Because clinical medicine represents a considerable part of the entire scientific output of a country (particularly in western countries), this language effect directly influences the overall impact position of universities (van Leeuwen et al.2001).

This effect has serious consequences for all citation-per-paper based rankings, but also for citations per staff rankings. The most striking result is a too low position of most German and France universities in these rankings, particularly universities with a medical school. As citations play an important role in university rankings such as the Times Higher Education, QS, Shanghai (ARWU) and the Leiden Ranking, all these rankings suffer from this language effect.

A. F. J. van Raan (&)
T. N. van Leeuwen
M. S. Visser

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, The Netherlands e-mail: vanraan@cwts.leidenuniv.nl

DOI 10.1007/s11192-011-0382-1

Thus, for ranking and comparison with countries such as the US, UK and all countries where almost all WoS-covered publications are written in English, calculation of biblio- metric indicators based on only-English publications is a more fair procedure.

We show results in Fig.1for in total 69 German and French universities. We see that the ‘English-only’ citation impact is systematically higher than the ‘all publications’

Citation Impact German and French universities All versus English-only Publications

y = 1,08x R² = 0,81

0,00 0,20 0,40 0,60 0,80 1,00 1,20 1,40 1,60 1,80 2,00

imp-all

imp-eng

Citation Impact Ranking German and French universities All versus English-only Publications

y = 1,02x - 38,31 R² = 0,83

0 50 100 150 200 250 300 350 400 450 500

0,00 0,20 0,40 0,60 0,80 1,00 1,20 1,40 1,60 1,80 2,00

0 50 100 150 200 250 300 350 400 450 500

r-all

r-eng

Fig. 1 Language effect in citation impact of all German and French universities. The upper figure shows the difference in impact (numerical values of the field-normalized citation score) in the case of all WoS (imp-all, horizontal axis) and only-English WoS publications (imp-eng, vertical axis). The lower figure shows the difference in citation impact ranking position in the case of all WoS (r-all, horizontal axis) and only-English WoS publications (r-eng, vertical axis)

496 A. F. J. van Raan et al.

Citation Impact 25 German universities

0,00 0,20 0,40 0,60 0,80 1,00 1,20 1,40 1,60

UNIV HEI DELBERG UNIV MU

NCHEN

UN IV FRAN

KFO RT

UNIV ERL ANGEN T U

MU NC

HEN

UNIV GOTTIN GEN

UNIV D USSELDOR

F

UNIV M AIN

Z

UN IV A

ACHEN

KARLSR UH

ER INST F TECH

N

MED UNIV HANN

OV ER

UNIV WURZ BU

RG UN

IV BO NN

HUMBOLDT UNIV BERLIN UN

IV F REIBU

RG UNIV KIEL

UN IV KO

LN

UNIV P OTSD

AM

UNIV DORTM UND

UNIV ULM UNIV HAM

BURG

UN IV R

EGEN SBUR

G

UNIV STU TTG

ART

UN IV SA

ARLAN DES

UN IV TUBIN

GEN

impact

impact english only impact all

Citation Impact Ranking 25 German universities

0 50 100 150 200 250 300 350

UNIV HEI DELBERG UNIV MU

NCHEN

UNIV F RANKF

ORT

UNIV ERL ANGE

N

T U MUNC

HEN

UNIV GOTTIN GEN

UNIV D USSELDOR

F

UNIV MAINZ UN

IV AACHE N

KARLSRUHER INST F TECH

N

MED UNIV HANN

OV ER

UNIV WURZBU RG UNIV B

ONN

HUMBO LDT U

NIV BERLIN UN

IV FRE IBURG

UNIV KIEL UN

IV KO LN

UNIV POTSD AM

UNIV DOR TM

UND UN

IV UL M

UNIV HAMBU RG

UN IV REG

EN SBU

RG

UNIV ST UTTG

ART

UNIV SAA RLANDES UNIV TUBI

NGEN

ranking

ranking english only ranking all

Fig. 2 Language effect in citation impact of the 25 highest impact German universities. The upper figure shows the difference in impact (numerical values of the field-normalized citation score) in the case of all WoS and only-English WoS publications. The lower figure shows the difference in citation impact ranking positions in the case of all WoS and only-English WoS publications

impact, and the ranking position based on ‘English-only’ citation impact is systematically lower (thus, ‘better’) than based on ‘all publications’ impact, with a few exceptions due to differences in inter-group changes of ranking positions. In Fig.2 the language effect is shown in a slightly different way for the 25 German universities with the highest citation impact. A comparison of the rankings for the 500 largest universities in the world based on all WoS publications as well as on English-only publications is given in our ranking website (Leiden Ranking2010).

In the above context, we focus on the language effect of, particularly European uni- versities in the German and French language areas, but a similar effect will also be present for Italian and Spanish universities. The issue of publication language is related to the problem of ‘local journals’ (but covered in the Web of Science). These journals play only a marginal role in the scientific communication process at an international scale. The pub- lications in these journals mainly transfer knowledge to local or national end-users. For instance, medical knowledge transferred to non-academic hospitals to improve clinical practice. In the technical sciences we find transfer of new knowledge to the engineering practice. As most fields contain local journals, these journals should ideally be excluded from the sets forming the basis for bibliometric studies in international comparison. This is not a simple task and it requires further bibliometric studies.

One can expect that similar language effects will occur when using Scopus data and that this effect will seriously aggravate by the extension of the coverage of the WoS and Scopus with national language journals.

Open Access This article is distributed under the terms of the Creative Commons Attribution Noncom- mercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

References

Leiden Ranking (2010).http://www.socialsciences.leiden.edu/cwts/products-services/leiden-ranking-2010- cwts.html.

van Leeuwen, T. N., Moed, H. F., Tijssen, R. J. W., Visser, M. S., & van Raan, A. F. J. (2001). Language biases in the coverage of the science citation index and its consequences for international comparisons of national research performance. Scientometrics, 51(1), 335–346.

498 A. F. J. van Raan et al.