Comparisons of altmetric, usage and citation indicators in chemistry disciplines

(1)

STI 2018 Conference Proceedings

Proceedings of the 23rd International Conference on Science and Technology Indicators

All papers published in this conference proceedings have been peer reviewed through a peer review process administered by the proceedings Editors. Reviews were conducted by expert referees to the professional and scientific standards expected of a conference proceedings.

Chair of the Conference Paul Wouters

Scientific Editors Rodrigo Costas Thomas Franssen Alfredo Yegros-Yegros

Layout

Andrea Reyes Elizondo Suze van der Luijt-Jansen

The articles of this collection can be accessed at https://hdl.handle.net/1887/64521 ISBN: 978-90-9031204-0

This ARTICLE is licensed under a Creative Commons Atribution-NonCommercial-NonDetivates 4.0 International Licensed

(2)

Comparisons of altmetric, usage and citation indicators in

chemistry disciplines

Pei-Shan Chi^*, Juan Gorraiz^** and Wolfgang Glänzel^***

*peishan.chi@kuleuven.be

ECOOM, KU Leuven, Naamsestraat 61, Louvain, 3000 (Belgium)

** juan.gorraiz@univie.ac.at

University of Vienna, Vienna University Library, Bibliometrics and Publication Strategies, Boltzmanngasse 5, Vienna, 1090 (Austria)

***wolfgang.glanzel@kuleuven.be; glanzw@iif.hu

ECOOM and Dept MSI, KU Leuven, Naamsestraat 61, Louvain, 3000 (Belgium)

Library of the Hungarian Academy of Sciences, Dept. Science Policy and Scientometrics, Arany János u. 1, Budapest, 1051 (Hungary)

Introduction

Altmetrics indicators provide a new perspective beyond the scholarly impact of research on research itself, but also of the impact of research on other segments of society (Bornmann, 2015; Moed, 2017;), drawing scholars’ attentions to categorizing metrics according to different types of communication, although the particular user groups are not always specified and use is often not fully formalised. Nonetheless, Plum Analytics provides insights into the ways people interact with individual pieces of research output in the online environment¹, dividing the altmetrics service PlumX Metrics into five categories, Usage, Captures, Mentions, Social Media, and Citations. In a summary, most of the previous studies have found some degree of correlation between altmetrics and citations indicators, suggesting that these two approaches are somehow related but not the same, and support the hypothesis that they should rather be considered as complementary sources providing different points of view (Costas, Zahedi &

Wouters, 2014; Gorraiz et al., 2018).

Since the presence and density of social media altmetric counts are very low and not very frequent among scientiﬁc publications (Costas et al, 2014; Zahedi, Costas & Wouters, 2014;

Peters et al. 2017), in this paper we extend the perspective of the scholarly impact from the traditional citations to other forms of scientific communication among scholars, which are not necessarily connected to knowledge production directly, such as Mendeley Readership and usage counts. There are in total six indicators out of three categories including Captures, Citations, and Usage analysed in the study. The focus of this study is to identify those measures that are suited to supplement traditional impact metrics and to investigate the differences between those altmetric indicators and the robustness of the parameter-free distribution-based method Characteristic Scores and Scales (CSS; see Glänzel et al., 2014) among all kinds of

1 https://plumanalytics.com/learn/about-metrics/

(3)

indicators. The special cases among different indicators with contradicting strength of impact will also be discussed in the second part of the study. In short, we attempt to outline an overview of all the available metrics based on scholarly communication in the mirror of the PlumX and WoS data.

Methodology

Data sources and data processing

For this pilot study, we extracted all papers in two subfields of chemistry from the WoS Core Collection. According to the modified Leuven-Budapest classification system (see Glänzel, Thijs & Chi, 2016), there are 7 subfields within the major field Chemistry. The disciplines analytical, inorganic & nuclear chemistry (analytical chemistry in short) and organic &

medicinal chemistry (organic chemistry in short) were selected to represent the major field and compare the differences between each other. The reason for this selection was a pragmatic one:

On the one hand, documents do need a DOI to combine data extracted from citation indexes with altmetrics. In addition, the PlumX dataset was restricted to about 40.000 papers per set and the annual collections of WoS Core Collection indexed journal papers assigned to the two chemistry disciplines met this condition. Table 1 shows the detailed definitions of those subfields. In this study we analyse only the publications published in 2013 to ensure a similar period for accumulating citations, readership counts and usage counts simultaneously.

Furthermore, the choice of 2013 allowed the collection of citations in a four-year window. We have extracted all papers with a valid DOI number. Most so-called citable articles had a DOI number, which, in turn, was the basis for the matching with PlumX data. Only seven unmatched DOIs and ten wrongly-indexed DOIs by WoS occurred during the matching process. In total, 39,736 papers in analytical chemistry (C1) and 27,531 papers in organic chemistry (C3) could be matched with PlumX Metrics and WoS to obtain their altmetric indicators.

Table 1. The definitions of analysed subfields in the study.

As already mentioned above, all the collected signals are categorised automatically in PlumX into five separate dimensions: Usage, Captures, Mentions, Social Media, and Citations (Torres Salinas et al., 2017). This categorisation may be subject to criticism, but one big advantage of PlumX is that the results are differentiated in the resulting dataset for each measure and its origin and can be aggregated according to the user criterion.

Indicators

Based on our research objective and due to the very low percentages of data availability for some indicators, we had to limit the analysed PlumX indicators in this study into four indicators out of three categories: Captures, Citations, and Usage. The reason was as follows. Several

‘altmetric’ indicators are requiring full text, the availability of which is often not granted.

Because of this, bias we have removed those metrics that had more than 50% zero values. In turn, we added citation and usage counts from the Web of Science platform, both for a four-

Code Subfield WoS Subject category

C1 Analytical, Inorganic & Nuclear Chemistry

Chemistry, Analytical

Chemistry, Inorganic & Nuclear Spectroscopy

C3 Organic & Medicinal Chemistry Chemistry, Medicinal Chemistry, Organic

(4)

STI Conference 2018 · Leiden

year window. As a result we were able to “synchronise” all metrics for the same observation period. The complete description of the applied indicators is summarised in Table 2.

 Captures

Mohammadi and Thelwall (2014) suggests that Mendeley readership data could be used to help capture knowledge transfer across scientific disciplines, as well as giving impact credit at an earlier stage than citation counts. In their study, low and medium correlations between Mendeley bookmarks and citation counts in the social sciences and humanities disciplines were found, indicating that these measures reflect different aspects of research impact. The use of captures is reinforced by previous studies reporting that the highest coverage or degree of data availability is provided by the number of readers in Mendeley independently of the knowledge area (Zahedi et al, 2014; Gorraiz et al. 2018).

Table 2. The definitions of used indicators in the study.

Source: plumanalytics.com and webofknowledge.com  Citations

Citations are seen as a valid indicator of research quality in quantitative studies, assuring an “unobtrusive measure” of the impact of cited documents (de Bellis, 2009, p.14).

Compared to other altmetric indicators, citation counts reflect the highest scholarly impact because of the usage remarks shown in scientific publications. Different platforms provide different citation counts based on their coverages, and are worthy being compared with the differences.

 Usage

Usage indicators can be considered as a supplement to citations accounting for early discovery as opposed to formal recognition by other scholars (Bollen et al., 2008;

Gorraiz et al., 2014; Chi & Glänzel, 2017). Even though the definitions of ‘usage’ in the WoS and abstract views in EBSCO are more peripheral than, e.g., that of citations or full-text downloads, the usage indicators are potential measures showing the interests and motivations of scholars. This measure provides a different perspective of knowledge transfer from the users of interdisciplinary databases.

The low or almost inexistent number of signals traced in mentions and social media are in good agreement with recent results reporting a very low percentage of data availability for these

Category Metric Source Description

Captures Readers Mendeley The number of people who have added the artifact to their library/briefcase on Mendeley

Citations Citation Indexes

CrossRef The number of articles that cite the artifact according to CrossRef

Scopus The number of articles that cite the artifact according to Scopus

WoS The number of articles that cite the artifact according to WoS

Usage

Abstract Views EBSCO The number of times the abstract of an article has been viewed on EBSCO

Information

needs WoS

The number of times the full text of a record has been accessed or a record has been saved on WoS since 1 February 2013

(5)

categories or dimensions in the natural sciences in comparison to the life sciences or to the social sciences (Gorraiz et al., 2018).

Results Overview

Before we examine the data, we show the mean values of each indicator for the two subfields as analysed in the study in Table 3, to provide an overview of our dataset. Among the citation indicators, chemistry publications have higher citation counts in Scopus than CrossRef and WoS. The abstract views calculated on EBSCO are higher than the “usage” counts on WoS.

Table 3. Average values of six indicators for the two subfields in chemistry Code Papers Mendeley CrossRef Scopus WoS_C EBSCO WoS_U

C1 39,736 7.57 10.94 12.27 8.27 47.90 33.95

C3 27,531 8.06 11.66 13.39 9.06 37.89 24.00

Regression analysis

In first step we have applied a linear regression analysis to study the relationship between selected altmeric indicators themselves and the traditional impact measures. Due to the highly skewed distributions of all indicators, Spearman correlation was applied in this study instead of Pearson correlation.

Figure 1: Spearman correlations between six indicators in analytical chemistry (C1)

(6)

Figure 2: Spearman correlations between six indicators in organic chemistry (C3) Figure 1 and 2 show that the correlations are very similar in both samples but slightly higher in analytical than in organic chemistry. The three citation indicators are highly correlated with each other, and have low and medium correlations with Mendeley readership and usage at WoS Core Collection platform. The WoS usage count measuring bibliographic records saving and full-text clicking is the closest indicator to the citation ones. The low correlation between citations and Mendeley readers is basically in line with the results of Mohammadi and Thelwall (2014) in the social sciences and humanities. The abstract view at EBSCO correlates with other indicators at very low degree. It provides the most distinct perspective of the impact of publications from other scholarly communication.

Characteristic Scores and Scales

Characteristic Scores and Scales (CSS; e.g., Glänzel et al., 2014) have originally been used to reduce the distribution of citations over scientific papers to a standard of a small number of classes instead of the theoretically possible unlimited number of events (i.e., citations). In practice, the use of three of four classes has proved to be successful. CSS classes were very robust with regard to subject, publication time and citation window. Although CSS is a parameter-free approach and independent of pre-set percentiles, the distribution over the classes from the lowest class (Class 1 = poorly cited) over the moderate one (Class 2 = fairly cited) to the highly cited classes (Class 3 and 4) with about 70%–21%–6.5%–2.5% proved quite stable.

The method could also be applied with similar results in different contexts, e.g., to usage counts (cf. Chi & Glänzel, 2017). The scores defining the thresholds for the class intervals, do – as all impact measures – depend on subject and time specific factors. In this subsection we apply CSS to all selected metrics to test the robustness of the method and to identify metric-specific peculiarities and to find possible concordance between the respective metric standards.

(7)

Table 4 and Table 5 show the similar distribution patterns of all the indicators in the two subfields. The four classes of different indicators are divided into different sizes but keep some robustness among different indicators and disciplines. The most distinct indicator is the abstract view at EBSCO, which has the most skewed distribution. Even more interesting is the concordance of scores. The first one is 0 by definition as it defines the lower bound for the

“poor” class allowing also for zero captures, usage or citation. Interestingly, the score for Captures and Citations are of the same magnitude, while Usage shows rather similarities with full-text download statistics and can grow up to a higher order of magnitude than citations (cf.

Glänzel & Heeffer, 2014).

Table 4. Comparison of the six indicators of papers in 2013 in analytical chemistry (C1).

Captures Citations Usage

Class Mendeley CrossRef Scopus WoS_C EBSCO WoS_U Score % Score % Score % Score % Score % Score % 1 (poorly cited) 0.00 70.6% 0.00 66.1% 0.00 68.2% 0.00 68.1% 0.00 76.6% 0.00 65.8%

2 7.57 20.7% 10.94 23.3% 12.27 22.2% 8.27 21.4% 47.90 17.5% 33.95 24.0%

3 22.84 6.4% 24.48 7.5% 28.05 6.7% 18.98 7.4% 156.98 4.3% 72.16 7.0%

4 (highly cited) 45.17 2.4% 43.44 3.2% 49.84 2.9% 32.24 3.2% 380.18 1.5% 128.19 3.2%

Table 5. Comparison of the six indicators of papers in 2013 in organic chemistry (C3).

Captures Citations Usage

Class Mendeley CrossRef Scopus WoS_C EBSCO WoS_U Scor

e % Scor

e % Score % Scor

e %

1 (poorly

cited) 0.00 69.9

% 0.00 65.9

% 0.00 66.4

% 0.00 68.0

% 0.00 75.6

% 0.00 62.5

%

2 8.06 21.6

% 11.66 23.1

% 13.39 23.3

% 9.06 21.6

% 37.89 17.9

% 24.00 24.9

% 3 23.08 6.1% 24.73 7.5% 28.07 7.1% 19.81 7.1% 117.4

6 4.5% 45.95 8.6%

4 (highly cited) 45.52 2.4% 41.84 3.5% 48.07 3.1% 32.92 3.3% 265.7

7 2.0% 72.42 4.0%

Special Cases

In the previous sections, we have looked at general patterns of and similarities among the selected metrics. Now we will have a look at deviations and extreme values that are not necessarily shared by all metrics.

In Table 6 and Table 7, we see those articles with frequent abstract views at EBSCO rarely with other high indicator values in the two subfields. For example, the article 10.1016/j.poly.2012.12.037 has the largest number of views among all the 39,736 papers in analytical chemistry but its values of other indicators are all very small. To explore more among those highly viewed articles on EBSCO, we found that they are mainly from the journals Separation and Purification Reviews (C1), Journal of Ethnopharmacology (C2) and on the topics of adsorption and pharmacology, respectively. Some articles specially have lots of

(8)

Mendeley readers or WoS Usage counts and are not cited or viewed a lot at other platforms. In contrast, three citation indicators are more or less associated with each other, showing similarity among the small group.

Table 6. Special cases have extreme performance in a specific indicator in analytical chemistry (C1).

MD = Mendeley, CR = CrossRef, SC = Scopus, WC = WoS Core Collection Citations, EB = EBSCO and WU = WoS Usage.

Table 7. Special cases have extreme performance in a specific indicator in organic chemistry (C3).

Conclusions

The altmetric indicators covering three dimensions related to scholarly communications are compared in this study and significant correlations at different degrees among each other were reported. Three citation indicators of different platforms are highly correlated with each other.

Mendeley readership and WoS usage counts have low and medium correlations with other indicators. Strikingly, the two usage metrics (WoS and EBSCO) do not strongly correlate either.

The abstract view at EBSCO shows a distinct perspective of the impact of publications from other scholarly communications, with very low correlations with other indicators. All these results are in very good agreement to the ones reported for all Austrian publications published in the years 2014, 2015 and 2016 independently of the subject area (Gorraiz et al., 2018).

The CSS method was applied to different types of altmetric indicators for the first time. The result proves the robustness of the method again. Compared to other indicators, abstract view at EBSCO has the most skewed distribution. This method was also successfully used in order

DOI MD CR SC WC EB WU

10.1146/annurev-anchem-062012-092628 1,169 212 237 130 60 136 10.1016/j.jqsrt.2013.07.002 479 1,109 1,527 788 340 227 10.1016/j.poly.2012.12.037 0 14 16 13 15,769 15 10.1080/15422119.2012.758638 0 13 10 7 6,354 0 10.1080/15422119.2013.795902 0 16 14 10 5,538 0 10.1080/15422119.2013.821996 1 64 83 40 7,919 0 10.1016/j.ccr.2013.01.012 92 372 404 286 37 233 10.1021/ic401215x 1,435 1,391 1,394 803 66 1,730 10.1016/j.bios.2013.04.034 149 196 217 146 430 1,116

(9)

to identify special cases, that means publications with extreme values in one of the metrics but not in the other ones. For these “outliers”, especially in analytical chemistry, the number of abstract views was extremely high reaching even 15,000 views, while the values of the other indicators never surpassed 1,500 scores.

The results of this study suggest that different metrics are somewhat related to each other but they also their own characteristics. Captures and Usage indicators measure very different aspects of research impact, although both of them show some similarity from the perspective of citations. Once again we can conclude that we are dealing with different forms of information use and impact as it has already been stressed by Glänzel & Heeffer (2014) in the context of download statistics. While citations are integral part of scholarly communication and intimately connected with the process of knowledge creation - as citing literature results in publishing new literature -, downloads are not closely related to documented or published scholarly communication. While the user group underlying citation metrics obtained from bibliographic databases is clearly delineated and comprises of citers, which are at the same time peers, documents might be downloaded by any user with access but without using or incorporating downloaded information in own and new published research. Thus downloads and ‘clicks’

certainly express interest and attention but they do not specify user groups and the actual use of information, and, therefore, its possible context remains obscure. Thus part of downloads and usage might even be ‘deadlocked’ as nobody knows what is done after all with downloaded documents or information available after clicking links.

Some limitations were encountered by performing this study. In the first place, matches between altmetrics and bibliographic information are based on correct DOI information² but not all documents indexed in bibliographic databases like Web of Science Core Collection have a DOI, although the share of documents without DOI is constantly shrinking (Gorraiz et al., 2016). Furthermore, in this study we only focused on the field of chemistry. The results for two subfields in this major field were very similar but it would be necessary to broaden the scope of the study to other disciplines in order to gain reliable and consolidated new insights of more general validity. A further limitation is due to the required availability of full text for the calculation of several metrics. Hence we had to omit full-text based metrics with overwhelming zero frequencies. We hope that open science and open access initiative will improve the situation in the near future. Finally, the two usage-related indicators applied in this study all have their limitations in terms of the usage definition. WoS usage only counts bibliographic records saving, full-text clicking, EBSCO abstract-views and calculates the usage of the abstract page on the website. The distance between this type of “usage” and document “download”

needs to be taken into account.

Acknowledgement

Altmetrics data for the two chemistry disciplines was provided by courtesy of Plum Analytics.

The authors thank Christina Lohr, Stephanie Faulkner and Tina Moir from Elsevier for granted trial access to PlumX. WoS Citation data were sourced from Clarivate Analytics Web of Science Core Collection.

References

2 Of course urls and other non-permanent identifiers can also be used, but they might reduce considerably the accuracy of the matches and the correctness of the traced scores.

(10)

Bollen, J., Van de Sompel, H. (2008) Usage impact factor: The effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59(1), 136-149.

Bornmann, L. (2015). Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123-1144.

Chi, P.S., & Glänzel, W. (2017). An empirical investigation of the associations among usage, scientific collaboration and citation impact. Scientometrics, 112(1), 403-412.

Costas, R., Zahedi, Z., & Wouters, P. (2014). Do “Altmetrics” Correlate With Citations?

Extensive Comparison of Altmetric Indicators With Citations From a Multidisciplinary Perspective. Journal of the Association for Information Science and Technology, 66(10), 2003- 2019.

de Bellis, N. (2009). Bibliometrics and citation analysis: from the Science Citation Index to Cybermetrics. Lanham, MD, USA: Scarecrow Press.

Glänzel, W., Thijs, Debackere, K. (2014). The application of citation-based performance classes to the disciplinary and multidisciplinary assessment in national comparison and institutional research assessment. Scientometrics, 101(2), 939–952.

Glänzel, W., Heeffer, S. (2014), Cross-national preferences and similarities in downloads and citations of scientific articles: A pilot study. In: E. Noyons (Ed.), “Context Counts: Pathways to Master Big and Little Data”. Proceedings of the STI Conference 2014, Leiden University, 207–215.

Glänzel, W., Thijs, B. & Chi, P.S. (2016). The challenges to expand bibliometric studies from periodical literature to monographic literature with a new data source: the book citation index.

Scientometrics, 109(3), 2165-2179.

Gorraiz, J., Gumpenberger, C., Schloegl, C. (2014). Usage versus citation behaviours in four subject areas," Scientometrics, 101 (2), 1077–1095.

Gorraiz, J., Melero-Fuentes, D., Gumpenberger, C. & Valderrama-Zurián, J.C. (2016) Availability of digital object identifiers (DOIs) in Web of Science and Scopus. Journal of Informetrics, 10 (1), 98-109.

Gorraiz J., Blahous B., Wieland M. (2018) Monitoring the Broader Impact of the Journal Publication Output on Country Level: A Case Study for Austria. In: Erdt M., Sesagiri Raamkumar A., Rasmussen E., Theng YL. (eds) Altmetrics for Research Outputs Measurement and Scholarly Information Management. AROSIM 2018. Communications in Computer and Information Science, vol 856. Springer, Singapore

Moed, H.F. (2017). Applied Evaluative Informetrics. Springer International Publishing. ISBN:

978-3-319-60521-0

(11)

Mohammadi, E., & Thelwall, M. (2014). Mendeley Readership Altmetrics for the Social Sciences and Humanities: Research Evaluation and Knowledge Flows. Journal of the Association for Information Science and Technology, 65(8), 1627-1638.

Peters, I.; Kraker, P-; Lex, E.; Gumpenberger, C. and Gorraiz, J. (2017). Zenodo in the spotlight of old and new metrics. Frontiers in Research Metrics and Analytics;

https://doi.org/10.3389/frma.2017.00013.

Torres-Salinas, D., Gumpenberger, C., Gorraiz, J. (2017). PlumX As a Potential Tool to Assess the Macroscopic Multidimensional Impact of Books. Frontiers in Research Metrics and Analytics, 03 July 2017. DOI:10.3389/frma.2017.00005 (2017).

Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross- disciplinary analysis of the presence of ‘alternative metrics’ in scientiﬁc publications.

Scientometrics, 101(2), 1491-1513.