Eight clusters : a dynamic perspective and structural analysis for the evaluation of institutional research performance
Thijs, B.
Citation
Thijs, B. (2010, January 27). Eight clusters : a dynamic perspective and structural analysis for the evaluation of institutional research performance. Retrieved from
https://hdl.handle.net/1887/14617
Version: Not Applicable (or Unknown)
License: Licence agreement concerning inclusion of doctoral thesis in the Institutional Repository of the University of Leiden
Downloaded from: https://hdl.handle.net/1887/14617
Note: To cite this publication please use the final published version (if applicable).
1 GENERAL I
NTRODUCTION
1.1 Background
In the last decade several national systems or exercises to assess research performance of institutes of higher education were created. Policy makers nowadays are more and more faced with voters and taxpayers who ask for accountable government spending. One of the reasons is the limited availability of funds at government level (Hodges, et al. 1996) and policy makers have to take responsibility for the choices they make. Another reason is the demand for more competition and transparency in scienti c research (Fisch and Daniel, 1986, 1988). Then, assessment of quality of research performance can be a fruitful tool in supporting the policy decisions and a transparent mechanism for distribution of research funding.
In the UK, the results of the 2008 Research Assessment Exercise (RAE) have been published recently. This RAE is a framework for the evaluation of research activities at UK higher education institutes using peer review. From 2010 onwards metrics will become an important element in the assessment of research in the elds of natural science, technology, engineering and medicine. In other
elds, such as mathematics, social sciences, arts and humanities metrics will be used to inform peer review. Up to 2008 it was a qualitative exercise based on 67 units of assessment . The results, quality pro les, are used by the funding bodies.
Since 2004, the Research Quality Framework was developed in Australia and at the beginning of 2008 it was replaced by the Excellence in Research for Australia (ERA) initiative. In this new initiative both quantitative measures and peer review are combined. One of the leading Australian centres for systematic evaluation and mapping of research, the Research Evaluation and Policy Project (REPP) at the Australian National University in Canberra, recognises the value of both qualitative and quantitative approaches to research assessment.
The Performance Based Research Fund in New Zealand is a program to ensure excellence in research at institutes at higher education. The rst Quality Evaluation was completed in 2003, and then a partial round was held in 2006. The 2012 round will be a full round again. These research performance assessment are used as a base for the funding to reward quality of researchers.
In the Netherlands, CWTS (Centrum voor Wetenschaps- en Technologiestudies) is taking the lead in quantitative studies of science and technology and application of these studies in research assessment projects.
The Norwegian Research Institute for Studies in Innovation, Research and Education (NIFU-STEP) receives grants from the Research Council of Norway to manage a comprehensive institutional publication database (FRIDA) which is used for institutional funding.
In 2004 the Italian government started the Three-Year Assessment Exercise (VTR). This exercises uses peer review to assess the research performance of academic institutes (both universities as well as public research agencies) (Reale et al., 2007)
In Flanders the BOF-key was developed to support research funding and policy making (BOF: Bijzonder Onderzoeksfonds, Special Research Funding).
This key for the allocation of money to the Flemish universities includes publications and citation counts among other indicators. The rst time that funds were distributed based on this key was for scal year 2003 (Debackere & Glänzel, 2004).
The theme of research evaluation has already a long history in the eld of scientometrics. In the 1970s Francis Narin coined to concept ‘Evaluative Bibliometrics’ (Narin, 1976) and Gar eld discussed whether citation analysis is a legitimate evaluation tool (Gar eld, 1979). In the eighties and nineties many papers and books were published on this issue. The journal ‘Research Evaluation’, completely devoted to the topic, was founded in 1998. In 2006 the rst volume of the Scientometrics Guidebook Series was published. In this Guidebook 66 papers, all published in Scientometrics, about evaluations of individual scientists and research institutions were compiled. This extensive knowledge has been used in many evaluation projects at the national level or for individual institutions.
All the above mentioned developments focus on research performance at institutes of higher education. But these institutes do not have a monopoly on scienti c research. Also hospitals, companies, government agencies, specialized institutes do publish and information on their performance can be relevant. Many of them even receive public funding for their research activities and are at some point eligible to assessment. Several other problems surface when looking in detail at some of the above mentioned methodologies. In the next section the most common problems are identi ed. This thesis presents a framework for quantitative research evaluation that helps to solve these problems.
1.2 Common problems with research assessment
1.2.1 Field dependence.Characteristics of research ‘culture’ and the related communications practices are strongly correlated with the elds in which the institutions are active.
This has been extensively reported in literature (e.g., Schubert & Braun, 1986, 1996;
Schubert et al, 1987, Glänzel and Schoep in, 1999). Other authors investigated also the relation between institutional rankings and eld differentiation. For instance,
Calero et al (2008) analyse the research performance of 386 institutes and ranked them using bibliometrics indicators both on all elds combined and, separately, in oncology. They show that the average number of positions that universities move from one ranking to the other is 103 and they conclude that it would be appropriate to rank universities per research eld.
Only few of the above mentioned exercises use different methodologies for the assessment of performance in different elds in the sciences. Some of them, like BOF nowadays, include weighting schemes to correct for eld differences.
Others apply different methodologies for the social sciences and humanities.
However, the performance of institutions in sciences and life sciences is assessed without making a distinction between eld specialization of these institutions.
Therefore the use of eld normalized indicators is crucial in the assessment of individual institutes. The Leiden ranking (CWTS, 2009) uses the CWTS crown indicator which is a eld normalized citation indicator (see Moed et al., 1995; van Raan, 2004).
1.2.2 Comparability
The national systems of research assessment have one big disadvantage in common. They are all just national systems. Trans-national comparison between institutes is dif cult because as each country has its own system. This is especially a problem within Europe where the creation of the European Research Area (ERA) and the Bologna process try to develop a completely integrated European system of higher education which makes trans-national comparisons between institutes crucial to policy makers and stakeholders (Bonaccorsi et al. 2007).
Also the above mentioned relation between eld and research activity has in uences on the validity of comparisons between institutes. A comparison of bibliometrics indicators between institutes can only lead to valid results if these institutes are active in similar elds. This requirement is not often met both in national exercises and in ranking schemes.
1.2.3 Quantitative versus qualitative approach
In research evaluation there is an ongoing debate about quantitative versus qualitative approach. Up to 2008 the RAE had a qualitative approach with peer review. The next round, in 2010, standard quantitative indicators for the assessment of performance in the elds of natural sciences, technology, engineering and medicine will be applied and will play an important role. For elds in the Social Sciences, Arts and Humanities a combination of both approaches will be used while keeping the role of peer review central.
Contrary to the above, the Australian Research Quality Framework was an indicators-only approach but has been altered towards peer-review system using metrics as important, additional information.
This is in line with recommendation by many authors. For instance, Debackere and Glänzel (2004) state that qualitative and quantitative methods in science studies should complement each other. Moed (2007) makes suggestions for the combination of bibliometric indicators and the peer-review process. Butler (2007) held also a plea for a balanced approach
In this thesis, I focus on the quantitative component of the evaluative process which will be embedded in a international context using state of the art bibliometric indicators. It has to be stressed that the presented classi cation system and dynamic perspective alone cannot serve as a standalone system for the evaluation of individual researchers or research institutions but that it should be combined with qualitative approaches like peer review or others.
1.3 Introduction in bibliometrics
Because of the focus on the quantitative component in this dissertation a short introduction on bibliometrics is given.
1.3.1 Defi nition and history
Several de nitions of the term ‘Bibliometrics’ are proposed. The French word ‘Bibliometrié’ was introduced by Paul Otlet in 1934. The English term appeared in the Journal of Documentation in 1969. It was proposed by Alan Pritchard and also used in the title of a paper by Robert Fairthorne. Pritchard (1969) de ned bibliometrics as
‘...the application of mathematical and statistical methods to books and other media of communication’.
Around the same time Nalimov and Mulchenko (1969) introduced the term ‘Scientometrics’ as
‘...the application of those quantitative methods which are dealing with the analysis of science viewed as an information process’.
Many others gave de nitions of the term ‘Bibliometrics’, eg. Fairthorne (1969), Hawkins (1977), Burton (1988), Egghe (1988), Tague-Sutcliffe (1992).
Broadus (1987) wrote a paper on the de nition of bibliometrics. He examined and evaluated several de nitions and proposed a new one:
‘(It is) the quantitative study of physical published units, or of bibliographic units or surrogates of either’.
Ten years after Pritchard and Nalimov introduced their de nitions and Nacke (1979) coined the term ‘Informetrics’. Tague-Sutcliffe (1992) de ned this as ‘the study of quantitative aspects of information in any form’ which makes it more general than the two other terms. In the last decades both terms bibliometrics and scientometrics have become more or less synonyms (Braun et al., 1985).
The discoveries of distributions or laws in statistical analysis of scienti c literature were of great importance to the development of the eld (Hood, 2001).
Lotka stated in 1926 that (Lotka’s Law):
‘the number (of authors) making n contributions is 1/n² of those making one’.
A few years later Bradford (1934) wrote:
‘a given subject, they may be divided into a nucleus of journals more particularly devoted to the subject and several groups or zones containing the same number of articles as the nucleus when the numbers of periodicals in the nucleus and the succeeding zones will be as 1: b : b² …’.
Zipf’s Law (Zipf, 1949) concerns word frequency but can be seen as a more general form of the two other laws. rf=C, where r is the rank of a word, f is the frequency and C is a constant. Lotka’s Law can be rewritten as a special case of Zipf’s Law as YXn=C with Y=the number of authors with X articles, C=number of authors with only 1 article and n=2.
The publication of Derek de Solla Price’s book ‘Little Science – Big Science’ (1963) indicated a revolution. Science as it was known before came to its end. It had taken such a sharp rise after World War II that existing systems for information retrieval failed and funding and evaluation based on personal knowledge and peer reviews became more and more dif cult (Glänzel, 2008
From the seventies onward the growth of the eld really took off.
Scientists active in all kinds of elds got interested in bibliometrics and applied their knowledge to the study of science and scienti c communication. The input from these researchers with such a diverse background (Mathematics, Sociology, Psychology, History of Science or even Chemistry) made bibliometrics to the multidisciplinary eld it is today. With the creation of the journal Scientometrics in 1979 and rst international conferences at the beginning of the eighties a distinct discipline was established (Glänzel, 2008).
1.3.2 Bibliographic databases
Crucial in the development of the eld was also the availability of bibliographic data on scienti c publications. In such a database all information necessary to identify unambiguously a speci c publication is mandatory. This includes names of authors, title, publisher in case of a book or source title and volume or issue for a paper in a periodical. Additional information such as abstract, references, keywords, author af liations makes the database much richer and opens up a wide range of opportunities for bibliometric analysis. It is important to notice that none of the available databases were intentionally created for bibliometric research but as a tool for individual researchers to easily nd valuable information relevant to their own research.
The rst important database containing bibliographic data –the Science Citation Index, SCI– was created by the Institute for Scienti c Information (ISI) in Philadelphia (PA, USA). It had several advantages over other bibliographic indexes at that time. It was a multi-disciplinary database, it contained information on all authors and all addresses in the by-line of a paper. But most important, it contained all references included in a paper. This information allows to track down all the cited papers and create links between citing and cited publications. Other databases like MEDLINE, INSPEC or Chemical Abstracts lack this completeness of information. Upon its conception the founder of ISI, Eugene Gar eld, created metrics to measure the impact of journals enabling him to decide whether journals were important or relevant enough to be indexed in the database. This database was later expanded (SCI-E) and additional sources were added with the Social Science Citation Index and the Arts & Humanities Citation Index. Nowadays, ISI is part of Thomson Reuters and the database is accessible through a web interface Web of Science (WoS) which itself is part of a larger research information platform Web of Knowledge (WoK).
In 2002 the publisher Elsevier started with the creation of its own multi- disciplinary database ‘Scopus’ which was launched at the end of 2004. It claims to be the largest abstract and citation database of research literature and quality web sources. It does not only include papers published in journals but also results from web pages and patents. Elsevier aimed at creating a database with a broad coverage. This is con rmed by a study of Lopez-Illescas et al. (2008) in which they found that oncology journals covered in the Web of Science is a subset of the journals covered in Scopus. Scopus covered 106 additional oncology journals. Despite this broader coverage, the SCImago Journal Rank correlates high with WoS-based impact factors. So it remains unclear what the bene ts are of this broader coverage for bibliometric studies. Furthermore, because the erce competition between Elsevier and Thomson Reuters, it remains to be seen which of the databases (Scopus or WoS) will end up with the broadest coverage.
Google Scholar is a freely available alternative to the above mentioned fee- based databases. It was launched in the same month as Scopus (November 2004) (Payne, 2004; Bar-Ilan, 2008). Data is indexed if an abstract is online available to non-subscribers of the journal who come from Google and Google Scholar.
However, Jasco (2006) pointed out several shortcomings and inconsistencies of Google Scholar, at that time for instance ‘I Introduction’ was indexed as an author of a huge number of papers.
Several authors compared these databases on their appropriateness for bibliometric studies. Bar-Ilan (2008) showed that the choice of citation tool has an impact on computations of the h-index. Bornmann et al. (2009a) compared references in three databases (WOS, Scopus, Chemical Abstracts) to papers published in Angewandte Chemie or other chemistry journals with the citations
based on Google Scholar data. Their study shows a convergent validity of the citation analysis from the fee-based databases but a lack for the analysis based in Google Scholar.
Based on the existing literature and our own long experience with the use of the Web Of Science we are con dent that it is a legitimate choice to use this database for evaluative purposes. Its features comply with the requirements of bibliometric methodology.
1.3.3 Indicators and bibliometric tools
The rst unit of analysis in bibliometric studies is the published paper.
For one paper several variables can be collected: number of authors, number of addresses (institutions, cities, countries) involved, number of citations received within a certain time frame, expected citation rate based on journal or eld, discipline classi cation. These variables can be aggregated at three subsequent levels:
• Micro level: Publications of individual researchers or research groups
• Meso level: Publications of institutions or large departments or faculties.
Studies of individual journals are also regarded as belonging to the meso level
• Macro level: studies at regional or national level or higher aggregations,
eld or discipline studies.
Each of these levels demands for its own approach. It is obvious that evaluative studies at the micro level require retrieval and cleaning procedures with the highest possible accuracy while at the macro level these studies are less prone to data retrieval errors. At each level a different implementation of bibliometric concepts can be appropriate. A self-citation occurs whenever an entity refers in one of its papers to another paper that is assigned to the same entity. This de nition can be applied to authors or journals or institutions but evidently these choices require other computations.
For the calculation of research productivity different counting schemes can be applied. These schemes refer to the rules used to count the publications assigned to the entity under study (Glänzel, 2008).
Fractional counts: In this scheme the contribution of one entity is fractionated by the total number of involved entities, they share one publication or citation. The advantage of this scheme is that the share of entities at one level can be added up at a higher level of aggregation. However this does not hold in all cases and can be problematic sometimes, e.g. one single author with appointments at two different institutions in different countries both mentioned on one publication.
First address or rst author count: Only the rst author or the institution or country mentioned in the rst address is taken into account. This scheme implements the notion that the rst author is the main contributor to the publication. However
in some elds or institutions it is common to list the authors in alphabetical order which makes the notion and thus the counting scheme invalid. The scheme is unavoidably used when the bibliographic database does not include all addresses.
Full or integer counting: The publication is counted as full for each of the unique contributing entities, which can be institutions, institutes, authors and countries. One could say that with this scheme contributions to just the paper itself are counted. If someone’s name (or an institution) appears in the heading of a paper this means that he (it) has contributed. This is a binary variable. As it is impossible to measure the exact share of contribution of each entity involved in an automated manner many authors apply this scheme in their studies nowadays.
Once publications are assigned to entity different indicators can be used to present and compare publication activity. These indicators can be used to express the number of publications per author, institution, country, eld or journal. As special cases, the number of uncited as well as the number of highly cited papers (during a speci c time window) can also be reported. It also allows ranking the entities with respect to their output. Different types of indicators are possible.
Absolute indicators give the mere number of output produced by the unit.
Sometimes they can be quite impressive but they are hardly informative on the position of the entity in comparison to its peers or competitors. Relative indicators relate a entity’s performance with a speci c reference. This reference can be located either within or outside of the entity under study. For instance the share of international publications of an institution relates the number of internationally co- authored papers to the total output of that institution which is an internal reference.
A special relative indicator is the Activity index introduced by Frame (1977) and later applied in macro studies by Schubert et al. (1989). This indicator is de ned as the ratio between the share of the given unit in publications in the given eld and the overall share of the unit in publications.
This enables the creation of publication pro les of a unit by discipline and indicates whether the unit has a higher or lower activity in a eld compared its overall share in the total output. This improves the comparability between units with respect to their activities across elds.
Another possibility to improve the comparability between units is to use ranks instead of (absolute or relative) frequency counts. The above mentioned distributions (Lotka, Bradford and Zipf) are attempts to rank on authors, journals or words. However, ranking easily becomes problematic when using multiple indicators to describe and compare publication activity. Thus, a multidimensional space could be used to plot the units or indicators could be combined or aggregated into one single indicator using different weightings. But none of these solutions enhances the understanding of the relation between the units.
Citations and citation indicators have always taken a prominent role in bibliometrics. Price (1970) proposed this convention for citations:
‘If Paper R contains a bibliographic footnote using and describing Paper C, then R contains a reference to C, and C has a citation from R.’
Next to this operational de nition of citations there was a long and still is an ongoing discussion on the meaning of citations. Also in the discussion on citations the truly multidisciplinary nature of the eld is revealed. Merton (1973) and Kaplan (1965) gave a sociologically explanation to citations as the giving of recognition to which the scientist is obliged (Wouters, 1999).
Moed (2005) added a cognitive dimension to the discussion by interpreting citations as:
‘…manifestations of intellectual infl uence, even though such infl uence may not directly be traced from the citing texts. They can be viewed as instances of citing authors’ socio-cognitive location that refl ect their awareness of what are the important groups or programmes that must be included in their reference lists.’
In his book on citation analysis Moed (ibid.) outlines the different views on citation and citation analysis and he concludes that all these perspectives are valid as they all point out different aspects of referencing practices. It is therefore impossible to have one single theoretical concept to express what citations measure.
Cozzens (1989) referred to the primarily rhetoric nature of citations where the reward comes only secondarily. This discussion on opposite viewpoints on citations is still lively going on in an online mailing list (Sigmetrics, see http://
web.utk.edu/~gwhitney/sigmetrics.html).
More pragmatically was the interpretation of citations by Glänzel and Schoep in (1999):
‘one important form of use of scientifi c information within the framework of documented science communication’
As part of this debate on the meaning of citations some authors investigated the reasons for citing published research. Gar eld (1965) and somewhat later Weinstock (1971) compiled a list of 15 reasons for citing publications:
1. Paying homage to pioneers 2. Giving credit for related work
3. Identifying methodology, equipment, … 4. Providing background reading
5. Correcting one’s own work 6. Correcting the work of others 7. Criticizing previous work 8. Substantiating claims
9. Alerting to forthcoming work
10. Providing leads to poorly disseminated, poorly indexed or uncited work
11. Authenticating data and classes of facts
12. Identifying original publications in which an idea or concept was discussed
13. Identifying original publications or other work describing an eponymic concept or term
14. Disclaiming work or ideas of others (negative claim) 15. Disclaiming priority claims of others (negative homage).
Some of the reasons listed above have indeed a rhetoric nature as claimed by Cozzens but other reasons are easily understood as rewarding. The list thus stresses the complementarity of the two viewpoints. It also helps to understand that citation indicators have to interpreted much broader than mere ‘quality measures’.
It is more the reception or use of information that can be captured. Braun et al.
(1985) stated this as follows:
‘if a paper receives 5 or 10 citations a year throughout several years after its publication, it is very likely that its content will become integrated into the body of knowledge of the respective subject fi eld; if, on the other hand, no reference is made at all to the paper during 5 to10 years after publication, it is likely that the results involved do not contribute essentially to the contemporary scientifi c paradigm system of the subject fi eld in question’
Also van Raan (1998, 2004) sees enough evidence to conclude that references motives as mentioned above are not randomly given and thus the phenomenon of citation can play its role as reliable measure of impact.
It is this pragmatic perspective on citations that is taken in this thesis and which permits the use of citation indicators in the quantitative component of a research evaluation exercise. A broad range of indicators is chosen with the emphasis on relative indicators. Relative citation indicators compare the total citation count of a unit with a reference standard set by a journal or a eld.
1.3.4 Applications of bibliometrics
A rst application of bibliometric tools is the description of the structure or development of science using quantitative techniques. Merton (1938) used bibliometric tools to investigate the dynamics of research activities in the seventeenth century. Price (1963) described the exponential growth of scienti c output from the seventeenth century until the early sixties. Later this application became known as ‘Mapping of Science’ as more advanced statistical techniques like clustering were applied to the bibliometric data. Specialized bibliometric tools were developed like co-citation analysis, bibliographic coupling and co-word analysis. Co-citation and bibliographic coupling are often confused. The rst one
analyses papers cited by the same paper while the latter one starts from papers that share at least one reference. Small and Sweeney (1985) used single linkage clustering to sequentially link co-cited papers. This results in sets of clustered publications –core papers- and publications in a cloud citing at least one in a cluster (Oberski, 1988, Braam et al. 1988). Using multidimensional scaling and visualization techniques these clusters and relations can be plotted in maps which can enhance the understanding of the structure and dynamics of science.
A second application of bibliometric tools is in description and evaluation of research performance. It was Price (1963) again who rst pointed out the possibilities of quantitative studies of science in evaluation and policy. A few years earlier, in 1957, had the launch of the Sputnik increased the pressure on western governments, scientists and military: the ‘Sputnik shock’ (Leydesdorff, 2005). A consequence was the reorganization of both the education and the science system. NASA was created and the budget of the National Science Foundation was increased by nearly $100 million. One decade later, at the end of the sixties, funding for research was decreasing drastically again in a response to economic dif culties and the huge costs of the Vietnam War. This created the need to provide both policy makers and the general public with objective data on the health, quality and competitiveness of research in the US (Braun, 1985). In a response to this need the National Science Foundation published the book ‘Science Indicators 1972’
(NSF, 1973). These volumes were published bi-annually and SCI data provided by ISI was used to calculate the scientometric indicators. It was Narin (1976) who introduced the term ‘Evaluative Bibliometrics’. Both publication and citation indicators are valuable in the evaluation of science. Publication indicators measure the scienti c activity while scienti c achievement or merit can be expressed by citation indicators (Moed, 1989). Recently Bornmann (2009b) proposed the use of pairs of indicators to cover both aspects as a meaningful approach for comparing scientists. The discussion on the role of citations is described above.
A third application is the use of bibliometric indicators in library management. The indicator most widely used in library science is the Impact Factor as introduced by Gar eld (1955). It is de ned for a journal X in year n as
The total number of citations from SCI-covered journals in a given year n to all [citable] papers published in journal X in years n-1 and n-2 divided by the total number of [citable] papers in journal X published in years n-1 and n-2.
This indicator can be seen as a measure for the usage of a journal in scienti c literature and as such it is an important tool in the management of a collection with respect to the needs of its audience. It was constructed by Gar eld as a tool to help in the selection of journals to be included in the Science Citation Index. Even after half a century the indicators published in the JCR are widely applied by librarians, scientists, policy makers or journal editors. Van Hooydonk
et al. (1994) related the bibliometric data available in the Journal Citation Report (JCR) with what they called bibliotheconomic data (subscription prices, prices per article, holdings, etc.).
Although the main theme of this thesis focuses on the area of research assessment and evaluation, the classi cation model applied on research institutions also can help in the understanding of the structure of science and the relation between different institutions.
1.4 A dynamic perspective based on a structural analysis of institutional research profi les
This thesis presents a new methodology that can be used in quantitative assessment exercises of individual institutions and in institutional evaluation at national level. It aims at solving the problems mentioned above and implementing state of the art bibliometric methodologies..
For individual assessment of research performance we recommend also a qualitative counterpart which could confront the management with quantitative
ndings.
First, a classi cation model was built in order to create groups of likewise institutes. This grouping tackles the issues of eld dependence of indicators and inter-institute comparability. This clustering of research pro les will be called structural analysis throughout the thesis.
The other crucial element in the methodology is the dynamic perspective.
In this perspective four principles of good evaluative practice are combined.
The idea for the creation of this methodology originated after two successful applications of the clustering of institutes. A rst paper appeared in 2006 on the in uence of author self-citations at the meso-level (see part II, chapter 1) . Later, in a paper on Brazilian institutes this clustering was used again to give better understanding of the structure or pro le of research in this country.
At the S&T Indicators Conference in Leuven (sept. 2006) a rst step towards the nal version of the classi cation model was presented. A fully elaborated description appeared in 2008 in Scientometrics (see part II, chapter 2).
Before this classi cation could be used in assessment the grouping was subject of a study on indicator differences. This study was presented at the ISSI conference in Madrid 2007 and is published in Scientometrics in 2009 (see part II, chapter 3). These two papers on the structural analysis of institutional research pro les proved the validity and usability of the classi cation model in assessment and benchmarking studies.
At the S&T Indicators conference in Vienna, (july 2008), the rst application of this dynamic perspective was presented. The research performance of ve Israeli institutes was examined and discussed. This paper will be publish in Research Evaluation (see part II, chapter 4).
1.5 Framework of the thesis
This thesis consists of the following three parts: I Backgrounds, Models and Applications; II Core papers; and III Future Prospects. The rst part (in which we are now) contains four chapters. First, a general introduction on current research evaluation studies and on quantitative studies of publications and citations –particularly bibliometrics- is discussed in this chapter. Then, the next chapter presents the classi cation model. It also describes the address cleaning procedure that is used as this is an absolute necessary condition in order to get valid results on institutional performance. A detailed description of the three versions of the clustering procedure is provided and the last section deals with the discriminant analysis used for the creation of the classi cation model.
The main goal in mind when developing this model was to improve the comparability and assessment of institutional research performance. This application is presented in the third chapter on the dynamic perspective on research performance.
In the fourth and last chapter of the rst part we will particular focus on important issues such as the improvement of bibliometric indicators; patterns of changing research pro les; and differences in collaborative behavior.
In the second part, the core of this thesis, ve papers published (or accepted for publication) in international refereed journals are included:
• The in uence of author self-citations on bibliometric meso-indicators.
The case of European universities; published in Scientometrics, 66(1), 71-80
• A Structural Analysis of publication pro les for the classi cation of European research institutes; published in Scientometrics, 74(2), 223- 236
• A structural analysis of benchmarks on different bibliometrical indicators for European research institutes based on thbeir research pro le; published in Scientometrics, 79(2), 377-388
• Israeli Research Institutes: A Dynamic Perspective; accepted for publication in Research Evaluation.
• A structural analysis of collaboration between European research institutes; accepted for presentation at the ISSI 2009 conference, Rio de Janeiro, Brazil and submitted to Research Evaluation.
In the last part some topics for future research are presented. Several of these topics concern the extension of published papers.
References
Bar-Ilan, J. (2008), Which h-index? – A comparison of WoS, Scopus and Google Scholar, Scientometrics, 74(2), 257-271.
Bonaccorsi, A., Daraio, C., Lepori, B., Slipersaeter, S. (2007), Indicators on individual higher education institutions: addressing data problems and comparability issues, Research Evaluation, 16(2), 66-78.
Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., Daniel, HD. (2009a), Convergent validity of bibliometric Google Scholar data in the eld of chemistry-Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts, Journal of informetrics, 3(1), 27-35.
Bornmann, L., . (2009b), Do we need the h-index and its variants in addition to standard bibliometric measures, Journal of the American society for information science and technology, 60(6), 1286-1289.
Braam, R.R., Moed, H.F., Van Raan, A.F.J. (1988), Mapping of Science:
Critical Elaboration and new Approaches, A Case Study in Agricultural Biochemistry, Informetrics, 87/88, 15-28.
Bradford, S.C. (1934), Sources of information on speci c subjects, Engineering, 137: 85-86.
Braun, T., Glänzel, W., Schubert , A. (1985), Scientometric indicators. A 32-Country Comparative Evaluation of Publishing Performance and Citation Impact. World Scienti c Publishing, Singapore.
Broadus, R.N. (1987), Towards a de nition of ‘bibliometrics’, Scientometric, 12, 373-379.
Burton, H.D. (1988), Use of a virtual information system for bibliometric analysis, Information Processing and Management, 24, 39-44.
Butler L. (2007), Assessing university research: a plea for a balanced approach, Science and Public Policy, 34(8), 565-574.
Calero-Medina, C., Lopez-Illescas, C., Visser, M.S., Moed, H.F. (2008), Important factors when interpreting bibliometric rankings of world universities: an example from oncology, Research Evaluation, 17(1), 71- 81.
Cozzens, S.E. (1989), What do citations count? The rhetoric- rst model, Scientometrics, 15, 437-447.
Cronin, B. (1981), The need for a theory of citation, Journal of Documentation, 37, 16-24.
CWTS (2009), The Leiden Ranking 2008, (accessible via http://www.cwts.nl/
ranking/LeidenRankingWebSite.html ).
Debackere, K., Glänzel, G. (2004), Using a bibliometric approach to support research policy making: The case of Flemish BOF-key, Scientometrics, 59(2), 253-276.
Egghe, L. (1988), On the classi cation of the classical bibliometric laws, Journal of Documentation, 44, 53-62.
Fairthorne, R.A. (1969), Empirical hyperbolic distributions (Bradford-Zipf- Mandelbrot) for bibliometric description and prediction, Journal of Documentation, 25, 319-343.
Fisch, R., Daniël, H.D. (1986), Messung and förderung von forschungsleistung, Konstanzer beiträge zur sozialwissenschaftlichen forschung, Band 2, Universitätsverlag, Konstanz.
Fisch, R., Daniël, H.D. (1988), Evaluation von Forschung, Beiträge zur socialwissenschaftlichen forschung, Band 4, Universitätsverlag, Konstanz.
Frame, J.D. (1977), Mainstream research in Latin America and the Caribbean, Interciencia, 2, 143-148.
Gar eld, E. (1955), Citation Indexes for Science. A New Dimension in Documentation through Association of Ideas. Science, 122, 108-111.
Gar eld, E. (1965), Can citation indexing be automated? In Stevens, M.E. and Giuliano, V.E. (Eds.), Statistical association methods for mechanized documentation, National Bureau of Standards, Washington, DC.
Gar eld, E. (1979), Is citation analysis a legitimate evaluation tool?
Scientometrics, 1(4), 359-37 .
Glänzel, W., Schoep in, U. (1999), A bilbiometric study of reference literature in the sciences and social sciences, Information Processing and Management, 35(1), 31-44.
Glänzel, W. (2008), Bibliometrics as a research fi eld. A course on theory and applications of bibliometric indicators. KU Leuven, Leuven.
Hawkins, D.T. (1977), Unconventional uses of on-line information retrieval systems: on-line bibliometric studies, Journal of the American Society for Information Science, 28, 13-18.
Hodges, S., Hodges, B., Meadows, A.J., Beaulieu, M., Law, D. (1996), The use of an algorithmic approach for the assessment of research quality, Scientometrics, 35(1), 3-13.
Hood, W.W., Wilson, C.S. (2001), The literature of bibliometrics, scientometrics, and informetrics, Scientometrics, 52(2), 291-314.
Jasco, P. (2006), De ated, in ated and phantom citation counts, Online Information Review, 30(3), 297-309.
Kaplan, N. (1965), ‘Prolegomena to the footnote’, American Documentation, 16, 179–187.
Leydesdorff, L. (2005), Evaluation of research and evolution of science indicators, Current Science, 89(9), 1510-1517.
Liu, N.C., Cheng, Y. (2005), Academic Ranking of World Universities – Methodologies an Problems, Higher Education in Europe, 30(2).
Lopez-Illescas, C., Moya-Anegon, F., Moed, H.F. (2008) Coverage and citation impact ofoncological journals in the Web of Science and Scopus, Journal of Informetrics, 2 (4), 304-316.
Lotka, A.J. (1926), The frequency distribution of scienti c productivity, Journal of the Washington Academy of Science, 16(12), 317-323.
Merton, R.K. (1973), The normative structure of science, University of Chicago Press, Chicago.
Moed, H.F. (1989), The use of bibliometric indicators for the assessment of research performance in the natural and life sciences (PhD-thesis), DSWO Press, Leiden.
Moed, H F., De Bruin, R.E., Van Leeuwen, TH.N. (1995), New bibliometric tools for the assessment of national research performance: database description, overview of indicators and rst applications, Scientometrics, 33 (3), 381-422.
Moed, H.F. (2005), Citation Analysis in Research Evaluation, Springer, Dordrecht.
Moed, H.F. (2007), The future of research evaluation rests with an intelligent combination of advanced metrics and transparant peer review, Science and Pulic Policy, 34(8), 575-583.
Nacke, O. (1979), Informetrie: Ein neuer Name für eine neue Disziplin, Nachrichten für Dokumentation, 30, 212-226.
Nalimov, V.V., Mulchenko, Z.M. (1969), Naukometriya. Izuchenie Razvitiya Nauki kak Informatsionnogo Protsessa. [Scientometrics. Study of the Development of Science as an Information Process], Nauka, Moscow, (English translation: 1971. Washington, D.C.: Foreign Technology Division. U.S. Air Force Systems Command, Wright-Patterson AFB, Ohio. (NTIS Report No. AD735-634)).
Narin, F. (1976), Evaluative Bibliometrics: The use of Publication and Citation Analysis in the Evaluation of Scientifi c Activity, Computer Horizons, Inc., Cherry Hill.
Oberski, J.E.J. (1988), Co-citation Cluster Analysis and Dutch Physics. In:
van Raan, A.F.J (Ed.), Handbook of Quantitative Studies of Science &
Technology, Elsevier Science Publishers bv, Amsterdam.
Otlet, P. (1934), Traite de Documentation. Le livre sur le livre. Theorie et pratique, Van Keerberghen, Brussel.
Payne, D. (2004), Google Scholar welcomed, News from The Scientist, 5(1):
20041123 01. Retrieved May 13, 2009, from http://www.the-scientist.
com/news/20041123/01/.
Pritchard, A. (1969),Statistical bibliography or bibliometrics?, Journal of Documentation, 25, 348-349.
Price, D. de Solla, (1963) Little Science, Big Science, Columbia Univ. Press, New York.
Price, D. de Solla, (1970), Citation measures of hard science, soft science, technology and non-science, D.C. Heath & Co., Lexington, MA.
Price, D. de Solla. (1976), A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27, 292–306.
Reale, E. Barbara A., Costantini, A. (2007), Peer review for the evaluation of academic research: lessons from the Italian experience, Research Evaluation, 16 (3), 216-228.
Schubert, A., Braun, T. (1986), Relative Indicators and Relation Charts for comparative assessment of publication output and Citation Impact, Scientometrics, 9 (5-6), 281-291.
Schubert, A., Glänzel, W., Braun, T. (1987), Subject Field Characteristic Citation Scores and Scales for Assessing Research Performance, Scientometrics, 12 (5-6), 267-292.
Schubert, A., Glänzel, W., Braun, T. (1989), World ash on basic research:
Scientometric data les. A comprehensive set of indicators on 2649 journals and 96 countries in all major science elds and sub elds, 1981- 1985, Scientometrics, 16(1-6), 3-478.
Schubert, A., Braun, T. (1996), Cross-Field Normalization of Scientometric Indicators, Scientometrics, 36(3), 311-324.
Small, H., Sweeney, E. (1985), Clustering the science citation index using co- citaions. 1. A comparison of methods, Scientometrics, 7(3-6), 391-409
Smith, L.C. (1981), Citation Analysis, Library Trends, 30(1), 85.
Tague-Sutcliffe, J.M. (1992), An introduction to informetrics, Information Processing & Management, 28, 1-3.
Van Hooydonk, G., Gevaert, R., Milis-Proost, G., Van De Sompel, H.,
Debackere, K. (1994), A bibliometric analysis of the impact factors of scienti c disciplines, Scientometrics, 30(1), 65-81.
van Raan, A.F.J. (2004), Measuring Science. Capita Selecta of Current Main Issues. In: Moed, H.F., Glänzel, W., Schmoch, U. (Eds.), Handbook of Quantitative Science and Technology Research, Kluwer Academic Publishers, Dordrecht.
Weinstock, N. (1971), Citation Indexes, In: Kent, A. (Ed.), Encyclopedia of Library and Information Science, Marcel Dekker, New York.
Wouters, P. (1999), The Citation Culture (Doctoral Thesis), Private edition.
Zipf, G.K. (1949), Human behavior and the principle of least effort, Addison- Wesley, Reading, MA.
