Metric Monsters
Citation Metrics in the Sociologist´s Life
Johanna M. TaufnerMaster´s programme: Sociology: Gender, Sexuality and Society
University: Graduate School of Social Science – University of Amsterdam
Supervisor: Sarah Bracke
Second Reader: Margriet van Heesch
Student, Student ID: Johanna Taufner, 11709308
E-Mail: j.taufner@gmail.com
Date, Place: 5th July 2018, Amsterdam
Metric Monsters
Table of Contents
Summary ... 4
Introduction ... 6
When Monsters Meet Ants ... 11
The Birth of Metric Monsters ... 18
Approaching Monsters Methodologically ... 27
Mapping Metric Monsters ... 33
Discussion ... 50
Bibliography ... 54
Acknowledgements ... 61
“Cyborg imagery can suggest a way out of the maze of dualisms in which we have explained our bodies and our tools to ourselves. This is a dream not of a common language, but of a powerful infidel heteroglossia. It is an imagination of a feminist speaking in tongues to strike
fear into the circuits of the supersavers of the new right. It means both building and destroying machines, identities, categories, relationships, space stories. Though both are
bound in the spiral dance, I would rather be a cyborg than a goddess.” (Haraway 2004a, p. 39)
Metric Monsters
Summary
In this thesis, I investigate science-in-the-making by analysing bibliometric measures in the lived experience of sociologists, and drawing from Science and Technology Studies (STS) scholars, such as Bruno Latour (2017), Donna Haraway (1997), and María Puig de la Bellacasa (2011). In the contemporary European academic landscape, citation metrics are broadly used to measure the impact of scientific research. The journal impact factor (JIF) ranks scientific journals according to the average frequency of cited articles. The JIF is calculated on the basis of the Web of Science database and published annually in the Journal Citation Report by Clarivate Analytics, which is owned by the private equity firms Onex Corporation and Baring Private Equity Asia. The international publishing company Springer and Google Scholar provide citation alert tools, allowing scholars to track the impact of their articles on a daily basis. Frequently used as a standardized tool to measure quality, impact factors can affect which journals sociologists choose to publish in, and who is granted research funds. Due to their ubiquitous presence, they exemplify what Bruno Latour (2017) characterizes as matters of fact. But what conditions allow(ed) the JIF to become such a matter of fact?
In this research, I re-visit contemporary social science literature that discusses the origin of citation metrics, and I investigate on the JIF, using Actor-Network Theory. Initially, identifying research gaps in the current discussion surrounding citation impacts, I endeavour to discover what elements form relations in the lived experience of sociologists that make the durable assemblage of citation metrics possible. By doing so, I make the various fluid connections between textual, social, technological, topographical, and corporal elements visible, that create the heterogeneous network that we come to know as the JIF.
In order to localize this durable assemblage, I conducted fifteen semi-structured interviews with European sociologists, with an emphasis on PhD students, over a four-week period in spring 2018. Using the qualitative data analysis method of Abductive Analysis in the tradition of Timmermans and Tavory (2012), I theorized from the collected data in conversation with contemporary social science literature.
As this thesis will show, although the use of JIFs is frequently criticised, they are matters of fact for the interviewed sociologists. Even though the PhD students´ viewpoints on metrics vary from positive to negative, they all identify the JIF as a frequently used indicator to measure scientific quality. Furthermore, by identifying a gap in the research on the relations between affects, bodies and metrics, I illustrate in this study the interactions between
emotions, calculations, and texts that form the assemblage of citation metrics. By theorizing on emotions, I show that the investigation of affects is essential in order to understand how forms of quantifications are stabilized in specific moments of time and place. Lastly, I highlight the decisions sociologists make about their work, because of metrics. By doing so, I theorize on the effects of the JIF on epistemic diversity within European sociological knowledge production.
I understand this qualitative research as an addition to contemporary STS literature, the investigation of forms of quantification in general, and the JIF in particular. By describing and mapping citation metrics in the situated and lived experiences of sociologists, I endeavour to contribute to a better understanding of the durable assemblages – the metric monsters – that shape contemporary Western knowledge production.
Metric Monsters
Introduction
I begin this thesis by re-visiting three events.
Event one: May 28, 1936, three years before WWII the mathematician Alan Turing submits his paper On Computable Numbers (1936) to the London Mathematical Society. In this mathematical description he introduces the universal computing machine. Inspired by the cognitive capabilities of humans such as remembering and problem solving, he designs this abstraction to solve any mathematical problem, based on stored program instructions. His paper introduces central concepts on which modern computers will be based.
Event two: this is the year of the oil crisis, 1973. At the Institute for Scientific Information in Philadelphia, Pennsylvania, Eugene Garfield and Irving H. Sher sit in front of their computers, which operate on the basis of Turing´s theory. Drawn to statistical calculations, and in an attempt to create a tool to organize the growing amount of scientific publications, they have been digging around the huge amount of stored data covering physics and life science journals. In the search for a calculation that could measure a journal´s impact, they create a formula that divides “the number of times a journal is cited by the number of articles that journal has published” (Garfield & Sher, 1963, p. 199). The created metric will be known as the journal impact factor (JIF).
Event three: this is the year 2018. A young Austrian PhD candidate sits in front of her computer. She visits the website of a sociology journal, in which her first article will be published. Her eyes drift along the screen until they perceive two symbols, a letter and a number, which she immediately identifies as the JIF, the citation metric based on Garfield and Sher´s formula. It is a high rank. In an instance, her body releases a neurochemical cocktail of endorphin, dopamine, serotonin, and oxytocin. Her heartbeat accelerates. She feels proud (Interview XIV, 2018).
I have entered this investigation by re-visiting these three lived experiences that are separated through time (1936, 1973, 2018) and space (Great Britain, USA, Austria) for two reasons. First, to introduce the subject of this research and the matter of my concern in the European scientific landscape in which I am situated: the journal impact factor (JIF). Second, I want to draw the reader´s attention, not to what separates these events, but rather what connects them. The JIF exist in 2018, because Alan Turing´s research made the invention of computers possible, Eugene Garfield and Irving Sher created a metric, and the Austrian PhD student uses this metric by visiting the journal´s website. All of these connections between different
scientists, computers, numbers, the Internet, and a heartbeat become part of the citation metric. It is this heterogeneous mixture of human and non-human elements and their relations that make its existence possible.1
Today, in contemporary Western knowledge production, citation metrics, and most popularly the JIF, have become a standardized tool. The bibliometric measure is used on a daily basis to evaluate scientific journals, papers, researchers, and institutes (cf. Archambault & Lariviére, 2009). However, over the last decade, scientometricians, scholars, and publishers have started to turn against the metric, due to growing critique and allegations of misuse (Callaway, 2016; Diana Hicks et al., 2015). Here, studies problematize for example the practice of self-citation (Smith, 1997), the method of naming honorary authors, and superfluous citing to manipulate the metric (Fong & Wilhite, 2017). Although challenged, scholars argue that the JIF remains durable in the evaluation of scientific output and research performance (Desrochers et al., 2018; Fleck, 2013; Haustein & Larivière, 2015).
As a standardized tool in academia, the JIF can be identified as what Bruno Latour calls a matter of fact. But to be considered as a matter of fact means to become a “black box” (Latour, 1987, p. 1) that disguises all the aforementioned connections between human and non-human elements that make its existence possible. Yet according to Bruno Latour (2017) it is precisely these connections that should be followed and described, because they will help the sociologist to understand how these heterogeneous ontologies came to be and how they shape our social world. This is the endeavour of my thesis: opening up the black box of the JIF.
Previous studies on the JIF and its derivatives have addressed different issues surrounding the bibliometric measure. For instance, Glänzel and Moed (2002) discuss the JIF´s accuracy, by investigating technical and methodological aspects, and highlighting distortion for calculation in the metric. Focusing their work on statistical reliability, they argue that statistics can be applied for a comparative analysis of journal´s impact.
Picking up on the calculation´s distortion, another strand of scholars such as Archambault and Lariviére (2009) and Fleck (2013) have investigated the measure´s limitations by re-visiting its origins. They argue that the metric emerged in a specific situated American and natural science context and was later implemented in other fields. According to the authors,
1 To express this human/non-human formation, Science and Technology Studies (STS) authors introduced the metaphor of monsters (Law, 1991a; Haraway, 1988). Following this tradition, this thesis title was chosen accordingly. I further elaborate on this in the theoretical chapter of this paper.
Metric Monsters
not sensitized to other national or disciplinary contexts, its application slowly changed citation patterns and publishing cultures.
Other scholars have further addressed the JIFs institutionalization in contemporary scientific practices. To mention a few, Rushforth & de Rijcke´s research (2015) on the use of the JIF to assess scientific papers that are produced by the Dutch University Medical Centres. Further, Fuyono and Cyranoski (2006) investigated the metric´s application in Pakistan, China and South Korea, where researchers receive bonus salary payments for articles published in highly ranked journals. Finally, Franzoni, Scellato and Stephan (2011) discuss the sexenio evaluation in Spain that is partly based on the JIF, which assesses the researcher´s productivity.
A different group of researchers addresses the JIF´s effects on contemporary knowledge production and its misuse. For instance, Willmott (2011) highlights in his study of the UK Association of Business Schools list, which ranks journals based on their impact, that researchers adapt their reading habits, and also create syllabi and reference lists according to the JIF. According to him, this fosters a “monoculture” in scientific output (p. 429). Stephan, Veugelers and Wang (2017) show that bibliometric measures discourage risky studies. Since reviewers rely on the proxy to judge impact and quality, this results in a risk-averse research culture. Fong and Wilhite (2017) problematize the practice of honorary authorship and superfluous citations to manipulate the JIF. Also, Smith (1997) addresses self-citation practices exemplified on the case of Leukemia´s publisher, who have been accused of asking authors to reference papers published in their journal in order to increase its rank.
Overall, previous literature on the JIF has touched on various aspects. Foregrounded in these studies are national and disciplinary implications (Franzoni, Scellato & Stephan, 2011; Fuyono and Cyranoski, 2006; Rushforth & de Rijcke, 2015) and questions surrounding the JIF´s accuracy in general (Archambault and Lariviére, 2009; Fleck, 2013; Glänzel & Moed, 2002). Since these investigations are institutionally, nationally, and disciplinarily framed, they do not take account of the lived experience of scholars themselves. In an attempt to fill in pieces of this research gap, in this thesis, I open up the “black box” (Latour, 1987, p. 1) of the JIF by analysing the situated and lived experience of fifteen European sociologists. By doing so, I describe the various connections that are formed between bodies, metrics, texts, companies, and governments in the durable assemblage of the JIF.
With my three introductory examples – Alan Turing´s paper, Eugene Garfield and Irving Sher´s calculation, and the Austrian PhD candidate´s bodily response when she saw the JIF –
I have already highlighted various human and non-human elements that assemble to make the JIF´s existence possible. Guided by Bruno Latour´s (2017) suggestion to follow these connections in order to make them visible, I locate these relations in the lived experience of fifteen European sociologists. To orient myself in this endeavour, I use Actor-Network Theory (ANT) by drawing from Science and Technology Studies (STS) scholars such as Bruno Latour (2017), Donna Haraway (1997), John Law (1991a), and María Puig de la Bellacasa (2011). I do so to make the various fluid connections between textual, social, technological, topographical, and corporal elements visible, that constitute the heterogeneous network that we come to know as the JIF. I argue that this theoretical approach is beneficial to gain a better understanding of why the JIF has become such a universal tool and why it remains durable, despite growing critique.
This analysis was directed by two research questions: First, what elements assemble in the lived experience of sociologists that make the human/non-human network of the JIF possible? Second, what decisions do PhD candidates make about their work, because of citation metrics?
To answer these questions, I interviewed fifteen European sociologists, with an emphasis on PhD students, over a four-week period in spring 2018. I initially started this investigation in an explorative manner. Hence, I applied Abductive Analysis in the tradition of Timmermans and Tavory (2012), which guided me through my fieldwork and my analysis. By no means do I attempt to portray a representative account of European sociologists in general in this study. Rather, this research theorizes, in conversation with social science literature, from the lived experiences of fifteen individual situated cases.
As this thesis will show, the JIF is a matter of fact in the PhD students´ lives. Although the respondents have very contrasting perceptions of it and its ability to assess scientific knowledge output, they all consider it a standardized tool to measure scientific quality. Additionally, by highlighting a research gap on the relations between bodies, affects and metrics, I show in this study the affective flows between texts, calculations, and bodies that form the durable assemblage of the JIF. By doing so, I illustrate that the investigation of affects is essential if we want to understand how the JIF is stabilized in specific situations, as well as what questions we have to ask if we want to change its dominant position in the evaluation of scientific output and the assessment of researchers. Finally, I discuss what decisions the respondents make about their work because of bibliometric measures. Here, I
Metric Monsters
theorize the JIF´s effects on epistemic diversity within European sociological knowledge production.
I understand this qualitative research as a contribution to existing STS literature, research on new forms of quantification in general, and as an extension of scholarly discussions surrounding the JIF in particular. By investigating and describing the JIF in the lived experience of situated sociologists, I endeavour to contribute to a better understanding of the durable assemblages – the metric monsters – that shape contemporary Western knowledge production.
This thesis is structured as follows: In the first chapter, When Monsters Meet Ants, I outline my theoretical approach, Actor-Network Theory. Initially starting with Bruno Latour (2017), I draw heavily from other STS scholars such as Donna Haraway (1997), John Law (1991a), María Puig de la Bellacasa (2011), and Susan Leigh Star (1991). I discuss three main theoretical concepts that guided my empirical operationalization: matters of concern, modest witness and matters of care.
In the second chapter, The Birth of Metric Monsters, I initially outline the historical conditions that enabled the creation of bibliometric measures. For this, I re-visit the emergence of social statistics and their connected history with the development of the modern European nation state. Here, drawing from Michel Foucault (2008), I discuss new forms of governmental techniques, and the concept of biopower, which is crucial to the understanding of human/non-human assemblages such as the JIF. Following this general historical introduction, I zoom in on the emergence of bibliometric measures in the second half of this chapter. Here, I elaborate on the history of the JIF, by pointing to heterogeneous elements that became assembled for its existence. Furthermore, I summarize contemporary discussions surrounding the JIF. This chapter is the theoretical-historical basis from which my own investigative journey through the lived and situated experience of fifteen sociologists takes off.
In the third chapter, Approaching Monsters Methodologically, I discuss my methodology. In order to locate the durable assemblage of the JIF in the sociologist´s life, I chose to conduct fifteen semi-structured interviews, with an emphasis on PhD candidates, in the spring of 2018. In the first part of this chapter, I elaborate on my choice of method, semi-structured interviews. Following this, I argue for and describe my case selection. In the last part, I discuss the method of Abductive Analysis in the tradition of Timmermans and Tavory
(2012). Lastly, I address ethical issues, my own situatedness within this research, and I mark limitations of this research design.
In the fourth chapter, Mapping Metric Monsters, I enter the analysis of this thesis. This chapter is divided into three subchapters. In the first subchapter, Reflecting on Citation
Metrics, I map out the wide range of different discourses – from rejection to acceptance – on
bibliometric measures, from which the respondents draw. In the second subchapter, Affects,
Bodies, and Metrics, I focus on affective flows between the calculation and bodies. Here, I
illustrate how the analysis of affects provides answers to the question of how the JIF is stabilized in situated experiences. In the last subchapter, Making Decisions, I discuss what decisions respondents make in order to maximize their work´s ranking. Following on from this, I theorize the effects of the JIF on epistemic diversity in scientific knowledge production.
In the last chapter, Discussion, I recapitulate central findings of this analysis, limitations of this study, and conclude with final remarks.
When Monsters Meet Ants
Situated in the field of knowledge production, this thesis investigates the relations between metrics, machines, and bodies in the example of citation metrics as experienced by sociologists. To theorize about this heterogeneous ensemble of human/non-human entities, all in one breath, I started this investigation with Actor-Network Theory (ANT) in the tradition of Bruno Latour (2017). I highlight two reasons why I did so: First, as I show in this thesis, citation metrics like the impact factor (JIF) are heterogeneous arrangements, or metric
monsters, consisting of and depending on formulas, computers, the internet, statisticians,
scholars etc. ANT enables us to investigate these human/non-human relations in one theoretical approach. Second, ANT emerged from the field of Science and Technology Studies (STS) that examines science-in-the-making. Thus, conducting research in the area of sociological knowledge production is a perfect fit. Although starting with ANT in the tradition of Bruno Latour, I enriched this theoretical approach by drawing heavily from other STS scholars such as Donna Haraway (1997), John Law (1991a), María Puig de la Bellacasa (2011), and Susan Leigh Star (1991). In the following chapter I will map out in three sections the main theoretical assumptions and concepts that guided this research: (1) From Matters of
Metric Monsters
Facts to Matters of Concern, (2) Sociologists as Modest Witnesses, (3) From Matters of Concern to Matters of Care.
From Matters of Facts to Matters of Concern
In his book Reassembling the Social, Latour (2017) argues that what traditional sociology has considered as social, in the sense of inter-connections between human beings, is in fact not social at all. According to him, the social expands into heterogeneous arrangements between human/non-human networks. Therefore, sociological theory needs to develop concepts that are able to theorize on what Donna Haraway calls the “wealth of connections that constitute a specific, finite, material-semiotic universe” (Haraway, 1997, p. 3). ANT enables us to make these heterogeneous material-semiotic assemblages visible.2
In order to reflect about these heterogeneous networks that seem to create a life of their own, Donna Haraway (1997) introduces in her work the metaphors of the cyborg or the (techno-) monster to capture these fluid human/non-human organisms (p. 12). She understands cyborg monsters both as an analytical tool to investigate human/non-human assemblages and as a concept to define “different political possibilities and limits” beyond dualist Western stories of origins, and “a way out of the maze of dualisms” (Haraway, 2004b, p. 39). This metaphor illustrates the living, fluid existence of assemblages, in contrast to the static notion of networks. Latour (2017) himself dislikes the term network, due to its static connotations and its ubiquitous use both inside and outside the scientific discourse. However, because of its historic popularity within STS, and as it suggests the flattened analytical conceptualization of human/non-human actors, he continues to use this term throughout his theorization (p. 112). Similar to Haraway, John Law uses the monster metaphor to describe durable assemblages or networks, for instance in his book A Sociology of Monsters (1991a), that initially inspired the title of this thesis.3
When we take the previous assumption – the social is not social – as our starting point, this thesis itself is an assemblage of various human and non-human relations. At first glance, merely a text, a second look uncovers the various connections that come together for its
2 This analytically flattening of human/non-human actors is specific to ANT. However, other theoretical approaches have done this before to some extend, for instance Marxist theory investigates the relationship between workplace organisation and production technologies (cf. Law, 1991a, p. 8).
3 This definition differs from Susan Leigh Star´s (1991) description of monsters. In her work, she uses the metaphor to analyse on the outcast in standardized worlds, the experience of marginalised selves and misfits (p. 54).
existence: a keyboard, microchips, the internet, a library card, other authors, texts, respondents, two supervisors, and, of course, myself. These relations are the sociologist´s concern, because it is the relations that explain the social. Following this, it is the researcher´s mission to describe these connections, make them visible, and ask how they are made durable (Latour, 2017, pp. 10, 131 & 410).
Following the assumption of human/non-human assemblages, the hierarchy between the social and the object world becomes dismissed. Every element in the assemblage becomes analytically relevant and has to be taken into account (p. 100). Latour urges us to de-objectivy matters of facts. Objects are no longer lifeless things that are simply used. They themselves become actors. Each element of the assemblage is able to affect the next one, and is in turn affected by them. This affective capacity of each element is expressed in the term
actant. The inter-actions in this fluid connection between textual, social, technological,
topographical, and corporal actants become the centre of ANT investigations (pp. 70, 81 & 88). In order to conduct research on these assemblages Latour wants to open up the “black box” (Latour, 1987, p. 1) and to unveil the fluid arrangements of what is called matters of facts, and define them as matters of concern (Latour 2017, p. 199). For this, the author introduces the metaphor of an ant. Inspired by this diligent insect, the researcher should travel along the different paths, following every trace within these heterogeneous ensembles (Star, 1991, p. 43).
Furthermore, like an ant that explores the world from the ground, the researcher should always localize the global. Every assemblage is created, duplicated or stabilized in certain times and places, through specific activities. Thus, through localization the researcher can describe how certain monsters are made durable, while others will become extinct (Latour, 2010, pp. 299n; Haraway, 1997, p. 25).
Following this, in order to ask how certain matters of concerns are stabilized, ANT dismisses the analytical division between the macro, (meso), and the micro level. According to Latour, micro and macro actants have the same analytical position, as a macro actor can act as one actant in an assemblage. Thus, the division between agency and structure becomes irrelevant as well. However, this analytical flattening does not imply that we should ignore different levels of access, and asymmetries of power and dominance (Star, 1991, p. 43). When researchers make human and non-human connections visible, when they ask how specific assemblages grew to become their current durable forms and what conditions allowed them to do so, they are enabled to theorize how asymmetrical power relations are created. As
Metric Monsters
Latour argues, becoming a durable matter of concern means to be composed of endless actants, to transcend time and space. It means to affect steadily other actants in a vast complex conjunction between various assemblages (Latour, 2017, pp. 109n; cf. Star, 1991, p. 43). Hence, not only do asymmetrical power relations transform into stabilized human/non-human assemblages, but they also result in their creation (Latour, 2017, p. 110f). To continue the discussion on how power is woven into durable assemblages, I elaborate Michel Foucault´s concept of biopower a few pages later.
Sociologists as Modest Witnesses
Following the first assumption that the social is not social, but in fact a human/non-human intermingling, analytical dichotomies such as society vs. nature and object vs. subject have to dissolve (Latour, 2017, pp. 74 & 192). Coming from modernity this dichotomous thinking is critiqued by Bruno Latour. In his text We have never been modern (1991), Latour argues that the scientific and cultural movement of modernity made us believe ourselves to be “modern” by artificially separating nature and society as two oppositional spheres that were constructed to explain one another. By doing so, the enlightened scholar disguised and made it impossible to take account for all the relations that create the social. In opposition to this modern scholar, he calls his position and scientific practice amodern, a definition that is picked up by Haraway (2004c, p. 77). In his amodern framing, nature and culture are not to be explained, but considered to be the consequences of modernity (Latour, 1991). According to Haraway, by doing so “technoscience extravagantly exceeds the distinction between science and technology as well as those between nature and society, subjects and objects, and the natural and the artificial that structure the imaginary time called modernity” (Haraway, 1997, p. 3).4 As Haraway points out in this quote, this dichotomous European tradition of thought created nature, society and technology as separated analytical categories. Furthermore, subject and object became two distinct spheres. This artificial disconnection was essential as it enabled modern scientists to disguise their situated subject position. Only by doing so were they able to separate bodies from minds, situatedness from scientific knowledge and produce universal truth claims. This modern, masculine and European subject position is what Haraway describes as the “modest witness” (1997). According to the author, it is the virtue of modesty which guaranteed the modern scholar – or witness – the authority to make accounts for the
4 In this thesis, I follow Latour´s (1991) understanding of modernity. In the following text, I use the term „modern“ to refer to this specific European thought tradition.
object world, while simultaneously denying their “biasing embodiment” (p. 24). By doing so he became “endowed with the remarkable power to establish the facts. He bears witness: he is objective; he guarantees the clarity and purity of objects. His subjectivity is his objectivity.” (ibid.) By artificially disconnecting object and subject the modest witnesses built the grounds “of several centuries of race, sex, and class discourses as objective scientific reports” (p. 32).5
Bruno Latour and other STS scholars contribute successfully in unveiling this self-invisible modest witness, i.e. Bruno Latour´s research on Louis Pasteur (1988). Yet, although Latour (2017) dismisses the notion of object and subject as two dichotomous categories he neglects to acknowledge his own subject position within the assemblages he uncovers. Even though his research is built on describing the work of scientists, by meticulously examining their subject positions and the multiple actants that made their discoveries possible, he never critically assesses himself. As he acts like an outsider, he falls into the trap of reproducing oppositional thinking of subject vs. object in his own research (Haraway, 1997, p. 35). Haraway (1997) concludes her reflections on the modest witness by re-conceptualizing it and adopting it into her own theoretical thinking. According to her, a modest witness insists on an ANT that demands the reflection on its own history and situatedness, that traces all relations between human/non-human, and that engages in the needs and projects of those that are not able to inhabit the subject positions of the scientific sites and “laboratories” of the modern, European, white academic (p. 270). The modest witness recognizes his/her embodied European enlightened history, and embraces the complexity of the heterogeneous arrangements that shape and make our social world. As Haraway summarizes: “I remain a child of the Scientific Revolution, the Enlightment, and technoscience. My modest witness cannot ever be simply oppositional. Rather, s/he is suspicious, implicated, knowing, ignorant, worried, and hopeful” (p. 3).
For my own research I will apply her concept of the modest witness in two ways. First, in order to situate myself within this research project and to reflect on my own witnessing, I make my own subject position, my epistemological starting point, and the theoretical decisions that shaped this research visible. I take account for this in various moments of this thesis. Second, Haraway´s concept guided this research from beginning to end, as I made the
5 Whereas Haraway (1997) and Latour (2017) position the distinction between subject and object in the centre of Western scientific debates, Fujimura (1998) approaches this science war differently. She argues that academic debates are structured around the question of scientific authority. “What kind of science should be practiced, and who gets to define it?” (p. 347)
Metric Monsters
witnesses, the sociologists themselves, the centre of my investigation to “unveil the self-invisible modest” sociologist (pp. 23 & 33).
From Matters of Concern to Matters of Care
With the introduction of the notion of concern in the investigation of matters of fact, Latour opens the discussion on ethico-political agendas in science. “Science is not politics. It is politics by other means”, he argues (Latour, 1988, p. 228). To reflect on ethico-political aspects within epistemic practices does not mean to reduce science from moves towards truth to merely power moves (Haraway, 1988, p. 576). Rather, the concept of matters of concern allows us to ask how and what interests are woven into human/non-human relations, how objects and scientific facts become problematic and politicized, but also “what political issues science and technology take part in creating” (Asdal, Brenna & Moser, 2007, p. 9). Yet, sensitizing the researcher´s theoretical lens to interests does not enable her to ask why specific relations are formed in the first place, and why actors take part in making human/non-human networks durable, even though they do not serve their own interests or might be in opposition to them.
Picking up on this problem, STS scholar María Puig de la Bellacasa (2011) extends Latour´s ANT approach, by drawing from feminist research on the notion of care. Here, she follows Joan Tronto and Berenice Fisher´s (1990) definition of care that is defined as “a species´ activity that includes everything that we do to maintain, continue and, repair our ‘world’ so that we can live in it as well as possible” (p. 40). Following this broad definition of care as an activity of creation and maintenance, Puig de la Bellacasa (2012) argues that every relation within heterogeneous networks inevitably involves care (p. 198). This means that human/non-human networks are made durable, because actors learn to care about them, as well as take care of them. Latour´s definition of concern, she argues, lacks this notion of doing or practices that are important to make heterogeneous networks possible. Here, Puig de la Bellacasa stresses the different meanings of care. First, care means to be engaged in something. It is an affective-ethical state, an obligation or responsibility situated in the individual body, which is practiced by actors in situated contexts (Puig de la Bellacasa, 2017, p. 42).
Second, care also entails practices of maintenance. Thus, the invisible work of human labour is essential to keep the heterogeneous technomonsters alive (Puig de la Bellacasa, 2011, p.
92). As the author summarizes, “this version of caring for technology carries well the double significance of care as an everyday labour of maintenance that is also an ethical obligation: we must take care of things in order to remain responsible for their becomings” (Puig de la Bellacasa, 2011, p. 90).
By further developing matters of concern to matters of care, Puig de la Bellacasa provides analytic tool to investigate how both “interests” and “affectively animated forces” are entangled in the assemblages of human/non-human actants that shape our world (p. 87). Overall, I apply Puig de la Bellacasa´s concept of matters of care in this thesis for three reasons: First, bibliometric measures are created and stabilized because PhD students learn to care about them from the beginning of their career. Second, situated in daily practice, sociologists take care of the human/non-human network that we come to know as the JIF. I further elaborate on both of these meanings of care woven into the JIF in my analysis. Third, the endeavour of STS, including writing this thesis, becomes a matter of care itself in a two-fold way. First, STS itself and the notion of thinking of diverse species in a non-hierarchical way is a practice of care about these species in and of itself (Puig de la Bellacasa, 2017). Second, writing this thesis represents the care of studying the lives of human beings, fifteen European sociologists, entangled in epistemic and technological processes: the monsters of citation metrics (Puig de la Bellacasa & Bauchspies, 2009, p. 227).
In conclusion, in this chapter I discussed three theoretical concepts that lay the ground for this research. Starting with a discussion of Actor-Network theory (ANT) in the tradition of Bruno Latour (2017), I first explored the concept matters of concern, which enables us to investigate the inter-action between human/non-human assemblages. Second, I described the concept of modest witness by Donna Haraway (1997), as a theoretical critique of modern European thought tradition, and also an analytical tool for ANT studies. Third, I introduced María Puig de la Bellacasa´s (2017) concept matters of care. By discussing and further developing the concept of matters of concern, it highlights how interests and affects are woven into the creation and maintenance of durable assemblages.
Following this introduction to the ANT approach that manoeuvred me through this investigation, I shift my focus in the next chapter on this thesis´ subject: the journal impact factor. Here, looking back in history, I initially draw my attention to the conditions that enable the development of the bibliometric measure. I do so to highlight various corporal, textual, governmental, technological, and social actors that assemble for the metric monster´s existence. Further, as an introduction to this topic, I summarize and discuss contemporary
Metric Monsters
literature surrounding the use of JIF in academia. Finally, addressing research gaps on citation metrics, this historical chapter leads to my empirical research, and the analysis of the lived experience of fifteen European sociologists.
The Birth of Metric Monsters
In this chapter, I explore the creation of bibliometric measures in academia by illustrating the origin of the journal impact factor (JIF). Before I do this, I briefly outline the conditions that enabled its emergence, by describing the history of social statistics and its connection with the development of the modern European nation state. In relation to this, I introduce Michel Foucault´s (2007) concept of biopower that is closely linked to the production of metric knowledge in the eighteenth century. Building on this, in the second part of this chapter I zoom in on the origin of the JIF. As I follow an ANT approach, I highlight different social, textual, topographical, technological, and corporal actants that created the durable assemblage of this metric monster. In the last section of this chapter, I discuss contemporary literature on the JIF and the growing critique on its (mis-)use.
The Birth of Statistics
Social numbers have a long history. For centuries, church leaders and state rulers have calculated births, deaths or the volume of crops for taxation. Such numbers were often treated as secrets of the church or the state (Porter, 1986, p. 18). However, due to the emergence of modern European nation states, the nature of social numbers became a public issue, as the government began to draw from social statistics to collect information on their growing populations (Espeland & Sauder, 2008, p. 4).
In his work, Michel Foucault (2008) explores this connected history of the modern nation state, statistics, and new forms of power techniques.6 According to Foucault, in the eighteenth century, along the development of modern bureaucracy, a new form of governance emerged:
governmentality. Replacing a solely disciplinary form of state power, governmentality refers
to all forms of technologies, including new forms of power techniques that are not repressive, but productive. In this understanding, power is not something that can be possessed and
6 This intertwined history can already be traced in the terms´ etymology: the word statistic derives from Statistik (German) and Statistikum (New Latin), which means „of the state“ (cf. Michalitsch, 2014).
exercised over subjects, nor a sovereign power, but a domain of relations that produces a (governable) subject. Thereby, individuals become active participants in their own governance (Foucault, 2008, p. 187). Before I elaborate more on Foucault´s definition of power and the process of subjectification, I first discuss the production of metric knowledge and the creation of the governable collective subject: the population.
The collective subject of the nation state is the population, a modern idea and a central concept in Foucault´s theorization of power techniques. In order to identify, monitor and manage the population, the nation state relies heavily on the production of knowledge about its citizens. Directed by the aforementioned modern concept of objectivity, this knowledge was expressed in statistics and “neutral” facts that described and identified individuals. On the basis of generated metric knowledge, such as birth, marriage and death statistics, individuals were shaped, managed, and learned to govern themselves (Foucault, 2008, p. 13). The rise of social statistics allowed the creation and identification of the nation state´s subject, the population, while simultaneously defining the sociologist´s research field.
Further examining this connected history between social science and new forms of governmental techniques is Alain Desrosières (1998). In his research on the history of statistical reasoning in Germany, France, Great Britain, and the USA, he illustrates how statistical knowledge production became implemented within both state and academic institutions. According to him, modern statistics derives from the recombining of scientific and administrative practices that were initially far apart (p. 9). He asserts that in seventeenth-century Germany, statistics was mainly engaged in taxonomy, the practice of classification and the description of states. However, with the creation of modern nation states, the so-called table-statistician emerged. Now, statistics became an activity of quantification, which consequently gave birth to biometrics, enabling the description and identification of human beings in numerical categories (pp. 327-329).
Similar to this, Hans Erich Bödeker (2001) asserts that social science benefited from the state´s demand for information on the population. This condition gave rise to the “statistical gaze” in academia, the desire to systematically describe the social world according to mathematical logic (p. 169). Yet, as a consequence of the statistical gaze, conducting research became a means of quantifying relations of measure. For this, the table statisticians “defined their objects not as quantitative “state peculiarities”, instead they oriented themselves largely
Metric Monsters
on the methodological criteria of measurability” (p. 179). In other words, only phenomena that were quantifiable were worth studying.7
According to Gabriele Michalitsch (2014), in the process of the scientific production of metric knowledge, statistics created the collective subject, the population, which subsequently formed the basis of new governmental techniques: biopolitics (p. 2; Foucault 2008, p. 20). Biopolitics refers to the regulation and governing of the population and individual bodies through the management of life processes. The exercise of biopolitics, thereby biopower, is a means of expanding self-rationalization of every aspect of human existence, through the development of techniques and technologies that operate through the body. These techniques are directed by the capitalist virtue of productivity. As Foucault (1983) states, biopower means the adjustment of human bodies to economic processes (p. 140). Bodies can be shaped, trained, regulated, monitored, and punished, for the purpose of governance. Drawing from Foucault´s theorization of biopower, Donna Haraway (1997) applies this concept in her ANT investigation of techno-monsters. She argues that human/non-human relations inhabit time-space-regimes that she identifies as technobiopower (Haraway, 1997, p. 12; Haraway, 2004a, p. 11). By adapting Foucault´s concept, she enables us to address power-relations within heterogeneous ensembles such as the JIF. A metric that aspires to calculate, select and monitor, and optimize scientific knowledge output and – as I will discuss further – has become a measure to assess researchers themselves.
In this section, I discussed the conditions that allowed the emergence of the production of metric knowledge to measure and manage life processes, the connected history of modern European nation states, administrative practices, new forms of power techniques and statistics. I did so by introducing the concept of biopower, the exercise of biopolitics, which refers to the management of all aspects of human existence and the production of govermentable subjects. At the beginning of the twentieth century, bibliometric measures emerge on the basis of this. Leaving this general introduction of the origin of social statistics behind, in the following section I zoom in on the emergence of bibliometrics, exemplified by the history of the JIF, this thesis´ matter of concern.
7 Reflecting on this phenomenon in a contemporary setting is Sally Merry Engle (2016). In her study The
Seductions of Quantification, she identifies that metric knowledge claims are in contrast to qualitative forms of
knowledge output emphasised in current policy making. In her research, she analysis global indicators on human rights, gender violence, and sex trafficking, which are used as the basis for policy making. Here, she shows that the creators of these indicators systematically ignored qualitative research on battered women´s treatment by the courts. Due to this, the globally implemented indices are not able to measure or address a range of problems that were raised by affected women themselves (p. 1).
The Birth of Bibliometrics: The Impact Factor
Following this introduction of the emergence of statistics, I devote this section to the history of quantitative metric assessment tools that are used to monitor and evaluate scientific knowledge output: bibliometric measures. According to Alan Pritchard (1969), bibliometry refers to the quantitative study of written communication.8 Complementing – but also
replacing (Henderson, Shurville & Fernstrom, 2009) – the traditional method of peer-review, these research evaluation tools have been implemented across fields since the 1990s (Desrochers et al., 2018, p. 3). An important milestone in the development of bibliometrics was the creation of the JIF by Eugene Garfield. In the following paragraphs I, first, point out contemporary bibliometric indicators in general. Second, I describe the history of the JIF and the Institute for Scientific Information. Lastly, I discuss limitations of the JIF and growing critique of its usage.
Citation Metrics
Today, next to the JIF, a broad set of metric tools are used to assess scientific knowledge production. One of these methods is the h-index, which was developed by Jorge Hirsch (2005). It focuses on researchers themselves by indexing the quantity of the individual´s scientific output in relation to citation frequencies.
Furthermore, as a complementary tool to citation metrics, altmetrics are used to indicate scientific output and a researcher´s online recognition, and audience attention beyond academia (Desrochers et al., 2018). Provided by platforms such as Impactstory or Altmetric, altmetric measures count citations on websites such as Scopus, but they also track online media coverage, Wikipedia mentions, bookmarks on reference managers such as Mendeley, and social media coverage on Twitter, blogs, Youtube, Vimeo, Facebook, Google+ etc. (cf. Altmetric, 2018 & Impactstory, n.d.)
However, none of the aforementioned bibliometric measures are used as widely as the JIF (Archambault & Lariviére, 2009). The official JIF is traditionally calculated on the basis of the Web of Science database and annually published in the Journal Citation Report. The
8 In this thesis I use the term bibliometry or bibliometric. However, several terms have been used since the twentieth century to describe the quantitative studies in library science and scientometrics, the quantitative study of science and scientific research. This is due to the development in the field and its transdisciplinary context. For instance, in the 1940s the term librametry emerged, whereas bibliometrics has been used since the 1960s, and informetrics since the 1980s. For a full discussion on the history of terminologies surrounding the statistical and mathematical formalization of written communication and research output cf. Ravichandra Rao (1994) and Smith (2012).
Metric Monsters
database was initially owned by the Institute for Scientific Information (ISI).In 1997, Web of Science was bought by the Thomson Reuters corporation and later named Clarivate Analytics. In 2016 it was sold again and is now property of the private equity firms Onex Corporation and Baring Private Equity Asia (Grant, 2016). In the following section, I will elaborate on the JIFs history, by highlighting various human/non-human relations that made its creation possible.
The Journal Impact Factor
Due to the growing amount of scientific journals after WWII, US university and college librarians were in search for an objective method of journal selection (Archambault & Lariviére, 2009). In response to that, Eugene Garfield, founder of the Institute for Scientific Information (ISI) in Philadelphia, Pennsylvania, and the journal Scientist, began to develop the idea of an indicator to measure a journal´s impact since 1955 (Garfield, 2006). To work on this mathematical formalization, he used data that were stored on his institute´s computer, covering life science and physics journals (Fleck, 2013, p. 338). To measure impact, he followed Robert Merton´s (1988) work, which emphasised the importance of citations within scientific communities, as they play an essential role in both the symbolic appraisal of previous knowledge claims and in the representation of peer recognition (p. 620).
Finally, in collaboration with Irving H. Sher, Garfield designed a formula that divided “the number of times a journal is cited by the number of articles that journal has published” (Garfield & Sher, 1963, p. 199). The JIF was launched in the Journal Citations Report in 1973. In the first decades it did not receive much attention. However, since 1995, the use of the JIF and its derivatives exploded after it was made available on CD-Rom, and in 1998 online via the institute´s website, allowing it to be instantly accessible around the world (Minnick, 2017). The increase in numbers of papers indexed in Web of Science further enhanced the JIF´s success (Archambault & Lariviére, 2009, p. 635).
Furthermore, the growing use of bibliometric measures was enhanced by the proliferation of evaluation tools to measure and evaluate performance that swept across public institutions since the 1990s. This social phenomenon that has been described as the “audit society” (Power, 1997) or “evaluation society” (Dahler-Larsen, 2012) likewise spread across academia, leading to the blossoming of the fields of bibliometrics and scientometrics.9
9 The establishment of the International Society for Informetrics and Scientrometrics (ISSI) in 1993 exemplifies this (ISSI, 2018).
Additionally, in the European Union the implementation of citation metrics was further enhanced due to the Bologna Declaration (European Higher Education Area, 1999), the joint declaration of the European Ministers of Education. In this, the harmonization of the European system of higher education, the comparability of scientific knowledge production, and its increase in international competitiveness were promoted.
Also, according to Desrochers et al. (2018), the rise of the JIF in particular was enabled by the monopolist market position of ISI, which allowed the JIF to be imposed as a transnational and transdisciplinary standard during the 1990s. Although Clarivate Analytics still remains a central actor in the creation of the JIF, other companies contributed to the expansion of citation indexes over the last years as well, for example Elsevier´s Scopus, Google Scholar, and Microsoft Academic (p. 10).
Today, the JIF is used as a basis for tenure and funding decisions and institutionalized in various ways (p. 11). I want to point out a few examples: in the Netherlands the Dutch University Medical Centres, that specialize in biomedicine, and produce one third of all journal articles published in the country are assessed and monitored according to the JIF (Rushforth & de Rijcke, 2015).
As stated by Fuyono and Cyranoski (2006), in Pakistan, China and South Korea researchers receive bonus salary payments, when they publish in highly ranked journals. Similar to this, in Spain, a salary increase based on productivity, the sexenio evaluation, is partly based on the JIF (Franzoni, Scellato & Stephan, 2011).
Another example is the British Research Assessment Exercise (RAE) that bases their recommendations on bibliometric measures (Fleck, 2013, p. 329). Likewise, according to Fleck (2013), the Times Higher Education World University Rankings, which is powered by Elsevier and before that by the Thomson Reuters company, bases their assessment to a great extent on the Web of Science database.
Funding agencies in Flanders, Italy, and the Czech Republic request a listing of the JIF alongside the applicant´s publications. The government in Flanders distributes tax money to universities in a similar manner (Stephan, Veugelers & Wang, 2017). In Poland, the Ministry of Science and Higher Education allocates funds for research on the basis of a list that ranks scientific journals according to the JIF (Ministerstwo Nauki i Szkolnictwa Wyższego, 2017a). Overall, the JIF has become a standard bibliometric measure across national and disciplinary fields. In this human/non-human network various actors assemble. These actors are corporal (Elsevier, Microsoft, Onex, Baring Private Equity Asia), textual (news articles, scientific
Metric Monsters
papers, journals, applications forms, guidelines), governmental (ministries, public funding agencies), and social (researchers, public officials, shareholders, applicants). Hence, the JIF can be defined as a matter of concern in the Latourian sense par excellence.
Although broadly used, during the last decade critical voices (Desrochers, 2018; Fleck, 2013; Archambault & Lariviére, 2009) have been pointing out the JIF´s limitations and flaws that originate in the metric´s history, as well as its misuse. In the following section, I summarize and highlight their central arguments.
The JIF´s Misuse and Growing Critique
According to Archambault & Lariviére (2009), the JIF was initially intended for US librarians as an objective method to select journals. However, during the last decades it evolved into a tool to assess the impact of researchers themselves, as well as institutions and countries. Over the last years, the JIF´s implementation as a universal standard assessment tool has become more and more criticised (p. 636; cf. Desrochers et al., 2018, p. 10):
First, the JIF was developed in a specific US context. Archambault and Lariviére (2009) argue that the method of focusing on US journals, created a selective promotion loop – thus, the measure became biased towards these journals. The same journals were read and cited again and again by researchers. US based journals are still higher ranked than others on average (p. 638). Adding to this, Fleck (2013) asserts that journals from the margin cannot reach higher ranks, simply because of the fact that non-English articles will not be read and, thus, cited as often (p. 352). Likewise the Web of Science database, Google Scholar and Scopus emphasise on English-language journals (Desrochers, 2018, p. 10).
Second, a further bias remains that is highlighted amongst critics. This refers to the focus on natural science journals (ibid.). Although the JIF was originally created on the basis of life science and physics journals, it was later, without being adapted, implemented in other fields. Because the JIF was separated from its specific context, it could be established as a seemingly universal tool to assess other fields such as social science and humanities (Archambault & Lariviére, 2009, p. 636). Yet, different fields developed different scientific communication cultures and different citation habits, something the JIF is not sensitized to. For instance, German sociology traditionally emphasises the publication of books, not articles. However, monographies are not indexed in the Web of Science database (ibid.). Third, a second biasing consequence of the JIF´s origin in natural science is shown in the two-year citation window in which the journals are monitored. Debates and ideas in for
example history, philosophy, and sociology are developed over time and not referenced as quickly as in, for instance, biomedical science. Some articles are only recognized a few years after their publication. Some articles are not cited as often (or at all) in other scientific journals. This does not mean they do not have an (public) impact. In response to this, Clarivate Analytics provides an optional five-year JIF since 2008 (Minnick, 2017). However, this tool is not used as frequently as the well-established two-year JIF. Furthermore, the JIF is not sensitized to the sizes of the different research communities and, therefore, the audience reach of journals (Archambault & Lariviére, 2009, p. 636; Fleck, 2013, p. 393).
Fourth, all citations are considered as equal in the JIF calculation. For instance, self-citations and honorary citations are counted, which has led to misuse in the past (Desrochers et al., 2018, p. 11). For instance, publishers of the journal Leukemia have been accused of asking authors to cite their journal´s papers to maximize their average JIF (Smith, 1997). Additionally, the metric does not distinguish between negative and positive citations or the type of citation. For instance, to reference a paper might mean to draw from its core ideas, but it can also mean to simply point to one specific statement the author made. The JIF does not distinguish between these very different forms of citations. They are all counted in the same way (Archambault & Lariviére 2009, p. 640). Finally, the JIF is highly influenced by a small number of often-cited papers that push the journal´s rank (Callaway, 2016).
Fifth, critics address the metric´s performative effect. For instance, Willmott (2011) shows in his research on the UK Association of Business Schools list, that researchers create reference lists and adapt their reading habits according to the JIF, which fosters what he calls a “monoculture” (p. 429). Stephan, Veugelers and Wang (2017) show that, because of bibliometric measures, panel reviewers eschew risky studies more and more. In their analysis, the authors tracked more than 660,000 scientific papers over a fifteen-year period. Their findings suggest that the JIF undervalues highly novel papers and research with a long-term and high impact (p. 412).
Due to the growing critiques and allegations of misuse, scientometricians, scholars and publishers have started to turn against the JIF: For instance, in 2010, the German Research Foundation (2010) made a press release under the motto “quality not quantity”. Here, they declared that future funding proposals and final reports will only include a limited number of significant publications, in order to reduce the importance placed on bibliometric measures and publication lists.
Metric Monsters
In 2012, during the Annual Meeting of the American Society for Cell Biology in San Francisco, The Declaration on Research Assessment (DORA) was developed. This declaration highlights the limitations and the misuse of the JIF and emphasises the need to improve research performance. They call for the elimination of journal-based metrics in promotion considerations, funding, and appointments. Since its creation, the declaration has been signed by approximately 500 organizations, including universities, research societies, associations, and institutes around the world (Dora, n. d.).
In 2015 Diana Hicks et al. (2015) published The Leiden Manifesto in Nature. In this, the authors underline the danger of research evaluation that is led by metrics rather than by judgement. Following their critique, they propose ten principles that should guide future research assessments (Leidenmanifesto, n. d.).
In this vein, the American Society for Microbiology announced in Washington in July 2016 that it will remove the JIF from its website and journals, as well as from advertising (Callaway, 2016).
2016 in response to some parts of the growing critique, Heidi Siegel, spokesperson for Thomson Reuters (today Clarivate Analytics), defends the JIF as a measure for a journal´s output that “should not be used as a proxy for the quality of any single paper or its authors” (Callaway, 2016). Yet, the aforementioned power relations woven into the metric between natural science and social science, as well as different geographical contexts and languages are not addressed, and Clarivate promotes on their website the “Web of Science Core Collection the most accurate, objective, and complete resource available” today (Clarivate Analytics, 2018).
In spite of the rise of critical voices and Clarivate´s responses to them, according to contemporary studies, the JIF remains as widely used as ever. In his 2013 study, Christian Fleck shows that all 53 first ranked journals in the Journal Citation Report use their ranking for advertisement and marketing, by announcing it on their website or in the journals (p. 329).
According to Cruickshank (2016), the use of JIFs was further promoted in the Green Paper of 2015, which was published by the House of Commons. This governmental proposal, which initiated a reform in the UKs higher education system along the line of market rationality, underlined the use of metrics to encourage academics to increase their efficiency and to produce scientific output, which has “benefit to the economy, society, culture” (p.5).
This resonates with a recent survey of British institutions, which was conducted by the Forum for Responsible Research Metrics, a partnership supported by five UK research agencies. This study showed that only four out of 96 UK research organizations have taken comprehensive action against the (mis-)use of the JIF since the foundation of DORA (Gaind, 2018).
Similar to this, other authors (Desrochers et al., 2018; Haustein & Larivière, 2015; Stephan, Veugelers & Wang, 2017) highlight the continued application of bibliometric measures like the JIF and its effect on epistemic practices as well as academic careers.
To conclude, in this chapter I introduced the JIF and summarized contemporary discussions surrounding its use. I started this introduction by re-visiting the emergence of statistical thinking and its interconnected history with the development of the modern nation state. In the second half of this chapter, I described the emergence of bibliometric measures, by focusing on the history of the JIF. Here, I described how the JIF, which was developed in a specific geographical and disciplinary context, was de-contextualized and implemented as a universal and objective research assessment tool across fields and countries. Furthermore, guided by ANT I highlighted various social, technological, corporal, textual, governmental, and topographical actors that assemble(d) to create the JIF. I did so to emphasise that the JIF is precisely what Bruno Latour (2017) considers a matter of concern. Although the aforementioned existing studies discuss the JIF´s effects on contemporary scientific knowledge production, it lacks account of the lived experience – meaning making, forms of embodiment, affective responses etc. – of situated researchers. Building on and expanding these existing discussions, this thesis focuses on the lived experience of European sociologists, to gain a better understanding of the inter-relations and inter-actions between metrics, bodies, affects, and technologies that assemble to the metric monster of the JIF.
Approaching Monsters Methodologically
In this study, I conducted fifteen semi-structured interviews with European sociologists, with an emphasis on PhD candidates. For this, I chose the qualitative data analysis method Abductive Analysis in the tradition of Timmermans and Tavory (2012), in order to analyse the interviews. In this chapter, I outline my methodological approach in five sections. First, I argue for my method of choice, semi-structured interviews, and describe my sampling process. Second, I outline my case studies, European sociologists, who I interviewed over the
Metric Monsters
course of four weeks in spring 2018. Third, I elaborate on Abductive Analysis, my qualitative data analysis approach, which guided my fieldwork from the very beginning. Fourth, I discuss ethical principles that directed my research. In the last section, I highlight the limitations of this research design.
Interviewing as Data Collection
Following Latour´s conceptualization of ANT I made two decisions concerning the methodological design of this thesis. First, I chose to conduct interviews, to localize the global. I initially entered the field in an explorative manner with a broad set of interview questions that concerned the topic of quantification in academia (see appendix). My interview-guide was semi-structured and addressed a broad set of topics. Here, I followed Joseph Hermanowicz´ (2002) instructions on how to create a guideline and how to conduct interviews successfully.
Second, I did not constrain my case selection to one sociology department. I made this decision for two reasons: First, ANT argues a researcher should travel along the many relations that make an assemblage. Since citation metrics are not confined by national borders I, too, did not limit this research to a specific department, university or country. Second, academia is an international field with high occupational and student mobility, a trend that increased since the aforementioned Bologna Conference in 1999 (European Higher Education Area, 1999), and due to precarious working conditions (Geppert & Hollinshead, 2017). Therefore, academics themselves are often not wedded to one department either during their studies or their working lives (Metcalfe, 2017, p. 132). Scholars can be employed by more than one department and research agency at the same time, they co-write articles with other social scientists around the world, they publish in international journals, apply for conferences in other countries etc. Because of this, I did not reduce this research to one department, and made a call for respondents across Europe. However, as I previously mentioned, I am aware of national differences in publishing cultures. I address this in my analysis chapter, whenever it is contextually necessary.
I began the search for respondents in my own network, by making a call through my Facebook page and my mailing list. Additionally, I wrote to various sociology departments and associations in Europe. Here I started from my own standpoint, my language skills, and my personal experience as an Austrian student, who studied and worked in Poland and is
