What’s really the difference? Developing machine learning classifiers for
automatically identifying Russian state-funded news in Serbia
Student: Ognjan Denkovski
Student number: 10841482
Master’s Thesis, Graduate School of Communication
Master’s programme Communication Science
Supervisor: Dr. Damian Trilling
Abstract
Democratic nations globally are experiencing increasing levels of false and misleading information circulating on social media and political websites, often propagating alternative socio-political realities. One of the main actors in this process has been the Russian state, whose organized disinformation campaigns have influenced elections and public discourse throughout the Western world. A key element of these campaigns is the content produced by outlets like RT and Sputnik – content shared and republished by underfunded and sympathetic local media, as well as state-coordinated social media groups, which attempt to shape mainstream political narratives abroad. In response to a lack of comprehensive research examining the characteristics of this content, this paper examines whether, and if so how, content produced by Russian state-funded outlets is structurally distinct from U.S. state-state-funded outlets, taken as representative of Western media. Through text-as-data methods the study examines the structural, thematic and linguistic differences in content produced by U.S. and Russian state-backed outlets in Serbia, a key geopolitical interest for both states. More relevantly, the study demonstrates how these features can be used to automatically distinguish the country source of an article – findings with potential practical applications. The paper contributes towards an understanding of the structural characteristics of disinformation and online political polarization in a novel context - Balkan online news - while also forwarding the application of text-as-data methods in Serbian.
Introduction
The creation and active spread of misleading, polarizing or simply false information on social media and news websites has become an increasingly prominent topic in academia and public debate. While the most widely covered example about the influence and potential of “fake news” was the 2016 U.S. election, such media events consisting of highly charged public opinion manipulation accompanied by extensive polarization in (online) political discourse, have been increasingly visible across the globe (Galante & EE, 2018; Bradshaw & Howard, 2018). In many cases, these events are directly tied to state or state related actors operating with the goal of shaping public opinion at home or abroad. One of the most prominent actors in these types of operations is the Russian state (Helmus et al., 2018). Notable examples of Russian media influence campaigns include the narrative surrounding the MH17 flight which state-owned outlets like Russia Today ( RT) and Sputnik dominated in Google’s Top Stories, as well as the infamous Lisa case in 2015 Germany (a fabricated rape story), which resulted in protests in front of Chancellor Merkel’s chancellery (Hanlon, 2018; NATO, 2019). The perceived effectiveness and potential of these campaigns is demonstrated by the recent establishment of EU and NATO task forces for countering (Russian) disinformation in Europe (Helmus et al., 2018)
In recent years, numerous studies have examined the possibility of countering Russian influence campaigns in the Western world, by identifying and tracking the activity Russian state-coordinated or sympathetic groups on social media – a search for bots, trolls and their influence (Bradshaw & Howard, 2018, Helmus et al., 2018; Wooley & Howard, 2016; Baddaway, Ferara & Lerman, 2018; Broniatowski et al., 2018). However, few of these studies have focused on the source of much of the content which these groups propagate and discuss – content produced by
state-funded outlets like RT and Sputnik (Helmus et al., 2018; Stronski, 2019). Studies which have analyzed the coverage by these outlets have relied on qualitative methods and a curetted sample for analysis, while offering little insight into potential solutions for effectively
recognizing and responding to this content – solutions which if effective, must be automated (Yablokov, 2015; Galante & EE, 2018; Richter, 2017).1 Automatically detecting content produced by Russian state-funded outlets can contribute towards the identification of extremist social media communities, such as the Sputnik linked group recently shutdown by Facebook for spreading anti-NATO propaganda, as well as help monitor for the development of new
disinformation narratives promoted by the Russian state (Cerulus, 2019; Wooley & Howard, 2016).
The automatic detection of Russian state-funded news implies an approach with which this content can be distinguished from other types of news, and in most practical applications, from Western media. This study combines text-as-data methods, machine learning and manual content analysis to compare and contrast the profiles of articles from Russian state-funded outlets and Western content, represented by U.S. state-funded and mainstream outlets. A total of 10,132 articles are analyzed from three U.S. and two Russian state-funded outlets, covering a period of four months. The articles are analyzed on the basis of fourteen features, represented by three feature sets, namely: structural and thematic frames in articles, and linguistic properties of text. These features are used firstly to examine whether it is possible to create distinct profiles of U.S. and Russian state-funded news and secondly, whether these differences can be used to
automatically distinguish the source of an article.
1 To the extent of the authors knowledge, at the time of writing this paper, Ramsay and Robertshaw (2019) published the first robust automated study of the content of RT and Sputnik during key events in the UK.
RQ: Can Russian state- funded news in Serbia, a key element of Russian disinformation
campaigns, be automatically distinguished from Western news, represented by U.S. state-funded and mainstream outlets?
Recognizing disinformation as a contextual process, with differing characteristics across nations, this study examines the case of U.S. and Russian state-funded news in Serbia, while developing a methodological and theoretical framework which can be applied to a broader context (Bennet & Livingston, 2018). Serbia is an “ideal” case for the study and characterization of Russian state-funded news due to the low media freedom rates within the country, the existing pro-Russian sentiment in a majority of the population and the relevance of Serbian media in the region, entailing an analysis of content with impact on a regional level (Klepo, 2017; PSSI, 2019). All analyzed content is in the Serbian language.
Theoretical framework, context and feature selection
The automated detection of Russian state-funded news, a key element of Russian
disinformation campaigns, requires a consideration of three theoretical perspectives. Firstly, this study examines the roots of disinformation and the theoretical distinctiveness of disinformation, followed by a discussion of the potential impact of Russian disinformation in media systems such as the one in the Western Balkans, characterized by political parallelism and selective reporting (Hallin & Mancini, 2004). Finally, the study provides an overview of past research in automated text classification and presents the fourteen features used for the development of distinct profiles of U.S. and Russian state-funded news and machine learning classifiers which automatically distinguish this content.
Roots and theoretical distinctiveness of disinformation
In comparison to traditional media systems, characterized by few channels of information flow in which “effective gatekeeping against dangerous narratives” was possible, social media platforms and the dominance of online news sources has made it increasingly difficult for any one actor to control the flow of local or global narratives (Bennet & Livingston, 2018, p.4). In this media environment, the creation, spread and consumption of news has become an almost entirely horizontal process, eroding the gatekeeping role of traditional media and allowing verified or unverified news content produced by anyone to easily reach millions across the globe (Nemr & Gangware, 2019). State actors globally are actively utilizing these affordances of modern media and the availability of numerous, and often contradictory, interpretations of political and social events to pursue geo-political interests by targeting radical segments of foreign populations. Russia, China, Iran and Turkey are among the most innovative in this process (Janda, 2016; Polyakova & Boyer, 2018; PSSI, 2019). However, this ease of producing and spreading content online, combined with increasing popular demand for alternative
interpretations of political and social events, has also led to the development of a wide range of a) local “alternative” news sources, promoting ethnic nationalism and anti-globalist conspiracies b) for-profit fake or sensationalist news thriving on an attention economy (Bennet & Livingston, 2018; Bandeira, Barojan, Braga, Peñarredonda & Argüello, 2019; Garimella & Weber, 2017; Beam, Hutchens & Hmielowski, 2018; Morales, Borodno & Losada, 2015).
This variety of actors and diversity of type of content produced has also inspired a wide range of concepts and frameworks for describing information which misleads, deceives and polarizes, including: fake news, hyperpartisan content, junk news and clickbait, to name a few.
The ambiguity and diversity of these concepts leaves researchers hard pressed in practical applications where content delineation is required (Nemr & Gangware, 2019). Two increasingly useful and well-defined catch-all terms for researchers based in political communication are misinformation and disinformation (Jackson, 2018; Bennet & Livingston, 2018). Misinformation describes the “inadvertent” process of sharing “false information”, which is not intended to cause harm though often tied to reckless journalism, while disinformation describes the “purposeful dissemination of distorted and false information” created with the intention to mislead, harm or promote foreign interests (Nemr & Gangware, 2019, p.14; Bennet & Livingston, 2018; Schudson & Zelizer, 2017). Disinformation thus understood implies process, both organized and
continuous, rather than any isolated media events. Where state actors are involved this process frequently implies the slow building of a legitimate following based on attractive content and a utilization of this following to shape significant narratives during key social events (Bradshaw & Howard, 2018). For the current study, disinformation is more useful than terms such as fake news and hyperpartisan, as it indicates both the type of content that should be considered and the intentions of the actors behind this content. For instance, the term hyperpartisan, generally applied in studies of Western right-wing media cannot be applied to the current study, as research shows that Russian state-funded news rarely follows a consistent ideological stance, rather promoting content in-line with views of various political fractions, such as European right-wing nationalists swayed by an anti-EU message, but also European far-left actors catered to with stories regarding U.S. hegemony (Sanovich, 2017; Richter, 2017; Barbashin, 2018). The term fake news, which specifies neither the type of content nor the intentions of the actors behind it, is also not applicable in the current study, as research shows that much content from Russian
outlets cannot be considered false (Yablokov, 2015; Sanovich, 2017; Galante & EE, 2018). Finally, in accordance with this definition, disinformation is largely socio-political in nature.
This study uses this definition of disinformation which suggests that disinformation is created and spread with the intention to manipulate public opinion and is thus distinct in agenda from Western legitimate journalistic content, which aims to truthfully inform the public and hold authorities accountable (Bradshaw & Howard, 2018; Bechev, 2018). This assumption implies that Russian state-funded outlets follow distinct news production patterns and routines, which can be used to distinguish this content from Western outlets. This assumption leads to the follow research question.
RQ1: Can statistically significant differences be observed in news production routines and linguistic habits (operationalized as structural, thematic and linguistic features) of Russian and U.S. outlets in Serbia?
Disinformation across media systems
The repercussions of Russian disinformation are tangible in stable democratic media spaces in Western societies as made evident by the establishment of both EU and NATO task forces specifically created for developing approaches to detect and counter Russian
disinformation (Janda, 2016). Both task forces were preceded by the Lisa case, one of the first incidents in Europe where the potential impact of Russian disinformation became clear, as 700 protestors of Russian and German origin protested in front of Chancellor Merkel’s chancellery holding anti-immigrant banners due to a rape incident which was proven as fabricated, but extensively covered by RT and Sputnik (Hartmann & Alder-Niessen, 2018). As these campaigns
can have a significant impact on public opinion and political behavior across the Western world, regions like the Western Balkans where media and democracy remain underdeveloped and where “fake news and disinformation are the norm” are particularly susceptible (Klepo, 2017; Bechev, 2018, p. 5).
Western Balkan media is best described by the polarized pluralist ideal in the seminal work by Hallin and Mancini (2004), an ideal described by high levels of political parallelism, overtly biased reporting, gatekeeping bias and selective emphasis reporting (Haselmayer, Manger & Meyer, 2017; Andresen, Hoxha & Godole, 2017; Klepo, 2017). This ideal is particularly suitable for describing Serbia’s media environment since the (re)election of Aleksandar Vučić, former Prime Minister and current President of Serbia, who has not responded kindly to critical media (Stronski, 2019). In Serbia, critical media is largely
underfunded and not infrequently under threat of violence, while government sympathetic and alternative news outlets dominate news consumption. The state of Serbian media partially motivated Serbia’s lengthy “1in5 million” protests, the country’s largest since the fall of
President Milošević and on-going since November 2018, protests which public broadcaster RTS has largely neglected (Al Jazeera, 2019; Bechev, 2018; Fidanovski, 2019). At the same time, journalists in Serbia are facing many of the issues faced by journalists globally, such as a decrease in advertising revenue, an increased demand for rapid content production, reliance on social media and for-profit news production patterns (Andresen, Hoxha & Godole, 2017; Schauster, Ferrucci & Neill, 2016). This media environment, combined with the existing anti-Western attitudes among Serbs, as well as the division between pro-European and pro-Russian
public opinion, makes Serbia an “ideal” space for promotion the promotion and success of geopolitically relevant narratives by foreign states (Eisentraut & de Leon, 2018; Stronski, 2019).
Russia has been particularly successful in this respect. Research from the U.S. Senate Foreign Affairs Committee shows that Russian media influence in Serbia is notable, with Russian popularity among Serbs increasing from 47.8% in 2015, the year Sputnik Serbia was launched, to 60% in June 2017 (PSSI, 2019). President Putin’s popularity in Serbia, who mainstream media portrays as a great supporter of Serbia against Kosovo’s independence, is second only to that of President Vučič (EU vs. Disinfo, 2018). This successful penetration can be explained by two factors: the existing (or at least perceived) connection between Russia and Serbia combined with existing anti-Western attitudes in the population, and perhaps more relevantly, the free-for-all policy of Russian outlets, which do not charge a fee for republishing their content. Consequently, an increasing number of sympathetic, underfunded or for-profit local outlets have begun utilized this possibility and either actively republish content from Russian state outlets in Serbia or base their own reporting according to their narratives. In 2016, researchers showed that one third of outlets in Serbia publish content about international actors without noting sources or authors, much of which characterized by pro-Russian and anti-U.S. attitudes (Center for Euroatlantic Studies, 2016).
However, U.S. state-funded and mainstream media also maintains a strong presence in Serbian online news, with outlets such as Radio Slobodna Evropa (RSE), Glas Amerike (VOA) and N1 (a regional informative partner of the CNN), being particularly popular. However, U.S. media has not achieved the level of market penetration allowed by Russia’s free content policy, even though RSE and VOA in many ways mirror outlets like RT and Spunik, in that they are
state-funded and promote U.S. interests and topics to foreign populations. A qualitative analysis of the coverage by N1 and Sputnik Serbia showed “noticeable differences in the selection of topics and interlocutors”, as well as ideological stances towards the local government, foreign-policy and social issues, leading to the conclusion that the presence and content of U.S. and Russian media in Serbia is a reflection of the existing geopolitical divisions in the country (Klepo, 2017, p.3). Similar findings have been reached by analyses thereafter, with Bechev noting the increasing need for Western outlets which would counter Russian state-funded narratives in Serbia (Bechev, 2018; Eisentraut & de Leon, 2018, Stronski, 2019). This study builds on these findings and assesses their validity with fourteen theoretically relevant features which are tested for their potential in automatically distinguishing between U.S. and Russian state-funded news.
RQ2: Can differences in distribution of the features (structural frames, thematic frames and linguistic properties of text) in articles from Russian and U.S. state-funded outlets be used to automatically determine article country source?
Automatically identifying Russian state-funded news – feature selection
Recent advances in computational methods for text analysis offer numerous possibilities for tool-development which can meaningfully contribute to the process of automated
identification of content linked to disinformation, including automated frame prediction and text classification with machine learning (Jain, Sharma & Kaushal, 2016; Conroy, Rubin & Chan, 2016; Burscher, Odijk, Vliegenthart, De Rijke & De Vreese, 2014). This literature suggests three relevant feature types which can be used for automatically detecting article source, namely: structural frames, thematic frames and linguistic properties of text. The latter features are
extracted automatically; while to obtain the former, a small, manually coded data set is used for training a supervised machine learning classifier which automatically predicts frame presence in the entire data set. This study builds on the success of the holistic frame identification methods proposed by Burscher et al. (2014), which uses human ‘Yes’/’No’ responses to indicator
questions related to each frame for training the supervised classifier. This procedure is elaborated in detail in the methodology section of the study. In line with past automated framing literature, this study considers that “word co-occurrence“, “repeated patterns” and “related topics” found by automated text analysis techniques are representative of frames in news production (Matthes & Koring, 2008, p.263; Greussing & Boomgarden, 2017, p.1755; Van der Meer, Kroon, Verhoeven & Jonkman, 2019, p.791).
The first feature set (structural frames), generally referred to as generic frames, is based on theories such as mediatization which suggest that news production and presentation is an increasingly institutionalized process, guided by routines and standards of news production developed to optimize content appeal (Stromback, 2008). These frames “transcend thematic limitations” and refer to the format of news presentation and journalistic routine (Matthes & Koring, 2008; Hertog & Mcleod, 2001; Neuman, Just & Crigler, 1992; Burscher et al., 2014). The structural frames considered in this study are human interest, conflict and morality (De Vreese, 2005). The conflict frame emphasizes conflict between individuals, groups or
institutions, and is one of the most prevalent frames in news coverage (Semetko & Valkenburg, 2000). The human interest frame adds a “human face to news coverage” and attempts to
personalize an issue in order to increase audience appeal (Iyengar, 1994, p.152). Finally, the morality frame treats religious or moral prescriptions as central when discussing an issue or
event (e.g. in public discourse about gay marriage or euthanasia) (Semetko & Valkenburg, 2000). Literature has consistently demonstrated that journalists across the world frequently use these frames to simplify social issues and improve content appeal (Van der Meer et al., 2019; Stromback, 2008; Matthes & Koring, 2008; De Vreese, 2005). As the use of these frames is indicative of routine and institutionally dependent content production patters, this study expects that differences in the use of these frames can be observed in U.S. and Russian state-funded news due to the differing media systems which they operate from and the distinct agendas which they pursue.
The second feature set (thematic frames) refers to journalistic practices in reporting about specific topics. Just as the structural frames, the thematic frames aim to capture common
practices and routines in reporting, but allow for a “profound level of specificity” regarding the narrative surrounding the examined issue (De Vreese, 2005, p. 54; Matthes & Koring, 2008). These frames are frequently referred to as issue-specific and rarely apply in other contexts. The thematic frames examined in the study build on past findings regarding Russian narratives in Serbia, including anti-Western sentiment, Russian opposition to Kosovo’s independence, and the promotion of a pan-Slavic/Orthodox connection between Russia and Serbia (Klepo, 2017; Bechev, 2018; Eisentraut & de Leon, 2018). The first thematic frame, Serbian victimhood, refers to the presentation of Serbia as a victim of Western geo-politics, through references to the NATO bombing of Serbia or the “loss” of Kosovo (Eisentraut & De Leon, 2018; Bechev, 2018). The Anti-West frame refers to the presentation of Western actors as aggressive or as a threat to Serbia or “global order” (Eisentraut & De Leon, 2018; Klepo, 2017). The Pro-Russian frame
(Stronski, 2019). Finally, the Russian Might frame emphasizes the relevance of the Russian state in global geo-politics, economy and international relations, as well as its ability to “defend” Serbian interests from Western actors (Bechev, 2018; Stronski, 2019).
The third feature set (linguistic properties of text) captures variations in linguistic habits of journalists. A recent strand of text analysis research demonstrates that linguistic properties of text are often stable habits with significant variation based on author ideology (Schoonvelde, Brosius, Schumacher & Baker, 2019). For instance, Brundidge et al. (2014) demonstrate that conservative political bloggers in the U.S. use significantly simpler language than their liberal counterparts (Brundidge, Reid, Choi & Muddiman, 2014). More direct properties of text, such as article and title length, and punctuation patterns, have been used for automatically distinguishing between fake news, clickbait content and legitimate journalistic content (Pennebaker, Mehl & Niederhoffer, 2003; Horne & Adali, 2017; Charkraborty, Paranjape, Kakarla & Ganguly, 2016). Horne and Adali (2017) demonstrate that fake news has longer titles and uses simpler, more repetitive content in the text body, while Chakraborty et al. (2016) demonstrate that clickbait content can be automatically distinguished from legitimate news through features such as word length and common bait phrases. Working within the limitations of automated text analysis for Serbian, this study tests the utility of seven linguistic properties of text, namely: title length, article length, ratio of unique words, ratio of substantive words in article text, ratio of substantive words in article title, average word length and named entities in article text (references to people and institutions, e.g. “the EU”, “government”, “the people”).
Together, these features sets cover a wide range of potential metrics for creating distinct profiles of Russian and U.S. state-funded news and for automatically distinguishing between the
two. The structural and thematic frames attempt to capture distinctions in generic and issue-specific routines and institutional practices, while the linguistic features of text attempt to capture stable linguistic habits of journalists. The study tests the utility of each of the three feature sets and any combination thereof for the automated distinction of article country source.
RQ3: Which feature set (structural frames, thematic frames or linguistic properties of text) or combination of features can best inform the automated identification and distinction of Russian and U.S. state-funded news?
Table 1
All features (N=14)
Thematic and structural features Linguistic features Structural frames Thematic frames Title length (in characters)
Article length (in words)
Ratio of unique words in article text Ratio of substantive words in article text
Ratio of substantive words in article title
Average word length in text (in characters) Named entities Human Interest Conflict Morality Serbian Victimhood Anti-Western Pro-Russian Russian Might
Methodology
This study consists of four stages of data analysis and preparation. Firstly, the study presents and justifies the data set analyzed, including criteria for outlet and content selection. Thereafter, the study discusses the manual content analysis procedure (N=1108), which produced the data set used for training supervised machine learning (SML) classifiers for frame prediction in the entire data set (N=10132). The following section details the development of classifiers for
each frame, including classifier evaluation and relevant metrics. Finally the study details the automated classification of articles on the basis of country source.
All data scrapping was conducted in Python, primarily relying on the packages Selenium and BeautifulSoup (Salunke, 2014; Nair, 2014). Data processing and wrangling was conducted with the packages Pandas and NumPy (McKinney, 2011; Van Der Walt, Colbert & Varoquaux, 2011). Text processing was conducted with the NLTK library, while some Serbian-language specific tasks are conducted with the packages Polyglot and Transliterate (Loper & Bird, 2012; Al-Rfou, Perozzi & Skiena, 2013; Qian et al., 2010). All machine learning was conducted within the Python Scikit-Learn framework (Pedregosa et al., 2011). Refer to appendix A for access to the Github repository containing all relevant scripts from this study.
Dataset overview
The dataset analyzed consists of 10,132 articles from the period between the 1st of November 2018 and the 1st of February 2019, 5860 (57%) of which from three U.S. outlets and 4272 (43%) from two Russian outlets.2The five analyzed outlets are: N1, Radio Slobodna Evropa (RSE) and Glas Amerike (VOA) representing U.S. state funded and mainstream outlets and Sputnik Serbia and Vostok Vesti representing Russian state-funded outlets. The outlets considered were chosen on the basis of data accessibility, reliability of archives, the connection between the outlets and U.S. or Russian state-funding, as well as outlet impact, examined through the number of followers on Facebook. N1 is a regional informative partner of the CNN with 199 thousand followers on Facebook and while not funded by the U.S. government it does represent news
2 10,800 articles were collected, however 668 were dropped on the basis of conditions such as minimum text length or invalid URL’s.
production patters of mainstream U.S. media. RSE and VOA are U.S. state-funded outlets with 215 thousand and 125 thousand followers on Facebook, respectively (Klepo, 2017). Sputnik Serbia is the official Serbian branch of Sputnik, while Vostok, whose funding and ownership are not disclosed, republishes RT and Sputnik international content in Serbian, with 125 thousand and 80 thousand followers on Facebook, respectively (Klepo, 2017). As literature suggests that disinformation is typically socio-political in nature, only categories such as world news, politics, economy and culture are considered. All articles from these categories are scrapped for the time period studied. Refer to Appendix B for the categories associated with each outlet and the number of articles from each outlet.
Frame prediction with SML and manual content analysis
In a study comparing several approaches for automated frame detection and prediction, Burscher et al. (2014), show that the best way to teach supervised classifiers to recognize and predict frames is the holistic approach. With this approach, human coders identify the presence or absence of frames through ‘Yes’/’No’ responses to a set of indicator questions associated with each frame. Each indicator question is taken as representative of different but equally important aspects of a frame and a ‘Yes” response to at least one indicator question indicates the presence of a frame (Burscher et al., 2014). The indicators used for the structural frames have a long tradition in framing research with established potential in several seminal publications (Iyengar, 1994; Semetko & Valkenburg, 2000). These indicators were also used by Burscher et al. for comparing automated frame prediction approaches, with a high classifier success rate (2014). The thematic frame indicators are original contributions of the author developed on the basis of past literature regarding Russian content in Serbia and a manual review of past publications by
the considered outlets. For instance, coders identify the presence of the conflict frame with the following indicators:
1. Does the item reflect disagreement or conflict between parties, individuals, groups or countries?
2. Does the item refer to two sides or more than two sides of a problem?
The presence of the Serbian victimhood frame is identified with the following indicators: 1. Does the item make reference the NATO bombing of Serbia?
2. Does the item make reference to Kosovo as illegitimate or as a threat to Serbs or Serbia? 3. Does the item make reference to the oppression of Serbs by other ethnic groups in the
region?
This procedure produces a series of single ‘Yes’/’No’ responses for each frame in each article. These responses are used by the SML classifier to replicate human decision making by establishing probabilistic associations between properties of text and human-indicated frame presence or absence. In this study, SML classifiers are trained to replicate human decisions about the presence or absence of structural and thematic frames based on 1108 manually coded articles, then applied to predicting frame presence in the remaining data sets (N=9024). In this way, a small data set with human input is used for the classification of an amount of data which would otherwise have taken substantially longer, exceeding both budget and time constraints of the study.
Manual coding was conducted by three recent graduates in communication science and related disciplines and the author, all of which fluent in Serbian.3 Coders were provided with instructions and a codebook containing a delineation of units of analysis, frame definitions and frame indicators questions. Coders were instructed to treat the title and text of articles as a single
3
unit of analysis, such that a frame could be present in either, both or neither, where single and multiple presences of a frame within an article were coded with a single ‘Yes’ response. Coders underwent two training sessions and received feedback thrice, after which each coder was provided with a double stratified sample accounting for both country source and month of publication of the data. Refer to Appendix C for the codebook and for the indicators associated with each frame. Inter-coder reliability was assessed with Krippendorff’s alpha (kalpha) and a double-stratified sample (N=31). Kalpha scores for each frame are presented in Table 2.
Table 2
Inter-coder reliability scores with Krippendorrf’s alpha (Kalpha)
Frame Kalpha Human interest .52 Conflict .72 Morality .58 Serbian victimhood .80 Anti-West .78 Pro-Russian 1 Russian might .69
The suggested kalpha level for considering an item reliable is .80, with .60 as a recommended minimum (De Swert, 2012). Five out of seven frames have kalpha scores which meet the minimum recommended value of .60, with the human interest and morality frames performing below this threshold. Despite the low reliability scores for these frames, all structural frame inter-coder reliability scores are a significant improvement from those achieved Burscher et al., which use this data to develop classifiers performing with an accuracy of up to 90%.4 Due to the
explorative nature of this study, data for all frames is used for the development of SML
4
prediction classifiers, the performance of which determines which frames will be used for country source classification.
Once data by human coders was obtained, four pre-processing steps were necessary for the training of SML classifiers for recognizing and predicting frame presence in unlabelled data (Burscher et al., 2014). Following standard practice in text classification, all stopwords (e.g. he, she, it), and punctuation were removed from the articles. The remaining words were stemmed (i.e. reduced to their roots), and converted to lower case, so as to standardize the data and to drop “white-noise” attributes of text (Boumans & Trilling, 2016; Graaf & van der Vossen, 2013). This study develops a customized stopword list by combining a pre-existing list developed by Jason Champion and a new list developed on the basis of the most commonly occurring stop words in the data set used for this study (Champion, 2019). The two lists were combined and manually reviewed by the authors, leaving 251 unique stopwords. Refer to Appendix A for the Github repository of this project, where the stopword list is uploaded. For stemming, the study relies on a custom Serbian stemmer developed by Milosevic, the code for which is available on the Github repository of the project (Milosevic, 2012). Thereafter, the content of each article is represented quantitatively as a vector of document features, where each word is assigned with a score on the basis of its frequency in the data set and each article is related to its corresponding frame
responses by manual coders (Matthes & Koring, 2008; Boumans & Trilling, 2016; Grimmer & Stewart, 2013). Finally, the SML model statistically analyses the features of documents and establishes connections between the vectorized representations of articles and the frame
of word “weights” and their association with ‘Yes’/’No’ labels for each frame (Grimmer & Stewart, 2013).
Evaluating classifier performance for frame and country source prediction
The performance of machine learning classifiers is highly dependent on the chosen model and text vectorizer, as well as their hyperparameters (Grimmer & Stewart, 2013). Scikit-Learn’s grid search is used to test 20 model-hyperapameter combinations to determine the optimal classifier for detecting and predicting each frame. The performance of the optimal model-hyperparameter combination is tested in combination with two text vectorizers, namely: Scikit-Learn’s count vectorizer, which produces a simple frequency score for each word in the corpus, and the Tfidf vectorizer, which assigns each word with a score based on the number of times it occurs in the documents as weighted by the inversed frequency of that word across the entire corpus (Pedregosa et al., 2011; Russell & Norvig, 2004). In total, some 600 parameter
combinations are used for determining the optimal vectorizer to use with each model for each frame.
Classifier performance is evaluated with the weighted f1 metric, a “weighted harmonic mean” of precision and recall, where precision indicates the percentage of correctly classified cases, while recall indicates how many of the cases that should have been identified were actually identified (Forman, 2003, p.1294).5 The metric ranges from 0 to 1, with 1 indicating perfect performance. With the weighted f1 score, the weight of each class (in this case the proportion of ‘Yes’/‘No’ responses) is accounted for when producing the averaged metric. The weighted f1 score is a more appropriate metric than accuracy, which can be misleading in cases
5
with unbalanced data (Godbole & Sarawagi, 2004). As most automated text classification studies achieve classifier accuracy rates between 70% and 90%, this study considers those classifiers which perform with a weighted f1 score greater than 70% as fit for frame prediction on unlabelled data (Grimmer & Stewart, 2013; Horne & Adali, 2017; Chakraborty et al., 2016; Burscher et al, 2014). Performance metrics for each model are obtained with five-fold cross validation, where each model-hypeparameter combination is fitted five times on five separate subsamples, producing five scores which are averaged to obtain a single weighted f1 metric (Burscher et al., 2014). This approach ensures that classifier performance rates are not solely due to the particular sample used.
The manually coded and automatically predicted frames are combined with the
automatically extracted linguistic features of articles to predict the country source of an article. First, the variance of each feature across the two data sets is examined, so as to eliminate those features which do not vary significantly and which consequently are not likely to contribute to the country source classification task. The remaining features are used for country source
classification with a set of 119 model-hyperparameter combinations, and weighted f1 metrics are obtained through five fold cross-validation. To optimize classifier performance, a total of nine feature combinations are considered, including: all remaining features, all remaining frames, each feature set individually, and four feature combinations suggested by selection techniques available in Sckit-Learn. The feature reduction section techniques used are univariate feature reduction, based on a Chi-square analysis, Recursive Feature Elimiation (RFE) which recursively removes attributes and builds a model with the remaining attributes, Random Forest Bogged Decision Trees which use Random Forest structures to determine feature relevance and variance
analysis which assesses which variables vary most across the two data sets (Khalid, Khalil & Nasreen, 2014; Brownlee, 2016). Refer to Appendix D for the models and hyperparameters used in the grid search for frame prediction and country source classification.
Results
The results sections reports three distinct analyses. Firstly, the study demonstrates the distribution of frames per country source based on manually coded data and the implications of this distribution for automated frame classification. Consequently, the performance of frame prediction classifiers is evaluated. In the second stage, the study demonstrates the level of variance of frames and linguistic features across U.S. and Russian news, thus examining the distinctiveness of the profiles and answering RQ1. Those features for which variance across U.S. and Russian news is significant are used for the development of country source classifiers, answering RQ2. At the same time, the study evaluates the best performing feature combination and answers RQ3.
Manually coded frame distribution and SML frame prediction
Figure 1 shows the distribution of human coded ‘Yes’/’No’ responses for each frame across the two data sets as a proportion of total articles in U.S. and Russian news. The figure demonstrates that all structural frames except the human interest frame are nearly equally distributed across the two data sets. With the exception of the Serbian victimhood frame, all thematic frames are unique to Russian news items, with the anti-West frame present in nearly 20% of Russian articles. This distribution suggests that all thematic frames and the human interest frame can help distinguish between the two data sets, while the morality and conflict
0,00% 10,00% 20,00% 30,00% 40,00% 50,00% 60,00% 70,00% 80,00% 90,00% 100,00% USA-Yes USA-No RUS-Yes RUS-No
frames likely will not have high discriminatory potential due to similar distribution across U.S. and Russian news.
Figure 1
Proportional frame distribution by country source in manually coded data (N=1108)
The optimal classifier score for each frame is presented in Table 3. With the exception of the morality frame, all frames meet the 70% cut-off for weighted f1 scores. For structural frames, precision and recall rates for both ‘Yes’ and ‘No’ responses range between 50% and 87%, with a maximum precision score of 85% and a maximum recall score of 87%.
Table 3
Optimal classifier performance scores for frame prediction
Frame Optimal classifier Weighted f1 ‘Yes’ precision ‘Yes’ Recall ‘No’ precision ‘No’ recall
Human Interest MultinomialNB 75% 57% 50% 81% 85%
Conflict MultionomialNB 75% 81% 87% 52% 42%
Morality MultinomialNB 60% 61% 47% 69% 80%
Serbian Victimhood AdaBoost 89% 69% 30% 90% 98%
Pro Russian Russian might Logistic Logistic 90% 90% 80% 50% 40% 5% 97% 91% 100% 100%
The very positive high f1 scores of the thematic frames are qualified by their precision and recall rates. While precision varies from 40% to 98%, recall rates for some thematic frames are quite poor with the Russian might frame having a recall rate of only 5%. These scores are explained by the distribution of frames in Figure 1 which shows that thematic frames are highly unbalanced, with each frame having significantly more ‘No’ responses. Thus, while precision rates for ‘Yes’ responses for thematic frames range between 50% and 80%, indicating that the classifiers are capable of identifying the characteristics of frames, the recall rates are never higher than 40% meaning that the classifier returns a high number of false negative responses. Based on the performance of the classifiers, the morality and Russian might frame are not used for country source classification. The morality frame is dropped due to an f1 score below 70%, while the Russian might frame is dropped due to the extremely poor recall rate of 5%. A detailed overview of the classification reports for each frame is available in Appendix E.
Feature variance for country source prediction
The five remaining frames are combined with the automatically obtained linguistic features, a total of twelve features characterizing 10132 articles. The study reports results from a Chi-square analysis of the variance of frames in U.S. and Russian news, as well as a Wilcoxon rank sum test for the variance of linguistic features. These analyses answers RQ1 and inform the feature selection for country source classification by showing which features are unlikely to have discriminative power for article source (Horne & Adali, 2017). The outcome of the Chi-square
analysis for frame distribution is presented in Table 4, which shows that all frames vary
significantly in their distribution across U.S. and Russian news. All five frames are kept for the country source classification task.
Table 4
Chi Square (X2) scores – frame distribution by country source
Frame X2 (2, N = 10132) Human interest 211.97*** Conflict 63.39*** Serbian victimhood 11.43*** Anti-West 95.96*** Pro-Russian 32.71*** Note: ***p < 0.001
These distributions are visualized in Figure 2.
Figure 2
Proportional frame distribution by country source in automatically and manually coded data (N=10132)
As suggested by the distribution of ‘Yes’/’No’ responses in the manual data and the performance of the classifiers, ‘Yes’ response rates are underrepresented in the entire data set
0,00% 10,00% 20,00% 30,00% 40,00% 50,00% 60,00% 70,00% 80,00% 90,00% 100,00%
Human Interest Conflict Serbian Victimhood Anti-West Pro-Russian USA-Yes USA-No RUS-Yes RUS-No
and in particularly in the thematic features, as classifiers conservatively assign ‘Yes’ responses based on the unbalanced distribution in the manual data. However, the distribution presented above is consistent with that suggested by manually coded data, with the human interest frame significantly more prevalent in U.S. news, while all thematic frames are significantly more prevalent in Russian news.
For the linguistic features, the study reports the results from a Wilcoxon rank sum test, the non-parametric equivalent of a t-test (used when data does not conform to a Gaussian distribution), which demonstrates that all automated features vary significantly in their
distribution across the two data sets. Based on this analysis, all automated features are considered for country source prediction.
Table 5
Wilcoxon rank sum test for linguistic features
Feature Z-score
Article length Title length
Ratio substantive words - text Ratio substantive words - title
-23.23*** -14.90*** -17.74*** -17.73*** Average word length
Ratio unique words
-27.48*** -17.34*** Note: ***p<0.001
Figure 3 demonstrates this distribution visually and shows that U.S. articles are longer on average, while Russian articles have longer titles and are more substantive (with a higher ratio of non-stop words) and more complex (with longer words and a higher ratio of unique words).
Figure 3
Automated features by country source, z-score
Finally, figure 4 presents word cloud rendering of the named entities in article texts, the variance of which cannot be tested statistically. These word cloud renderings show the most discussed entities by outlets from each country, with entity size corresponding to the frequency of mention. Figure 4
Word Cloud representation of named entities (NE) per country source
The entities referenced by journalists shows that U.S. news in Serbia has a strong focus on Serbia and Serbian affairs as indicated by references to Serbia[‘srbija/srbiji/srbije’],Belgrade[‘beograd’] and President Vučić. In comparison, Russian news while focused on Serbia[‘srbije/srbiji’] also extensively discusses the U.S.[‘sad’], the EU[‘eu’] and Russia [‘rusije/rusija’].These
visualization indicate substantial differences in actor focus in the reporting by outlets, an observation confirmed by an analysis of the five most frequent named entities in each data set, shown in Table 5. For a listing of the fifteen most common named entities and the fifteen most common words in the articles, refer to appendix F.
Table 5
Top five named entities per country source
Five most referenced - entities U.S. news
Frequency Five most referenced entities - Russian news
Frequency
Serbia[‘srbija’] 1698 USA [‘sad’] 1109
USA [‘sad’] 1241 Serbia [srbija’] 1061
Serbia [‘srbiji’] 1017 Russia [‘Rusije’] 869
Serbia [‘srbija’] 839 Russia [‘rusija’] 607
EU [‘eu’] 815 Kosovo [‘kosova’] 594
The analysis of human coded data, frame prediction on the basis of this data and feature variance of all fourteen features indicates that twelve of these features can aid the distinction between Russian and U.S. news. The analyses show that the automated features article length, title length and named entities can be particularly informative, along with the human interest,
pro-Russian and anti-West frames. This analysis answers RQ1 and shows that twelve of the
Country source classification
In the development of a country source classifier, the study tests the classifying ability of 9 combinations of features, namely: all twelve remaining features, the five remaining frames, each feature set individually, and four feature combinations suggested by feature selection techniques available in the Scikit-Learn framework (Khalid, Khalil & Nasreen, 2014; Brownlee, 2016). The output of each feature selection analysis presented in Table 5.
Table 5
Features selected by feature selection analyses
Feature selection output
Univariate Recursive Random forest Variance threshold
Article length Article length Average word length Article length Title length Average word
length
Ratio substantive words
Average word length
Human interest frame Title length Article length Title length Anti-West frame Thematic frames Ratio unique words Conflict frame Ukraine - NE Structural frames Title length Anti-West frame Note: Ukraine NE represents the named entity Ukraine and its presence or absence in an article. Thematic frames
indicates all remaining thematic frames and structural frames indicates all remaining structural frames.
The selected features from each feature reduction technique consistently indicate article length, title length, average word length and other linguistic features as the most relevant features for distinguishing article country source. As suggested by the distribution of frames in Figures 1 and 2, the anti-West and human interest frame are also selected as relevant by two feature selection techniques. Optimal model-hyperapameter scores for each feature set are reported in Figure 5, along with the corresponding weighted f1 score of a dummy classifier, which uses
simple rules for classification and provides a baseline classification rate against which model performance can be evaluated (Pedregosa et al., 2011).
Figure 5
Country source classification in descending order of feature performance
The classifier performance rates show that the best performing classifier uses all the remaining features (N=12), producing a weighted f1 score of 75%, along with three more feature combinations with scores approaching 70%, all of which significantly outperform their
corresponding dummy classifiers.6 This analysis answers RQ2 and shows that the automated distinction of Russian and U.S. state-funded news is possible with a relatively high degree of accuracy. These findings also answer RQ3, showing that the most useful feature set, out of the three initially posited, is the linguistic feature set, in particular the variables article length, title length, average word length and ratio of unique words. The worst performing classifiers are
6 Classifiers performed with scores up to 86% in classification tasks with a smaller training set (N=1300/4000). These scores could not be replicated with the entire data set.
Thematic frames (N=3) Structural and thematic frames (N=5) Structural frames(N=2) Univariate feature reduction(N=4) RandomTrees feature reduction(N=4) Variance feature reduction (N=4) Recursive feature reduction (N=4) Linguistic features (N=7) All features (N=12)
Dummy weighted f1 Weighted f1
based on the structural and thematic frames, with classifiers performing no better than the dummy classifiers. However, all features perform somewhat better than only the automated features, indicating that the frames do contribute to the country source classification. Figure 6 shows the precision and recall rates of each feature combination.
Figure 6
Recall and precision rates per country source for each model
These recall and precision rates indicate that the classifiers are equally suited for recognizing both U.S. and Russian articles with precision scores of 76% and 73% in the best model. However, recall rates for U.S. articles are consistently higher, with an optimal score 82%, whereas Russian recall ranges between 10% and 70%, with 63% in the optimal model. The higher recall scores for U.S. news may indicate that U.S. news has more consistent unique characteristics which allow for high recall or that these findings are the result of the slight imbalance of data, with U.S. articles representing 57% of the data set.
0,00% 10,00% 20,00% 30,00% 40,00% 50,00% 60,00% 70,00% 80,00% 90,00% 100,00% U.S. precision U.S. recall Russia precision Russia recall
Conclusion and discussion
Building on literature on framing, disinformation and modern media systems, this study examines the potential of novel automated methods in political communication research and demonstrates that the automated detection and distinction of U.S. and Russian news in Serbia is possible. The study also demonstrates the potential of the proposed methodological and
theoretical framework for the identification of content linked to Russian disinformation campaigns – a framework which can easily be applied in numerous contexts with slight
adaptations of the thematic frames used, while all remaining features can be directly applied and expanded on to new data in foreign contexts. Perhaps the most relevant findings of this study is the demonstration that the distinctions between U.S. and Russian state-funded news in Serbia are best presented by simple linguistic properties of text, suggesting that a large scale cross-country analysis of the differences between Russian news and local media could be conducted easily without consideration of language barriers, while potentially suggesting robust approaches for detecting and countering Russian disinformation.
Firstly, the study demonstrates that theoretical feature selection was effective, as all three feature sets show relevant differences across the reporting of outlets from both countries, with U.S. news characterized by longer, simpler and more humanized news, while Russian news uses longer titles, more complex language, promotes anti-Western narratives and discusses Russia and Russian affairs extensively. Conversely, U.S. media seems quite extensively focused on internal Serbian affairs, more so than on internal U.S. affairs or Russia. In line with past research, this study answers RQ1 and demonstrates observable differences in both styles and content of reporting from U.S. and Russian outlets (Klepo, 2017; Bechev, 2018; Yablokov, 2015). More
generally, the findings support past literature which suggests that news production values and linguistic habits are significantly varied across media systems, ideology and authors, suggesting that quantitatively profiling outlet news production values, as well as linguistic habits of
journalists, bloggers and even politicians has extensive merit for political communication research (Horne & Adali, 2017; Schoonvelde, et al., 2019).
Secondly, the study shows that while Russian state-funded news does contain anti-Western and divisive elements, these are present in no more than 20% of articles. This finding is supported by the high inter-coder reliability scores for these frames (all above 70%) and these findings are quite possibly representative of Russian news in other contexts. This finding qualifies past research which suggests that RT, Sputnik and related outlets are far more
concentrated on promoting Russian geo-political interests, than on reporting about relevant social events and fulfilling traditional media roles (Stronski, 2019; PSSI, 2019; Bechev, 2018; Klepo, 2017). While this study cannot make claims about the objectiveness or quality of the reporting of Russian outlets, it clearly demonstrates that most Russian content does not contain the divisive elements suggested by past qualitative research, even though these narratives do contribute to a part of their reporting (Klepo, 2017; Stronski, 2019; Galanate & EE, 2018; Richter, 2017).
The study also shows that the differences in content of Russian and U.S. outlets can be used to automatically predict the source of an article with a weighted f1 score of 75%, a relatively high performance score, answering RQ2. The classification scores achieved by this study are in-line with that of past automated text classification research with scores typically ranging between 70% and 90% (Horne & Adali, 2017; Chakraborty et al, 2016; Burscher et al., 2016). The high precision rates for both U.S. and Russian articles (76% and 73%) demonstrates
that the differences in feature distribution across articles from both countries are significant and in many cases apparent to classifiers. From a practical perspective, these findings demonstrate that further quantitative research regarding the distinctions between Russian state-funded and Western media has merit and is called for. For instance, a machine learning classifier such as the one developed in this study could easily be applied to the discourse of social media groups to identify concentrated networks which exchange and discuss Russian state-funded news, as well as automated accounts designed to spread Russian news – detection mechanisms which Western governments and researchers are increasingly in need off (Bradshaw & Howard, 2016; Wooley & Howard, 2016; Helmus, 2018; Hanlon et al., 2018). Helmus et al (2018) apply a similar analysis to identify pro-Russian communities in Eastern European social media networks based on the language used and linguistic properties of the discourse, achieving high rates of success in identifying high-risk groups. This study offers an additional approach to this type of research, by suggesting that group or suspicious user identification can be conducted on the basis of the content shared and potentially on the corresponding linguistic properties in the discourse about this content.
In line with past findings, this study demonstrates that simple and easily obtained linguistic properties of text, such as article length or average word length can be particularly informative for automatically distinguishing between U.S. and Russian news (Bakker & Arcenaux, 2014; Brundidge et al., 2014). This finding has significant implications for text classification tasks in widely spoken languages for which numerous automated approaches are available for detailed linguistic property extraction which could allow for extremely precise text classification in a wide range of fields, from political communication to private market interests
(Grimmer & Stewart, 2013). However, the study also demonstrates that text classification research is possible in languages with very limited standardized tools and only basic text-processing capabilities.
Limitations
It is also relevant to reflect on the limitations of this study. Firstly, this study does not identify Russian disinformation, nor does it inform us about how to counter it directly. This task would require an entirely different methodological and theoretical approach and would prove far more challenging than the current study. However, the study provides a profile of Russian state-funded news which does share disinformation and allows for a proxy approach to detecting suspicious communities or content. Moreover, the binary framework which guides the analysis prevents a nuanced representation of the media systems considered. U.S. media is diverse and complex and the outlets used in this study do not reflect U.S. journalism as a whole, nor can U.S. journalism be taken as entirely representative of Western journalism. Adding European outlets to the analysis would provide a more nuanced and convincing contrast of Russian news and
Western journalism and properly demonstrate whether RT and Sputnik have succeeded in their self-proclaimed mission of counterbalancing the “information monopoly of Western media” (EU vs. Disinfo, 2017). However, other significant foreign media is underrepresented in the Serbian context, while the Serbian branch of the BBC, an excellent choice for inclusion in the analysis, does not permit access to its archives.
Secondly, both inter-coder reliability scores and frame classifier scores were not optimal for all frames, placing the validity of the final frame distribution into question. Indeed, in the distribution of the five frames in the entire data set, ‘Yes’ responses for all frames were
proportionally underrepresented when compared with the manually coded data, while human interest inter-coder reliability was no better than chance (at 51%). Moreover, inter-coder reliability is assessed with a relatively small sample (N=31), due to budgetary limitations in the study and coder availability. Improved classifiers are called for in future studies, as the predicted frames in the current study likely under represent the presence of all frames considered and do not inform the country source classifier as well as they could about the relevance of these frames in Russian content. This is particularly the case for the thematic frames, most of which have recall rates no higher than 50%. Ultimately, this issue is directly linked to the small manually annotated data set used in this study (N=1108). Indeed, Burscher et al. (2014) achieve high frame prediction rates with supervised classifiers trained with a data set nearly ten-fold the one used in this study. Still, the findings of this analysis show that automatically detecting both structural and thematic frames is possible with a relatively high degree of precision even when the sample size is small, provided well delineated frames and properly trained coders. The holistic approach proposed by Burscher et al. (2014) was shown as effective both for the more equally distributed structural frames, but also for the thematic frames which are present in no more than 10% of articles (Burscher et al, 2014).
Finally, the linguistic features used in this study are relatively simple, despite their
effectiveness. With the exception of the named entities, all of these features can be obtained with simple text processing and they do not represent the full range of possibilities for automated linguistic feature extraction. Studies for which such tools are available, are encouraged to also consider the sentiment of article titles, the sentiment associated with named entities, more formal
language complexity scores such as the Flesch reading score, and various other NLP tools available to researchers.
References
Al Jazeera. (2019, Feb 3). 'One in five million': Protesting Serbia's muzzled media. Al Jazeera. Retrieved from: https://www.aljazeera.com/programmes/listeningpost/2019/02/million-protesting-serbia-muzzled-media-190202102248693.html
Al-Rfou, R., Perozzi, B., & Skienna, S. (2013). Polyglot: Distributed word representations for multilingual nlp. arXiv preprint arXiv:1307.1662.
Andresen, K., Hoxha, A., & Godole, J. (2017). New roles for media in the Western Balkans. Journalism Studies, 1-15.doi: 10.1080/1461670X.2016.1268928.
Badawy, A., Ferrara, E., & Lerman, K. (2018, August). Analyzing the digital traces of political manipulation: The 2016 russian interference twitter campaign. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) (pp. 258-265). IEEE.
Bandeira, L., Barojan, D, Braga, R. Penaredonnda, L.J., & Arguelo, P.F.M. (2019).
Disinformation in democracies: Strengthening Digital Resilience in Latin America. Atlantic Council Report. Retrieved from:
https://www.atlanticcouncil.org/publications/reports/disinformation-democracies-strengthening-digital-resilience-latin-america
Beam, M. A., Hutchens, M. J., & Hmielowski, J. D. (2018). Facebook news and (de)
polarization: reinforcing spirals in the 2016 US election. Information, Communication & Society, 21(7), 940-958. doi: 10.1080/1369118X.2018.1444783.
Bechev, D. (2018). Understanding Russia’s Influence in the Western Balkans. Strategic Analysis, 9-2018.
Bennett, W. L., & Livingston, S. (2018). The disinformation order: Disruptive communication and the decline of democratic institutions. European journal of communication, 33(2), 122-139. doi: 10.1177/0267323118760317.
Berinsky, A. J. (Ed.). (2015). New directions in public opinion. Routledge. doi:10.4324/9781315774367. doi: 10.4324/9781315774367.
Boumans, J. W., & Trilling, D. (2016). Taking stock of the toolkit: An overview of relevant automated content analysis approaches and techniques for digital journalism
scholars. Digital journalism, 4(1), 8-23.doi: 10.1080/21670811.2015.1096598.
Bradshaw, S., & Howard, P. N. (2018). Challenging truth and trust: A global inventory of organized social media manipulation. The Computational Propaganda Project.
Broniatowski, D. A., Jamison, A. M., Qi, S., AlKulaib, L., Chen, T., Benton, A., & Dredze, M. (2018). Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate. American journal of public health, 108(10), 1378-1384.
Brownlee, J. (2016, May 20). Feature selection for machine learning in Python. Machine Learning Mastery. Retrieved from: https://machinelearningmastery.com/feature-selection-machine-learning-python/
Brundidge, J., Reid, S.A., Choi, S., & Muddiman, A. (2014). The “deliberative digital divide:” Opinion leadership and integrative complexity in the US political blogosphere. Political Psychology, 35(6), 741-755. doi: 10.1111/pops.12201.doi.
Burscher, B., Odijk, D., Vliegenthart, R., De Rijke, M., & De Vreese, C. H. (2014). Teaching the computer to code frames in news: Comparing two supervised machine learning approaches to frame analysis. Communication Methods and Measures, 8(3), 190-206.doi: 10.1080/19312458.2014.937527.
Cellan-Jones, R. (2017, September 22). Fake news worries ‘are growing’ suggests BBC poll. BBC. Retrieved from: https://www.bbc.com/news/technology-41319683
Cerulus, L. (2019, Jan 17). Facebook takes down two Russian disinformation networks in Eastern Europe. Politico. Retrieved from: https://www.politico.eu/article/facebook-takes-down-two-russian-disinformation-networks-in-eastern-europe/
Chakraborty, A., Paranjape, B., Kakarla, S., & Ganguly, N. (2016, August). Stop clickbait: Detecting and preventing clickbaits in online news media. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) (pp. 9-16). IEEE.doi: 10.1109/ASONAM.2016.7752207
Champion, J. (2013). WbSrch Engine, extra stopwords. Github repository available at: https://github.com/Xangis/extra-stopword
Conroy, N.J., Rubin, V.L., & Chen, Y. (2015). Automatic deception detection: Methods for findings fake news. Proceedings of the Association for Information Science and Technology, 52(1), 1-4.doi: 10.1002/pra2.2015.145052010082.
De Graaf, R., & van der Vossen, R. (2013). Bits versus brains in content analysis. Comparing the advantages and disadvantages of manual and automated methods for content
analysis. Communications, 38(4), 433-443. doi: 10.1515/commun-2013-0025. De Swert, K. (2012). Calculating inter-coder reliability in media content analysis using
Krippendorff’s Alpha. Center for Politics and Communication, 1-15.
De Vreese, C. H. (2005). News framing: Theory and typology. Information design journal & document design, 13(1). doi: 10.1075/idjdd.13.1.06vre.
Eisentrat, S., & de Leon, S. (2018). Propaganda and Disinformation in the Western Balkans: How the EU Can Counter Russia’s Information War. Konrad Adenauer Stiftung Report. Retrieved from: https://www.Eisentraut & de
Leon.de/c/document_library/get_file?uuid=33dbbc29-eb30-e4ec-39c8-a4c976319449&groupId=252038
EU vs. Disinfo (2018, Aug 3). Disinformation Analysis on the Western Balkans: Lack of Sources Indicate Potential Disinformation. EU vs. Disinfo. Retrieved from:
https://euvsdisinfo.eu/disinformation-analysis-on-the-western-balkans-lack-of-sources-indicates-potential-disinformation/
Fletcher, R., Cornia, A., Graves, L., & Nielsen, R. K. (2018). Measuring the reach of “fake news” and online disinformation in Europe. Factsheets Reuters Institute. Retrieved from: https://reutersinstitute.politics.ox.ac.uk/our-research/measuring-reach-fake-news-and-online-disinformation-europe
Forman, G. (2003). An extensive empirical study of feature selection metrics for text classification. Journal of machine learning research, 3, 1289-1305.
Galante, L., & EE. S. (2018) Defining Russian election interference: An analysis of select 2014 to 2018 cyber enabled incidents. Atlantic Council Report. Retrieved from:
https://www.atlanticcouncil.org/images/publications/Defining_Russian_Election_Interfer ence_web.pdf
Galeotti, M. (2018). Do the Western Balkans face a coming Russian storm?. European Council on Foreign Relations Report. Retrieved from:
https://www.ecfr.eu/publications/summary/do_the_western_balkans_face_a_coming_russ ian_storm
Garimella, V. R. K., & Weber, I. (2017, May). A long-term analysis of polarization on Twitter. In Eleventh International AAAI Conference on Web and Social Media.
Godbole, S., & Sarawagi, S. (2004, May). Discriminative methods for multi-labeled
classification. In Pacific-Asia conference on knowledge discovery and data mining, 22-30. Springer, Berlin, Heidelberg.doi: 10.1007/978-3-540-24775-3_5.
Grimmer, J., & Stewart, B. M. (2013). Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political analysis, 21(3), 267-297.doi: 10.1093/pan/mps028.
Hallin, D. C., & Mancini, P. (2004). Comparing media systems: Three models of media and politics. Cambridge university press.doi: 10.1017/CBO9780511790867.
Hanlon, B. (2018). A Long Way To Go: Analyzing Facebook, Twitter, and Google’s Efforts to Combat Foreign Interference. Alliance for Securing Democracy Executive Summary. Retrieved from: https://securingdemocracy.gmfus.org/a-long-way-to-go-analyzing-facebook-twitter-and-googles-efforts-to-combat-foreign-interference/
Hanna, A., Wells, C., Maurer, P., Friedland, L., Shah, D., & Matthes, J. (2013, October). Partisan alignments and political polarization online: A computational approach to understanding the French and US presidential elections. In Proceedings of the 2nd workshop on Politics, elections and data (pp. 15-22). ACM.doi: 10.1145/2508436.2508438.
