How are scientific papers on
COVID-19 discussed on
Twitter and in newspapers?
Layout: typeset by the author using LATEX.
How are scientific papers on
COVID-19 discussed on Twitter
and in newspapers?
Looking for fake news concerning the Corona virus
Sarah Bosscha 11291486
Bachelor thesis Credits: 18 EC
Bachelor Kunstmatige Intelligentie
University of Amsterdam Faculty of Science Science Park 904 1098 XH Amsterdam Supervisor dhr. G. Colavizza Faculty of Humanities University of Amsterdam Turfdraagsterpad 9 June 26th, 2020
0.1
Abstract
Since december 2019, the COVID-19 pandemic captivated the global scientific world. Twitter and newspapers are primarily responsible for spreading scientific findings on the pandemic and the virus. With the high medical threat of the virus it is important that the articles are discussed correctly. Thus, it is essential to do research on the accuracy of media concerning the COVID-19 scientific findings and the sentiment of the message.
Until now, not much research has been done on the spreading of medical news. This thesis explores sentiment analysis and the linguistic features that are able to distinguish between fake and real news. It compares the main topics of scientific articles with those of news articles and tweets.
It appears that the news articles and the tweets are mostly facts and beliefs, so the sentiment is called ‘indicative’. In the news articles there are some conjectures, which make this sentiment called ‘conditional’. In the end, the used methods in this thesis are not sufficient enough to achieve the desired result. It is important to have a human view on the reliability of fact checking the scientific article.
0.2
Preface
This thesis was written under the supervision of G. Colavizza. The process of collecting data was done in collaboration with fellow student Stefan Houben (11000341), parts of the code (mainly in the process of collecting data) have also been written in collaboration with Stefan Houben. G. Colavizza has been fully aware and supportive of this throughout the project.
Contents
0.1 Abstract . . . 4 0.2 Preface . . . 4 1Introduction
1 2Theoretical framework
4 2.1 Infodemic . . . 4 2.2 Fake news . . . 42.2.1 Fake news in Tweets . . . 5
2.3 Newsworthiness . . . 5
2.4 Sentiment analysis . . . 6
3
Data collecting
7 3.1 Dimensions and Altmetrics . . . 73.2 Tweets collecting . . . 8
3.3 News articles collecting . . . 8
4
Method
10 4.1 TextBlob . . . 104.2 Main topics analysis . . . 10
4.2.1 Comparing Concepts with highest word frequencies . . . 11
4.2.2 Comparing Concepts with tweet texts . . . 11
4.2.3 NLP-techniques: POS-tagging . . . 12
4.2.4 Comparing Concepts with POS tagged nouns . . . 12
4.3 NLP: Sentiment analysis . . . 13
4.3.1 Patter.en library . . . 13
4.3.2 Sentiment from Patter.en library . . . 13
4.3.3 Modality from Patter.en library, Fact checking . . . 15
4.3.4 Moods from Patter.en library . . . 16
5 Results 18 5.1 Full results of DOI : 10.1002/jmv.25805 Newspaper: "The
Conver-sation" . . . 18
5.1.1 Concepts . . . 19
5.1.2 Modality . . . 20
5.1.3 Moods . . . 21
5.1.4 Sentiment and Subjectivity . . . 22
5.2 Notable results . . . 22
5.3 comparing different Newspapers . . . 24
5.4 Evaluation . . . 25
6 Discussion 27 7 Conclusion 29 7.1 Future work . . . 30
Chapter 1
Introduction
"And then I see the disinfectant where it knocks it out in a minute. One minute. And is there a way we can do something like that, by injection inside or almost a cleaning?" These were the words of the president of the United States, Donald Trump, during the White House Coronavirus Task Force briefing in April 2020 [16]. The Coronavirus, COVID-19, is gripping the Netherlands and the entire world. After the outbreak of the virus in Wuhan at the end of 2019, the virus has spread widely and infections have been counted all over the world. Because of this for everyone totally new threat, it is important that everyone receives the correct information and stays safe.
The global scientific world is heavily attracted by the COVID-19 pandemic. Med-ical researches have their full priority on the virus and on the management of the crisis, at this moment often from an epidemiological view. A large part of the med-ical community is looking for a vaccin against the Coronavirus. Besides a lot of research about the vaccin, thousands of other articles are published to help to stop the pandemic. Many research communities, funding agencies and third-parties are taking action to support the fight against the pandemic. All these scientific articles with weekly new information around the pandemic are quickly spread through so-cial media and news articles. There is a lot of soso-cial media attention for COVID-19 reserach [6] and the opportunity for miscommunication of the real threat of the article is present.
It is well known that spreading of misinformation regarding the pandemic can be very dangerous. Health crises can be fueled by information crises. [5] The World Health Organisation has recently launched a platform that aims to combat misinformation concerning COVID-19, called "Myth busters". Sharma et al. [24] concludes: “The misinformation on the COVID-19 pandemic is especially
ing, since any missteps can pose a serious public health risk by leading to exponen-tial spread of the disease and accidental death due to self-medication." [24]. The current COVID-19 outbreak dominates attention from social media, in particular from Twitter, highlighting the public interest for scientific results at the time of the pandemic. [6] Social media platforms provide direct access to an unprecedented amount of content and may amplify rumors and questionable information. [5]
The misinformation concerning COVID-19 comes in many forms, Swiney [27] gives an example: "from conspiracy theories about the virus being created as a biological weapon in China to claims that coconut oil kills the virus".
Also the misinformation in news articles can lead to fake news. This is a phe-nomenon where the content of a news-article does not correspond to the actual truth being a mix of false and true statements or lies. Fake news in media can cause problems. These information bubbles or news manipulation and the lack of trust in the media are growing problems with hugh ramifications in society. There is trust in the scientific community, but when science is translated into the media, people are suspicious.
The research question for this thesis is: ‘How are scientific papers on COVID-19 discussed on Twitter and in newspapers?
In order to give a proper answer to the question, it is essential to address the following subquestions that will help to shape up the result.
1) In what way are the articles discussed?
2) How reliable are the social media and newspaper discussions with respect to the original paper?
3) How can fake news be detected?
This project focuses on textual analysis of social media concerning the COVID-19 crisis and how scientific papers on COVID-19 are discussed in online newspapers and on Twitter. It observes how reliable these discussions are with respect to the original paper, and what part social media is focused on. The dataset will consist of tweets and news articles on scientific papers from Altmetrics, Dimensions and Twitter. The study seeks to establish whether or not there is a correlation. Various data analysing techniques will be combined to obtain results. Presenta-tion language modelling and sentiment analysis will be used in the project. The goal is that the Python program and the literature study on the linguistic word-ing of fake news and infodemics provide a clear overview of sentiment concernword-ing the COVID-19 articles in the news and on twitter and fake news concerning the COVID-19 crisis.
This thesis shall first explain about the data collecting process and with it the used application programming interfaces (API). Then it will discuss the related
work that will assist in enhancing the methodology. After that the methodology will be elaborated and the results of the thesis will be shown. The aim is to find methods to determine the sentiment of the news articles and the tweets concern-ing the COVID-19 publications and to search for a method to see how reliable the tweets and news articles are comapring to the original mentioned publication.
Chapter 2
Theoretical framework
2.1
Infodemic
“We’re not just fighting an epidemic; we’re fighting an infodemic”, said World Health Organization (WHO) Director–General Tedros Adhanom Ghebreyesus at the Munich Security Conference on 15 February 2020 [12]. The term infodemic has been coined to outline the insecurities of misinformation phenomena during the management of virus outbreaks. Furthermore, Cinelli, in this [5], clearly indicates what can cause a rumor or fake news and is informative about an “infodemic”. [30] It can be seen that spreading information strongly influences people’s behavior. For example the rumor spread by CNN that Lombardy (a northern region in Italy) would find itself in possible lockdown, caused an overcrowd of people trying to leave the region in trains and going to supermarkets to buy nutrition. The Corona-virus is mostly spread by people being too close to each other and by overcrowding, thereby showing the critical impact of this new information envi-ronment [5]. Zarocostas [30] also points out that fake news may spread faster and more widely than fact-based news.
The growing COVID-19 pandemic is generating a high proportion of publications on the themes of the coronaviruses and the management of epidemics, including an instantaneous spread of the publications online and in the news. So on, it is important to always be aware of fake news detection; and have a look at the change a news article is intentionally deceptive. [20] [6]
2.2
Fake news
‘Fake news’ is defined as: fabricated information that mimics news media content in form but not in organizational process of intent. The rise of fake news highlights
the erosions of long-standing institutional bulwarks against misinformation in the internet age. [14] Since the rise of fake news during the US Presidential Election in 2016 [17], fake news has become a controversial topic and the consequences of it are feared by many. [15] During that specific election were several voters influenced by fake news. It is important to find a way to control the spread of such misinformation, in order to prevent people from believing falsity and influencing many important factors such as their attitude towards certain Concepts. Fake news is intentionally written to mislead readers to believe false information, which makes it nontrivial and problematic to detect based on news content. [25]
2.2.1
Fake news in Tweets
Tweets can comprise a lot of fake news. Twitter has a huge role in today’s society, with the ability to spread quick short messages around the world. The social media platform is one of the biggest sources for information for citizens. In the occasion of a tragedy, Twitter is the main platform to spread information. The social media platform has also been used to track epidemics. [13]
In research of Gupta and Kumaraguru [10] linguistic features such as swear words and pronouns were good predictors for the credibility of tweets. [10] Research of Tan et al (2014) showed a different approach on this problem, he explored the wording and propagation of tweets. It turned out that the linguistic properties of a tweet influenced the degree of propagation of a this message. [28] A highly conscious subject comparable to the COVID-19 virus is possible to get a lot of propagation.
2.3
Newsworthiness
Newsworthiness is defined as the prediction of news values. [9] People read the news to get aware and learn more about any topic. Besides news has also an important role in the translation from the academic community to the rest of the world. It is a job for journalists to correctly explain the newest inventions of researchers. In times of the current pandemic every step of research in COVID-19 is blown up to big news, especially in the search for the vaccine. Because there is so much research, journalist must be very quick in their reporting. Partly because of possibility they translate the research incorrectly, fact checking is important, particularly in the scientific domain. This allows the source to be traced and the veracity of scientific claims to be measured. However, a journalist must also have trust in science and not make unnecessary accusatory claims, which can lead to fake news [29]
percentage has kept growing. People trust the internet as a news source as much as other media, with the the most trust in (online) newspapers. [8] Additionally, Schmierbach and Oeldorf-Hirsch (2010) conducted an experiment in which the headline of a news item was presented to users in different ways. Users found the same news headline significantly less credible when presented on Twitter. [23] Furthermore, this causes a lot of confidence in news sites, people have doubts about Twitter more quickly, but sometimes they blindly trust the news. Potthast et al. [18] did research to classifying fake news and concluded it was a complicated task. They concluded that the accuracy was less than chance. However, they did find that making a distinction between hyper partisan news and mainstream news is possible and can be used as fake news classifier. [18]
2.4
Sentiment analysis
Sentiment analysis is, in essence, the process of determining the emotion of the writer, whether it is positive or negative or neutral. Sentiment analysis are all credibility related, they are good to compare with semantic features. [11] In order to understand the emotion of a text, sentiment analysis is used. This is a widely used Natural Language Processing(NLP-) technique, where given some text, the sentiment of it is extracted. Most methods do this by giving scores to words that are highly indicative, for example "horrible" or "amazing". With the total score of a whole sentence, the sentiment of the text is determined.
Chapter 3
Data collecting
The data sources are three large datasets full of articles concerning the COVID-19 pandemic: CORD-19 Dataset, Dimension API Dataset and WHO-Dataset. The first datasource is the COVID-19 Open Research Dataset, 19 [1]. CORD-19 is a fast growing dataset, updated every week with COVID-CORD-19 publications, capturing recent as well as older research on COVID-19 and the coronavirus fam-ily of viruses published by the global research community. [6]. Recent research from Colavizza et al. [6] stated that a large part of the dataset is filled with ar-ticles around viruses in general. Publications from CORD-19 mostly focuses on a few, well-defined areas including coronaviruses primarily SARS, MERS, COVID-19. This research focuseses mainly on articles discussed on Twitter and in news sites concerning the COVID-19 pandemic, whereby all articles written before 2020 are dropped out of the dataset. The second datasource is a large amount of data received from the WHO concerning COVID-19; the last dataset is from the Di-mensions API company.
3.1
Dimensions and Altmetrics
Dimensions is an innovative linked research data infrastructure tool, re-imaging discovery and access to research: citations, clinical trials, grants, publications and patents are all in one place.[3] Dimensions data also comprises the Altmetrics data. Altmetrics give scores to the public and policy engagement attention, including social media, traditional media and policy attentions. In the Dimensions database the previously mentioned data are brought together with the academic, innovation and clinical attentions.[3] Therefore, Altmetrics and Dimensions are very suitable as a database for this research.
By using a private Altmetrics ID an database full of articles is received from 7
Altmetrics. The CORD-19 publications have also been explored using Altmetrics data, with the aim of describing their reception on among other things social me-dia and news. The Altmetrics company gives to full access users the opportunity to “fetch”. With this possibility to fetch for every article, which is most easy to find with by Digital Object Identifier (DOI), a large amount of information will be received. (for example:
https : //api.Altmetric.com/v1/f etch/doi/10.1001/jama.2020.4031?key = xxx) With the written code the topics for the research could be selected. The selected parts are in the ‘Counts’ category counts for news and twitter; in the ‘Posts’ category the used information is the News and the Twitter ID and Tweeter ID.
3.2
Tweets collecting
The final Altmetric collected database comprises thousands of unique articles. Each Altmetric API contains information about the amount of tweets mentioning the article. The database will be filtered by selecting only articles written in 2020, and by sorting articles with the highest amount of tweets, and after that selecting the highest ten percent. The article with the fewest tweets has 186 tweets and the article with the most tweets has 85911 tweets.
After selecting the ultimate database, the Altmetrics “fetch” gives a Tweeter ID and a Tweet ID. This combination leads to the webpage of the tweet mentioning the article. With the modules Request, JSON and BeautifulSoup the HTML page is loaded and the tweets can be collected in full text.
3.3
News articles collecting
The Altmetric Database gives a selection of articles written in 2020 and has a positive number of ‘News counts’. ‘News counts’ in Altmetrics provides the num-ber of mentions and unique authors in each relevant source type. After select-ing the database a rankselect-ing of the most popular used news websites who spoke about COVID-19 articles, is performed. To get as much news articles as relevant, the journals that cite articles from the database the most are selected. In total 2128 uniuqe journals als citating or mentioning one of the publications from our database, the ten websites with the most citations were: Yahoo, MedicalXpress, MSN, Infosurfhoy, the Conversation, Foreign Affairs New Zeeland, Medscape, Eu-rekalert and the Dailyhunt.
A unique code is necessary to scrape the text from the different news webpages, because every web page has a different HTML layout.
The news articles concerning the papers are collected by the fetching link including the DOI of the article. A search is performed in the fetch page for news articles from the ten most popular news sites (Yahoo news, Medical Xpress, MSN, Info Urhoy, The Conversation, Foreign Affairs New Zealand, Medscape, The Medical News, Eurekalert and Dailyhunt;) the received URL is going through the text re-ceiving program of the news website. On the fetch webpage an article is displayed like this:
Chapter 4
Method
4.1
TextBlob
Almost all text mining programs work with words and their context, Tokenization divides the large texts (of the news papers and the tweets) in to separate words. In context of the future analysis it is necessary to use Tokenization. The newspaper is split by tokenization done by TextBlob. TextBlob is a text processing Python Library and is built upon NLTK providing a well-designed interface to the NLTK library. Tokenization refers to a word in a text document; it can split the text in sentences and in words. Tokenization is necessary to do analysis on the text.
4.2
Main topics analysis
One of the most frequently used methods to verify the accuracy of a news article in comparison to a scientific article is to have a look at the Dimensions given ‘Concepts’. These are words describing the main topics of a publication; they are automatically derived, by Dimensions, from the publication text using machine learning. These words give a good indication of the core truth of the article. The Concepts can be compared with information received from the news article. The Concepts are a good measurement to check the correctness of the information and in what way it is comparable to the original article or paper. If the news article or tweet contains a lot of similarity with the Concepts, it can be concluded that the truth core of the article is properly described in the news article or tweet. The process of of the total method of comparing, and later on sentiment analysis, is all at a time done per DOI. There is one DOI selected which is cited in a specific journal, for example Yahoo or The Conversation, after that the tweets are loaded concerning this DOI and the full text is loaded of the news article mentioning this
DOI.
4.2.1
Comparing Concepts with highest word frequencies
To see if the Concepts have a correct notice in the text of news or tweet, different comparison methods will be used. First, the Concepts are compared to the words with the highest probability in the text.
This probability is calculated by summing up the counts of appearance of a word divided by the amount of words in the text of the news article. In text mining it is common to look at word frequencies [26] and to compare frequencies across different texts. Comparing the 10 percent words with the highest probability in a news article gives a good indication if the text is similar to the article or not. Concepts are such important that, in a good representation of the scientific article, the occurrence of the Concepts words should be higher than the other words.
Image 2: Concepts appearance in most frequent words in Yahoo news article men-tioning article DOI: 10.1186/s13578-020-00404-4
2.2 % of the most frequent words in the news article is a Concept.
4.2.2
Comparing Concepts with tweet texts
The Concepts will be compared to the entire Tweet text. All the tweets collected by the Data collecting program matching the same DOI will be compared to the given Concepts. There will be a good indication which Concepts are the most attractive to writers to put upon, their 140 character limited, tweet.
Image 3: Concepts appearance in tweets mentioning article DOI: 10.1186/s13578-020-00404-4
2.7 % of the words in all the tweets is a Concept.
4.2.3
NLP-techniques: POS-tagging
Words can vary in meaning, depending on their context or usage. A word can be categorized by Part-of-speech (POS) tagging. This Natural Language Processing task will return the POS-tag of each word. In this thesis the by POS-tag selected nouns will be examined. The noun selection will be done to get more focus on the for analysis necessary parts, the Concepts are all almost nouns.
4.2.4
Comparing Concepts with POS tagged nouns
The Concepts can also be compared to all the nouns in the news article selected with Part-of-speech (POS) tagging. POS-tagging, also called word-category dis-ambiguation, is the process of marking up a word in a text by corresponding to a particular part of speech. All the Concepts are nouns, so when all the nouns are extracted from the news article a good comparison can be made. POS-tagging is a supervised learning solution that uses features like the next word, the previous word, if a first letter is capitalized etcetera. POS-tagging is based on the definition of a word and its context. For POS-tagging Natural Language Toolkit (NLTK) is used; it has a function to get POS-tags and it works after a tokenization process. Comparing all the Concepts gives a clear indication on whether certain topics are discussed well from the truth core of the article.
Image 4: Concepts appearance in POS-tagged nouns in Yahoo news article men-tioning article DOI: 10.1186/s13578-020-00404-4
9.1 % of the POS-tagged nouns in the news article is a Concept.
4.3
NLP: Sentiment analysis
Sentiment analysis can give answer to the question in what way articles are dis-cussed. Sentiment analysis is a wide range of possible analysis. For example Moods, Modality, Subjectivity and Sentiment can be calculaded by scores or in classes.
4.3.1
Patter.en library
Patter.en is a web mining Python module and is useful in Natural Language Pro-cessing. The Pattern library offers functionality to find sentiment from a text string. It contains functions as "Moods" and "Modality" who can help with defin-ing the sentiment of the sentences of the tweets and news articles. The Pattern package is a complete package for NLP, text analytics and information retrieval. The package is developed by CliPS (Computational Linguistics Psycho linguis-tics), a research center associated with the Linguistics Department of the Faculty of Arts of the University of Antwerp [21].
4.3.2
Sentiment from Patter.en library
Sentiment refers to an opinion or feeling towards a certain concept. Sentiment analysis and opinion mining are important tasks in text processing. When hu-man readers evaluate a text, we use our understanding of the emotional meaning of words to infer whether a section of text is positive or negative, or, perhaps, characterized by some other more nuanced emotions like surprise or disgust. [26] We can use the tools of text mining to approach the emotional content of text
programmatically. [26]
The Sentiment contains scores like polarity, subjectivity, intensity, and confidence, along with other tags as the part of speech tagging and WordNet identifier. The sentiment object is used to find polarity (negativity or passivity) of a text along with its subjectivity. The most indicative positives are good, best, excellent et cetera. The most known negatives are bad, awful, pathetic et cetera. The score of sentiment is between 1 and -1 and is assigned to the text.
In addition to the sentiment score, subjectivity is also assessed. The subjectivity value can be between 0 and 1. Subjectivity quantifies the amount of
personal opinion and factual information contained in the text. The higher the subjectivity score, means that the text contained personal opinion rather than factual information. [2] There is done some K-means clustering over the results
of the Sentiment and Subjectivity analysis to see if there are some clear structural cluster. K-means clustering is an unsupervised machine learning
algorithm and can help labeling results.
Sentiment and subjectivity from a Yahoo news article and tweets concerning DOI 10.1186/s13578-020-00404-4 plotted toward each other and K-means clustering
applied on results gives:
Image 5 and 6: Sentiment and Subjectivity of the news articles and tweets concern-ing DOI: 10.1186/s13578-020-00404-4 and K-means clusterconcern-ing applied on results
There is a detectable difference between the tweets and the newspaper concern-ing the same scientific article. Tweets have a much higher Subjectivity and much wider spread Sentiment. This is explained by the face that tweets are individual texts without citation and expected correctness.
4.3.3
Modality from Patter.en library, Fact checking
The sentiment score is also called the polarity score. It is important to fact check this sentiment, in which the modality function can help. This function can be used to assess the degree of certainty in a text string. It gives a value between -1 and 1. If the text is a fact, it will give a value higher than 0.5 [2]. It is important to have a modality score close to 1 to ensure the correctness of the news articles.
Image 7: modality of sentences in Yahoo news article mentioning article DOI: 10.1186/s13578-020-00404-4
Sentences in this news article with negative modality: score: -0.027
Carrying out these tests would enable scientists to predict the true number of pa-tients, as well as how many did not develop the tell-tale fever and cough.
score: -0.5
How contagious the coronavirus is has been up for debate. score: -0.25
Creating a vaccine could lead to herd immunity, preventing the coronavirus taking hold if it re-emerges.
Tweets in this news article with negative modality: score: -0.05
How the CDC Could Prolong the Lockdown for 3 More Years What’s funny is the CDC isn’t a government organization. score: -0.25
I would say at this point it’s closer to the flu than then. score: -0.5
I was curious if you would catch that.
4.3.4
Moods from Patter.en library
The Mood function from Pattern helps in determining the mood expressed by particular part of a text document. [21] The mood function can give back four optional moods: Indicative, Imperative, Conditional or Subjunctive for any text based on its content. Indicative sentences are facts of beliefs, Imperative sentences are commands or warnings, Conditional sentences are conjectures and Subjective sentences are wishes or opinions. For the research on checking if news articles and tweets are comparable to scientific articles and do not spread fake news, it is important to have a majority of Indicative sentences. [21]
Table 1: Examples of the different types of moods [21]:
Mood Example Use
Indicative It is contagious. Belief, Fact Imperative Don’t be contagious! Command, Warning Conditional It might be contagious. Conjecture Subjunctive I hope it is contagious. Opinion, Wish
10.1186/s13578-020-00404-4
image 10: moods of the of tweets mentioning article DOI: 10.1186/s13578-020-00404-4
Clearly there in the tweets and in the news article both a large amount of Indica-tive sentences. Furthermore, it is striking that there are zero subjecIndica-tive sentences in the tweets.
Chapter 5
Results
The results are made per DOI. Every DOI in the database will have results made from, Altmetrics given, tweets and news articles. Some results are already visible in the previous part of the report, but here will be shown some more and notabele results.
5.1
Full results of DOI : 10.1002/jmv.25805
News-paper: "The Conversation"
Subsequently here is a result of DOI: 10.1002/jmv.25805 with mentions in a news article form ’The Conversation’. This Result shows that in the news article, as well in the POS-tagged nouns as in the most frequent words, there are is a big variation of Concepts used in the text. With the tweets there are so many different Concepts used and some clearly more then the others. The moods are almost similar to each other, comparing the Tweets and the news article. In the Sentiment and Subjectivity graph there is a same kind of shape visible, such as a upside down letter ’V’. This result is an example of a regular result with no massive anomalies.
5.1.1
Concepts
Image 11: Concepts appearance in POS-tagged nouns and in POS-tagged nouns in The Conversation news article mentioning article DOI: 10.1002/jmv.25805
Image 12: Concepts appearance in most frequent words and in POS-tagged nouns in The Conversation news article mentioning article DOI: 10.1002/jmv.25805
5.1.2
Modality
Image 14: modality of The Conversation news article mentioning article DOI: 10.1002/jmv.25805
Sentences with negative modality: score: -0.25
Almost 1,5 month since Indonesian President Joko Widodo’s call for the public to adhere to social distancing, avoid crowds and self-isolate at home, there remains no sign of COVID-19 transmission reducing.
Image 15: modality of tweets mentioning article DOI: 10.1002/jmv.25805 Tweets with a negative modality score:
score: -0.375
I don’t know how to teach the @WHO. score: -0.05
Kinda like HCTZ, only with evidence that it might help! score: -0.25
This might help explain why Trump doesn’t like to wear a mask in public. score: -0.125
Which also draws your hand to your face to adjust the mask the germs you keep with you not others then get on your hands and other surfaces.
score: -0.16666666666666666
As a good assumption is that most of us have been infected and not necessarily AFFECTED.
score: -0.16666666666666666
Wonder what all the doctors and nurses all over the world would think of your statement.
score: -0.08333333333333333
After finishing making another TikTok video, because they’re so “busy”, most of them would agree with me.
score: -0.25
Depends on what study you want to align with.
It is interesting to see the difference between the tweets and the news article sen-tences with a negative modality score. What clearly similar are doubtful language, words as: "Almost", "I don’t know", "might", "most".
5.1.3
Moods
Image 17: moods of The Conversation news article mentioning article DOI: 10.1002/jmv.25805
5.1.4
Sentiment and Subjectivity
Image 18 and 19: Sentiment en Subjectivity of The Conversation news article and tweets mentioning article DOI: 10.1002/jmv.25805 and K-means clustering applied on results
5.2
Notable results
Some sentiment analysis are difficult to read because of the difference between the sizes of the datasets. The graphs visible here, sentiment en sensibility from DOI 10.1101/2020.04.23.20076042, have a tweet database from 3355 tweets and the news articles just 51 sentences. Again there is an upside down V-shape visible.
Image 20 and 21: Sentiment en Subjectivity of The Conversation news article and tweets mentioning article DOI: 10.1101/2020.04.23.20076042 and K-means clus-tering applied on results
This same article had also a big amount of Concepts selected by Dimensions (132 Concepts), which causes a confusing histogram as well as a big percentage of com-paring with the POS nouns in the news article. It is debatable whether this is a good or a bad thing.
In this example below it is clearly visible that just one Concept is used (Coro-navirus), it is obvious to conclude that this newspaper is not reliable to the sci-entific article. The other not used Concepts in this case are: ’pathogens’, ’biol-ogy’,’infection’, ’test’, ’drugs’, vaccine’ and ’samples.
Image 22: Concepts appearance in most frequent words in The Conversation news article mentioning article DOI: 10.1038/d41586-020-00262-7
Image 23: Concepts appearance in POS-tagged nouns in The Conversation news article mentioning article DOI: 10.1038/d41586-020-00262-7
0.21 % of the most frequent words in the news article is a Concept, only the Concept ’coronavirus’ appears in the text.
2.68 % of the POS-tagged nouns in the news article is a Concept, only the Concept ’coronavirus’ appears in the text.
5.3
comparing different Newspapers
Below there are the percentages of Yahoo and The Conversation.
’POS’ = the percentage of Concepts that are the same as the POS-nouns in the news article
’freq’ = the percentage of Concepts that are the same as the most frequent words in the news article
Image 24 and 25: The percentages of comparing the Concepts with POS-nouns and most frequent words of Yahoo (left) and The Conversation(right) news
ar-ticles. The numbers in the first column doesn’t mean that it is about the same article (DOI)
With an average for Yahoo of POS = 12.38 percent and freq = 3.22 percent and for The Conversation POS average = 12.50 and feq = 2.80 percent, there is not a significant difference between the two newspapers. They both treat the scientific articles as the same reliability. Clearly is that the POS-nouns have more the same nouns as the Concepts than the most frequent words, which is explicable because with the POS-nouns there has been no selection and are also more words than the highest ten percent frequent words.
5.4
Evaluation
In text mining it is common to look at word frequencies [26] and to compare these frequencies across different texts. In this project the, by Dimensions given, Con-cepts are compared with POS-tagged nouns, the most frequent words and the full tweet text.
Because for every analysis on reliability and sentiment the dataset is filtered for only publications written in 2020 and text related to a specific journal, the compar-ing process has been applied on just a small dataset of articles and tweets. There are some similarities visible between the Concepts and the POS-nouns, most fre-quent terms or the tweet text. The POS-nouns have on average more in common with the Concepts, i.e. around 12%, then the most frequent words, i.e. around 3%. The tweets show an overlap of 3% with the Concepts. Furthermore, it varies a lot how many of the Concepts appear in the text. Some tweets and news articles only compare with one word of the Concepts.
When a news article shows a similarity with the Concepts of 3% or less, there has been looked manually at the news article and the publication. The news articles never discussed the article incorrectly or spread fake news.
The sentiment analysis, from the Pattern package for Python, do give a good indication of the moods concerning the COVID-19 articles. The Indicative mood rises above the other three moods, conditional, subjective and imperative. The detected difference between the news article and the tweets sentiment is the fact that the imperative mood is sometimes present in the tweets, but not that much in news articles. (as visible in Image 9, 10, 16 and 17).
The Sentiment en Subjectivity scores doesn’t have structural specific values, i.e. they do seem to occur in a similar pattern. This Pattern has the most tweets in the centre with a 0.0 score for Sentiment and a 0.5 score for Subjectivity. With
a Subjectivity score around 1.0 the Sentiment score is widely spread between -1.0 and 1.0 (Image 20 and 21).
The modality score does select, with a negativity score, some insecure sentences. Furthermore it is clear that the size of the tweet database and the news article database are very spread in size, which makes it challenging to compare these two. (Image 22 and 23).
At last, there is no significant difference between Yahoo and The Conversation comparing Concepts with POS-nouns and most frequent words.
Chapter 6
Discussion
The major finding of this thesis is that it is still hard to detect fake news using the techniques described. In contrast, it seems possible to score the sentiment of the text reflecting the scientific article.
Ultimatly, the database was reduced in its size since there was a selection of ar-ticles written only in 2020 and of arar-ticles only mentioned in one of the top ten articles.
Furthermore, the data were scraped from webpages, but these webpages change over time. The current code is very likely to be outdated and not completely workable with the recent HTML pages. The differing and changing HTML pages are a big obstacle for working effectively. Within one news company, for example "Yahoo!", there are several sub sites with different HTML layouts.
Some URL-newslinks are not in English, these parts of the datasets are not useful because it is too complicated to add translation to the text and analyse the relia-bility and sentiment after that.
However, even if the data is in English, tweets can still contain inappropriate or incorrect words. The informal language can cause problems analysing it. When looking at sentiment, Emoticons, pictorial representations of human facial expres-sions [7], have a large influence on the mood of the message but aren’t used in the analysis.[22]
The methods used in this thesis are also not without their faults.
In the Sentiment analysis, only the Patter.en Python package is used. It is known that there are more interesting packages to give a sentiment score to a text, such as: VADER, SentiWordNet and SVM [21].
Furthermore, the method used to check the accuracy of the news articles and
the tweets comprised purely the by Dimensions provided Concepts. This requires great confidence in Dimensions.
There is also the possibility that there are no Concepts from Dimensions attributed to the article, so there is nothing to compare.
Chapter 7
Conclusion
Social media has become a major place for campaigns of misinformation affecting the credibility of the entire news ecosystem.[19]
This thesis looked at the different types of Sentiment in a news article or a tweet. This has been analysed by several methods from the Patter.en package: Sentiment, Subjectivity, Moods and Modality are used functions. Next to that there has been an analysis on the comparing of Concepts, the main topics of the scientific article, with the text of the tweet or the news article.
The articles concerning COVID-19 as in the CORD-19 dataset were not all about the COVID-19 virus, but as well on other Coronaviruses like SARS. Therefore, this project only looks at COVID-19 related articles. It is understandable that the possible deadly consequences of COVID-19 create an emotional atmosphere around the scientific articles; news articles, tweets are therefore sensitive.
It appears that the news articles and the tweets are mostly facts and beliefs, so the sentiment is labeled as ‘indicative’. In both the news articles and the tweets there are some conjectures, which make this sentiment called ‘conditional’. The indicative moods suggest that the tweets and news articles are mostly facts, so not likely to be fake news.
The modality, i.e. the certainly, gives only a few negative scores for sentences that should be not true. The Sentiment and Subjectivity score does not give a significant result.
Checking the reliability of the news articles and tweets, compare to the scien-tific articles, it is hard to conclude something significant. The Newsworthiness cannot be properly identified. The percentages of comparing the Concepts with POS-nouns, tweets-texts and highest frequency words are not significant enough to conclude that. In this project no significant results are found with the method,
when the Concept comparing method returned a low percentage is, it suggested that the, for example, news article fake news and discussed the scientific article incorrect. When these news articles and scientific articles were manually checked, there was never concluded that the news article discussed the article wrong or spreaded fake news. So on, the Concepts method does not detect incorrectness. After all, there is always a human view necessary to fact check the news articles or the twitter message, but these comparing methods and especially the sentiment analysis can help to do a pre-selection.
7.1
Future work
For future work it is interesting to make the program work automatically with new publications, news articles and tweets; by sending an alert when there is a contra-dictory score or improper use of Concepts. With human help a machine learning program can be build. If the program guesses that a text has a low modality score, an overall Subjunctive mood, a large Subjectivity score and is not reliable (after the Concepts camparing), it will be fact checked by a human.
Further more, another promising direction could be to compare the different sen-timent analysis on this specific COVID-19 dataset. Sarkar (2016) [21] already compared Pattern, VADER, SentiWordNet and SVM. Applying this comparison to specific COVID-19 articles is interesting to analyse.
Further studies would be needed to conclude anything about the integration be-tween the two main methods in this thesis. For example, a subjunctive text could have less change to be perfectly reliable to the original paper. Combining the Concepts comparing with the sentiment analysis would be interesting; mostly the combination of Concepts comparing with the Modality score.
A good dataset is important for these future projects. After all, the program can learn and in the end classify possible fake news by itself.
Bibliography
[1] Covid-19 open research dataset (cord-19).
[2] Python for nlp: Introduction to the pattern library.
[3] Christian Bode, Christian Herzog, Daniel Hook, and Robert McGrath. A guide to the dimensions data approach. Dimensions Report. Cambridge, MA: Digital Science, 2018.
[4] Pew Research Center. the new news landscape: rise of the internet, 2010. URL: http:// pewresearch.org/-pubs/-1508/-internet-cell-phone-users-news-social-experience.
[5] Matteo Cinelli, Walter Quattrociocchi, Alessandro Galeazzi, Carlo Michele Valensise, Emanuele Brugnoli, Ana Lucia Schmidt, Paola Zola, Fabiana Zollo, and Antonio Scala. The covid-19 social media infodemic. arXiv preprint arXiv:2003.05004, 2020.
[6] Giovanni Colavizza, Rodrigo Costas, Vincent A Traag, Nees Jan Van Eck, Thed Van Leeuwen, and Ludo Waltman. A scientometric overview of cord-19. BioRxiv, 2020.
[7] Montserrat Comesaña, Ana Paula Soares, Manuel Perea, Ana P Piñeiro, Is-abel Fraga, and Ana Pinheiro. Erp correlates of masked affective priming with emoticons. Computers in Human Behavior, 29(3):588–595, 2013.
[8] Andrew J Flanagin and Miriam J Metzger. Perceptions of internet information credibility. Journalism & Mass Communication Quarterly, 77(3):515–540, 2000.
[9] Daniel Gayo-Avello, Panagiotis Takis Metaxas, Eni Mustafaraj, Markus Strohmaier, Harald Schoen, Peter Gloor, Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. Predicting information credibility in time-sensitive so-cial media. Internet Research, 2013.
[10] Aditi Gupta and Ponnurangam Kumaraguru. Credibility ranking of tweets during high impact events. In Proceedings of the 1st workshop on privacy and security in online social media, pages 2–8, 2012.
[11] Momchil Hardalov, Ivan Koychev, and Preslav Nakov. In search of credible news. In International Conference on Artificial Intelligence: Methodology, Systems, and Applications, pages 172–180. Springer, 2016.
[12] Jinling Hua and Rajib Shaw. Corona virus (covid-19)“infodemic” and emerg-ing issues through a data lens: The case of china. International journal of environmental research and public health, 17(7):2309, 2020.
[13] Vasileios Lampos, Tijl De Bie, and Nello Cristianini. Flu detector-tracking epidemics on twitter. In Joint European conference on machine learning and knowledge discovery in databases, pages 599–602. Springer, 2010.
[14] David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gor-don Pennycook, David Rothschild, et al. The science of fake news. Science, 359(6380):1094–1096, 2018.
[15] BBC news. Facebook must tackle fake news, 2017. URL: https://www.bbc.com/news/technology-39718034.
[16] BBC news. Outcry after trump suggests injecting disinfectant, 2020. URL: https://www.bbc.com/news/world-us-canada-52407177.
[17] PolitiFact. 2016 lie of the year: Fake news., 2016.
[18] Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. A stylometric inquiry into hyperpartisan and fake news. arXiv preprint arXiv:1702.05638, 2017.
[19] Julio CS Reis, André Correia, Fabrício Murai, Adriano Veloso, and Fabrício Benevenuto. Supervised learning for fake news detection. IEEE Intelligent Systems, 34(2):76–81, 2019.
[20] Victoria L Rubin, Niall J Conroy, and Yimin Chen. Towards news verification: Deception detection methods for news discourse. In Hawaii International Conference on System Sciences, pages 5–8, 2015.
[21] Dipanjan Sarkar. Text analytics with python: A practical real-world approach to gaining actionable insights from your data. new york: Apress; 2016.
[22] Aliza Sarlan, Chayanit Nadam, and Shuib Basri. Twitter sentiment analysis. In Proceedings of the 6th International conference on Information Technology and Multimedia, pages 212–216. IEEE, 2014.
[23] Mike Schmierbach and Anne Oeldorf-Hirsch. A little bird told me, so i didn’t believe it: Twitter, credibility, and issue perceptions. Communication Quar-terly, 60(3):317–337, 2012.
[24] Karishma Sharma, Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, Aastha Dua, and Yan Liu. Coronavirus on social media: Analyzing mis-information in twitter conversations. arXiv preprint arXiv:2003.12309, 2020. [25] Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. Fake news detection on social media: A data mining perspective. ACM SIGKDD explo-rations newsletter, 19(1):22–36, 2017.
[26] Julia Silge and David Robinson. Text mining with R: A tidy approach. " O’Reilly Media, Inc.", 2017.
[27] Lauren Swiney. Intuitive biology, moral reasoning, and engineering life: essen-tialist thinking and moral purity concerns shape risk assessments of synthetic biology technologies. Cognition, 201:104264, 2020.
[28] Chenhao Tan, Lillian Lee, and Bo Pang. The effect of wording on mes-sage propagation: Topic-and author-controlled natural experiments on twit-ter. arXiv preprint arXiv:1405.1438, 2014.
[29] David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. arXiv preprint arXiv:2004.14974, 2020.
[30] John Zarocostas. How to fight an infodemic. The Lancet, 395(10225):676, 2020.
