I May Click but May I Trust You? Exploring the Effect of Clickbait Headline on Media Credibility: An Experimental Study
Shengchao Wang University of Amsterdam
Author Note
This research was supported and supervised by ASCoR research institute, University of Amsterdam. Sincere gratitude to my supervisor Annemarie van Oosten for the scrupulous guidance during the long period
Abstract
A classical experiment is conducted to assess the effect of clickbait headline on individuals’ perception of media credibility. Media credibility is measured including source credibility and message credibility. Participants (N=89) are randomly assigned to either an article with clickbait headline or an identical article with non-clickbait headline and required to judge the credibility of the content publisher (source credibility) and of the communicated message itself (message credibility). The result shows that exposure to clickbait does not have an effect on either source or message credibility judgement, though an exploratory analysis shows that participants’ own perception of headline sensation level has a negative relation with message credibility. The mediating role of arousal and moderating role of personal relevance and need for cognition are not observed.
Keywords: clickbait, experiment, media credibility, source credibility, message credibility, personal relevance, need for cognition, arousal
I May Click but May I Trust You? Exploring the Effect of Clickbait Headline on Media Credibility: An Experimental Study
Headlines are traditionally regarded as brief and attractive summaries of the news item (Dor, 2003). However, as the media moves online, the click-based advertisement revenue gradually becomes the pillar income for media producers which leads to a trend on creating misleading titles and exaggerating the content on the landing page. The primary goal of headlines becomes enticing users to click in order to monetize the landing page (Biyani, Tsioutsiouliklis, & Blackmer, 2016). Despite clickbait was proved to effectively attract clicks (Kuiken, Schuth, Spitters, & Marx, 2017), its detrimental effect on journalism was stressed because clickbait was often associated with misleading and spreading fake news (Elyashar, Bendahan, & Puzis, 2017). Currently not only fake news and rumor sources but even the reliable traditional news outlets with high reputation are applying clickbait strategy to boost views in their online practice. For instance, Chen, Conroy and Rubin (2015) argued that on Facebook, an article from the highly reputed newspaper New York Times highly resembled an article from the Onion which specialized in publishing satire articles on international, national and local news.
The downside of the clickbait technique is seldomly stressed especially the harm to the communicated message and source. This concern is driven by the long-term association between clickbait and unreliable content like fake news and rumors (Doyle, 2014; Shu, Sliva, Wang, Tang & Liu, 2017). The potential side effect requires empirical substantiation because if clickbait headlines have negative effect on readers’ trust, it will directly lower their loyalty to the brand (Lau & Lee, 1999), which might have considerable adverse effect upon the reputation and
development of the media outlet. A question regarding its effect on media credibility is raised by the current research that even if the content is identical, will the readers treat the content and source of an article with clickbait headline as less credible compared to that with non-clickbait headlines?
Within communication research, credibility is commonly discussed in three subsets, medium credibility, message credibility and source credibility (Kang, 2010). Medium credibility is the perceived credibility of specific media form like print media (newspaper, magazine), television or Internet. Source credibility is commonly defined as perceived trustworthiness and expertise of the source (Hovland, Janis, & Kelley, 1953; Self, 1996). Message credibility is defined as the credibility of the communicated message itself (Li & Suh, 2015). This study focuses on source credibility and message credibility as two outcome variables, medium credibility is left out since only an article published in the online context was assessed. The research question is illustrated as:
RQ: What’s the influence of clickbait headline on media credibility?
Regarding practical contribution, the study result is informative for online article publishers to consider the potential negative impact by clickbait headlines on readers’ credibility judgement of both the source and the communicated message. This technique could be a two-side sword which may generate distrust against the source and might lead to negative consequences in both financial and reputational concern.
In terms of scientific relevance, this research adds to the few studies assessing the relation between clickbait headlines and media credibility. Besides, inspired by Hurst (2016)
where mere source credibility was assessed in his work, the current study aims to broaden the perspective and pioneers to include both source credibility and message credibility as the two dependent variables. Moreover, previous studies didn’t take any moderation or mediation effect into consideration, the current study is also the first study to consider personal relevance (Petty & Cacioppo, 1979) and need for cognition (Cacioppo & Petty, 1982) as moderating effect and emotional status (i.e. arousal) as mediating effect.
Theoretical Background
Clickbait: The Effectiveness of Sensational Headlines in an Online Context
Previous research regarding clickbait are divided into three main areas. One area focuses on developing computational models to detect clickbait headlines (e.g. Chen et al., 2015; Anand, Chakraborty & Park, 2017). Though technological issues are not discussed in the current study, these papers collectively provide a comprehensive illustration of the linguistic difference between clickbait headlines and non-clickbait headlines, for example, hyperbolic words and Internet slangs were found to be commonly used in clickbait headlines but rarely appear in the traditional non-clickbait headlines (Chakraborty, Paranjape, Kakarla, & Ganguly, 2016).
The second area of research focuses on content-analyzing the prevalence of clickbait in the current media landscape. For instance, Orosa, Santorun and Garcia (2017) analyzed 140 editions of European newspapers and a total amount of 840 texts and found that 60% of the news headlines prioritized sensation over information and only a quarter of the news headlines favored information over emotion. Beleslin, Njegovan and Vukadinovic(2017) analyzed Serbia’s most-read daily newspapers and discovered that clickbait titles were present in one thirds of the
news and regarding the different sections, clickbait headlines were found to appear more frequently in entertainment sections and lifestyle section than politics section.
The third area lies in the media effect of clickbait. Clickbait headlines have been widely proved to be efficient to attract clicks. In an experimental study, Kuiken et al. (2017) rewrote traditional headlines into clickbait headlines and assessed the effectiveness of the headlines by comparing the original headlines and clickbait headlines indicated by CTR(Click-Through-Rate). They found that several features like inclusion of sensational words could significantly predict higher CTR.
Lai and Farbrot (2014) conducted two experiments and concluded that headlines written in question framing, which were commonly used in clickbait headline yet barely used in declarative headlines, were more effective in generating readership than declarative headlines. Milkman and Berger (2014) also focused on the relation between specific clickbait headline rhetoric and article popularity. They found that sensation and interestingness of the headline was associated with more shares among the content shared on social networks. Lockwood (2016) argued in alignment with Milkman & Berger (2014) that article titles phrased more interestingly attained more attention. The attractiveness of clickbait headline could also apply to academic publications. Jamali and Nikzad (2011) found that question-framed titles in articles in PLOS journals received more downloads.
The underlying mechanism of the clickbait effectiveness on attracting clicks was explained (Chakraborty, et al., 2016; Potthast, Kopsel, Stein, & Hagen, 2016) using the curiosity gap model (Loewenstein, 1994). The researchers stated that by using some rhetoric techniques,
some crucial information could be withheld in the headline. A gap of knowledge would then be created among the readers after exposing to the headline and this gap could only be filled by advancing to the full content. These researches focus on the positive side of applying clickbait headlines and they share the same conclusion that roughly the clickbait tricks could indeed generate more clicks, however, the dark side of this technique such as potential side effect on media credibility are comparatively less researched.
Effects of Clickbait Headlines on Media Credibility
Mackay and Bailey (2012) found that articles written in a sensational way were perceived as less credible than the traditional stories, though they focused on the overall writing style thorough the article instead of mere headlines. Yoo (2013) examined whether the inset of ‘breaking news’ into the headline affect credibility evaluation and no significant difference was found. This finding was congruent with Munger et al. (2018) where they found no evidence on the effect of clickbait headline on trust in media. However, there were also two studies that indicating a negative relation between clickbait headline and media credibility. For instance, Beleslin, Njegovan and Vukadinovic (2017) found that the majority of the participants expressed negative attitudes toward clickbait headlines and considered it untrustworthy. Hurst (2016) found the same source would receive significantly lower credibility rating when applying clickbait headlines compared to non-clickbait traditional headlines.
Theoretical framework for Direct Effect of Clickbait Headline on Media Credibility: Media Priming
be inherited in the current study as well. The main idea of media priming theory is that the media content would have effect on people’s mentality, attitude and behavior (Ewoldsen & Ewoldsen, 2009). Hurst (2016) argued that the headline information would have a priming effect beyond the headline text itself to the whole processing of the article, thus people would form a suspicious attitude toward the source under the influence of the clickbait headlines. Based on the long-term association between the clickbait headline and fake news and spams (Doyle, 2014; Shu et al., 2017), it could be assumed that participants exposing to clickbait headline would be primed to regard both the article content and the article publisher as less credible than those exposing to non-clickbait headlines.
Based on the finding of Hurst (2016) and media priming theory (Ewoldsen & Ewoldsen, 2009), the hypothesis is proposed as follow:
H1: Online articles with clickbait headlines receive lower media credibility than that with non-clickbait headlines.
Though the first hypothesis has been researched by previous studies (e.g. Mackay & Bailey, 2012; Munger et al.; Yoo, 2013; Hurst, 2016) and seems replicated, the specific contribution of this study also takes moderating and mediating mechanism into account.
The Mediating role of Arousal Level
Hurst (2016) demonstrated that though a negative relation between clickbait headline and source credibility was found, the underlying mechanism was unclear and needed further exploration. This study assumes one mediator ‘arousal’. Arousal refers to the intensity of emotional activation ranging from dejection to excitement (Mehrabian & Russell, 1974).
Regarding the causal relation between clickbait exposure and arousal, several studies indicated that clickbait headline exposure led to subsequently increased arousal level. It was found that exposing to clickbait headline led to higher level of arousal compared to normal news headlines (Pengnate, 2016; Pengnate, Young & Chen, 2018). Though focusing on writing style of the whole article instead of the headline, Grabe, Zhou, Lang and David (2000) also argued that participants exposing to more sensational stories self-reported higher arousal level.
This mechanism beneath the clickbait technique and arousal level is connected to sensationalism which refers to the activation of irrational emotional responses (Grabe et al., 2000) and clickbait evokes the emotion by using sensational writing style (Blom & Hansen, 2015). Based on the theoretical and empirical evidence, the assumption is made as:
H2: Exposing to clickbait headline leads to higher level of arousal.
Regarding the relation between arousal level and media credibility judgement, the Cognitive-Affect Process Model (Bagozzi, 1982) underlines the emotional states such as arousal (Uribe & Gunter, 2007) influence the subsequent information processing and intention. This theory implies that the arousal level can predict individuals’ perception of the media credibility.
Moreover, empirical evidence showed that individuals viewing emotionally-arousing articles rated the article as less credible compared to those exposing to less sensational ones (Grabe et al., 2000). Though the referred study focused on the article sensation instead of headline sensation, it could still imply a negative correlation between arousal level and credibility evaluation. Based both on the theoretical model and finding of the previous study, the predicting effect of arousal on media credibility is assumed:
H3: Higher level of arousal leads to lower media credibility.
Moderating Effect: Personal Relevance and Need for Cognition
Previous studies like Hurst (2016) merely assessed the direct effect of clickbait headline on the credibility judgement without taking any potential moderating effect into account, the current study speculates that the extent of information relevance and need for cognition could moderate the effect. This assumption is supported by the Heuristic-Systematic Model (Chaiken & Eagly, 1989) and Elaboration Likelihood Model (Petty & Cacioppo, 1986).
Heuristic-Systematic Model (Chaiken, 1980) proposes a systematic and a heuristic view of persuasion. The Elaboration Likelihood Model (Petty & Cacioppo, 1986) shares the similar idea, the minor difference is that the heuristic route in Heuristic-Systematic Model is termed peripheral route and systematic route is termed central route in the context of Elaboration Likelihood Model. When media consumers adopt the systematic route, considerable cognitive effort will be spent in comprehending the information as well as judging the validity of the conclusion. While within a heuristic route, the recipients tend to search for and base their judge on more accessible information such as the source factors and other non-content factors (Chaiken & Eagly, 1989). When people are taking a heuristic route for information assimilation, the clickbait headline is more likely to be used as a heuristic cue which leads to more negative priming effect of clickbait which lower their credibility judgement.
Whether people decide to take systematic or heuristic view depend on several factors. One of the factors can be personal relevance (Petty & Cacioppo, 1979; Chaiken, 1980). If individuals regard the topic of the message as highly personally relevant, a systematic approach
will be employed while if the topic is treated as insignificant, a heuristic approach will be adopted (Chaiken & Eagly, 1989). For instance, Srivastava (2013) asked participants to perform a memory task and found that highly personally relevant content did lead to more thorough and comprehensive cognitive processing. This result implies the higher level of information relevance may avert individuals’ attention from surface cues like headlines which could help weaken the negative influence of the clickbait headline thus led to more positive attitude toward the media credibility.
H4. Higher level of personal relevance moderates the effect of clickbait headline on media credibility.
Besides externally situational factors such as personal relevance, the individual difference in the motivation to engage in effortful information processing, which is termed need for cognition (Cacioppo & Petty, 1982), could be the internal factor to decide which of the two routes individuals are willing to take when encountering new information (Haugtvedt, Petty & Cacioppo, 1992). Individuals with high need for cognition are more influenced by the quality of arguments, while individuals with low need for cognition are more likely to be affected by surface cues like headline text (Haugtvedt et al., 1992).
Similar to the argument of personal relevance, the more an individual is intrinsically willing to perform arduous work in comprehending the content, the less priming effect would be expected of surface cues like clickbait headlines, therefore the third hypothesis is:
H5: Higher level of need for cognition moderates the effect of clickbait headline on media credibility.
Method Research Design and Procedure
To evaluate the potential causal effect, a classical online between-subjects experiment was adopted. For the convenience of experiment conduction, the whole experimental context and material used were all in Chinese. This experiment used two groups with one manipulation group exposing the participant to a clickbait headline and one control group presenting the participants with a non-clickbait headline. These two types of headlines were generated based on distinctive features identified by Chakraborty et al. (2016) (see “Stimuli”). The whole experiment was conducted through the computational online survey tool Qualtrics.
There were two items designed in the survey to identify the participants as inappropriate in the sample. The first item was the consent form following the fact sheet asks for participants’ consent in participating in this experiment. If participant was unwilling to participate, he or she would be directed to the end of the experiment. The other question was Have you read the article before? set after the article presentation. If participant had read the article before he or she would be excluded from the sample because that would seriously bias their attitude toward the article and damage the internal validity.
Participants accessed the experiment via a hyperlink. In the beginning of the experiment a fact sheet illustrated the synopsis of the related matters of the research. Participants were also well informed that their anonymity was safeguarded, they could quit at any time according to their own willingness and they had to give active consent before participating. The researcher’s contact was also provided in case the participants have issues to ask.
Participants were then asked to answer questions about control variables, including age, sex, educational level, and frequency and skillfulness of WeChat use, as well as personal relevance and need for cognition as moderators
Participants were then randomly assigned into each of the two groups by Qualtrics system. They were first presented with a headline and were asked to click on the headline to access the whole article. This was a deliberately designed step to lead participants allocating more attention on the headline to enhance the effect. The article page resembled the WeChat article interface, the page was set to freeze for 100 seconds to avoid participants skipping the article and giving irresponsible answers for the following answers related to the article. After reading the article participants were directed back to the survey. The mediator arousal level was then first assessed. Then source credibility and message credibility were separately assessed. To avoid the possibility of making participants realize the purpose of the experiment, a manipulation check was added as the last question of the survey. After the experiment participants were rewarded a small amount of money(0.5EUR).
Participants
A total of 168 participants were included in the experiment and only 89 valid responses were recorded. The criteria of valid judgement was based on the completeness of the scale, only a complete response with no questions missing was judged as valid. Most of the participants were the researcher’s acquaintance and relatives, making the sample a convenience sample. The snowball sampling technique was also applied since participants were asked to invite their friends to participate.
Fourty-three females, 42 males and 4 people who were unwilling to specify their gender constituted the whole sample. In the respect of age, 62.9% of the participants were younger than or equal to 30 and the rest 37.1% reported older than 30. The youngest participant was 21 years old and the oldest was 57 years old. Among them 3 participants (3.4%) received middle school education, 2 people (2.2%) had PhD diploma, the majority were doing college/bachelor study (47.2%) or master study (47.2%).
Upholding the principle of random assignment, 43 participants were assigned into non-clickbait headline group and 46 were assigned into clickbait headline group. In terms of age, participants in the non-clickbait headline group (M=30.51, SD=9.28) were not significantly different from the participants in the clickbait headline group (M=31.67, SD=10.63), t=-0.55, p= 0.59. The average educational level in clickbait headline group (M=3.53, SD=0.67) was not statistically different from that in non-clickbait headline group (M=3.43, SD= 0.44) as well, t=0.78, p=0.44. That indicated the randomized assignment was successful that the demographic characteristics didn’t differ in the two groups.
Stimuli
Headline type. A clickbait headline and a non-clickbait headline were used in the experiment. The two headlines were written based on the finding of Chakraborty et al. (2016) as well as Blom and Hansen (2015). This study generated the clickbait headline using an Internet slang with extreme sentiment, three types of punctuation patterns (?, !!! ,!) (Chakraborty, et al., 2016) and forward-referencing technique in discourse deixis form (Blom & Hansen, 2015) which referred to the part of the forthcoming discourse relative to the location in the headline text (e.g.
‘He is the best athlete in all sports of all time’, while the name of the athlete is concealed in the pronoun ‘he’). The length of the clickbait headline was deliberately was set to include 21 Chinese characters since the clickbait headline was discovered typically containing more words than non-clickbait headlines (Chakraborty et al., 2016).The clickbait headline was generated in Chinese, the English translation was “Turned my world upside down!!!Our health is ruined by this food that we grow up eating?!” The first part of the sentence was a Chinese Internet slang, the researcher elaborately concealed the name of the food in the pronoun ‘this food’ where forward-referencing was applied. Punctuation patterns (?, !, !!!) were embedded in two places in the sentence.
The non-clickbait headline was generated in a declarative framing. The English translation should be appropriately described as “Grain is proved to potentially harm health.” It was written in a factual style illustrating the main idea of the article in an objective way. The whole headline contained 13 Chinese characters in order to stay shorter than the clickbait headline (Chakraborty, et al., 2016)
Manipulation check was conducted at the end of the experiment to test whether the level of sensation of the clickbait headline is higher than the non-clickbait headline. This question measured the sensation using 7-point Likert scale (Likert, 1932) ranging from ‘Very Objective’ to ‘Very Exaggerating’.
Article and Source. The articles presented in the two groups were identical. For the convenience of experiment conduction, the article was based on an article excepted from a health information account on WeChat, one of the largest social platforms in China (DMR, 2018) with
modest modification. The main idea of the article was the traditional Chinese diet which emphasized the nutritional value of grain was problematic, as excessive carbohydrates, which are brought by grain intake, may be responsible for causing obese. A health themed article was selected because health information on WeChat was frequently associated with rumor and fake news, around 15% of rumors identified by WeChat are health themed and the rumors always feature clickbait headlines (Tencent, 2016). The exposure to the clickbait headline on health information may lead participants to raise more suspicion on the authenticity of the article and the source, thus the headline effect may be enhanced. A fabled source of the article was used in the two groups in case that participants may hold preconception if an existing source was applied.
Measures
Demographic variables.
Demographic variable including age, educational level and gender were collected. Age is asked by an open question “How old are you?”, gender measurement was borrowed from Hurst (2016) “What’s your gender?” and three options were “Male” “Female” and “Not Specified”. Educational level is also asked, and an ordinal scale was used with “Primary school or below”, “Middle School”, “College/Bachelor”, “Master” and “PhD”
Dependent variables
Source credibility. The source credibility was measured in three facets, namely reliable, knowledgeable and influential, which were adopted from the five-item measurement of source credibility from Kang (2010). Knowledgeable referred to the extent the source exhibits
knowledge on some specific topic, influential referred to the authority of the source and reliable referred to the possibility of the source providing credible information. Three declarative statements were given: “I think the publisher of this article is knowledgeable” “I think the publisher of this article is reliable” and “I think the publisher of this article is influential”. Participants rated their level of agreement with the above statements in a 7-point Likert scale (Likert, 1932) ranging from totally disagree to totally agree. Factor analysis was performed on the three items and they all load on one dimension which indicates this measure is uni-dimensional and this factor explains 68% of the total variance. With respect of reliability, the Cronbach’s Alpha was 0.77 therefore the whole scale was reliable. A new variable MeanSourceCredibility was calculated as the average score of the three items (M= 4.13, SD=0.75).
Message credibility. Adopting the truncated version of message credibility scale developed and tested by Appelman and Sundar (2016), participants were required to rate their level of agreement on 3 items in a 7-point Likert scale: “I think the opinions conveyed in this article are believable”, “I think the information cited in the article is accurate”, and “I think this article is authentic”. However, due to the vagueness and overlapping between the ‘accurate’ and ‘authentic’ especially in the Chinese language context, only the first two items were kept in the scale. Correlational analysis was conducted on these two items and the result showed they were highly correlated (r=0.87, p<0.001). The mean of the two items were stored into a new variable MeanMessageCredibility (M=4.32, SD=1.11).
Personal Relevance. Personal relevance was measured with two items. Participants indicated in 7-point Likert scale to what extent they agreed with the two statements: “I think diet-related issues are important to me” and “I think health-related issues are important to me”. A correlational analysis was conducted on the two items and they are highly correlated (r=0.87, p<0.001). A mean score of the two items were contained by a new variable MeanPersonalRelevance (M= 6.12, SD=0.96).
Need for cognition. Need for cognition scale was borrowed from Cacioppo, Petty and Cao (1984) which contained eighteen statements. Participants indicated to what extent did they endorse on these statements in a seven-point scale ranging from totally disagree to totally agree. 9 of the 18 statements were reversely framed. After recoding the reverse-coded items, factor analysis was conducted, and it showed that 15 of the 18 items loaded on one factor which explains 34.67% of the total variance. The scale containing the remaining 15 items was highly reliable(α=0.89). The mean score of the 15 items were stored in a new variable MeanNeedForCognition (M=3.42, SD=0.60).
Mediator
Arousal. Arousal was measured by the arousal component of self-assessment manikin (Lang, 1980). The arousal scale of SAM was anchored by Calm/Relaxed/Sleepy on one side of the dimension and Aroused/Excited/Agitated on the other. This research deduced the original 9-point scale to a 7-point scale ranging from totally disagree to totally agree to uniform with other scales and avoid respondent fatigue. The result of the factor analysis showed two of the three items (Calm/Aroused, Relaxing/Excited) loaded on the same factor and these two items
were highly correlated. (r=0.89, p<0.001). A new variable MeanArousal (M=3.54, SD=2.12) was computed as the mean of the two items.
Ethics
This study examined closely on General Data Protection Regulation (Voigt & Bussche, 2017). Individuals invited were asked to consent on their participation in the study and well informed of their right in the beginning of the online experiment. This study didn’t require participants’ real name. In the data analysis part, the participants were indexed by serial number and their demographical information were only accessible by the researcher.
Analytic Approach
Identifying Outlier. By directly observing the dataset, no outliers were spotted.
To measure the moderating and mediating effect, An SPSS add-on PROCESS was applied in this study. Regression analysis would be conducted separately on the two dependent variables MeanSourceCredibility and MeanMessageCredibility with dichotomous variable Group as the independent variable. Model 10 in the PROCESS macro was used to test the moderation and mediation.
Result Manipulation Check
The last question in the questionnaire was set to test whether the clickbait headline was more sensational. The result showed that participants in the clickbait headline group perceived the headline significantly more sensational (M=3.59, SD=1.02) than their counterparts in the non-clickbait headline group (M=3.09, SD=0.68), t (79) = 2.69, p = 0.009, indicating that the
manipulation was successful.
The Effect of Clickbait Headline on Media Credibility
The first hypothesis speculated that an online article with a clickbait headline will receive lower media credulity than that with a non-clickbait headline. Equal variance across the two groups was assumed regarding both source credibility (F=0.11, p=0.75) and message credibility (F=1.21, p=0.27). MANOVA test showed that there was no statistically significant difference in media credibility judgement based on the two different groups, F(2, 86) = 0.17, p=0.85; Wilk's Λ = 1.00, partial η2 = 0.004.
Regarding the source credibility as part of the media credulity, there was no difference between the clickbait headline group (M=4.08, SD=0.82) and the non-clickbait headline group (M=4.17, SD=0.67), F (1,87) =0.27, p=0.60, partial η2= 0.003.
Regarding message credibility, the group distinction also had no effect, the message credibility score in the clickbait headline group (M=4.26, SD=1.20) was not significantly lower than the score in the comparative group (M=4.38, SD=1.02), F (1,87) = 0.27, p=0.61, partial η2= 0.003.
The Mediating Role of Arousal Level
The second hypothesis predicted that exposing to clickbait headlines led to higher arousal level than the non-clickbait headline. According to the t-test result, equal variance was assumed (F<0.001, p=0.99), the participants in the clickbait headline group (M=3.43, SD=2.17) were not significantly more aroused than their counterpart in the non-clickbait headline group (M=3.65, SD=2.09), t(87) =0.48, p=0.63, CI [-0.68, 1.11]. The third hypothesis predicted that the higher
arousal level was associated with lower media credibility. The regression analysis was conducted using the model 10 of PROCESS analysis tool including the two moderators personal relevance and need for cognition as well. In terms of source credulity, the regression model was not significant, R2=0.06, F (5,83) =1.09, p=0.37. The arousal level failed to significantly negatively predict lower source credibility (b= -0.01, SE=0.04, p=0.88, CI [-2.12, 4.20]). In the respect of message credibility, the regression model was also not significant, R2=0.07, F (6,82) =1.05, p=0.40, the arousal level also had no significant negative effect on message credibility (b= -0.06, SE= -0.06, p=0.26, CI [-0.18, 0.04]).
Moderation Analyses: The Role of Personal Relevance and Need for Cognition
Hypothesis 4 proposed that the level of personal relevance of the message would moderate the relation between clickbait headline and credibility, the more personally relevant the information were perceived by the participants, the weaker relationship between clickbait headline exposure and message or source credibility judgement. This hypothesis was also tested using the same model used in the previous hypothesis. The result showed that personal relevance has no moderation effect influencing the direct effect of clickbait headline on neither source credibility (b=-0.02, SE=0.15, p=0.91, CI [-0.31, 0.28]) nor message credibility (b=-0.01, SE=0.22, p=0.96, CI [-0.45, 0.43]).
Hypothesis 5 proposed that the higher level of need for cognition weakened the relation between exposure to a clickbait headline and message or source credibility. However, the result showed that need for cognition had no moderation effect on the direct relation between clickbait headline exposure and source credibility (b=-0.01, SE=0.20, p=0.98, CI [-0.41, 0.40]) nor on
message credibility (b=-0.30, SE=0.31, p=0.33, CI [-0.91, 0.31]).
Figure 1. The direct and indirect effect of clickbait headline on source credibility, as tested in the PROCESS model (model 10).
Note. The numbers illustrated in the figure represent the unstandardized regression coefficient with standard error between the brackets.
Figure 2. The direct and indirect effect of clickbait headline on message credibility, as tested in the PROCESS model 10.
Note. The numbers illustrated in the figure represent the unstandardized regression coefficient with standard error between the brackets.
No asterisk, non-significant.
Additional Exploratory Analysis: Negating the Manipulation
The analysis built around the two comparative groups showed non-significant results. However, it’s noticeable that the difference of the perceived sensational level tested by the manipulation check shows the mean difference of the headline sensation perception is marginal though significant (Mdifference=0.5).
That result may imply the manipulation is problematic to some degree and that led the author to revise the model by using the regression analysis with the manipulation check question (participants’ own perception of the sensation level of the headline) as the independent variable. The revised model served as an exploratory analysis to retest all the above hypotheses based on
participants’ own perception of the headline sensation instead of the researcher’s manipulation, using model 10 of the PROCESS analysis tool with the perceived headline sensation level was set as the independent variable.
The findings showed that for the prediction of message credibility the model was significant (R2=0.29, F (6,82) =5.65, p<0.001). The direct effect of headline sensation on the message credibility was marginally significant (b=-2.14, SE= -1.82, p=0.07, CI [-4.48, 0.20]). No significant prediction of source credibility was found. Moreover, the results showed that the model with the arousal measure as the outcome variable was not significant (R2=0.08, F (5,83) =1.39, p=0.24), however headline sensation level had marginally significant impact on arousal level (b=-4.44, p=0.08). Arousal level, in turn, failed to predict lower source credibility (b=-0.01, SE=0.03, p=0.76, CI [-0.07, 0.05]) but had marginally significant effect on message credibility (b=-0.10, SE= 0.05, p=0.06, CI [-0.20, 0.005]). This result implied arousal mediated the effect of headline sensation on message credibility though not source credibility. Moderation effects of personal relevance (b=-0.28) and need for cognition
Figure 3. The direct and indirect effect of perceived headline sensation level on source credibility Note. The numbers illustrated in the figure represent the unstandardized regression coefficient with standard error between brackets.
Figure 4. The direct and indirect effect of perceived headline sensation level on message credibility
Note. The numbers illustrated in the figure represent the unstandardized regression coefficient with standard error between brackets.
No asterisk, non-significant. *, significant at 0.10 level.
Discussion and Conclusion
The effect of clickbait headline on media credibility remains under-researched in the literature before the current study. The main aim of this experimental study is to explore what change in credibility judgement would happen if individuals are exposed to clickbait headlines compared to non-clickbait headlines .
It was expected that a clickbait headline exposure would lead participants to rate the experimental material as less credible toward both the source and message credibility. Besides, the mediating effect of arousal level and moderating role of personal relevance and need for cognition were taken into consideration.
The main findings were, however, in contrast to the expectations, asclickbait headline did not lead to readers’ deprecated perception of source or message credibility. Arousal level was not found to have significant mediation effect on the dynamics between clickbait headline and source or message credibility. Moreover, neither personal relevance nor need for cognition had significant moderating effect involved in the direct influence of clickbait headlines on source or message credibility.
Exploratory analyses on the manipulation check of sensationalism showed in terms of message credibility, higher headline sensation was associated with significantly lower message credibility with arousal level mediating the association. To be specific, higher headline sensation unexpectedly predicted lower arousal while higher arousal level predicted lower message credibility judgement as assumed. Neither the moderating role of need for cognition nor personal relevance were found.
Explaining the None Effects of Clickbait Headlines
The current study shares similar finding with Yoo (2013) and Munger et al. (2018) that exposing to clickbait headlines has non-significant effect on media credibility judgement. The finding of this study contradicts to Hurst (2016) and Beleslin et al. (2017) that in their studies the negative relation between clickbait headline and credibility was found. Detailing into the two studies, Beleslin et al. (2017) directly surveyed ‘Do you have confidence in information posted through news whose titles are written as clickbait?’ without any delicate indirect measures which might be too direct and simple, moreover, the survey method didn’t guarantee a direct and strong causal relation. The study of Hurst (2016) might be more insightful that he conducted a factorial
experiment (2 news types x 2 source types x 2 levels of source credibility), pre-existing sources were used in his study and they were distinguished as five highest-credible sources and five lowest- credible sources by a pretest. It implies that adopting existing sources instead of fabling one by the researcher may generate better effect.
However, the current study indicated the negative effect of the clickbait usage might be minor which may explain the overflow of clickbait usage (Agrawal, 2016) because the click-attracting benefits overwhelms the potential downside. The insignificance of the moderating effects may imply that regardless of how personally relevant the information is or how fond the person is of getting involved in deep thinking, the effectiveness of the headline remained the same, which contradicted to the expectations based on the heuristic-systematic model (Chaiken, 1980) and the ELM model (Petty & Cacioppo, 1986).
The non-significant effect of clickbait headline implies two possible explanations, the first one can attribute to the experiment material selection, the article message itself may trigger high suspicion so that the message and headline all lead to lower credibility judgement thus no matter if people take heuristic or systematic route of processing, they would rate credibility in the same direction. The second one could be the item order in the survey, it might be advisable to measure the sensation immediately following the exposure to the headline, however, the downside could be revealing the intention of the study to the participants which could endanger internal validity.
Practical Implications for Using Clickbait Headlines
their usual strategy. Though research on the long-term effects is lacking, the present result shows that at least in the short term, individuals might not perceive the content provider as less credible under clickbait headlines., media producers may discover that in practice, the short-term benefit of generating more clicks weighs far beyond the little source credibility concern. For the media producers, this result might imply that they can apply eye-attracting rhetoric techniques on headline generation without worrying excessively about its potential side effect on the source reputation.
However, it does not mean clickbait headline have no negative influence at all because at least perceived headline sensation has significant direct negative influence on message credibility. However, it can be argued that multiple factors other than the headline text can affect the perceived headline sensation level. When the participants reached the last question measuring headline sensation, they have already viewed the whole article so that some unexpected factors such as individuals’ perception of topic or the impression of the message may interfere the judgement. Moreover, unexpected negative relation between sensation level and arousal was found which might imply the abusive usage of clickbait headline resulted in subsequent meticulous attitude during media consumption.
Limitations and Suggestions for Future Research
There are some flaws limiting the validity of this study. The first was the inevitable downside of experimental design. An online experiment was chosen as the research method since it could assess the causal relationship between clickbait headline and the media credibility without the interference of other external factors, which guaranteed high internal validity of the
study. However, participants were required to click on the headline and read the following article, while in normally people would have the autonomy on what they were willing to consume. Moreover, they might not devote the same level of cognitive effort in a natural context as in an experimental environment, thus the ecological validity is challenged.
The sample recruitment could be the second limitation. Due to the practical concerns, the participants are either the acquaintance of the researcher or recruited by so-called snowball method, the sample have similar demographic characteristics. The majority of the participants are young adults (19-30 years old) so it’s reasonable to generalize the finding to this specific population but not to other age groups such as tweens (7-12 years old), teens(13-18 years old) and adults(older than 30) (Valkenburg & Piotrowski, 2017).
The final limitation concerns the clickbait headline manipulation. It was expected that the clickbait headline could be perceived much more sensational than the non-clickbait headline, however, the result of the manipulation check showed that the perceived sensation difference is minor (Mdifference=0.5) on a 7-point scale. The reason could be the survey took too long to finish and when the participants eventually got to the manipulation check question (the last question in the survey), their impression of the headline was already blurred, and they could randomly give a rating to finish or based their judgement on the actual article content instead of the headline itself. That could also be the reason why the perceived headline sensation level had significant effect on the media credibility. Individuals holding suspicious attitude toward the article might rate the headline as more sensational and exaggerated, even though they had little impression about the headline.
I have no intention to negate experimental design since considering media effect studies, experiment is the strongest design to test causal relation. However, besides experiment researchers could focus more on the long-term effect of clickbait headline on people’s cognitive status since the long-term study can provide a view of accumulative effect from clickbait. Secondly, I would suggest recruiting a more representative and larger sample with a roughly evenly distributed age range, educational level to make the result more applicable to a broader population. The third suggestion is improving the headline manipulation. One solution could be extending the headline exposure to strengthen its effect. Some participants reflect that they couldn’t even recall the headline when they were asked to indicate headline sensation as the last question in the online survey, which implied headline exposure may not be long enough for the priming effect to take place. Another solution could be adding more clickbait features into the headline such as stop words (e.g. I, she, he) and hyperbolic words (e.g. awe-inspiring, gut-wrenching), though the headline generation may need more work to varnish it as more sensational, however, extending the headline exposure might be a more effective issue compared to headline generation.
Conclusion
Given the booming rise of online media and declining subscription revenue, media producers depend more on advertisement revenue which are measured commonly by page views. The financial rewarding system and the decreasing journalistic professionalism drive the birth and prevalence of clickbait headlines which are created mainly for attracting clicks for financial consideration instead of briefing the news content as before (Dor, 2003). This study conducted a
classical experiment to assess the potential effect of clickbait headline exposure on readers’ perception toward the source and the news content.
The findings of study contradicted to Hurst (2016), the negative effect of clickbait headline on source credibility was not observed. It could be fairly concluded that the headline could have limited effect on the participants’ perception of the media credibility, which implied that individuals could base their judgement mainly on the article itself and the judgement was hardly influenced by the headline, or headline didn’t affect the article processing. Despite the potential damage to the message credibility, this study implied that clickbait headline might be an advisable choice for attracting attention and its side effect might be minor compared to its benefits.
References
Agrawal, A. (2016, October). Clickbait detection using deep learning. In Next Generation Computing Technologies (NGCT), 2016 2nd International Conference on (pp. 268-272). IEEE.
Anand, A., Chakraborty, T., & Park, N. (2017, April). We used Neural Networks to Detect Clickbaits: You won’t believe what happened Next!. In European Conference on
Information Retrieval (pp. 541-547). Springer, Cham.
Appelman, A., & Sundar, S. S. (2016). Measuring message credibility: Construction and validation of an exclusive scale. Journalism & Mass Communication Quarterly, 93(1), 59-79.
Bagozzi, R. P. (1982). A field investigation of causal relations among cognitions, affect, intentions, and behavior. Journal of marketing research, 562-583.
Beleslin, I., Njegovan, B. R., & Vukadinovic, M. S. (2017). Clickbait titles: Risky formula for attracting readers and advertisers. Proc. of the IS.
Biyani, P., Tsioutsiouliklis, K., & Blackmer, J. (2016, February). " 8 Amazing Secrets for Getting More Clicks": Detecting Clickbaits in News Streams Using Article Informality. In AAAI (pp. 94-100).
Blom, J. N., & Hansen, K. R. (2015). Click bait: Forward-reference as lure in online news headlines. Journal of Pragmatics, 76, 87-100.
Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of personality and social
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984). The efficient assessment of need for cognition. Journal of personality assessment, 48(3), 306-307.
Chaiken, S. (1980). Heuristic versus systematic information processing and the use of source versus message cues in persuasion. Journal of personality and social psychology, 39(5), 752.
Chaiken, S., & Eagly, A. H. (1989). Heuristic and systematic information processing within and. Unintended thought, 212, 212-252.
Chakraborty, A., Paranjape, B., Kakarla, S., & Ganguly, N. (2016, August). Stop clickbait: Detecting and preventing clickbaits in online news media. In Proceedings of the 2016
IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (pp. 9-16). IEEE Press.
Chakraborty, A., Sarkar, R., Mrigen, A., & Ganguly, N. (2017). Tabloids in the era of social media? understanding the production and consumption of clickbaits in twitter. arXiv
preprint arXiv:1709.02957.
Chen, Y., Conroy, N. J., & Rubin, V. L. (2015, November). Misleading online content: Recognizing clickbait as false news. In Proceedings of the 2015 ACM on Workshop on
Multimodal Deception Detection (pp. 15-19). ACM.
DMR (2018). 110 Amazing WeChat Statistics and Facts (December 2018). Retrieved from https://expandedramblings.com/index.php/wechat-statistics/
Dor, D. (2003). On newspaper headlines as relevance optimizers. Journal of Pragmatics, 35(5), 695-721.
Doyle, T. 2014. "Don’t Click That: The Clickbait Virus." Retrieved from http://www.ahundredmonkeys.com/clickbaitadvertising/
Elyashar, A., Bendahan, J., & Puzis, R. (2017). Detecting Clickbait in Online Social Media: You Won't Believe How We Did It. arXiv preprint arXiv:1710.06699.
Garcia Orosa, B., Gallur Santorun, S., & Lopez Garcia, X. (2017). Use of clickbait in the online news media of the 28 EU member countries.
Grabe , M. E., Zhou, S., Lang, A., & Bolls, P. D. (2000). Packaging television news: The effects of tabloid on information processing and evaluative responses. Journal of broadcasting &
Electronic media, 44(4), 581-598.
Grabe, M. E., Lang, A., Zhou, S., & Bolls, P. D. (2000). Cognitive access to negatively arousing news: An experimental investigation of the knowledge gap. Communication research, 27(1), 3-26.
Haugtvedt, C. P., Petty, R. E., & Cacioppo, J. T. (1992). Need for cognition and advertising: Understanding the role of personality variables in consumer behavior. Journal of
Consumer Psychology, 1(3), 239-260.
Hovland, C. I., Janis, I. L., & Kelley, H. H. (1953). Communication and persuasion; psychological studies of opinion change.
Hurst, N. (2016). To clickbait or not to clickbait? an examination of clickbait headline effects on
source credibility (Doctoral dissertation, University of Missouri--Columbia).
Jamali, H. R., & Nikzad, M. (2011). Article title type and its relation with the number of downloads and citations. Scientometrics, 88(2), 653-661.
Kang, M. (2010). Measuring social media credibility: A study on a Measure of Blog Credibility. Institute for Public Relations, 59-68.
Kuiken, J., Schuth, A., Spitters, M., & Marx, M. (2017). Effective headlines of newspaper articles in a digital environment. Digital Journalism, 5(10), 1300-1314.
Lai, L., & Farbrot, A. (2014). What makes you click? The effect of question headlines on readership in computer-mediated communication. Social Influence, 9(4), 289-299.
Lau, G. T., & Lee, S. H. (1999). Consumers' trust in a brand and the link to brand loyalty. Journal of Market-Focused Management, 4(4), 341-370.
Li, R., & Suh, A. (2015). Factors influencing information credibility on social media platforms: Evidence from Facebook pages. Procedia computer science, 72, 314-328.
Lockwood, G. (2016). Academic clickbait: articles with positively-framed titles, interesting phrasing, and no wordplay get more attention online. The Winnower, 3.
Mackay, J. B., & Bailey, E. (2012). Succulent Sins, Personalized Politics, and Mainstream Media’s Tabloidization Temptation. International Journal of Technoethics (IJT), 3(4), 41-53.
Milkman, K. L., & Berger, J. (2014). The science of sharing and the sharing of science. Proceedings of the National Academy of Sciences, 111(Supplement 4), 13642-13649.
Munger, K., Luca, M., Nagler, J., & Tucker, J. (2018). The Effect of Clickbait.
Pengnate, S. F. (2016). Measuring emotional arousal in clickbait: eye-tracking approach.
of Clickbait Headlines on User Responses.
Petty, R. E., & Cacioppo, J. T. (1979). Issue involvement can increase or decrease persuasion by enhancing message-relevant cognitive responses. Journal of personality and social psychology, 37(10), 1915.
Petty, R. E., & Cacioppo, J. T. (1986). The elaboration likelihood model of persuasion. In
Communication and persuasion (pp. 1-24). Springer, New York, NY.
Potthast, M., Köpsel, S., Stein, B., & Hagen, M. (2016, March). Clickbait detection. In European
Conference on Information Retrieval (pp. 810-817). Springer, Cham.
Roskos-Ewoldsen, D. R., & Roskos-Ewoldsen, B. (2009). Media priming: An updated synthesis. In Media effects (pp. 90-109). Routledge.
Russell, J. A., & Mehrabian, A. (1974). Distinguishing anger and anxiety in terms of emotional response factors. Journal of consulting and clinical psychology, 42(1), 79.
Self, C.C. (1996). Credibility. In M.B. Salwen & D.W. Stacks (Eds.), An integrated approach to communication theory and research (pp. 421-441). Mahwah, NJ.: Lawrence Erlbaum Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017). Fake news detection on social media: A
data mining perspective. ACM SIGKDD Explorations Newsletter, 19(1), 22-36.
Srivastava, J. (2013). Media multitasking performance: Role of message relevance and formatting cues in online environments. Computers in Human Behavior, 29(3), 888-895. Uribe, R., & Gunter, B. (2007). AreSensational'News Stories More Likely to Trigger Viewers'
Emotions than Non-Sensational News Stories? A Content Analysis of British TV News. European Journal of Communication, 22(2), 207-228.
Valkenburg, P. M., & Piotrowski, J. T. (2017). Plugged in: How media attract and affect youth. Yale University Press.
Voigt, P., & Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR) (Vol. 18). Springer.
Yoo, J. J. S. (2013). Real significance of online breaking news: examining the credibility of
