The BBC and Brexit: An analysis of the much-criticized coverage of
the EU membership referendum.
Master’s program of Communication Science - Graduate School of Communication Science Master’s Thesis - political communication
Supervisor: dr. Tom Powell Anne Mae van Tilburg, 10742948
29th June, 2018 Word count: 7460
Abstract
The EU membership referendum in June 2016 in the United Kingdom was a referendum with massive consequences. A leave vote could change the country drastically. Before referendum day, the public network of the UK (the BBC) received much criticism for their reporting on the campaign. Some critics argued that the BBC had put so much emphasis on being the impartial facilitator of a balanced debate between the remain- and the leave-camp that they failed to provide knowledge and substance. This master thesis aims to find out if there is a truth in the comments made by these critics. A content analysis of the articles of the BBC News website (N = 183) shows that the BBC indeed reported in a very balanced way about the referendum. The journalists took on a detached role and the content before the referendum was mainly focused on the electoral race instead of substance. The findings of this study contribute to the discussion on what is expected from journalism in today’s democracy, and during referenda in particular.
Keywords: referendum campaigns; political journalism; content analysis; EU membership referendum; Brexit; BBC; journalism.
The BBC and Brexit: An analysis of the much-criticized coverage of the EU membership referendum.
In spring 2016, the United Kingdom held a referendum that could transform their country drastically. The British citizens had to decide if their country had to leave or remain in the European Union. Leaving the EU could impact their economy, homeland affairs and the relationship with the world around them momentously. It was a choice with enormous risks, but also an opportunity to escape an institution where many citizens of the UK felt uncomfortable in. When such meaningful and complex political decisions have to be made by citizens, quality journalism is crucial.
In most modern democracies, the media form the bridge between citizens and politics. Multiple researchers discuss the importance of news media, and the values journalists should keep (for example Ferree, Gamson, Gerhard, & Rucht, 2002; Strombäck). However they represent sometimes different ideologies about journalism, they seem to agree with the following: the media should provide enough information to make sure citizens are sufficiently informed and the media should be impartial.
The goals to inform citizens well and act impartially are not only journalistic values by researchers like Strombäck (2005) and Ferree et al. (2002) but also values that stand at the base of many media organizations. For example, the BBC writes on its website that their goal as a public network is “To provide impartial news and information to help people understand and engage with the world around them” (BBC, n.d.). These two aspirations can be the basis of great journalism, but some argue that they can also hinder each other.
After the Brexit referendum, the BBC was criticized a lot for not offering enough qualitative information in the run-up to the referendum. Scholars Seaton (2016) and Payne (2016)
argued that the public media (BBC) exaggerated the focus on impartiality through balanced reporting and “had failed to provide the clarity that was so desperately needed” (Payne, 2016, para. 1). Also in the British media, the BBC got criticized (Bennett, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017). Historian Timothy Garton Ash (2016), specialized in the history of the European Union, stated in The Guardian: “You give equal airtime to unequal arguments, without daring to say that, on this or that point, one side has more evidence, or a significantly larger body of expert opinion, than the other” (para. 8). However, not everyone agreed with this. Others argued that the BBC has not been that impartial and was biased towards remaining in the EU (Panjwani, 2017).
This thesis aims to figure out if there is a truth in the criticism towards the BBC. The central research question is: Did the BBC focus on balanced coverage while their journalists took on a passive role, and did this lead to a lack of substance in the BBC coverage?
To investigate if the EU-referendum coverage of the BBC is genuinely impartial, the balance between content in favor of either remain or leave will be analyzed. Articles from the BBC News website will be used for this. To research the passivity of the coverage, the journalistic style of the BBC journalists will also be examined. This will tell if the BBC mainly functioned as a platform for campaigners to spread their message (Brody, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017; Payne, 2016; Seaton, 2016), or if the BBC has fulfilled its journalistic role of critically approaching the messages of those campaigners. Lastly, to say if there was indeed a lack of substance in the BBC content, the usage of frames in the articles will be analyzed too. Articles with a strategic game frame focus mainly on the ‘electoral race’ (Dekavalla, 2017) while articles with an issue frame focus on substantive information. The frequency of the usage of both frames can give an insight into the substantial value of the BBC articles.
Theoretical framework The role of the media at referenda
A referendum campaign differs from a regular electoral campaign. Whereas at regular elections citizens have to choose between people or parties, at referenda people have to make decisions about actual and often complex issues, usually simplified into a yes/no question. As a result, citizens need more substantive information to make a right decision, and it is the task of the media to provide this. The press is more powerful, as referendum campaigns are more about arguments than ‘regular’ electoral campaigns. (Wirth, Schemer, Wettstein, Friemel, Hänggly, & Siegert, 2010). Party and elite cues play less of a role at referendum campaigns than at ‘normal’ electoral campaigns (Schuck & De Vreese, 2008). Arguments and factual information, provided by the media, are therefore more important. Media coverage can influence the outcome of a campaign, but pre-existing beliefs also play a role (Schuck & De Vreese, 2008). When the media support one side in a referendum campaign, like in The Netherlands before a referendum about the European Constitution in 2005, people who support the other side felt more motivated in their choice (Schuck & De Vreese, 2009). People who were firmly against the European Constitution even saw ‘positive’ media frames that supported the constitution as frames that discouraged the emerge of a European Constitution (Schuck & De Vreese, 2012).
However the power of the media should, thus, not be overestimated, their ability to provide substantive information about the (sometimes complex) referendum question asks for responsibility. A critical assessment of the role of the journalist is therefore necessary. The next part will elaborate more on values and practices in (political) journalism.
Values in Journalism
In this thesis, the workings of the BBC before the EU membership referendum in the United Kingdom in 2016 are critically assessed. However, as pointed out by Althaus (2012), it is impossible to evaluate journalism without reflection on what is considered as ‘good’ or ‘bad’ journalism and to take into account that journalistic values are not identical for everyone. Some scholars argue that the role one sees for journalism can differ between the way people see democracy (Stömbäck, 2005) and the role of the ideal ‘public sphere’ in these democracies (Ferree, Gamson, Gerhard, Rucht, 2002).
The (news) media function as a public sphere, “where the exchange of information and views take place” and work as the place where “public opinion can be formed” (Habermas, Lennox, & Lennox, 1974, p.49). To inform people well, the media should function as a ‘marketplace of ideas’: a concept by John Stuart Mill who argues that people need access to a diverse spectrum of opinions to form their own opinion. Thus, to inform people well, various opinions and viewpoints should be represented in the media (Oster, 2013). Ferree et al. (2002) and Strömbäck (2005) both emphasize that this ‘marketplace of ideas’ can only work if all the different opinions and standpoints get equal attention. The media should provide enough information to make sure citizens have enough knowledge to make the right voting decision, and the media should be impartial.
The best way to describe the journalistic goal of the BBC is probably by the gatekeeper versus advocate model of Janowitz (1975). He distinguishes two types of journalists: Journalists who see themselves as ‘gatekeepers’ and journalists that see themselves as ‘advocates’. While gatekeepers find that journalists should function as experts in a field, need to act objectively and need to distinguish facts from opinions, advocates find the essential role of journalists to give groups in society a voice, especially the groups that need help. In a political discussion,
journalists that see themselves as advocates take part in the debate, while people who see themselves as gatekeepers report on the debate. Considering its mission statement, the BBC probably sees its journalists as gatekeepers. The BBC aims to be a platform for the various viewpoints in a discussion. However, the journalists stay detached.
The ideals as discussed above, of journalists as the arbitrators of a balanced political discussion, are also the values that while put into practice received much criticism during the EU referendum campaign (Brody, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017; Payne, 2016; Seaton, 2016). While it may seem like a proper journalistic ideal, these norms of objectivity and impartiality bring problems and disadvantages as well.
Objectivity, impartiality and balance in journalism
As discussed above, the gatekeeping role for journalists requires objective journalists (Janowitz, 1975), the BBC wants its journalists to be impartial (BBC, n.d.), and critics accused the BBC of focusing too much on balanced reporting (Brody, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017; Payne, 2016; Seaton, 2016). Objectivity, impartiality and balanced reporting are terms with different meanings but are closely connected.
Objectivity is a term with multiple definitions. However, there are roughly two ways of defining it. In the article of Skovsgaard, Albaek, Bro, & De Vreese (2013) a division is made between objectivity as a ‘moral idea’ and objectivity as a ‘practical tool’. Skovsgaard, et al. (2013) argue that full objectivity is not something a human being can achieve. Other research also shows that journalism as ‘an autonomous fourth estate’ is a hard ideology to put into practice. A survey study by Mellado and Van Dalen (2014), shows that journalists who have these kinds of ideals in their work have the most incongruence between their ideology and practice. Many external factors influence the construction of news (Hilgartner & Bosk, 1988),
like who is spreading the news (Molotch & Lester, 1975; Vos, 2014) and the routines of journalists (Reese & Shoemaker, 2016; Hilgartner & Bosk, 1988). Values can, thus, not always brought into practice.
Therefore, Skovsgaard et. al. (2013, p.32) operationalize objectivism in journalism in a more practical way. They describe objective practice with the following norms: ‘no subjectivism’ meaning that the opinions of a journalist may not influence the way a topic is presented; ‘balance’ meaning that all sides of a story are equally represented; ‘hard facts’ meaning that reporting should go ‘beyond the statements of the disputing sides’ and ‘gets to the concrete facts in the dispute’; and ‘value judgment’ meaning that ‘good journalism makes clear who has the better position in a dispute.’ Within this definition, objectivism means being impartial but intervening with value judgments when necessary at the same time.
As discussed before, the BBC finds being impartial an essential part of its organizational values. In the past, multiple studies have been done about impartiality and the BBC. For example, the BBC failed to be impartial in Islam-related topics (Flood, Hutchings, Miazhevich, & Nickels, 2011), and the BBC has had problems in retaining its values of impartiality through the rise of social media. The broadcaster saw that it was tempting to bring news found on social media during big events like terrorist attacks as breaking news, without checking if the BBC was exploited for advocating someone else’s agenda (Bélair-Gagnon, 2013). Also, another study on the BBC evening news focused on the expression of impartiality in the BBC content: balanced reporting. The emphasis on balance can mean that often the same parties and standpoints stand against each other, which leads to a ‘narrow view’ on the news. For example, when a member of the conservative party makes a statement, almost always a member of the Labour party is asked to react on this in the media. People with opinions that do not fit into this framework are not or less visible in BBC content. Also, regarding being impartial, the BBC often focuses mainly on
‘party-political conflict’ and fails to give context to stories. (Wahl-Jorgensen, Berry, Garcia-Blanco, Bennett, & Cable, 2016). This last point illustrates exactly the comments on the BBC during the EU membership referendum. Because the BBC wanted to be impartial, they focused too much on balance in their reports and neglected to interpret the statements of political stakeholders. Journalists did not intervene when necessary. Or, if you want to describe it with the norms of Skovsgaard et al. (2013): the journalists did not act subjective and reported in a balanced way, but they ignored the hard facts and the value judgment.
When a journalist or a journalistic organization sees itself mainly as a gatekeeper, they get similar kinds of criticism as the example mentioned above (Janowitz, 1975). For example, some argue that the impartial role of gatekeeping journalists can be seen as a way for the journalist to protect himself from criticism on the content of their messages. After all, they are just repeating someone else his opinion, and are therefore not accountable for the content (Janowitz, 1975). They fail to take on an active and responsible role in society (Stoker, 1995).
Because this study aims to find out if the BBC has been unable to take on this ‘responsible role’ before the EU membership referendum, we first have to find out if there was truly a focus on balanced reporting. Therefore, our first research question is:
RQ1: To what extent had the BBC an equal representation of arguments and voices for remaining and leaving the European Union?
To find out if the BBC did not take on a ‘responsible role in society’ (Stoker, 1995), it is aimed to find out if the journalists acted too passive. Did they fail to ‘go beyond the statements of the disputing sides’ and didn’t they make clear ‘who had the better position in the dispute,’ as mentioned by Skovsgaard et al. (2013) as norms for objective practice. To find this out, the
second research question is:
RQ2: To what extent did BBC journalists intervene in the coverage of the EU membership referendum?
Article substance: game- versus issue-framing
In earlier studies, researchers observed that when journalists emphasize impartiality and balance in their work, they sometimes use specific frames to deal with this. For example, to maintain balance, many journalists seem to use the conflict frame in their coverage. By utilizing a conflict frame, it is easier to represent the different sides of an issue equally, and to stay neutral (Bartholomé, Lecheler, & De Vreese, 2017). The conflict frame is also a standard frame in reports on the news on the European Union (Van Klingeren, Boomgaarden, & De Vreese, 2017).
A conflict frame emphasizes the disagreements between different groups or people in an issue. In the Brexit coverage, this means focusing on the differences between the remain- and the leave-camp. A frame that takes this one step further is the strategic game frame. Like the conflict game, the strategic game frame focuses on the differences between the parties, but besides that, also focuses on politics in terms of winning and losing. This frame can be defined as “A focus on questions related to who is winning and losing, the performances of politicians and parties, and on campaign strategies and tactics. This framing is often contrasted with a focus on political substance and issues.” (Aalberg, Strömbäck, & De Vreese, 2011, p.163).
The strategic game frame is also related to an impartial and balanced way of reporting. A study of Dekavalla (2016) found that to get a representation of the statements of each side of the spectrum, it is easy to present the different parties in an opposing way, and discuss their performance in ways of winning and losing. Because journalists judge the politicians on their
performance in the popularity race, and not based on their ideology or policy plans, there is less chance the journalists own personal opinions show through, and it is easier for journalists to act impartial (Dekavalla, 2016).
The strategic game frame is a very prominent frame in political reporting (Aalberg, Stromback, & De Vreese, 2011), mostly during election periods (Lawrence, 2000). That is logical, as elections are primarily about winning and losing. Especially at referendums where there are only two options: one side will win and one side will lose. However, the focus on competition often leads to the lack of substance. Just reporting on the game aspects of politics is superficial, and leads to less attention to the real important aspects of politics: policy and the ideas of politicians on how to deal with particular problems. In a ‘normative’ way, the usage of the strategic game frame is therefore wrong (Cappella, 2002). In an experimental study, it is also found that exposure to the strategic game frame leads to people who are less knowledgeable about politics (Valentino, Buhr, Beckmann, 2001). The use of primarily strategic game frames can also polarize policy attitudes (van Klingeren, Boomgaarden, & de Vreese, 2017).
The BBC has besides being impartial also the goal to inform people well (BBC, n.d.). The usage of the strategic game frame can, therefore, be an indication that the BBC fails to achieve this last aspect. It may means that the public network did not inform citizens to the maximum for a knowledgeable vote in the EU membership referendum.
An indicator of a substantive way of reporting can, on the other hand, be the issue frame. The issue frame focuses on "policy problems and solutions" and "the indications of this for the public" (Lawrence, 2000, p. 96). A frequent use of the issue frame in the EU membership referendum coverage can, therefore, indicate substantive journalism.
With research question one and two the journalistic style of the BBC is reviewed. However, the criticasters argued that as a result of this style, the articles about the EU
membership referendum had a lack of substance (Brody, 2016, Seaton, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017 Payne, 2018). The BBC “had failed to provide the clarity that was so desperately needed” (Payne, 2016, p. 100). To find out if the journalistic style of the BBC truly had an influence on the substance of the articles, the usage of the strategic game frame and the issue frame is compared.
RQ3: To what extent has the strategic game frame been used in the articles of the BBC before the EU membership referendum, compared with the issue frame?
Methods
To answer the research questions of this thesis, a quantitative content analysis has been done.
Sample
The BBC (The British Broadcasting Corporation) is the public audiovisual media organization in the United Kingdom. The BBC offers media on a broad spectrum of topics, including sports, entertainment, and news. They spread their content within various types of media, mainly via television, through radio and online.
Because not all the BBC content could be analyzed for this thesis, it is decided to focus on the news coverage of the BBC News website. The BBC News website is the most prominent news website in the UK, and has a market share of 30%. The website also gives us the possibility to research all the different formats in which the BBC presents its news: BBC News reports on the news while using texts, videos, and audio fragments. It is decided to analyze, therefore, all the linguistic content of the articles. This means that all the written text in the articles should be analyzed (including the titles, subheadings and picture and video captions), and that also all the
spoken text in the videos and in the audio fragments (if available) has to be analyzed too.
In the United Kingdom, there has been a discussion in the media about the pros and cons of European Union membership for years. However, it is chosen to only analyze the BBC News content on the EU membership referendum during the 'official campaign period' for the referendum. The government of the United Kingdom launched this specific time frame to give a start and an ending to the time in which politicians could campaign for leaving or remaining in the European Union. This period started ten weeks before the election day (23 June 2016) (House of Commons, 2016). The articles that are being selected for analysis are from the same period: from the 15th of April until one day before the referendum, the 22nd of June.
The website of BBC News does not contain a good search engine that makes it possible to collect all the articles about the EU referendum that have been published between the 15th of April and the 22nd of June. Therefore, the Google News search engine was used to select the sample. While using the search term ‘BBC News EU Referendum’ 196 articles were selected from the first 20 pages of the results. After these 20 pages, no BBC articles about the EU membership referendum were found for this specific period (pages 21 until 25 did not contain any news articles from the BBC News website). The sample in this thesis is, thus, in all probability a total population sample. It is not entirely sure that all the BBC news articles are present in the sample, but as Google News selects its articles in the order of relevance, it seems to be unlikely that significant articles are missing in the sample. Every week of the campaign period is well represented in the selection of articles (starting with 10 articles about the referendum in the first week of the campaign period, up to 37 in the last week before the referendum).
To be analyzed, the articles had to meet the following requirements: The article source is the BBC News website; the article is published between the 16th of April and the 22nd of June, and the article contains at least one element about the EU membership referendum or about the
UK leaving the EU. Of the 196 articles found through the Google News search, 183 articles met these requirements. The articles that did not meet these requirements were not analyzed.
Measures
The BBC news articles about the EU membership referendum are analyzed through the use of a codebook. The questions in the codebook are designed to answer the three research questions of this thesis. To mitigate the possibility of sub-optimal intercoder-reliability values, most research questions are measured with multiple variables. The first research question (To what extent had the BBC an equal representation of arguments and voices for leaving and remaining in the European Union?) is measured through question two to sixteen. The second and the third question of the codebook measure if the main message of the article is in favor of leaving or remaining in the EU. Questions four to six measure the balance in the number of arguments that are given in favor of leaving or remaining in the EU. Questions seven to 16 measure the balance in the presence of people who support the remain- and leave-side in the articles (and the differences in the types of people that are being mentioned by the BBC).
Questions 17 to 20 address the second research question: To what extent did BBC journalists intervene in their coverage of the EU membership referendum? Questions 17 and 18 measure if arguments in favor of leaving or remaining in the European Union are being dismissed by the authors of the articles. Question 19 measures if articles are predominantly descriptive or interpretive (Bartholomé, Lecheler, de Vreese, 2017). When an article is primarily descriptive, the article mainly describes someone else’s standpoints or opinions, without the interpretation of the journalist. When an article is mainly interpretative, it comprises mainly statements originated from the journalists’ own research. Question 20 measures if a journalistic evaluation of a politician is present in the article (Bartholomé, Lecheler, de Vreese, 2017). This captures whether
the journalist explicitly gave their opinion about a politician or campaigner.
The last research question measures if the issue frame and the strategic game frame are being used in the article. Question 21 measures the usage of the issue frame, by asking if there is a focus on policy problems and solutions, policy stands or legislation in the article (Aalberg, Strömbäck , & De Vreese, 2012). Question 22 measures if the strategic game frame is being used in the article, by asking if the article discusses the elections in terms of winning and losing, or if the article discusses strategies for winning the debates and election (Dekavalla, 2016).
Lastly, some control variables are being measured. Does the article contain a video (or audio fragment), what is the length of an article (measured in paragraphs) and is the article part of the category 'the EU referendum reality check’, in which statements made in the referendum campaign are fact-checked. For a full overview of the codebook, see appendix 1.
Coding procedure
The author coded all the articles of the sample. However, to guarantee the reliability of the variables, a random selection of approximately 10 percent of the articles (20 articles) was coded by another coder, to test the inter-coder reliability.
Before the second coder started with the 20 articles, she was given the training to get familiar with the codebook. She also had to do a trial of 5 articles, to see if some questions needed extra attention or explanation.
Through the analysis of the inter-coder reliability, it became clear that some questions are not reliable enough to be used. Questions two and three measuring the article balance, 17 and 18 measuring journalist intervention and question 20 measuring journalistic evaluations were not reliable enough for usage (see appendix 1 for the codebook.) All the other questions in the codebook had sufficient Krippendorff Alpha values (see Appendix 2 for the ICR values for each
variable), and can be used for analysis.
Analysis
To measure the balance in the articles, the mean of the number of arguments that are given in favor of remaining and leaving for each article are compared through a paired sample t-test. Also, the average number of people mentioned in the articles who endorsed remain or leave is compared through paired sample t-tests. To find out if the amount of arguments for each side have increased or decreased while coming closer to the referendum date, a single linear regression analysis has been done.
To be able to analyze the journalistic style in the articles of BBC news, the mean of the scale that tells if the behavior of the journalist was mainly interpretative or descriptive is calculated. As mentioned before, the BBC introduced the special ‘reality check’ section during the referendum campaign period. Because this section is specially designed for interpretation, we wanted to check the mean of the journalism style again with these reality check articles separated. The difference in the means of the article type (reality check or not) is compared through an independent sample t-test. Also, a regression analysis checked if the number of arguments given for remain or leave and the week of publishing influenced the journalistic style.
To analyze the frames that have been used in the BBC articles, the percentages of the articles in which the strategic game frame and the issue frame are present were compared through a McNemar test. To see if the frames cohered with the number of arguments that are given for remain or leave, four independent sample t-tests have been done. These analyses compared the average of the number of arguments between articles with a strategic game- or issue-frame and the articles without a strategic game- or issue frame. To check for the influence of article type (reality check or not) a chi-squared test is done.
Results Balance
The aim of the first research question is to find out if the BBC had equal representation of arguments and people supporting either leaving or remaining in the European Union in its articles. One answer to this question is provided by the number of arguments in the BBC News articles. A paired sample t-test shows that on average slightly more arguments in favor of remaining in the European Union (M=2.40, SD= 3.04) were given in the articles than arguments in favor of leaving the European Union (M=2.03, SD= 2.89). However, this difference in the amount of arguments for each side is not significant, t (182) = -1.64, p = .102, 95% CI [-.82, .07].
A regression analysis shows that there is a weak but significant increase in the number of arguments in favor of the remaining in the European Union as the referendum date drew closer. The model is significant, F(1,181) = 7.03, p = .01. Per week, there was an average increase of 0.22 arguments in favor of remaining in the articles, p = .01, 95% CI [.06, .38]. There is no significant regression model found for an increase or a decrease in the arguments in favor of leaving the European Union, F(1,181) = 1,72, p = .10. For the balance in the average of arguments given for each side per week, see figure 1.
Figure 1. The average amount of arguments given to each side per week.
To measure the balance in the BBC articles about the EU membership referendum, not only the number of arguments are taken into account. The amounts of people that are mentioned in articles for supporting remain or leave is also measured. The means are compared through a paired sample t-test. On average, slightly more people supported remain in the BBC articles than leave. However, this difference is not significant. Significantly more international politicians and experts that supported the remain side were mentioned in the BBC News articles. However, there is not a significant difference in the amount of UK politicians that support remain and leave in the BBC articles, and there is no significant difference between the amount of ‘other’ people (so people that are not (international) politicians or experts, such as celebrities and ‘normal’ citizens) mentioned in the articles (table 1).
Table 1.
Paired sample t-test on the average of endorsers per category for remaining and leaving in the European Union per article.
Category Leave Remain
M SD M SD t-test Politicians 1.09 1.34 1.30 3.32 -,87 International .05 0,35 .21 .81 -2,50* Politicians Experts .05 0,21 .38 .99 -4,68*** Others .40 1.33 2.10 20.97 -1,10 Total 1.73 3.86 3.85 21.20 -1,35 note. * p < .05, ** p<.01, *** p < .001. Df: 182.
Taken together, the findings show that the articles of the BBC News website about the EU referendum were on average pretty balanced. There is no significant difference in the average of arguments for the remain- and leave-side for each article, and in total, not significantly more people were named in the articles supporting one side. However, slightly more international politicians and experts were mentioned in the articles supporting remaining in the European Union.
To answer the second research question 'To what extent did BBC journalists intervene in the coverage of the EU membership referendum?,' the coders were asked if the style of the articles was predominantly descriptive or predominantly interpretative. On a five-point scale from -2 (very descriptive) to 2 (very interpretative), the articles of BBC News were predominantly descriptive (M= -.68, SD=1.58). Thus, journalists of the BBC behaved quite detached and passive in their reporting about the EU membership referendum.
In the campaign period before the European Union membership referendum, the BBC introduced the feature ‘The EU referendum reality check.’ In this section, the BBC checked (campaign-)statements about the (consequences) of leaving or remaining in the European Union. An independent sample t-test shows us that there is a significant difference in journalism style between articles that are part of the reality check section, or part of the ‘normal’ articles, t (38,27) = -11,93, p < 0,001, 95% CI [-2,98, -2,11], d = 0.16. Articles that are part of the EU referendum reality check score very high on the scale of journalism style (M=1.55, SD= .86), which means that they are very interpretative. All the articles that are not part of the EU referendum reality check section are on average much more descriptive (M=-1,00, SD= 1,38). These findings show us that journalists seem to be very careful about being interpretative in their regular reporting. Only in a section specially designed for interpretation, the journalists predominantly reported statements based on own research.
To see if the number of arguments in favor of leaving or remaining in the EU, and the time (in weeks) had an influence on the journalism style in the BBC articles, a regression analysis has been done. The model is significant, F(1, 179)= 3,82, p= 0,01. However, the amount of variance explained by the regression model is low, R²= 0,06, and all the results are not significant. Some trends that can be seen are that: the more arguments of others (by politicians, experts, etc.) that are used in an article the more descriptive the reporting style; arguments in
favor of leaving the EU have slightly more influence on the style of reporting than arguments in favor of remaining in the EU; and as the referendum date approaches articles become slightly more interpretative. As mentioned, however, these results were non-significant (see table 2).
Table 2.
Regression analysis on influence of arguments in favor of leaving the EU, arguments in favor of remaining in the EU and weeks on role of journalist.
B Std. error β
Constant -.52 .31
Arguments leave -.08 .05 -.15
Arguments remain -.07 .04 -.14
Weeks .04 .04 -.05
note. * p < .05, ** p<.01, *** p < .001. B = unstandardized regression coefficient. β = standardized regression coefficient.
In response to research question two, these results show that BBC journalists adopted a detached style. In the next section, the attention goes to the substantiveness of the articles, by analyzing the usage of the strategic game frame and the issue frame.
The aim of the third research question is to find out if the strategic game frame or the issue frame is more frequently used in the BBC coverage. An exact McNemar’s test shows a significant difference in the frequency of the usage of the strategic game frame and the issue frame, p=.002. In the BBC articles before the EU membership referendum, 48,90% of the articles contain a strategic game frame, and 31,10% of the articles contain an issue frame. Articles could contain both an issue frame and a strategic game frame. 13,50% of the articles have an obvious focus on policy problems, solutions, proposals and legislation, and a clear focus on winning or losing the election at the same time, and contained both frames.
To find out if the frames cohere with the number of arguments that are given for either remain or leave, an independent sample t-test is done. The differences are small and again not significant, but articles that contain no issue frame, are often (slightly) more in favor of remaining in the European Union than articles that do not contain an issue frame. Moreover, articles with a strategic game frame are slightly more in favor of remaining in the European Union than articles that do not contain a strategic game frame. Hence, although these differences are not significant, it seems like articles that contain more arguments in favor of remaining in the European Union are slightly less informative and substantial than articles that are more in favor of leaving the European Union (Table 3 and 4).
To see if there is a relationship between the usage of the frames and the type of article, a chi-squared test was performed between the two frames and the article type (part of the ‘EU referendum reality check’ or not). The relationship between the issue frame and the article section is significant X2 (1) = 29,83, p < 0,001. There is a difference in the number of times an issue frame is used for a reality check article and a ‘normal’ article. 77,3% of the articles that are part of the ‘EU referendum reality check’ contained an issue frame, while only 21,4% of the ‘normal’ articles contained an issue frame. There is a moderate association between the article section and
the usage of the issue frame, Cramers V = 0,41. There is also a significant relationship between article section and the strategic game frame, X2 (1) = 11,89, p < 0,001. There is a difference in the amount of time a strategic game frame is being used for a reality check article and a ‘normal’ article. 13,6% of the articles that were part of the EU referendum reality check contained a strategic game frame, while 52,8% of the ‘normal’ articles contained a strategic game frame. There is a weak association between the article type and the strategic game frame, Cramers V = 0,26.
To summarize, the strategic game frame is used in almost half of all the articles, the issue frame in less than a third. Whether the articles mainly favored remain or leave did not (significantly) affect these numbers. When the issue frame is being used, the article is most of the time part of a section specially made for fact-checks. In the ‘normal’ articles, the issue frame was hardly visible. This indicates that the reporting of the referendum of the BBC was very superficial: mostly focusing on the electoral ‘race,’ and not focusing on the substance, on what a Brexit would mean for the United Kingdom.
Table 3.
European Union while using the issue frame.
No issue frame Issue frame t-test
Leave arguments 2.03 2.02 .03
Remain arguments 2.48 2.19 .58
note. * p < .05, ** p<.01, *** p < .001, df: 181.
Table 4.
Independent sample t-test for the means of arguments given for remaining in and leaving the European Union while using the strategic game frame.
No strategic Strategic game t-test
game frame frame
Leave arguments 1.93 1.93 .03
Remain arguments 2.08 2.14 -.49
note: * p < .05, ** p<.01, *** p < .001. df: 181.
Discussion
balanced way between the leave- and the remain-camp. Some people argued that the BBC focused too much on balanced reporting and that this focus was at the expense of other journalistic norms of informing people sufficiently (Brody, 2016, Seaton, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017 Payne, 2018). Although we can’t say something about the motivations of the journalists with a content analysis, we can see that the BBC coverage was predominantly balanced regarding arguments and speakers for each side. It seems like the BBC indeed emphasized reporting in a balanced, and through that, objective and impartial way.
The second question focused on the extent of the journalistic interventions in the articles. Skovsgaard, Albaek, Bro, & De Vreese (2013) argue that objective practice as a journalist does not only demand being not subjective and balanced reporting but also requires to go ‘beyond the statements of the disputing sides’ and telling ‘which side has the better position in a dispute.’ So do the BBC news articles mainly contain out of statements of campaigners, or does the journalist contextualizes the campaign messages and complements the article with factual journalistic research? The findings of the content analysis show that the journalistic style is very detached: articles comprise mainly out of supporters of remain and leave reacting on each other, and journalistic research that gives statements context and substance are not very visible. The most journalistic intervention is visible in articles that are specially designed for it: the reality check articles. In the ‘regular’ articles, the BBC journalists struggle to achieve impartiality but be substantive at the same time. Although the answer to research question two gives us a clear image of the behavior of the journalists that work for the BBC, detached journalism does not necessarily mean ‘bad’ journalism. The critics of the BBC argued that the behavior of the journalists resulted in a lack of substantive coverage about the referendum (Brody, 2016, Seaton, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017 Payne, 2018). To find out if the detached style of the journalists goes hand in hand with not-substantive content, the usage of the strategic game frame versus the
issue frame is analyzed too.
When a strategic game frame is used in media coverage, it means that politics is portrayed as a ‘game’, and when an issue frame is used in an article, there is a focus on policy problems and solutions. In general, the strategic game frame is much more visible in the articles than the issue frame. The issue frame is mostly visible in articles that are reserved for journalistic context: the reality check articles. In the regular BBC content, there is an even more significant lack of substance about the referendum.
In the theory section I stated that the role conception of BBC journalists could be best described as impartial ‘gatekeepers’ (BBC, n.d.; Janowitz, 1975). The findings of this study draw an image of the BBC that is very much similar to the image of the gatekeeping journalist. The journalists have to stay detached, impartial and objective, and the BBC should function as the ‘public sphere’ (Habermas, Lennox, & Lennox, 1974) where various standpoints of society get equal attention. Skovsgaard, Albaek, Bro, & De Vreese (2013) argue, however, that when journalists try to bring ‘objectivity’ into practice, it should not mean passive journalism. It is essential to bring facts to the discussion and to be clear about who speaks the truth. The journalists of the BBC could have, in this sense, performed better before the EU membership referendum. They did not frequently intervene in the debate, and many articles were not very substantive. Especially regarding a referendum, access to adequate information to make a knowledgeable decision is crucial (Wirth, Schemer, Wettstein, Friemel, Hänggly, & Siegert, 2010).
This thesis aimed to find out if the comments on the functioning of the BBC were true (Brody, 2016, Seaton, 2016; Jack, 2016; Jenkins, 2017; Lusting, 2017 Payne, 2018). However some of the findings are small and not significant, some were taken into account anyway. In a full population study like this one, where all the material is being analyzed, these non-significant
findings can still be relevant. The outcomes of the study gave a very clear image of the functioning of the BBC, however, one can only speculate about the motivations of the journalists. For example, to find out if BBC journalists really see themselves as gatekeepers, other research, like survey studies or qualitative research on how BBC journalists perceive their role, is needed. Unfortunately, the codebook that was used is not perfect. Some variables could not be used due to low inter-coder reliability values. Some research questions could, because of this, only be answered with minimal information, like the second one. Further research should focus more on making the more interpretative questions of the codebook more reliable. Also, when interpreting the findings, one should keep in mind that only the online content of BBC News has been analyzed. The BBC has a major presence on other platforms to spread their news (TV and radio) and the results on these platforms can be different.
Despite these limitations, the findings of this study contribute to the discussion of what journalism means in a democracy and a developed media landscape. This study may be about the performance of specifically the BBC during the EU membership referendum, but the findings stand for a wider, and more global discussion about what ‘good’ journalism is. Many scholars, like Strombäck (2005), Ferree, Gamson, Gerhard, & Rucht, (2002), and Jonowitz (1975) have reflected on this and they all seem to agree that journalists are the people that decide how the public debate is conducted. However, media have changed over the years, and journalists are not the only ones who choose who gets a voice in the public debate or not. This developed media landscape may influence what people expect from journalists too.
What is the reason that the BBC has received so much criticism on their performance recently, while their values have been the same for a long time? Journalists are not the only gatekeepers of the (political) debate anymore. Through social media, everyone can speak publicly, and everyone has the opportunity to be heard. The journalist as an arbitrator of the
(political) debate is not very necessary anymore, because the debate has shifted to another place. And perhaps that is the exact reason why people have different requests for the added value of journalists. As mentioned by Strömbäck (2005), Ferree et al. (2002) and Althaus (2012), the ideal of ‘good’ journalism always depends on the shape of democracy and the public sphere. When the public sphere changes, what is expected from journalists changes too. In a public sphere where journalists are not the only facilitators of the debate, the focus of journalists should go further than only enabling a balanced discussion, in which each opinion gets an equal amount of attention. Instead of directing the debate, journalists should serve the debate with facts and substance. To stay relevant and to serve democracy, journalists should think critically about their role. Being impartial remains important, but it should not be at the expense of substance. Especially not in situations where being knowledgeable is essential, like at referenda.
References
Aalberg, T., Strömbäck, J., & De Vreese, C. H. (2011). The framing of politics as strategy and game: A review of concepts, operationalizations and key findings. Journalism, 13(2), 162-178. doi:10.1177/1464884911427799
Althaus, S. L. (2012). What’s good and bad in political communication research? Normative standards for evaluating media and citizen performance. The SAGE handbook of political communication, 97-112. doi: 10.4135/9781446201015n.9
Bartholomé, G., Lecheler, S., & de Vreese, C. (2017). Towards A typology of conflict frames Substantiveness and interventionism in political conflict news. Journalism Studies, 11(6), 1-23. doi: 10.1080/1461670X.2017.1299033
BBC (n.d.) Inside the BBC: Public Purposes. BBC. Retrieved from http://www.bbc.co.uk/corporate2/insidethebbc/whoweare/publicpurposes
Brody, J. E. (2016)/ The BBC’s fixation on ‘balance’ skews the truth. The Guardian. Retrieved from https://www.theguardian.com/commentisfree/2016/sep/03/bbc-impartiality-skewers-evidence-based-facts
Cappella, J. N. (2002). Cynicism and social trust in the new media environment. Journal of Communication, 52(1), 229-241. doi: 10.4135/9781446201015.n9
Bélair-Gagnon, V. (2013). Revisiting impartiality: Social media and journalism at the BBC. Symbolic interaction, 36(4), 478-492. doi: 10.1002/SYMB.72
Dekavalla, M. (2016). Issue and game frames in the news: Frame- building factors in television coverage of the 2014 Scottish independence referendum. Journalism, 38(6), 1-20. doi: 146488491667423. 10.1177/1464884916674231
Ferree, M. M., Gamson, W., Gerhard, J., & Rucht, D. (2002). Four models of the public sphere in modern democracies. Theory and Society, 31(3), 289-324. doi: 10.1023/a:1016284431021
Flood, C., Hutchings, S., Miazhevich, G., & Nickels, H. (2011). Between impartiality and ideology: The BBC's paradoxical remit and the case of Islam-related television news. Journalism Studies, 12(2), 221-238. doi: 10.1080/1461670X.2010.507934
Garton Ash, T. (2016). The BBC is too timid. Being impartial on the EU is not enough. The Guardian. Retrieved from https://www.theguardian.com/commentisfree/2016/apr/01/bbc-brexit-too-timid-impartial-on-eu-not-enough
Graham, T., Jackson, D., & Broersma, M. (2016). New platform, old habits? Candidates’ use of Twitter during the 2010 British and Dutch general election campaigns. New media &
society, 18(5), 765-783. doi: 10.1177/1461444814546728
Habermas, J., Lennox, S., & Lennox, F. (1974). The public sphere: An encyclopaedia article (1964). New German Critique, (3), 49-55. doi: 10.2307/487737
Hilgartner, S., & Bosk, C. L. (1988). The rise and fall of social problems: a public arenas model. American Journal of Sociology, 94(1), 53-78. doi: 10.1086/228951
House of Commons Library. (25 February, 2016). Briefing Paper: The EU referendum campaign (Report No. 7468). Retrieved from:
http://researchbriefings.parliament.uk/ResearchBriefing/Summary/CBP-7486
Jack, A. (2016). Brexit Briefing: Balance at the BBC. The Financial Times. Retrieved from https://www.ft.com/content/3236e982-5fb6-11e6-ae3f-77baadeb1c93
Janowitz, M. (1975). Professional models in journalism: The gatekeeper and the advocate. Journalism Quarterly, 52(4), 618-626. doi: 10.1177/107769907505200402 Jenkins, S. (2017). The best way to tackle BBC bias is make it plain for all to see. The Guardian.
Retrieved from https://www.theguardian.com/commentisfree/2017/apr/05/tackle-bbc-bias-make-plain-see-nick-robinson
van Klingeren, M., Boomgaarden, H. G., & de Vreese, C. H. (2017). Will conflict tear us apart? The effects of conflict and valenced media messages on polarizing attitudes toward EU immigration and border control. Public Opinion Quarterly, 81(2), 543-563. doi:10.1093/poq/nfw051
Lawrence, R. G. (2000). Game-framing the issues: Tracking the strategy frame in public policy news. Political Communication, 17(2), 93-114. doi:10.1080/105846000198422
Lusting, R. (2017). Robin Lustig: The BBC has confused balance with impartiality, it should never be impartial between good and evil. Press Gazette. Retrieved from
Mellado, C., & Van Dalen, A. (2014). Between rhetoric and practice: Explaining the gap between role conception and performance in journalism. Journalism Studies, 15(6), 859-878. doi: 10.1080/1461670X.2013.838046
Molotch, H., & Lester, M. (1975). Accidental news: the great oil spill as local occurrence and national event. American Journal of Sociology, 81(2), 235-260. doi:10.1086/226073
Oster, J. (2013). Theory and doctrine of ‘media freedom’as a legal concept. Journal of Media Law, 5(1), 57-78. doi:10.5235/17577632.5.1.57
Panjwani, A. (2017). BBC hits back at Sun's claim of anti-Brexit bias on Andrew Marr and Sunday Politics shows. Pressgazette. Retrieved from http://www.pressgazette.co.uk/bbc-hits-back-at-suns-claim-of-anti-brexit-bias-on-andrew-marr-and-sunday-politics-shows/ Payne, R. (2018). Brexit and the british media. The Round Table, 107(1), 109-110. doi:
10.1080/00358533.2018.1424083
Pedersen, R. T. (2014). News media framing of negative campaigning. Mass Communication and Society, 17(6), 898-919. doi: 10.1080/15205436.2013.858749
Reese, S. D., & Shoemaker, P. J. (2016). A media sociology for the networked public sphere: The hierarchy of influences model. Mass Communication and Society, 19(4), 389-410. doi: 10.1080/15205436.2016.1174268
Schuck, A. R., & De Vreese, C. H. (2008). The Dutch no to the EU constitution: Assessing the role of EU skepticism and the campaign. Journal of Elections, Public Opinion and
Parties, 18(1), 101-128. doi: 10.1080/17457280701858656
Schuck, A. R., & de Vreese, C. H. (2009). Reversed mobilization in referendum campaigns: How positive news framing can mobilize the skeptics. The International Journal of
Schuck, A. R., & de Vreese, C. H. (2012). When good news is bad news: Explicating the moderated mediation dynamic behind the reversed mobilization effect. Journal of
Communication, 62(1), 57-77. doi: 10.1111/j.1460-2466.2011.01624.x
Seaton, J. (2016). Brexit and the Media. The Political Quarterly, 87(3), 333-337. doi:
10.1111/1467- 923X.12296
Skovsgaard, M., Albæk, E., Bro, P., & de Vreese, C. (2013). A reality check: How journalists’ role perceptions impact their implementation of the objectivity norm. Journalism, 14(1), 22-42. doi: 10.1177/1464884912442286
Strömbäck, J. (2005). In search of a standard: Four models of democracy and their normative implications for journalism. Journalism Studies, 6(3), 331-345. doi:10.1080/14616700500131950
Valentino, N. A., Buhr, T. A., & Beckmann, M. N. (2001). When the frame is the game: Revisiting the impact of “strategic” campaign coverage on citizens' information retention. Journalism & Mass Communication Quarterly, 78(1), 93-112. doi:
10.1177/107769900107800107
Vos, D. (2014). Which politicians pass the news gates and why? Explaining inconsistencies in research on news coverage of individual politicians. International Journal of Communication, (8), 2438–2461. doi: 1932–8036/20140005
Wahl-Jorgensen, K., Berry, M., Garcia-Blanco, I., Bennett, L., & Cable, J. (2016). Rethinking balance and impartiality in journalism? How the BBC attempted and failed to change the paradigm. Journalism, 18(7), 781-800. doi: 10.1177/1464884916648094
Wirth, W., Matthes, J., Schemer, C., Wettstein, M., Friemel, T., Hänggli, R., & Siegert, G. (2010). Agenda building and setting in a referendum campaign: Investigating the flow of
arguments among campaigners, the media, and the public. Journalism & Mass
APPENDIX 1: Code Book
Note: It is required that all the coders have basic knowledge about the 2016 European Union membership referendum in the United Kingdom. However, if coders do not feel certain about specific terms or topics, they are allowed to use the Internet to help them.
STEP 1: Selecting the articles that need to be coded:
It is expected that all the provided articles are useful for our research. However, articles do only need to be coded if:
● The article contains at least one element about the European Union membership referendum in the United Kingdom.
OR/AND
● The article contains at least one element about the United Kingdom leaving the European Union.
AND
● The source of the article must be the BBC NEWS website
Q1: Does the article meet these requirements?
0. No 1. Yes
If it does not, you don’t have to continue coding this article.
If it does: Please read the article in full length first. If there are videos in the article, please watch them as well before you start the coding.
STEP 2: Media balance
Q2: To what extent is the main message of the article in favor of LEAVING THE EUROPEAN UNION?
If there is a main argument that gets most attention in an article that is in favour of leaving the EU, it is often placed in the title of the article, or in one of the first paragraphs. When there is not a main argument in the article, the main message is in favour of leaving the EU if most of the arguments are supporting arguments for leaving the EU, or if the analysis in the article supports leaving the EU.
(0) Not at all in the advantage (1) Slightly in the advantage (2) Very much in the advantage
Q3: To what extend is the main message of the article in the advantage of REMAINING IN THE EUROPEAN UNION?
If there is a main argument that gets most attention in an article that is in favour of remaining in the EU, it is often placed in the title of the article, or in one of the first paragraphs. When there is not a main argument in the article, the main message is in favour of remaining in the EU if most of the arguments are supporting arguments for leaving the EU, or if the analysis in the article supports leaving the EU.
(0) Not at all in the advantage (1) Slightly in the advantage (2) Very much in the advantage
The next couple of questions are about arguments.
In this codebook, arguments are statements that support leaving the EU or that support remaining in the EU, in terms of the consequences of leaving or remaining, or the pros and cons of being part of the European Union. Examples of arguments are:
- When the UK leaves the EU, there will be more money available for the NHS. (Supports leaving the EU)
- Leaving the EU would be a danger for the British economy. (Supports remaining in the EU)
- The EU is corrupt. (Supports leaving the EU)
Dismissing concerns of others also count as arguments. For example:
- Economic concerns about leaving the EU are fundamentally wrong. (Supports leaving the EU)
Simply telling who does and does not support Brexit is not an argument. So this is not an argument:
- International institutions like the IMF and NATO think the UK should remain in the EU.
This is not an argument: it only reflects standpoints, and does not tell why the IMF or NATO think the UK should stay in the EU.
Sub-arguments count as new arguments. For example:
When the UK leaves the EU, there will not be more money available for the NHS” could be a ‘main’ argument, and “the costs of separating from the EU will be far greater than the savings made” could be a sub-argument that support the other argument. Sub arguments count in this codebook as SEPARATE arguments. So in the example above two arguments are given.
Another example:
“On security, she said EU membership enabled the UK to access EU-wide information, such as criminal records, to allow the UK to turn away serious criminals and terrorists at the border, fast-track the extradition of offenders and simplify the deportation of prisoners.”
This paragraph contained four arguments
When the same arguments are repeated in an article, they do not count as new arguments. Do not forget to count the arguments in videos as well.
Q4: How many arguments does the article entail? ……… Arguments
Q5: How many arguments in the article are from people that are advocates of LEAVING THE EU?
………. Arguments
Q6: How many arguments in the article are from people that are advocates of REMAINING IN THE EU?
……….. Arguments
The next couple of questions are about endorsements:
In this codebook, we mean with endorsers people who are backing remain or leave. Examples of endorsements are:
- National Farmers' Union backs staying in EU.
- The IMF and Nato think the UK should remain in the EU.
- Boris Johnson has accused US President Barack Obama of "hypocrisy" over his support for the UK remaining in the EU. The London mayor, who backs EU exit, told the BBC the Americans "wouldn't dream of sharing [their] sovereignty" as the UK had done. Do not forget to code the endorsers in video’s as well.
Q7: How many endorsements in the article for LEAVING THE EU are from national politicians or EU referendum campaign groups (for example Vote Leave/LEAVE.EU)?
The UK politicians in this questions can be former politicians (like former foreign secretary’s, but they have to be alive. British politicians that have passed away (like Winston Churchill) have to be placed under others.
When a politician is cited in an article as part of a referendum campaign group, you only code the politician. The campaign-group does not count as a new ‘politician. If only the campaign group is cited (for example: Vote Leave thinks….) then you should count Vote Leave as a politician.
Q8: How many endorsements in the article for REMAINING IN THE EU are from national politicians or EU referendum campaign groups (Britain Stronger in Europe)?
The UK politicians in this questions can be former politicians (like former foreign secretary’s, but they have to be alive. British politicians that have passed away (like Winston Churchill) have to be placed under others.
When a politician is cited in an article as part of a referendum campaign group, you only code the politician. The campaign-group does not count as a new ‘politician. If only the campaign group is cited (for example: Vote Leave thinks….) then you should count Vote Leave as a politician.
Q9: How many endorsements in the article for LEAVING THE EU are from international politicians (so not from the UK)?
Q10: How many endorsements in the article for REMAINING IN THE EU are from international politicians (so not from the UK)?
Q11: How many endorsement in the article for LEAVING THE EU are from institutions or experts (think: the IMF, NATO, Universities, researchers, interest groups, etc.)
Q12: How many endorsement in the article for REMAINING IN THE EU are from institutions or experts (think: the IMF, NATO, Universities, researchers, interest groups, etc.)
Q13: How many endorsements in the article for LEAVING THE EU are from others (so no politicians or institutions?
Q14:How many endorsements in the article for REMAINING IN THE EU are from others (so no politicians or institutions?
Q15: How many people back LEAVING THE EU in this article in total? Q16: How many people back REMAINING IN THE EU in this article in total?
STEP 3: Journalistic interventions
Q17: Is (at least one) of the arguments made IN FAVOUR OF LEAVING THE EU dismissed by the author?
This is not done by a quote of someone else. An example could be that the author of the article states that a statement made by someone else is not true.
0. No 1. Yes, one 2. Yes, multiple
Q18: Is (at least one) of the arguments made IN FAVOUR OF REMAINING IN THE EU dismissed by the author?
This is not done by a quote of someone else. An example could be that the author of the article states that a statement made by someone else is not true.
0. No 1. Yes, one 2. Yes, multiple
Q19: To what extent do you think the article is predominantly descriptive, or predominantly interpretative?
Descriptive articles are articles that mainly comprise of statements of others Interpretative articles are articles that mainly comprise statements that originate from the journalist’s own research.
(-2) Predominantly descriptive (-1) Slightly descriptive
(0) Not descriptive / not interpretative (1) Slightly interpretative
(2) Predominantly interpretative
Q20: Is a journalistic evaluation of a politician present in the article?
In this codebook, a journalistic evaluation means that in the article a certain statement is made about the actions or functioning of a politician by the journalist. The statement is not made by someone else (so,for example, not by a quote of a politician).
For example: the journalist states that something a politician has said is not realistic. 0. No 1. Yes STEP 4: Frames
You say that an ‘issue frame’ is being used when:
● There is a focus in the article on policy problems and solutions,
● There is a focus in the article on descriptions of politicians’ policy stands
● There is a focus in the article on implications of different proposals or legislations,
An issue frame is not used when an article only repeats the opinion of a politician/stakeholder/campaigner, without elaborating on it.
0. No 1. Yes
Q22: Is the ‘Strategic Game’ frame used in the article?
The Strategic Game frame is used when:
● The author discusses the election in terms of winning or losing in elections
○ For example through discussing polls and politicians’ or parties ‘standing in the polls’.
● The article discusses strategies for winning the debates/election. ○ For example: campaign strategies are discussed.
0. No 1. Yes
STEP 5: Article features
For example: this part contains 7 paragraphs.
……
Q24: Does the article contain a video?
0. No 1. Yes
Q25: Is the article part of the ‘EU referendum reality check’?
0. No 1. Yes
Appendix 2 – inter-coder reliability
table.
Table 1.
Krippendorff Alpha for the inter-coder reliability for the analyzed variables.
Krippendorff Alpha Q1: Requirements 1 Q4: Arguments total .81 Q5: Arguments leave .81 Q6: Arguments remain .80 Q7: UK politicians leave .83 Q8: UK politicians remain .91
Q9: International politicians leave 1
Q10: International politicians remain .78*
Q11: Experts leave 1
Q12: Experts remain 1
Q13: Others leave 1
Q14: Others remain .90
Q15: Total endorser leave .82
Q16 Total endorser remain .94
Q19: Journalistic style .93
Q21: Issue frame 1
Q22: Strategic game frame .88
Q23: Paragraphs .90
Q24: Video 1
Q25: Reality check 1
Note. *The ICR value should me above .80. However, as this value is close and is part of a series of questions with sufficient ICR values, it is decided to use this variable anyway.
