New Public Management in the European Commission:
The EU International Cooperation and Development Results Framework
Name: Frank Tijsterman Student number: 10175121
Thesis supervisor: dr. G. Georgakopoulos Date: 26 June 2017
Word count: 17,639
MSc Accountancy & Control, specialisation Control
Statement of Originality
This document is written by student Frank Tijsterman who declares to take full responsibility for the contents of this document.
I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.
The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.
Abstract
As new public management reforms are being implemented in the European Commission, this thesis aims to gain an in-depth understand on how New Public Management ideas impacted the implementation of one such reform, the EU International Cooperation and Development Results Framework. Moreover, the effect of the Results Framework on governance and performance within the Directorate-General will be examined. This will be done by using a qualitative research method to answer the research question, conducting six interviews with professionals working with the Framework. The findings show that, although the reform had the dual objective increasing accountability and management opportunities, the actual implementation focused mainly on accountability, showing a limited impact of NPM during the implementation Further, the findings suggest that, while governance and performance have been relatively little impacted, due to the nature of the Results Framework as a hybrid this might change in the coming years.
Contents
1 Introduction ... 5
2 Literature Review ... 8
2.1 Governmentality and hybridisation ... 8
2.2 New Public Management ... 9
2.3 ‘Hard’ vs ‘Soft’ management styles ... 12
2.4 Accountability ... 15
2.5 European Commission reforms ... 17
2.6 Chapter Summary ... 19
3 Research methodology ... 20
3.1 Research method ... 20
3.2 Context case study ... 22
3.3 Research design ... 24
3.4 Chapter Summary ... 26
4 Findings and discussion ... 28
4.1 Governmentality and hybridisation ... 28
4.2 New Public Management ... 31
4.3 Management styles ... 34
4.4 Accountability ... 37
4.5 Summary Chapter ... 39
5 Discussion and Conclusion ... 40
5.1 Discussion ... 40
5.2 Conclusion... 42
5.3 Avenues for further research and limitations ... 43
1 Introduction
In 1991 Cristopher Hood published his ‘A Public Management for all seasons?’ in which he invented the term New Public Management (NPM), a term with which he describes the changes that happened in the 1980s in the public sector. During this period, a newfound focus was put on aspects commonly found in the private sector, such as efficiency, financial control and target setting (Hood, 1995). Following the invention of NPM, it has become a widespread theory, often used to explain reforms in the public sector, as well as assessing whether these reforms have been successful. It has both become a field of study in itself and, simultaneously, been used in many other fields to describe the economisation of government (Suzuki, 2003). In the research on NPM most studies have focused on the broad implementation of new public governance or theoretical underpinnings of NPM (Pollitt and Dan, 2011). Some researchers have tried to explore the topic more deeply by using the idea of hybrids (Kurunmaki and Miller, 2006; Thomson et al. 2014), looking at specific systems and their interaction with both the policy and action level. This has the benefit of both providing an in-depth research of the case they are studying, as well as clear empirical evidence on how NPM influences specific situations.
NPM ideas has often been a large influence on government reform. For example, New Zealand in the 1980s and 1990s implemented large public management reforms fashioned after new financial management ideas, private sector management tools and focusing on outcomes (Pallot, 1998). But while such overarching goals and objectives are often found in government reforms, it is the hybrids that are the link between the general theory and the actions taken on the ground (Kurunmaki and Miller, 2011). Kurunmaki and Miller (2006) argue that this “hybridisation of expertise, and of organisational forms and processes, is increasingly prevalent”, but that the performance assessment of such hybrids is still lacking. They, and others, have tried to fill this gap this, but additional research is still needed (Kurunmaki and Miller, 2011). This lack of empirical analysis is corroborated by Pollit and Dan (2011), who found in their meta-study that most research has focused on the broad implementation of new public governance or theoretical underpinnings of NPM, failing to provide substantive evidence on the NPM impact. Moreover, those studies that have done empirical analysis mostly discussed the implementation of NPM, failing to focus more deeply on analysis of output changes due to NPM, while even fewer concentrated on NPM implementation outcomes or results (Pollitt and Dan, 2011).
One of the government reforms in the last two decades was initiated in 1999 by Neil Kinnock, vice-president of the European Commission (EC), afterwards called the Kinnock reform (Trondal, 2012). The reform project has generally been seen as push towards NPM, which can already be found in the first theme of the Kinnock reform: to develop a ‘culture based
on service’ (Ellinas and Suleiman, 2010: 710). This new focus on service is one of the pillars of NPM, as citizens are now seen as a substitute for customers, which results in an increased importance on service. But while this may have seen as an efficient solution for Kinnock, the clash between the traditional Weberian-bureaucratic style of the EC and NPM could not be avoided, further increasing difficulties already arising from the difficult to combine objectives of both efficiency and accountability (Ellinas and Suleiman, 2010). Some research on these reforms has been done (Levy, 2006; Ellinas and Suleiman (2008); Cini, 2004; Knill and Balint, 2008), but calls for new reforms in the European Commission have been made and on those new public reforms few studies have focused. In these calls the ideas of NPM have continued to exist can be seen in how the Commission views its administration: modern, based on the principles of efficiency, transparency and accountability (EC, 2014).
One of the outcomes of these calls within the EC is the EU Results Framework within the Directorate-General of International Cooperation and Development, DG DEVCO. The Framework structured around three levels reporting and monitoring the results of the different projects and processes of DG DEVCO (EC, 2015). The Framework can be seen as a hybrid between the general, theoretical calls on accountability and a need for management tools in the European Commission on the one hand and the execution of projects and processes of the EC on the other. Therefore, the research question of this thesis is:
How have New Public Management ideas impacted the implementation of the EU International Cooperation and Development Results Framework and what has its effect been on governance and performance?
This paper will contribute to existing literature in multiple ways. First, this study will add to the increasing body of literature concerning hybrids, specifically their implementation process and performance. Moreover, this research will not only show the broad implementation of new public governance, but will go into detail into a specific reform. Additionally, this reform will be examined both for its theoretical underpinnings, as is often done before, and also in the impact on governance and performance. Finally, this research adds to the limited body of literature on European Commission reforms, especially reforms initiated after the Kinnock reforms in the early 2000s.
To answer the research question in-depth interviews with professionals working in DG DEVCO will be performed and supplemented by documentary analysis, to gain extensive knowledge on the workings of the Framework on different levels. Access was provided via a student colleague and person to person referrals.
To investigate this topic, the thesis will start in chapter two with a literature review providing an overview on New Public Management, the arguments discussed between scholars and its relation to governmentality and hybridisation. The next chapter will discuss the research methodology by first explaining the research method, before turning to the context of the case study and the research design. The fourth chapter will show the findings and discuss them, while the final chapter will conclude the research by answering the research question, showing the research limitations and suggesting possible avenues for further research.
2 Literature Review
This chapter presents the findings of previous literature written concerning New Public Management. In doing so it will first examining the link between governmentality, hybridisation and NPM. Then the chapter will provide an overview on the history of NPM and its characteristics. Finally, the chapter will discuss three recurring arguments in NPM literature: ‘hard’ versus ‘soft’ management style, accountability and EC reforms.
2.1 Governmentality and hybridisation
Elements of New Public Management have been researched by using the concept of governmentality (Foucault, 1991; Kurunmaki and Miller 2011). Foucault (1991, p. 102) originally defined it as “the ensemble formed by the institutions, procedures, analyses and reflections, the calculations and tactics that allow the exercise of this very specific albeit complex form of power”. In more general terms, governmentality means the forms of governing, either direct or indirect. In previous governmentality studies, three levels of analysis have been used to explain phenomena such as government reforms (Kurunmaki and Miller, 2011). The first level is the programmatic or discursive level, in which the specifics of the reform do not yet matter. At this layer, it is about overall idea of what the reform should accomplish, the goals, the objectives. This is usually discussed in government papers and commissions and leads to what ideals should be achieved. The second level, then, is where the focus is on the practices and processes that accompany the reform, the means through which the ideal should be met. In 2011, Kurunmaki and Miller suggested that the concept of ‘mediating instruments’ should be used, in order to focus more deeply on the specific processes and practices in the situation being researched. These ‘mediating instruments’ are the way in which distinct and sometimes even competing ideas are combined into a single operation, function or framework. The instruments then, according to Kurunmaki and Miller (2011), are both a representation and a means of intervention. They represent the ideal that should be achieved, but are also used to intervene in order to accomplish the ideal. Understanding what these instruments are and how they are connected to the reforms allows for a deeper understanding of the reform itself, because the instruments are what connects the broad rhetoric of the reform at the highest level and the actions made by the experts at the lowest level. The third and final level of the governmentality literature are the professional ‘enclosures’, or the expert knowledge that is the norm in a specific field, organisation or sector (Kurunmaki and Miller, 2011). While this knowledge is not necessarily uncontested, it often is used without critical thinking involved: it has become the standard way of
working. Using these three levels provides a deeper understanding of a specific government reform because it looks at the different inputs in the system, how they are connected and how they influence each other.
As this thesis is studying the EU Results Framework, it is useful to delve deeper into the second level, where the mediating instrument can be found and its accompanying practices and processes. Thomson et al. (2014) have linked the idea of mediating instrument with another idea already present in the accounting literature: that of hybrids. In their study they looked at accounting-sustainability hybrids as mediating instruments, suggesting that to enable change these hybrids were the link between the programmes and the execution. What a hybrid exactly is has been a case for contestation (Miller et al., 2008). While a hybrid is always defined as a new phenomenon created out of two or more aspects usually not combined, why it exists is less clear. The first idea was that there can only be two pure ‘states’, for example on and off, and that hybrid is the link if one switched to the other. Later it was found that hybrids can have their own clear characteristics and therefore do not necessarily have to move to one pole. However, the problem with this second notion was that across disciplines too many hybrids were found, ‘impurity’ as Miller et al. (2008) called it was rampant in all domains of life. This led to the third and, for now, last iteration in which a hybridising is seen as a process, that can go in different directions. It might resolve into one of the two states, it might stabilise and even get institutionalised or it may impact others outside its own domain, thereby initiating a new process of hybridisation. Especially the last outcome, where hybrids interact with each other and create new effects, is important to understand, as in doing so it means that the reform that originally spawned the mediating instrument, the hybrid, can develop into something different than what it was intended for. For now it is assumed that the EU Results Framework is such a hybrid, though this will be more deeply discussed in the fourth chapter on Findings.
2.2 New Public Management
Having discussed the governmentality and hybridisation, we now turn to New Public Management itself. It was coined by Cristopher Hood in 1991, who looked back at the 1980s and saw developments in public management. Hood was of course not the only researcher trying to explain this phenomenon, but his theory has been used as a building block by others, causing NPM to, at times, mean different things to different people (Hood, 1995). However, Hood himself saw some broad themes in NPM that most scholars also use. Some examples of these themes are an emphasis on output instead of process, from hierarchies to a more competitive basis in public services, from a stress on policy making to management skills and more variable
pay. More specifically, in 1995 Hood mentions seven elements that show the shift in focus of the public sector. First, public organisations are broken down into smaller ‘corporatised’ units by product and, second, competition between public sector organisations themselves and competition between the public and private sector is encouraged. Third, an increased focus has been on private sector management practices for the public sector while, fourth, the public sector has moved towards lowering resource use. Fifth, more hands-on management is present, just as, six, explicit standards of performance are encouraged and, seven, greater stress is put on output controls. These seven elements can be broken down into two main themes, in which the first four dimensions focus on the broad shift of management style towards the private sector, whereas the final three elements focus on the specific standards and rules that limit professional discretion (Hood, 1995).
Focusing solely on New Public Management, though, as the only form of government reform is too limited. Therefore, as this thesis looks at public management reform in the European Commission it is necessary to take a look at what public management reform is, before we can understand how New Public Management is connected to this. In their book, Pollit & Bouckaert (2011, p. 2), see public management reform as “[d]eliberate changes to the structures and processes of public sector organizations with the objective of getting them (in some sense) to run better”. This is of course very broad, but considering the wide range of government policies also necessary. In order to gain a deeper understanding, they provide an overview of how public management has changed over the last half century, showing that in the 1950s and 1960s it was a legal and technical matter, not economic and political. Moreover, public reforms were done on a national level. This changed in the late 1960s, when what is coined the ‘first wave’ of change in public management occurred: especially in the UK, USA and France a more rational form of strategic policymaking and evaluation was started. With the economic crises of the 1970s, public management reform experienced a new transformation that continued into the 1980s and even 1990s. Welfare states were becoming too expensive to uphold and a need to increase efficiency and to save money arose. Moreover, public management received an international dimension as all over the world the same type of programmes were introduced, ranging from the USA to Sweden and then to New Zealand (Groot and Budding, 2008). The main direction of these reforms was create a smaller and concise government with specific targets and goals in order to reduce problems of coordination and accountability. This was especially visible in the third wave, where notions of, among others, trust and transparancy rose up, talks started on governance, and networks, all as a response to outside pressure such as a terrorism and globalisation. It is contested though whether this third wave is a new iteration or
just a shift in thinking of the second wave, but that is difficult to assess (Alford and Hughes, 2007). New developments of public management reforms are always on top of the old ones, there is never a ‘clean sweep’ in which the new reform starts from scratch with its own ideas, implementations and actions.
With an understanding of how public management reform has developed over the last half century, we can turn back to how New Public Management fits into this. It is clear that in the first wave NPM does not fit well, because while NPM is built on the basis of rational decisions, the first wave was did not include the private business aspect of NPM. This however changed with Ronald Raegan and Margaret Thatcher, two leaders of the USA and the UK that tried to ‘re-invent government’, by introducing more competition in the public sector, focusing on performance management and quality. It were these changes that led Hood to coin the term New Public Management in 1991, and they are characteristic for the second wave. However, as mentioned the second wave changed in the late 1990s and New Public Management now does not remain the sole model after which government reform is fashioned. Because while NPM was very popular in the late 1980s and early 1990s, it also became controversial. For one it was argued that NPM was very Anglo-Saxon (Broadbent and Guthrie, 2008) and could not be used well for other countries such as Germany and Italy. Moreover, even in countries that had embraced NPM such as New Zealand and the UK it was argued that NPM might not be the best way to reform the government. For example, the idea of networks gained increasing popularity since the 1990s, which for public management entailed that the focus was not solely on the government itself anymore, but now also included other actors such as civil society. These stakeholders can help to make the government more effective and legitimate, which is how the New Public Governance model started (Osborne, 2010). Another model put forward by Pollitt and Bouckaert (2011) is the New-Weberian state, which is focused on modernising the existing bureaucracy. While both these models have similarities with NPM such as modernisation and increasing citizen-friendliness, the main difference is that NPM is the most business-like and focused on results. As the European Commission specifically called for more NPM in managerial reforms (EC, 2014) and its corresponding ideas such as accountability, efficiency and effectiveness, this chapter will focus only on NPM going further. While NPM is definitely not the only model in use for public management reform anymore, and while its ideas have changed over the years, it best fits the reforms in the European Commission, which will be further elaborated in the fifth paragraph of this chapter.
How NPM is defined exactly is still a topic of debate. Dunleavy et al. (2006, p. 96) stated that “[t]here is now a substantial branch industry in defining how NPM should be
conceptualized and how NPM has changed”. Pollit and Bouckaert (2011) assume NPM as a two-level phenomenon, where at the higher two-level it is a general theory or doctrine and at the lower level a combination of specific concepts and practices. The general theory, in their idea, is that management in the public sector can be improved by implementing management tools of the private sector by looking at their concepts, values and techniques. The specific concepts and practices, on the other hand, range from performance measures to changing citizens to customers to reducing government size into a lean organisational form. If you compare this to the three levels of governmentality, it can be argued that the two levels that Pollit and Bouckaert (2011) use are similar to the first two levels of governmentality, the discursive level and the level of practices, processes and instruments. However, Kurunmaki and Miller (2011) included an additional step by also examining the influence of the users of the practices, processes and instruments. Instead of using the different levels, Dunleavy et al. (2006) saw NPM as ‘disaggregation + competition + incentivisation’, looking at the general trend of the reforms and its goals and objectives. However, using only the highest level of NPM, its discursive goals and objectives as a general theory, there is the risk that the NPM is seen as necessarily internally consistent (Pollit and Bouckaert, 2011). One of the advantages NPM has had is that it does not always have to be internally consistent, it can contain multiple ideas on the same topic, or sometimes even, at least seemingly, inherent contradictions. In the following paragraphs we will discuss two of these, seemingly, contradictory statements of NPM: the ‘soft’ versus ‘hard’ managing styles and the accountability versus effectiveness and efficiency debate.
2.3 ‘Hard’ vs ‘Soft’ management styles
One thing that is often seen as part of NPM is performance measurement (Broadbent and Guthrie, 2008). By implementing performance measurement systems (PMS), government tries to improve incentivisation of civil servants, hoping they will work better and harder, thereby increasing the results achieved. This has clearly come from the private sector, which has already used PMS for a longer period of time and originally stems from the economistic, principal-agent way of thinking (Pollit and Bouckaert, 2011). In this theory agents, or civil servants for the government, are supposed to work only in their own interest and should not be trusted outside of that. Incentives and penalties are introduced to manage the employees and get them to do what is best for the company, or the government in this case. Control is with the managers as they measure as much as possible and reward and punish accordingly. However, this has not been the only stream of thought in NPM. A second counter idea arose which tried to give control not to the managers, but to the civil servants, who in this stream are thought to be
professionals. This ‘soft’ version is more trusting of the employees and, while leadership is important, leadership here is not about measurement, reward and punishment, but about developing creativity and innovation, a more intrinsic motivation of the staff instead of extrinsic (Levay and Waks, 2009). The difference is that in the ‘hard’ version managers have to manage, they must take control and tell their employees what to do, whereas in the ‘soft’ version more control is given to the professionals.
In their case studies in a Swedish hospital laboratory and quality registry, Levay and Waks (2009, p. 1) state that “Professional self-governance is currently challenged by market reforms and managerial control efforts, often under the banner of New Public Management”. As professions got increasingly scrutinised outside forces, research had previously shown that there were two main responses by professionals to deal with this: introducing a new managerial logic in which the central focus was self-monitoring and decoupling auditing with the work processes (McGivern and Ferlie, 2007). However, Levay and Waks (2009) find that this clear distinction between the work of the professional and the new management tools does not necessarily have to be that strong. In their studies they found a situation that they coined ‘soft autonomy’, in which auditing ideas to boost accountability and legitimacy were implemented, but where the evaluation criteria were controlled by the professionals. At first this was done voluntarily, with the hopes of influencing the process should they participate from the start and although this was successful, once the new system was implemented it became fixed and the transparancy systems became obligatory (Levay and Waks, 2009). This, eventually, mixed success is why it was called ‘soft autonomy’: the professionals negotiated their own terms for evaluation, but they could not work as carefree as previously possible when they had professional autonomy.
The increased outside pressure for accountability and transparancy has resulted in multiple difficult reforms, ranging from a strict management style to a more hybrid version. But even though these changes have been going on since the late 1960s, whether the reforms are successful is still being contested. These conflicting findings of the success of NPM implementation and its subsequent results are also found more generally in research. Pollitt and Dan (2011) conducted a meta-analysis on the impacts of NPM in Europe, combining almost 520 relevant studies. Though they found a relative lack of studies concerning outputs and outcomes, and called for more direct empirical analysis, the studies that do focus on outputs and outcomes vary widely. Outputs range from highly positive to barely any change to quite negative, suggesting contextual factors may play a role in the implementation of NPM. Moreover, concerning outcomes Pollitt and Dan (2011) found over half the studies mentioning outcomes
of NPM reforms saw no or a negative change, indicating that NPM implementation may not automatically lead to success.
This has been corroborated by Speklé and Verbeeten (2014). Looking at the performance of NPM in the public sector, they specifically focused on performance measurement systems (PMS). Though Speklé and Verbeeten (2014) do not disagree with the goals of NPM, such as efficiency, effectiveness and accountability, they do find two problems in NPM’s emphasis on performance measurement systems in regard to incentive provision, performance evaluation and target setting. First, NPM’s use of performance measurement systems is reduced to only accountability and incentive provision, whereas this can be expanded to also include feedback information. Second, NPM suggests that performance measurement systems in specific, and private sector style management in general, always have a positive outcome. However, Speklé and Verbeeten (2014) find that even in situations in which general theory suggests that PMS should be successful, so when contractibility is high, the incentive-oriented use of PMS negatively impacts performance. This suggests that, while public sector organisations that contains better pre-determined goals, better metrics for measurement and can more easily view the link between actions and outcomes, i.e. contractibility, will perform better, even that is not enough to improve performance if incentive-oriented use of PMS is applied. On the other hand, an exploratory use of PMS does lead to better performance, suggesting that the focus on understanding data and actions leads to better results in public sector organisations.
However, Propper et al. (2008) in their study on hospital care did find a positive influence of a strong management style on performance, just as Kelman and Friedman (2009) did in their research on the English National Health Service. In their study, Kelman and Friedman (2009) tried to find out whether focused attention on a specific measurement had unintended dysfunctional effects or that it would result in only positive results. Studying the effect of wait-time performance in English hospitals they find that these have reduced significantly over three years without any negative side effects. There was no substitution of work under the employees, so they did not reduce efforts in other areas than the one being measured, and they did not try to game the system by providing false data.
In conclusion, while it is certain that changes in public management have been and will continue to be made, the outcome is not. It seems that it depends on the specific context whether a more ‘soft’ or ‘hard’ management style fits better and can positively impact the reform. For some, such as the English hospitals, a strong management style focused on PMS worked really well, but in other situations such as Speklé and Verbeeten (2014) studied that might not be the case. Finally, it seems that professionals are successful in gaining influence in the process of
management reforms (Levay and Waks, 2009), although it is not clear whether that has a positive effect on performance.
2.4 Accountability
Another phenomenon associated with New Public Management is accountability. In the call for management reform in the European Commission (2014) accountability is the overarching topic, which is supported by ideas such as efficiency, effectiveness and transparancy. After entering the public debate in the third wave, accountability has remained incredibly important. For the EC it was the main topic of reform, whereas for example Canada implemented a specific Federal Accountability Act (Bilodeau et al., 2007) after multiple scandals had broken out, and they were by no means the only ones. Accountability is, in essence, quite simple: it is the fact or condition of being accountable, of responsibility. However, what this specifically entails for a government is much more difficult. Usually one answers for one’s actions to a higher authority (Ott and Russell, 2001), but as government is usually the highest authority they are accountable to those that give them that power: the citizens. Multiple accountability models exist, ranging from the relatively straightforward of Ministerial accountability to more complex models in which accountability is dispersed within a democratic system (Hodge and Coghill, 2007). However, as accountability for this study is merely focused on the European Commission, we will only use accountability definitions in so far as they only focus on such an institution instead of a definition that covers the whole of society. One final point that can be used from Hodge and Coghill (2007), is that they note that the state is currently “characterized by lower political accountability […] but with higher managerial and market accountability”. They used Stone’s (1995) perspectives on accountability, in which managerial accountability is seen as where the agency has to meet certain tests, performance measures and audits and market accountability in which the service providers, or government, has to be responsive to ‘sovereign’ consumers. But even though the old Ministerial accountability is not used anymore, the other options Hodge and Coghill (2007) suggest still put ultimate accountability at the highest level of government.
Hupe and Hill (2007) also looked at the topic of public accountability, but from the perspective of ‘street-level bureaucracy’, a concept which focuses on the relative autonomy of professionals as they work at the base, or street, level of the government, or bureaucracy. Instead of the standard political accountability that is established top-down, Hupe and Hill (2007) and other scholars tried to look at it from a wider perspective and saw “institutional practices of account giving” (Meijer and Bovens, 2005, p. 172). This was then defined as public accountability, in which multiple accountability relationships could be found: organizational,
professional, political, legal and administrative. Accountability for a government then is no longer just about accountability from the top level of government to the citizens and other stakeholders, but also throughout the complete organisation, suggesting a more bottom-up approach to accountability (Hupe and Hill, 2007).
The European Commission seems to follow a combination of these two forms. In principle they use Bovens (EC, 2014) who argues that accountability is “a social relationship between an actor and a forum in which the actor explains his conduct and gives information to the forum.” The forum can then judge and, if necessary, take action against the actor, which is aligned with the bottom-up accountability of Hupe and Hill (2007). However, they then continue to define accountability as something that gives stakeholders “confidence that the entity is providing correct and reliable information about its policies, operations and performance”, specifically mentioning the entity as a whole and no longer specific actors or professionals (EC, 2014). Moreover, they use the concept of Public Finance Management in which accountability is seen as an instrument that guarantees that resources given to a public institution are used correctly, relying on transparancy by sharing information on the use of the resources. This then suggests that the EC has a more top-down understanding of accountability. Finally, the European Commission (2014) mentions that a decentralised unit in the EC is answerable to both the hierarchically super-ordinate body of government through managerial accountability on the one hand and the citizens and the public on the other hand. The EC then tries to be accountable both in a traditional top-down setting and in a more modern bottom-up setting, but the definition changes on the situation and may not always be clear.
Talking about accountability is difficult to do in a vacuum because it is not apart from everything else. For example, the European Commission (2014) argues that in order to accomplish the accountability to other stakeholders, they need to focus on Efficiency, Effectiveness and Transparancy. However, at times it has been argued that while accountability is compatible with transparancy, it is contradictory to both efficiency and effectiveness. Accountability is associated with increased bureaucratisation through the need to take on the burden of transparancy and creating new forms of communication to stakeholders (Pollit and Bouckaert, 2011). Moreover, accountability and effectiveness may not necessarily work well together. For this it is important to understand the difference between outputs and outcomes: outputs are results that can be controlled by managers, and in so talk about the efficiency, while outcomes are results that are also influenced by factors outside of management control, but do talk about effectiveness. If one is accountable for the output, so how resources are used from the process from input to output, it is often relatively straightforward how efficient a manager has
been. However, when the focus lies on effectiveness and a manager is accountable for the outcome, this is more difficult as the final result may be outside of his influence. So not reaching the final goal or objective may be something that is outside the hands of the manager, raising questions of his accountability. This sometimes seemingly contradictory effect between accountability on the one hand and effectiveness and efficiency is one that is often overlooked.
2.5 European Commission reforms
The final paragraph of this chapter concerns the European Commission, specifically its reforms throughout the years. In the 1980s and even until the mid-1990s the European Commission resisted calls for management reforms (Kassim, 2008). Even though it had long been argued that reforms were necessary because while the Commission was growing, inefficiencies and a general lack of action palpable in the EC (Pollit and Bouckaert, 2011). As a response, the European Commission started the first wave of reforms in 1995 with the Sound and Effective Management 2000 programme, followed by two other programmes a few years later. However, as the Santer Commission leading these reforms collapsed under allegations of “fraud, nepotism and mismanagement” (Levy, 2006), the new Commission under leadership of Prodi was forced to initiate major reforms. This was done via the now famous Kinnock reforms, started with the White Paper on Reforming the Commission in 2000 in which a call was made to increase strategic priority setting and resource allocation and improve human and financial resource management (Knill and Balint, 2008). On the Kinnock reforms in the European Commission multiple studies have been conducted. Early studies have focused on the justifications the EC has claimed for the reform, concentrating on traditional NPM rhetoric: the public sector should move towards a private sector management style because of improved Economy, Efficiency and Effectiveness (Cini, 2004). An increased focus was placed on accountability through these three E’s, hoping to decrease the EU’s management deficit (Quinlivan and Schön, 2002). Later studies assessed the extent to which the reforms were implemented, showing conflicting results. Knill and Balint (2008) compared the European Commission with the Organization for Economic Co-operation and Development (OECD) in regards to their reforms on human resource management. As the OECD is a staunch propagator of NPM practices, the authors expected NPM to be more pronounced in the OECD compared to the EC. However, they found the opposite result, indicating that the Commission has a deeper NPM implementation than predicted. On the other hand, Ellinas and Suleiman (2008) suggest that the two types of improvements the Kinnock and subsequent reforms have tried to accomplish, both higher
accountability and efficiency, did not work together. The modernisation of the EC through the implementation of NPM ideas was combined with bureaucratisation of traditional Weberian ideas. This resulted in a combination of reforms focused on the one hand on efficiency, in which ideas such as individual responsibility and output controls were encouraged, and on the other hand accountability, strengthening internal rules and regulations as well as input controls. These policies were working against each other, resulting in poor results according to Ellinas and Suleiman (2008). Levy (2004) goes even further than Ellinas and Suleiman by stating that the implementation of the Kinnock reform has mostly failed as it provides more rhetoric than substance. This lack of substance is because the majority of actions proposed in the reform fall into ‘preliminary’ categories rather than final actions (Levy, 2004). Moreover, there has been an increase in institutions and systems rather than a decrease as proposed and the implementation failure rate of the intended actions have been twice as high as suggested. Finally, Pollit and Bouckaert (2011) argue that the focus of the reforms slowly shifted towards the human resource management aspect at the cost of financial management and resource allocation. Although there was now, in contrast to the reforms of the Santer Commission, clear leadership, performance oriented elements seemed to be ranked second, resulting in a rhetoric of decentralisation while it resulted in tighter control and increased bureaucratisation, for example through the new audit function.
Most research concerning the European Commission reforms have focused on the Kinnock reforms in the early 2000s, as after those reforms it has been silent for a long a time. Also, relatively few studies have focused on the EC specifically. Pollit and Bouckaert (2011) found in their meta studies only eight studies out of a total of 518, or only 1.6 percent. They do not however provide any insights into why this so few studies on the Commission have been conducted. This might change with the new reforms underway in the EC, as the call for increased internal control through NPM can be viewed (EC, 2014). However, while there exists an overarching idea of where the Commission should move towards, most changes made in the EC are individual reforms in the different Directorates-Generals (DGs). The EU Results Framework is such an example where only DG DEVCO has, as of yet, started with the implementation of such a reform.
2.6 Chapter Summary
This chapter has tried to provide an overview of the literature written on New Public Management. It started with the topic of governmentality and hybridisation, in which three levels were distinguished: first the discursive, or the general theory of NPM, second the level of
mediating instruments or hybrids, such as the EU Results Framework, and third the professional knowledges. The chapter then continued with the history of NPM and the contested meaning of New Public Management, though one important aspects remains: the focus on more business like management styles. The chapter then turned to a discussion on ‘hard’ versus ‘soft’ management styles, showing that while NPM changes in the government may not always be beneficial, it ultimately depends on the context of the reform itself, just as their does not seem to be one best style of managing, although that is argued by different scholars. Accountability was next, providing an understanding of the importance of accountability in NPM but also the difficulties in achieving it, especially in combination with, sometimes seemingly contradictory, other goals such as efficiency and effectiveness. Finally, the chapter examined the European Commission reforms in the last few decades, paying substantial attention to the Kinnock reforms and pointing to the lack of studies done in other EC reforms. The next chapter will turn to the research method used for this study, before turning to the findings and discussion in the final chapter.
3 Research methodology
Having discussed the literature review in the previous chapter, this chapter will turn to the research methodology of the case study. To do so the chapter starts with the research method chosen for this study and its corresponding components, such as data collection via interviews. It will then continue with an overview of the context case study, the EU Results Framework, as well as more detailed information on the interviews and interviewees. The chapter will conclude by showing how the analysis process is given form.
3.1 Research method
There are two streams of research: quantitative and qualitative. Quantitative research has a more taken for granted based outlook on the world, seeing phenomena as objective and measurable. As this is the starting point for quantitative researchers, they often look at numbers, generalisations and the larger picture, the macro aspect (Bryman, 2008). Qualitative research on the other hand looks out on the world and sees social constructions and a more subjective interpretation. Therefore, to understand phenomena in the world the researcher has to gain understanding through finding the interpretation of the participants (Bryman, 2008). For this an in-depth understanding of the research phenomenon is necessary, so a deeper insight can be obtained than through purely quantitative data.
This thesis aims to better understand the impact of New Public Management in public management reforms, specifically in the European Commission. It is argued by multiple authors (Pollit and Dan, 2011; Kurunmaki and Miller, 2011) that in-depth studies on performance assessment are necessary, which is best done via a specific example of a hybrid, the Framework in this case. Researching a single example is best done by performing a case-study, in which one tries to understand the specific situation and context both. This is necessary, because in human affairs there only is context-dependent knowledge (Flyvbjerg, 2005). Because of this need for research into hybrids that focuses on the context of the specific hybrid, this thesis will be a qualitative study.
The case-study EU International Cooperation and Development Results Framework of DG DEVCO is chosen for three reasons. Firstly, the Framework is a mediating instrument for which multiple, distinct ideas are combined, such as accountability, results, performance and management. As this is a phenomenon created out of two or more aspects usually not combined, it can be understood as a hybrid (Miller et al.?, 2008). Secondly, the EU Results Framework directly responds to calls both from an Internal Control perspective arguing for more and better
NPM reforms (EC, 2014), as well as the 2010-11 Agenda for Change, arguing for better monitoring and reporting on results (EC, 2015). Finally, while the First Report on Selected Results was published in 2016, the Framework is still changing (EC, 2016). This study hopes to see how this process continues to develop and might even show new processes of hybridisation (Kurunmaki, 2011).
In qualitative studies there are three methods that can be applied: interviews, observations and documentary analysis (Patton, 2002). In participant observation the researcher aims to gain intimate knowledge of a given group of individuals, as well as their practices and context. As this needs a prolonged period of direct contact and participation, due to time constraints this study has not focused on this method, other than what could be observed when the researcher was on location for a limited amount of time.
As for interviews, Patton (2005) argued that it allows the opportunity to better understand the thoughts of the interviewees and, in doing so, learn about the case characteristics the researcher could not otherwise obtain. As this thesis looks both at the development of the Framework as well as how it is understood by professionals working with it, the interview method will be the main research method used. In this method there are three approaches: structured, semi-structured and unstructured interviews (Patton, 2005). In structured interviews answers can only be ‘yes’ and ‘no’ to predetermined questions, while unstructured interviews are the opposite and no interview outline nor questions are set. The third approach is the semi-structured interview, in which a broad outline is determined, as well as multiple possible, open-ended questions. However, depending on how the interview is going the interviewer can deviate and give opportunity for the interviewee to provide more in-depth information, while still leading the interview if it goes too far off the track. As, according to Bryman (2008) this can provide understanding of a process, semi-structured interviews is the approach chosen for the interviews.
In the last method for qualitative studies, documentary analysis, the researcher tries to obtain documentary evidence in order to perform content analysis. Documents such as White Papers, the report on the Results Framework, official publications and websites can be analysed (Patton, 2005). In combination with the interviews documentary analysis provides an additional form of obtaining information which can be used to see whether the interviews are coherent with other data. Therefore, it helps with triangulation and a deeper understanding of the case being studied and will be used in this study as complementary to the interviews (Bryman, 2008).
3.2 Context case study
As Bryman (2008) argued, qualitative case studies are, in contrast to quantitative studies, not about generalisation but about contextual understanding. Therefore, this paragraph will provide the context of the EU Results Framework. As DG DEVCO is within the European Commission, we will start with the role of the EC within the European Union. The EC is an institution within the EU that represents the general interest of the EU, not the individual member states and is responsible for both proposing and monitoring EU laws and policies as well as conducting the day-to-day business of the EU (EC, 2017a). It was established in 1951 and consists of two levels: the first level is the leadership of the Commission with 28 Commissioners that change every five years, led by the Commission President, currently Jean-Claude Juncker (EC, 2017a). The second level is that of the administration of the EC, which is divided into policy departments, or Directorate-Generals (DGs). The Directorate-General for International Cooperation and Development (DG DEVCO) is one of these departments and is currently led by Commissioner Neven Mimica. Being built on the foundations of EuropeAid, it consists of nine directorates of which six be divided into two groups: thematic and geographical directorates. The other three directorates are concerned with organisation of resources, analysis and policy formulation and knowledge, performance and results (EC, 2017b). These directorates are different columns within DG DEVCO, which can be more closely seen in Appendix A. The Framework was implemented throughout DG DEVCO, though the directorate for knowledge, performance and results has been responsible for both the implementation, development and monitoring of the Framework. The Framework is structured around three levels and can be seen in figure 1.
Level one is about the development progress in EU partner countries, looking at the medium and long term impact resulted from not only the work of the EU, but also the partner countries and other development actors (EC, 2015). The second level is about specific development outputs and outcomes that can be ascribed to the efforts of EU projects and programmes. The third and final level is more internally focused, as it is about the DG’s organisational performance, looking at the input and processes to see how they are managed and used (EC, 2015). In these three levels an input-output-outcome-impact chain is present, corresponding to OECD-DAC results terminology (EC, 2015). This chain can be seen in figure 2, where an example of the education sector is used. Further, in order to look at the results achieved in the three levels, a limited set of indicators was chosen: for level 1 and 2 indicators 32 each and for level three thirteen indicators were used (EC, 2016). The published results were then also divided into three sections corresponding with the three different levels. Section one only looked at the development progress by sector, so thematically, whereas section two was arranged into results by sector and results by country. Finally, the third section showed the DG DEVCO’s organisational performance, although not all internal management information was included (EC, 2016).
3.3 Research design
This paragraph is divided into three sections: the first section will provide information on the interviewees, the second on the interviews and the final section will discuss the analysis process of the interviews.
3.3.1 Interviewees
The interviewees all work or have worked for DG DEVCO during the implementation and use of the Results Framework. The interviews were conducted in the work environments of the interviewees, number a total of six interviews. Of the six interviewees five still work at DG DEVCO, though one has since left and started working for the European External Action Service. Two interviews were conducted with employees working for thematic units, and another was working for a geographical unit. Of the remaining three interviewees one was currently still working directly on the implementation and development of the Framework, another employee had worked closely on the implementation for a couple of years but has since left DG DEVCO and the final interviewee provides a more supportive role for other directorates and works with the Framework in that capacity. The decision was specifically made to not only interview employees that were directly involved with the implementation and development of the Framework, but also employees that experience working with the Framework on a regular basis. This should provide a deeper understanding into the effects of the Framework throughout the organisation instead of focusing solely on quality and results department.
Gaining access to the interviewees was first achieved by contacting an old student colleague currently working for DG HOME. Contacting others working in the European Commission, she provided the first contact in DG DEVCO who was then asked for an interview. After conducting the first interview the interviewee was asked whether some of his colleagues would be willing to be interviewed, resulting in purposive snowball sampling (Maxwell, 1997), where the sample is gained by person to person referrals, while still being cognisant that the interviewees should be relevant to the study.
Table 1: Overview of interviewees Interview
number
Sector Function Location
of interview
Date Duration
1 Quality and results Monitoring systems and results Brussels 13-06-2017 55 minutes 2 Sustainable Growth and Development Trade and Regional Integration Brussels 16-06-2017 48 minutes 3 European External Action Service Development cooperation and coordination Brussels 15-06-2017 41 minutes 4 Sustainable Growth and Development
Anonymised Brussels 16-06-2017 24 minutes
5 Development
Coordination
Finance, Contracts and Audit
Brussels 16-06-2017 51 minutes
6 Quality and results Anonymised Brussels 13-06-2017 25 minutes
3.3.2 Interviews
As mentioned previously, the interviews were open-ended semi-structured to provide a better platform for discussion and interviewee input. At the start of each interview, the interviewer followed Turner’s (2010) advice by first choosing a setting with little distraction where the purpose of the interview was explained. Information on the thesis and New Public Management was provided. That was followed by addressing the terms of confidentiality by asking the interview to be recorded and whether the interviewee wanted to remain anonymous. Then the format was explained and future contact discussed, for example for sending the interviewees the transcript. Finally, the interviewees were asked if they had any remaining questions before starting the interview.
An interview guide was prepared beforehand in which four themes were found: first some general questions were asked to the interviewee to understand the position of the interviewee in the organisation and the relation to the Framework. In the second theme
questions concerning the development of the Framework were framed and the third theme was about the implementation process of the Framework. In the final theme the impact of the Framework was discussed. Other than the general questions, all interviews were allowed to develop naturally. This meant that sometimes not all questions or themes were discussed, but it did provide the opportunity for detailed answers as well as the possibility for the interviewee to discuss other experiences not included in the interview guide. The guide can be viewed in Appendix B.
Interviews ranged from 24 to 55 minutes and were conducted in the work offices of the interviewees. They were all held in English due to the international nature of the European Commission and five of the six interviews were recorded. One interview was via a phone call and due to miscommunication the interviewer expected the purpose of the phone call to be to plan the interview, but it developed into an interview on the spot. Therefore the interviewer could not record the interview but did take detailed notes during the interview.
3.3.3 Data analysis
In the analysis process three distinct yet also sometimes overlapping phases take place: data reduction, data display and data interpretation (Huberman and Miles, 1994). In data reduction the transcribed interviews are coded and patterns modelled in combination with the theory discussed in the literature review. In this process data is reduced to overarching themes via three steps: gaining a general overview, then record the initial themes and finally reflect on the previous two steps and, if necessary, include new themes (O’Dwyer, 2004). The data is then displayed in a matrix, summarising the different themes found in the transcripts and literature. The matrix can be seen in Appendix C. Here initial themes may be merged together as the initial coding changes into focused coding and finally thematic coding, and main themes and sub-themes are found. The last phase is data interpretation in which the conclusions are drawn and both a descriptive and interpretive summary are provided. This ‘thick’ analysis and employing the analytical lens as suggested by O’Dwyer (2004) is done in the next chapter of Findings and discussion.
3.4 Chapter summary
This chapter started with a discussion on the research method, showing the difference between quantitative and qualitative studies. The choice for the qualitative case study was explained, as well as the decision to research the Results Framework of DG DEVCO. Moreover, the importance of interviews in a qualitative case study was shown, especially in combination with
documentary analysis for triangulation. The chapter then turned to the context of the case study, providing an increasingly narrow overview of first the European Commission, then DG DEVCO and finally the Framework itself. Lastly, the chapter discussed the research design by first reviewing the interviewees and how access was gained, followed by an account of the interviews that were held. Finally, the process of data analysing was examined and this will be continued in the next chapter where the findings are presented.
4 Findings
Having shown how the data will be analysed, this chapter will present and analyse the findings. The findings have been organised into four broad themes that will structure this chapter. These themes will be supplemented by quotes from the interviewees to their personal experiences. First governmentality and hybridisation in relation to the Results Framework will be discussed, before turning to the characteristics of New Public Management in the Framework. Third the chapter will look at differences in management styles and finally accountability will be discussed.
4.1 Governmentality and hybridisation
Kurunmaki and Miller (2011) suggested that three levels of analysis in governmentality studies have been performed. In line with their observation, this chapter starts with this topic to understand how the interviewees experience the public management reform on these different levels. The first level is the programmatic or discursive level, which consists of general ideas what the reform should accomplish, which can also be seen in the objectives and reasons for implementation of the reform. As all interviewees have commentated on this level, it seems that the twofold objectives for the reform have been clear. As Andrea Alfieri said:
“[T]here was the idea of setting up the results Framework similar to what other donors have to enhance accountability on one side, but on the other side to use it as a management tool”
Mr. Hussain added:
“The rationale for that was two fault, two reasons: one is that at a time of tightening economic constraints, the Western World and the EU and the EU member states were facing tight budgetary constraints. […] The second purpose is broadly as more, as a management tool to say okay from each individual projects this is what was being delivered and then you may then also think at a more higher level from results being achieved in the countries” Something that also came up in a couple of interviews was how previously only attention was paid how much money was being spent and what activities were being deployed. There was no focus on results:
(Alfieri): “At the time the Commissioner […]wanted to have a greater emphasis on operational results or shift the focus of our reporting from financial issues, activities, what we do and move towards and emphasis on what kind of result we achieved, our actions, our funding.”
This change in focus has generally be seen as something positive, especially at a theoretical level. All interviewees have said that they liked the idea of having accountability towards other
stakeholders, especially citizens of the European Union. As one employee of DG DEVCO stated:
(5) “My opinion is that public administration manages public money, more than the private. The duty-- they must be accountable; in a very adequate, appropriate way. The management and the reporting outside should be of quality . In the past, this was not the case and we had to improve”
On paper these words still stand strong as he talks about duty, but speaking to him in Brussels showed the passion he had for this topic, as he strongly believed that there was more than just a need for PR marketing: the European Commission, and DG DEVCO specifically, have an obligation to show how they spend the money given to them by the citizens of the European Union. However, while all interviewees agree that providing more accountability to other stakeholders is positive, some feel that instead of a dual objective for the Framework it mainly consists of the accountability aspect. They state that in the rhetoric used in publications and official announcements the function of the Results Framework as a management tool is important, but that in practice this is not the case:
(Jolas): “[O]n this corporate result framework I think that these 32 indicators are really for accountability purposes and this is fine, but they are not meant to help in the operation as a management tool. Maybe in the paper it is, but I think in the practical terms it isn’t”
This seems corroborated by the fact that the third level of the Framework, which is on the organisational performance of DG DEVCO, is only eight pages long on a total of 195 pages. It is clear that the accountability aspect has been the most important reason for implementation, but the management function still remains important argues another interviewee:
(6) “To have gone this long without this kind of monitoring system, I think it’s significant and not in a good way.”
With the general idea of the management reform being more focused on the accountability side rather than as a management tool, we can turn to the second level of analysis and see which practices and processes have accompanied the reform. As previously mentioned, Kurunmaki and Miller (2011) see mediating instruments as something through which this the theoretical level of the reform combines the actions taken on the ground. It is clear that for DG DEVCO this mediating instrument is the Results Framework itself: it tries to provide accountability to other stakeholders by showing the results achieved in a yearly report and gives the opportunity for internal improvement by gathering data that can be used to improve internal processes. As the Framework combines the two aspects of accountability and management, it can also be viewed
(Alfieri) “We cannot have 100 or 150 indicators, than it would be imaginable and we would have a report of thousands of pages and I lose again, I miss my objective . We aggregate a subset and that’s what we report and that’s we use for let’s say guiding discussions at corporate level and strategic level, management level but at the same time we have all the other results that we do not strictly aggregate and which are there and can give you a picture of performance at project level and country level.”
These indicators are not always perfect, because although it is important to limit the total number of indicators, it also means that choices must be made on what can be included:
(Jolas) “In the end we select in the case of trade as result the proportion of trade to their GDP for developing countries. That’s a very broad indicator for which we can only say we have contributed. But how accountable are you when you’re talking about indicators that are so broad.”
Moreover, it is difficult to assess the influence of projects by DG DEVCO on outcomes and impacts. While assessing outputs is relatively straightforward, higher level analysis is far more complicated. Mr. Alfieri argues that:
“You have to take another step, you have to analyse whether your intervention really contributed to that result, taking in account all the other factors and how much. You have different techniques to do that , you have qualitative evaluation, say contribution analyses, quantitative approaches like attribution analyses, through experiment or experimental techniques”
But even with these techniques, the causal link between the implementation of the Framework and the results being achieved are not always clear. A final comment on this second level of analysis can be made on hybridisation. Miller et. al (2008) suggested that a hybrid might get institutionalised, which seems to be the case for the Results Framework as the Results Report will be published every year. Moreover, the Framework shows signs of continuing hybridisation as:
(Hussain) “Yes, this initially this was being much more used as a communications tool to fend to the various stakeholders that DEVCO has was being achieved with the spend . However by merely setting up the systems of reporting, by setting up the expectation that this is the information that we will be collecting, we hopefully set the expectations that this is what we will be doing going forward”
Other interviewees corroborated this point of view that by just setting up the tools of gathering data and monitoring results, the processes surrounding the Framework would change and more attention would be placed on the management aspect.
The third level of analysis in the governmentality literature concerns the professional ‘enclosures’ (Kurunmaki and Miller, 2011), knowledge of the professionals within the field that is
taken-for-granted. Kurunmaki and Miller (2011) warn that management reform may be hampered by professionals trying to translate the policy reform into their own interests. In the interviewees no such feedback was provided, but it seems that not all employees in DG DEVCO were accepting the changes:
(6) “It will take a little bit time, or at least some of their [other employees] time. They don’t understand it. Now we’ll go back to the point that if you don’t have the right support coming from above, then […] you follow again with the culture, we don’t know the culture, so we have some way to go in convincing people.”
Andrea Alfieri on the other hand argued that:
“[We] are getting there, yes it’s an investment, it takes time, a lot of changes, shifting resources, a shift in culture. What we see at the moment is the shift in the culture is there, is happening and it’s actually welcome. Colleagues also want to place emphasises on results and, finally, start talking about results.”
Depending on who is spoken to different experiences are shared, showing that while the culture is slowly shifting in some places towards a more results oriented mindset, it still has some way to go before the complete organisation is changed.
4.2 New Public Management
In this section a deeper analysis on the role and impact of New Public Management in the management reform will be provided. An interesting note to start with though, is that none of the interviewees were very familiar with the concept of New Public Management, while some even had never heard of it. Especially considering that the European Commission called for more reforms using NPM (EC, 2014) and the Results Framework’s objectives are very closely linked to NPM in accountability and management tools, it was unexpected that NPM was so unfamiliar. Still, while the concept itself may not be known to the interviewees, NPM impact on the Framework can still be discussed. This will be done by first showing how the Framework fits NPM theory, followed by a section on the effects of the Framework on performance. Finally, improvements for the Framework suggested by the interviewees will be discussed. While that may not completely correlate with NPM, the ideas suggested often call, indirectly, for furthering of NPM topics which is why it is included in this paragraph.
4.2.1 Fit between Results Framework and New Public Management Theory
One immediately obvious fact on how the Results Framework fits NPM theory is in the title itself: ‘Results’. This is in line with two of the elements Hood (1995) saw as explicit standards of
