A.K. Viera
S1603590
Word count: 22760
Thesis Supervisor: Dr. V. Pattyn
Second reader: V.P. Karakasis MSc
A thesis submitted in partial fulfillment of the requirements for the degree of
Masters of Science in Public Administration
UNDERSTANDING THE IMPACT OF THE EU ON
MEMBER STATES’ EVALUATION CULTURE
1
Understanding the impact of the EU on Member States’
evaluation culture
A single-case study of the evaluation culture of the EU Cohesion Policy in the Netherlands
Anna Viera
a.k.viera@umail.leidenuniv.nl
May 2020
Master Thesis of the Public Administration program
Track: International and European Governance
Faculty of Governance and Global Affairs
Leiden University
Supervisor: Dr. Valérie Pattyn
2
ndreader: V.P. Karakasis MSc
2
Index
List of abbreviations ... 3
List of tables and figures ... 4
Abstract ... 5
1. Introduction ... 6
Guide... 10
2. Theoretical framework ... 11
2.1. Definitions of and debates about policy evaluation in the European Union... 11
2.2. The analytical framework: Europeanization ... 16
2.4. Overview of the expectations and the conceptual model ... 20
3. Methods ... 22
3.1. Introduction ... 22
3.2. Research design ... 22
3.3. Data collection methods ... 23
3.4. Data analysis methods ... 26
3.5. Operationalization and procedure ... 27
3.6. Triangulation ... 36
4. Case study report ... 38
4.1. Introduction ... 38
4.2. Background information: Policy evaluation context of the EUCP in the Netherlands ... 38
4.3. Understanding Dutch evaluation culture ... 40
4.4. Analysis ... 50
4.5. Summary of the findings on all the expectations ... 55
5. Conclusion ... 56
5.1. Answer to the research question ... 56
5.2. Limitations ... 57
5.3. Relevance ... 59
Bibliography ... 61
Appendices ... 66
Appendix A: List of all the collected official EU documents ... 66
Appendix B: List of all the collected official Dutch documents ... 68
Appendix C: Interview questions (in Dutch) ... 70
Appendix D: Consent form for respondents ... 73
3
List of abbreviations
Commission
European Commission
EA
Economic Affairs and Climate
EAFRD
European Agricultural Fund for Rural Development
EMFF
European Maritime and Fisheries Fund
ERDF
European Regional Development Fund
ESF
European Social Fund
ESI
European Structural and Investment Fund
EU
European Union
EUCP
European Union Cohesion Policy
MA
Management Authority
NCA
Netherlands Court of Audit
NPM
New Public Management
OP
Operational Program
PA
Public Administration
PDSSD
Participation and Decentralized Social Services Department
SAE
Social Affairs and Employment
4
List of tables and figures
Tables
Table 1. Expectations of the research: the impact of different behavior of EU
institutions on the Dutch evaluation culture
Table 2. Operationalization of the dependent variable: Dutch evaluation culture
Table 3. Operationalization of the independent variables: Coercive behavior, mimetic
behavior, and normative behavior of EU institutions
Table 4: The four typologies of triangulation in social science research
Table 5: Summary of the findings on all expectations
Figures
Figure 1. Conceptual model explaining the impact of the EU on national evaluation
cultures
5
Abstract
European cohesion policy is one of the policy areas that is responsible for the largest
expenditures of the budget of the European Union (EU). This indicates the importance of
evaluation in this policy area by the EU institutions themselves and the EU Member States.
This thesis attempts to ascertain the impact of the EU institutions on Dutch evaluation culture
in the process of evaluating the EU Cohesion Policy since the 1990s. A single case study has
been carried out with the help of interviews and document analysis. In conversation with
policymakers and experts in the field of Dutch evaluation of EU policies, it was possible to
discuss what this Dutch evaluation culture entails and how the EU has impacted it over the past
three decades.
The analysis in this thesis demonstrates how adaptational pressure of the EU institutions,
such as coercive or normative behavior, could impact developments in Dutch evaluation culture
concerning evaluations of the EU Cohesion Policy. This research illustrates how European
integration theories, such as Europeanization, can provide academics with useful
understandings of how EU policy evaluation is carried out by a Member State. This research
offers the insight that improvements in the process of policy evaluation changed the Dutch
government into an entity with strong data collection methods and blossoming evaluators, in
which values such as legality and effectiveness of conducting evaluations are upheld.
Recommendations for further research are to perform research with other
complementary quantitative research techniques on the same sort of data and also to perform
this research on less developed EU countries, such as Poland, to determine whether these
countries show other effects in terms of the impact of the EU on the national evaluation culture.
Keywords: evaluation culture, policy evaluation, Netherlands, evaluation capacity, EU
Cohesion Policy, Europeanization, coercive, mimetic, normative
6
1. Introduction
Currently, the use of evaluations to review programs and policies is strongly integrated in
European governmental systems and influential institutions (Furubo & Stame, 2019; Jordan &
Liefferink, 2004). Some of these institutions direct the way evaluation develops globally, such
as the European Union (EU) and the World Bank (Furubo, Sandahl, & Rist, 2002, p. 5). For
example, the World Bank Group claims that evaluations could stimulate the use of evidence
from these evaluations to inform decisions in parliaments (Heider, 2014). Speaking of
evaluation, it is necessary to explain what this concept entails when discussing policymaking.
What exactly is meant by “evaluation” in this thesis falls in line with the definition that is given
by Furubo, Sandahl, and Rist (2002, p. 3), which they cited from Chelimsky; namely, “program
evaluation is the application of systematic research methods to the assessment of program
design, implementation, and effectiveness” (1985, p. 7).
It is important to place the EU in the right perspective before discussing evaluation in
the EU further (Furubo, Sandahl, & Rist, 2002, p. 6). In this thesis the EU refers to the
supranational institutions rather than the Member States themselves (Furubo, Sandahl, & Rist,
2002). Therefore, different policy programs are implemented and evaluated in the Member
States. Activities concerning evaluation are executed in every country and may be incorporated
into the political systems of nation states (Furubo, Sandahl, & Rist, 2002, pp. 5-6). Examining
these evaluation activities can provide insight into the development or maturity level of certain
evaluation activities in, for example, European Member States. When evaluation activities have
matured extensively, this can be seen as a “mature evaluation culture” (Furubo, Sandahl, &
Rist, 2002, p. 5).
Evaluation culture develops differently across nation states. Evaluation and
policymaking seem inextricably connected to each other, but evaluation has not always held
this important position. Developments in evaluation practices brought about policy evaluations
in EU Member States in the 1960s and 1970s (Furubo, Sandahl, & Rist, 2002). Scholars have
argued that the United States was a catalyst for putting evaluation on the political agenda in the
current EU countries (Furubo, Sandahl, & Rist, 2002). Therefore, these countries’ interaction
with American evaluation culture made it possible for them to establish their national evaluation
practices through exposure to American public administration theories and social sciences
(Furubo, Sandahl, & Rist, 2002). This can be seen as the first wave of evolution arriving in
Europe that changed the culture of policy evaluations. According to Furubo and Stame (2019,
p. 10), the revolution in evaluation has taken off since then. The study of evaluation practice
7
itself began in the 1960s, and scholars such as Weiss (1972) emphasized the importance of
having more evaluators who have the skills and equipment to carry out these evaluations
(Furubo & Stame, 2019). This renewed interest in evaluations can be seen as the “quantitative
expansions of evaluation” (Furubo & Stame, 2019).
The second wave of change in evaluation culture in the European nation states was
stimulated when Anglo-Saxon countries introduced more public sector reforms during the
1980s (Stame, 2003). Countries such the United Kingdom (UK) started to reform their
governments by introducing New Public Management (NPM) methods in government
institutions and policy evaluations (Stame, 2003). The third wave of change in evaluation
culture occurred when countries were encouraged by external factors to change their evaluation
culture in the public sector (Stame, 2003). For example, the EU could apply external pressure
to change evaluation culture in EU countries (Stame, 2003). These three waves have influenced
the way evaluations are carried out by different Member States in the EU. These sequential
waves have formed the context of the evaluation practice in the EU countries over the past few
decades, during which the practice of national policy evaluations was also developing.
Currently, there is an “obligatory pattern of evaluation” that is anchored in the
regulations of EU institutions such as the European Commission (the Commission) (Schwab,
2009, p. 116). The policy evaluations in member states are susceptible to these EU regulations.
This pattern entails three types of evaluation in the EU: 1) having ex-ante evaluations while
drafting legislation, 2) midterm evaluations during the implementation of policy programs, and
3) ex-post evaluations after the implementation phase and after funding policy programs.
Schwab’s (2009) findings suggest that evaluations of nation states’ EU-related policy programs
could be a driving force for the impact that the supranational EU institutions have on national
policy evaluations. Since the 1990s, it has been required for nation states to perform at least one
of these three types of evaluations per program or instrument of EU-related policies, such as
the Structural Funds of the EU Cohesion Policy (Schwab, 2009). It is evident that the
“quantitative expansions of evaluation” that had already started in the 1960s were reinforced
because of these EU requirements for policy evaluations (Furubo & Stame, 2019).
At the moment, these EU policy evaluation requirements for Member States have
changed into conducting both impact assessments and evaluations to improve the policymaking
process in at EU level (European Commission, 2020). “Impact assessments” here means
collecting evidence, such as findings coming from EU evaluations, to check whether future
actions of EU institutions are justified or are the best manner to achieve the policy objectives
being pursued (European Commission, 2020). As a supplement to these evaluations, there are
8
also so-called fitness checks that address how different legislative tasks have helped to achieve
certain EU policy goals (European Commission, 2020).
The impact of these evaluations on EU-funded policies may have a qualitative impact on
the evaluation of national policies, but so far this subject is under-researched. For instance,
Schwab (2009) has posed questions that have added value to the discussion about the qualitative
impact of policy evaluation on the national evaluation culture of a Member State, such as
whether the experience gained from these evaluations “contribute[s] to the development of
evaluation capacities for national instruments” (Schwab, 2009, p. 116). Another important
question derived from Schwab’s research is whether the importance of evaluation is growing
in national policymaking. The discussions that these questions produce are focused on the
impact that European policies have on national evaluation culture (2009). All of the countries
that have established their national evaluation culture since the 1990s, such as Southern
European nations like Italy, were required to do so due to some kind of external pressure such
the establishments of the EU (Furubo, Sandahl, & Rist, 2002).
However, it is not clear what impact the EU has on the qualitative aspect of policy
evaluation in Member States, namely the national evaluation culture (Furubo & Stame, 2019).
Therefore, this research will examine how the EU has impacted the national evaluation culture
in one of the Member States to better understand this research puzzle. The Netherlands is one
of the countries that has endured strong public sector reforms in recent decades, and it is
typically defined as a country in which that policy evaluation was placed on the political agenda
due to second-wave stimuli, which may lead to interesting outcomes of this research (Furubo,
Sandahl, & Rist, 2002, p. 94). This second-wave stimuli forced countries to adopt evaluation
and integrate evaluation policy evaluation in public governance (Furubo & Stame, 2019). Some
evidence has already been found for the impact of EU-related policies on evaluation, even in
domestic regional policymaking, and this thesis aims to build upon this kind of research
(Schwab, 2009; Furubo & Stame, 2019).
This thesis’s main research goal is to find out whether the EU has impacted the evaluation
culture in the Netherlands and if so, how. To reach this goal, it is important to choose a policy
field in which the EU has imposed certain regulations that may have changed the national
evaluation culture of Member States as a case to study.
9
To perform this research, the following research question was formulated:
Has the European Union impacted the evaluation culture of Dutch policies at the
national level, and if so, how?
The term “Dutch policies” refers to policies at the national level in the Netherlands that are
susceptible to EU regulations. These Dutch policies have to be evaluated by the Dutch
authorities within the regulations and directives of the EU evaluation framework. Also, the EU
has created regulations that the Member States have to follow while performing these
EU-related policy evaluations. To find an answer to this research question, is it important to gain
clarity about the evaluation practices of the Member States. This research attempts to discover
the impact of the supranational EU institutions on the national culture of policy evaluation in
the Netherlands. A secondary goal is to gain more knowledge about the EU’s impact on the
Netherlands as a second-wave country in the third-wave era, wherein the EU is seen as an
external pressure for possible change in the national evaluation culture of Member States.
Seeing the Netherlands primarily as a second-wave country provides the ability to determine
what changed in the evaluation practice of this Member State in the third-wave era. In sum, this
research aims to explore how the evaluation culture at the national level (in this case, in the
Netherlands) is affected by the EU.
The starting point of this thesis is using the theoretical lens of Europeanization to
examine the behavior of EU institutions and explain developments in the evaluation cultures of
Member States. Europeanization is a central concept in European Union policymaking
scholarship (Menz, 2011). The analytical top-down approach of Europeanization is suitable for
explaining how it impacts policy evaluation at the national level through different adaptational
pressures such as coercive, mimetic, or normative behavior of institutions (Menz, 2011, p. 438;
Featherstone & Radaelli, 2003, p. 30; Dimaggio, 1999).
In empirical terms, this research examines the impact the EU has had on the evaluation
culture of the EU Cohesion Policy (CP) in one Member State, the Netherlands. This thesis does
so by examining the evaluation process of the CP, the Structural Funds, in the most recent
program periods from 2007-2013 and 2014-2020. The policy area selected for this research is
the CP in the EU, with a focus on its Structural Funds as a policy instrument of the EUCP
(European Union Cohesion Policy) in the different Member States. The EUCP was developed
shortly after the introduction of the European Regional Development Fund (ERDF) in the
mid-1970s (Manzella, 2009). Thus, this policy area has developed and been evaluated over the past
few decades, which makes it suitable for the purpose of this thesis.
10
The scientific relevance of this research becomes apparent in the following arguments.
First, it builds upon the work of different scholars such as Schwab (2009, p. 116) and his quest
to find the impact of EU policies on EU Member States. He wished to discover whether there
had been Europeanization of national evaluation cultures in the EU Member States. This thesis
makes it possible to examine whether his assumption can be confirmed empirically because it
explores different mechanisms, such as coercive or normative behavior, to explain the top-down
and vertical lines of influence between the EU and Member States.
Second, there is a gap in knowledge about the qualitative effects of the EU’s current
impact on national evaluation culture. Decades ago Weiss (1972) stated that there are indeed
quantitative effects because there is government funding for evaluations at the national level in
the public sector. She also stated, decades later, that governments provide funding for
evaluation with restrictions to strengthen the standards and quality of policy research. In this
quest to improve policy evaluation, is it strange that government institutions can endure
pressure to account for policy evaluation expenditures (Msila & Setlhako, 2013). This almost
amounts to political pressure against having a well-established evaluation program and the
budget to carry out evaluations (Msila & Setlhako, 2013). Scholars have also indicated that
there is more to be discovered about the EU’s qualitative impact on policy evaluation and have
theorized about how this impact can be explained (Furubo & Stame, 2019). This thesis aims to
help close the knowledge gap surrounding these notions of qualitative effects of policy
evaluation by performing research about the EU’s impact on the evaluation culture in policy
evaluation.
Finally, by examining the evaluation culture of a nation state such as the Netherlands, it
is possible for other scholars to discover how the national policy evaluation cultures of similar
second-wave countries, such as the UK, work under the policy evaluation regulations enacted
by the EU institutions. Specifically, this thesis explores how understanding the evaluation
cultures of nation states can help to improve evaluation practices, for example by prompting
discussions and debates that stimulate the national discourse about evaluation.
Guide
This thesis is divided into five parts. The first chapter introduces the main research topic. The
second chapter examines the theoretical assumptions about the main research question in this
thesis and has two sections. The first section begins with a short literature study about
Europeanization in the EU and reveals how this analytical approach towards European
integration helps to build an evaluation culture in Member States’ policies. The theoretical
11
framework of this research is then presented in the second section. In the third chapter the
research design is clarified, and in Chapter 4 the case study is introduced. The fifth chapter
draws conclusions about the research, elaborates on the limitations of the research, and provides
some recommendations for further research.
2. Theoretical framework
This chapter presents the variables that are important to examine in this analysis. The first
section will briefly explain the history of policy evaluation in the EU and will also present the
definitions of the most important concepts in this research. This first consists of explaining the
history and function of policy evaluation in the EU. When considering national evaluation
cultures, this first section also illustrates how the key aspects of the research are related to each
other, so that it becomes clear how they are useful in answering the following research question:
Has the European Union impacted the evaluation culture of Dutch policies at the national
level, and if so, how?
The second section describes the theoretical framework of this research and presents the
expectations of the relationships between the key concepts derived from the theoretical
approach of Europeanization. The last section summarizes the expectations and details the
conceptual model that is derived from the entire discussion in Chapter 2.
2.1. Definitions of and debates about policy evaluation in the European Union
History of policy evaluation in the European Union
In all three waves of developments in evaluation practice, it is apparent that certain stimuli (e.g.,
American evaluation cultures and theories in the 1960s and 1970s, NPM in the 1980s and
1990s, and external pressure from the EU since the 1990s) created change in evaluation practice
in the EU. However, it remains unclear how evaluation practice has changed due to the latter
wave of external pressures and whether the first and second waves have left their traces in
current evaluation practice. Before addressing the theoretical aspects and approach to examine
policy evaluation in this thesis, this section presents a brief look at the history of policy
evaluation in the EU. This section will illustrate and explain what evaluation practice looked
like in 2019 before examining possible theoretical explanations for the impact of the EU’s
current evaluation practice on Member States.
In the twenty years from 1960 to 1980, evaluation developed in a remarkable way
(Furubo, Sandahl, & Rist, 2002, p. 226; Furubo & Stame, 2019, p. 9). According to Furubo and
12
Stame (2019, p. 10), the revolution in evaluation took off during the period from 1960-1980
(Furubo & Stame, 2019). A sudden awareness and increased study of evaluation practice started
in the 1960s, and scholars such as Weiss emphasized the importance of having more evaluators
who have the skills and equipment to carry out these evaluations (Furubo & Stame, 2019). This
renewed interest in evaluation can be seen as the “quantitative expansions of evaluation”
(Furubo & Stame, 2019). Weiss’s research is a perfect example of the quantitative
consequences of the renewed interest in evaluation (Weiss, 1972; Furubo & Stame, 2019, p.
10). According to Furubo and Stame (2019), the value of this excess burden would equate to
about 140,000 US dollars today. Due to this quantitative effect of the renewed interest in
evaluation in the 1960s, scholars view this as the formative period of evaluation in the US.
The diffusion of evaluation in the US was an idea that was developed in the 1960s
(Furubo & Stame, 2019). In the US, the government introduced the idea of “shared
responsibility” to solve policy problems locally with conservative measures, and to tackle
policy issues with new solutions in the form of experiments (Stame N. , 2008). These shared
responsibilities were the source of the spread of program evaluation in the US in the 1960s
(Stame N. , 2008). Some countries that later joined the EU adopted these American ideas in the
1960s and 1970s. When the EU countries were developing their own national evaluation
cultures, they did not build upon their longstanding European culture of carrying out evaluations
(Furubo & Stame, 2019). Instead, EU countries adopted the evaluation practice from the US
and therefore creating disconnection in the way how EU countries performed policy evaluation
and the new way of doing it. This disconnect between the new way of performing evaluations
coming from the US and the more original, more conservative way of doing so in Europe is
now seen as some heritage of the US (Furubo & Stame, 2019). Among scholars, this US
influence is seen as the first wave of developing national evaluation practices in what is now
the EU. The influence of the US in the 1960s and 1970s created a pessimistic attitude among
current EU Member States towards the new evaluation practice due to government policies that
failed in these years (Furubo & Stame, 2019, p. 43). These government failures meant that new
problems arose from the need to understand new programs that were introduced to solve societal
issues in current EU countries, especially because they were new programs that were
implemented in very complex and diverse contexts (Stame N. , 2008). Therefore, more
discussions emerged in the 1960s and ‘70s about, for example, taking the political context into
account when evaluating EU government policy programs (Stame N. , 2008).
The second wave emerged in the realm of evaluation practice because of this pessimistic
stance (Furubo & Stame, 2019). NPM transformed the public sector in the 1980s and 1990s by
13
following more private sector management styles and being focused on results instead of
improving the evaluation procedures (Furubo & Stame, 2019, p. 12). In other words, NPM is
more focused on managerial administration in making and evaluating policies and less focused
on the politics and bureaucratic administration that come with it (Furubo & Stame, 2019).
Examples of countries that introduced these NPM reforms were mainly those that had
Anglo-Saxon traditions, such as Australia, the UK, the Netherlands, New Zealand, and the
Scandinavian countries (Stame, 2003). Hood’s (1991) well-known article “A Public
Management for All Seasons?” prescribes the principles of NPM. These principles are
hands-on professional management in the public sector; explicit standards and measures of performance; greater emphasis on output control; shift to disaggregation of units in the public sector; shift to greater competition in the public sector; stress on private sector styles of management practice; and stress on greater discipline and parsimony in resource use. (Furubo & Stame, 2019, p. 43; Hood, 1991)
Of course, there are other scholars who have discussed which values prevail in policymaking
or in public administration itself.
The third wave began in the 1990s and was stimulated by the formation of the EU. In
fact, the foundations of the EU were established in the 1950s under the European Coal and Steel
Community/European Economic Community (European Union, 2019). This notwithstanding,
the EU as we know it was established in 1993 and created many changes, such as the completion
of the internal market and the Schengen agreement (European Union, 2019). In this wave, the
evaluation practice in the EU developed internationally in the form of networks, and the
communication between these networks became more important in the development of
evaluation practice in the EU.
Evaluation has become important to the EU in the present day. However, the Member
States have certain obligations to the EU regarding evaluation. Member States are obligated to
set up at least one ex-ante, midterm, or ex-post evaluation for EU policies (Schwab, 2009). One
policy to which this applies in Member States is the CP, which has better regulation guidelines
to assess employment, its social impact, and its funds (Schwab, 2009; European Commission,
2019). These types of evaluations provide learning experiences and possibilities to further
develop evaluation in the EU and its Member States (Schwab, 2009).
Evaluation culture, evaluation capacity, and institutions
Evaluation is such an integrated process in policymaking that it is almost taken for granted.
According to Furubo and Stame (2019, p. 14), there are multiple functions of policy evaluation
that must be clarified in order to understand the practice. The term “evaluation” is used in a
14
broad sense throughout the whole discussion in this thesis. Using this broad sense of this term
makes it possible to use this word throughout the thesis as some activity that secures
improvement of public policies without constantly having a predetermined position about what
evaluation exactly is in every sentence (Furubo & Stame, 2019, p. xiv).
Policy evaluation has multiple functions that have to be expounded on before diving
into the concepts of evaluation culture, evaluation capacity, and institutions. Evaluation has
received more attention by public sector policy makers and evaluators because of its ability to
increase effectiveness and efficiency in government expenditure (Rist, 1990). A large part of
the gross national product is used for government expenditures in most countries, and therefore
is it important to account for the funds coming from public funding (Rist, 1990). Another
function is strengthening government administration in order to help society. To do so, is it
necessary to have a systematic way of evaluating government with clear guidelines and enough
information about how to perform evaluations. Evaluation is also a useful tool for the feedback
process in government administration.
Throughout the years, there have been differences between the EU countries in their
evaluation cultures. According to Bachtler and Wren (2006), these differences are visible in
the evaluation capacities of the Member States. Evaluation capacity, according
to Bachtler and Wren, refers to “the availability of qualified and experienced evaluators, and
the presence of appropriate knowledge and skills among evaluators and evaluation
commissioners, and the institutional frameworks to manage evaluation and promote evaluation
practices” (2006, p. 149). Northern Europe, with countries such as the Netherlands, Germany,
and the Scandinavian countries, has strong national policies that have evolved have evolved to
become more effective over the past few decades (Bachtler & Wren, 2006). Southern Europe,
with countries such as Spain and Italy, has not been as successful in its development of national
evaluation cultures (Bachtler & Wren, 2006). In this thesis, the evaluation culture of a policy
domain is considered to be the more qualitative side of the evaluation practice, which falls in
line with Speer’s definition of it as namely “a mainstream preference for specific evaluation
practices, approaches, and systems” (Speer, 2012, p. 66). In this respect, it is important to keep
in mind that evaluations integrate certain values and orientations that evaluators or institutions
that commission evaluations, such as the European institutions, may have.
Schwab used the concepts of evaluation culture and evaluation capacity the two
concepts interchangeably and explains these terms as: “the ability of a political system to use
evaluation—and its affinity to do so” (2009, p. 117). In fact, has evaluation capacity often been
defined by other scholars in a narrower way. For instance, it can be the ability to carry out
15
evaluations in a more effective manner (Milstein & Cotton, 2000). Another narrower definition
of evaluation capacity is the scope of an organization to have resources or the motivation to
carry out, analyze, and deploy evaluations (Preskill & Boyle, 2008). However, this does not
mean that the concept of evaluation culture is difficult to use for research. If researchers are
interested in a concept that covers qualitative experiences in addition to evaluation capacity,
such as the “the role of ideas and attitudes” (Schwab, 2009, p. 117) or the norms and practices
in the evaluation practice of public sector programs, then evaluation culture serves those goals.
For this research, both concepts—evaluation capacity and evaluation culture—remain distinct
because this allows for a better understanding of how to separate certain aspects of evaluation
practice in the empirical world. In this respect, evaluation capacity is seen as the more
quantitative side of evaluation and is interlinked with evaluation culture. Some aspects of De
Peuter and Pattyn (2008) are used to explain the interlinkages between evaluation capacity and
evaluation cultures.
A third element that is helpful in explaining evaluation culture in empirical research is
the definition of institutions. After all, this research is focused on explaining the impact of EU
institutions on the evaluation cultures in the policy domains that are subject to their regulations.
Borrowing from the perspective of historical institutionalism, institutions are defined as “the
formal or informal procedures, routines, norms and conventions embedded in the organizational
structure of the polity or political economy” (Hall & Taylor, 1996, p. 6). This perspective views
institutions as political entities that were set on certain trajectories in the past. Due to their
regulations or decisions, these institutions have to face certain consequences. In line with other
historical institutionalists, this research sees institutions as “organizations and the rules or
conventions promulgated by formal organization” (Hall & Taylor, 1996, p. 7). This perspective
is suitable for explaining how the EU institutions, such as the European Commission or its EU
agencies, promulgate their rules and regulations to other institutions such as Member States.
This view also allows one to see regulation or policies as rules or certain conventions that the
formal EU institutions are trying to uphold. In this respect, the institutional arrangements and
the “supply of evaluators” are mapped out in certain policy sectors (Speer, 2012, p. 70).
Furthermore, these institutions create the necessary data for evaluation by at once pursuing their
organizational goals and “creating data infrastructures that support certain evaluation designs”
(Speer, 2012, p. 70). In the following paragraph, an analytical perspective is explained that will
help to formulate expectations about how the EU evaluation practice and institutions may have
an influence on the national evaluation culture of a Member State.
16
2.2. The analytical framework: Europeanization
Definition
The theoretical approach of Europeanization falls under the denominator of European
integration theories and is important in order to answer the research question. Europeanization
attempts to explain the system of how European integration, as in the EU institutions on a
supranational level, impacts the Member States (McGowan, 2007, p. 12; Featherstone &
Radaelli, 2003).
Scholars have debated for decades about how Europeanization can be conceptualized in
research and used as a conceptual framework (Featherstone & Radaelli, 2003, pp. 6-12).
However, Europeanization is not always used as the only pathway to arrive at the final answer
about EU-Member State dynamics. This analytical perspective is often used in combination
with other theoretical frameworks such as new institutionalism, multilevel governance, and
policy networks (Featherstone & Radaelli, 2003). This thesis only focuses on the theoretical
framework of Europeanization due to its applicability for investigating the impact of a
supranational institution such as the EU on the policies of Member States. As a
conceptualization of Europeanization, this thesis will uphold the following definition of
Radaelli and Featherstone:
processes of (a) construction (b) diffusion and (c) institutionalization of formal and informal rules, procedures, policy paradigms, styles, ‘ways of doing things’, and shared beliefs and norms which are first defined and consolidated in the EU policy process and then incorporated in the logic of domestic (national and subnational) discourse, political structures, and public policies (Featherstone & Radaelli, 2003, p. 17).
This analytical Europeanization perspective within European integration theories is
suitable because it is able to answer questions related to “the role of domestic institutions in the
process of adaptation to Europe” (Featherstone & Radaelli, 2003, p. 33). The kind of research
that is designed using Europeanization as an analytical approach always has the condition of
strengthening or limiting Member States’ space to make their own decisions. By
using Europeanization, the EU as a supranational institution can therefore be understood as a
venue involving multiple processes, policies, bargains, diffusions, and interactions with
national-level actors (Featherstone & Radaelli, 2003). Second, this perspective can be used for
organizations and individual actors even though it does not explicitly mention
them (Featherstone & Radaelli, 2003). It does so by explaining organizational behavior and
individual behavior and covering the interests of scholars by taking a closer look at the “political
structure, public policy, [and] identities” (Featherstone & Radaelli, 2003, p. 30).
17
However, there are other theories of European integration that seem suitable for this
research. For instance, one of these theories is neofunctionalism. Neofunctionalism is
applicable because of its ability to use spillovers to explain how the EU institutions could
influence policies at national level via spillovers (Tranholm-Mikkelsen, 1991; Niemann, 2017).
When taking a closer look at neofunctionalism, this theory is not suitable to answer the main
research question because this research is only focused on one specific policy area. In order to
use neofunctionalism properly, it would be necessary to understand the EU’s impact on the
evaluation culture in at least two policy areas in one Member State. By doing so, it would be
possible to explain the dynamics between these policy areas in terms of spillover effects.
In general, there are two directions of Europeanization, namely horizontal and vertical.
Vertical Europeanization refers to two different levels: the EU level, where policy is
established, and the national level where this policy has to be metabolized by Member States
(Featherstone & Radaelli, 2003, p. 41). In this model, Member States are pressured to adopt
policy that is made at the EU level. By contrast, in horizontal Europeanization there is decidedly
no pressure coming from the EU level to conform to a certain policy that is made at this
supranational level (Featherstone & Radaelli, 2003, p. 41). Due the fact that this thesis is
interested in theories that can explain the vertical rather than the horizontal process of
Europeanization, this vertical process of Europeanization is the basis upon which the theoretical
framework has been designed. In the quest to conduct an institutional analysis of the impact of
the EU on one of its Member States, the work of Featherstone and Radaelli (2003, p. 41) has
guided the design of the theoretical framework. This thesis can be placed under other research
that is interested in finding answers through the vertical process of Europeanization.
According to Europeanization and new institutionalism, adaptational pressures are
explained by the mechanisms of coercion and mimetic and normative behavior (DiMaggio &
Powell, 1983). This way of explaining how these EU institutions influence Member States looks
very similar to organizational analysis from the perspective of new institutionalism (DiMaggio
& Powell, 1983; Featherstone & Radaelli, 2003). Contemporary institutional theory is based on
new institutionalism but also incorporates certain aspects of older institutional economics from
authors such as DiMaggio and Powell (1983; 2012). These theories explain the impact of
institutional pressures on public sector organizations (DiMaggio & Powell, 1983; 1999;
Carpenter & Feroz, 2001). It promotes the idea that these pressures coming from the external
environment of public institutions in the form of “coercive legislation, professional norms, and
mimetic examples” may impact the practices inside a certain public sector organization
(Keerasuntonpong, 2018, p. 1171). Featherstone and Radaelli (2003, p. 41) provide a lens
18
through which to examine the mechanisms of coercion, mimetic and normative pressures to
explain the impact of the EU on a Member State. This becomes possible by seeing this impact
as a process of institutional change that occurs through these three mechanisms.
The first mechanism of adaptational pressure is coercion, which takes place when, for
example, institution A is dependent upon, is forced by, or is somehow culturally expected to
change because of institution B (Greenwood & Meyer, 2008). For example, because of coercive
behavior by EU institutions, Member States are obliged to carry out regulatory arrangements
before certain EU directive or regulatory deadlines. Thus, scrutinizing or reviewing the actions
of public officers could improve the accountability of the public sector in a Member State within
strictly coercive legislation (Keerasuntonpong, 2018). This behavior is very forceful top-down
steering by the EU institutions.
The second mechanism is mimetic behavior. This mimetic behavior between institutions
comes mainly from the institutions’ uncertainty, due to which they copy their role models that
appear more rational or advanced (Greenwood & Meyer, 2008). This insecurity about possible
solutions stimulates organizations to imitate successful examples from other organizations
(Keerasuntonpong, 2018). This amounts to EU countries adjusting to EU rules and legislation
to obtain greater economic, social, or other desired benefits, and other Member States join in to
have the same benefits (Featherstone & Radaelli, 2003). In this sense, institutional change
occurs with one institution because the institution itself or its entrepreneurs are imitating the
successful behavior of other institutions because this behavior seems attractive in solving their
own institutional problems (Beckert, 2010).
A third mechanism is normative pressures, which depends on how attractive one
institutional model appears for another institution. This is closely linked with the socialization
processes of institutions (Beckert, 2010; DiMaggio & Powell, 1983). When public sector
institutions fear losing their reputation or legitimacy, bad publicity could arise if they do not
address the criticism of the principal EU institutions to which they are subject. Professional
training and networks reinforce the mechanism of normative pressures coming from one
institution to another. Professional training influences the “cognitive and normative frames that
shape their perspectives on regulative goals and the likely means to achieve them” (Beckert,
2010, p. 156). Socialization creates the routines and institutionalization of practices in a certain
profession, policy area, or of an institution itself. Through professional networks, it is possible
for institutions to diffuse their norms and standards across boundaries, as in this case from the
EU level to the national level, and therefore create similar ideas about how to tackle regulative
19
problems or how to evaluate certain policy areas and aspects thereof (Beckert, 2010; DiMaggio
& Powell, 1983, p. 152).
Expectations
Radaelli’s (2003, p. 13) definition of Europeanization is central in this thesis. Radaelli (2003)
claims that there are three processes—construction, diffusion, and institutionalization—that are
integrated into the policymaking process at the EU level. These processes are then diffused to
the public policies, politics, and discourse at the (sub)national level. For example, the process
of institutionalizing certain formal regulations at the EU level can affect Member States’ public
policies. This thesis argues that the way the EU affects national policies could be through the
three causal mechanisms of diffusion.
Therefore, the EU institutions can have an impact on national politics, policies, or
discourse through diffusion. In EU policymaking, it is possible to speak of policy diffusion
through aspects that originate from sociological institutionalism (DiMaggio & Powell,
1983). These aspects refer to certain behavior or pressure of public institutions. National public
policies can change through “the mechanisms of coercion, mimetism, and normative
pressures” (DiMaggio & Powell, 1983; Featherstone & Radaelli, 2003, p. 41). In this way, it
becomes possible to see the similarity between the European and national evaluation cultures
because the national evaluation practice experiences certain coercive, mimetic, or normative
pressures from the European evaluation practice. For example, formal EU institutions can create
certain coercive pressures by setting up regulations that will change the way evaluations are
carried out. Based on the theoretical lens of Europeanization, this research aims to understand
whether and how EU institutions may have an impact on the Dutch evaluation culture through
diffusion. This means that this research explores in detail whether there are adaptational
pressures such as mimetic, coercive, or normative pressures coming from EU institutions on
Dutch evaluation culture. Three expectations are formulated and will guide this research to
better understand how the influence of the EU affects the evaluation culture of the Netherlands.
The first expectation entails explaining the impact of the EU on Dutch evaluation culture
by the mechanism of coercion. Via coercion, the formal and informal pressures coming from a
supranational EU institution on Dutch national institutions create a more similar evaluation
culture between these two levels in one policy area. This could happen because how a certain
policy area is evaluated in the Netherlands could be dependent on how the EU would like to
have its Member States evaluate this specific policy area. The first expectation is formulated as
follows:
20
1. EU institutions have an impact on Dutch evaluation culture in the policy area of CP
through the mechanism of coercive behavior.
The second expectation refers to mimetic behavior as an explanation for the impact of
the EU on Dutch evaluation culture via the mechanism of mimetism. As mentioned earlier,
uncertainty is a very important element in explaining this impact through mimetism. If certain
EU institutions evaluate a certain policy area using certain rules or techniques, this EU-level
evaluation culture become very attractive for Member States. Also, if EU institutions follow
certain advanced institutional evaluation models, it may become convenient for Member States
to copy this evaluation model in their own national evaluation cultures. To examine whether
this is happening in the Netherlands in the area of CP, the next hypothesis is formulated as
follows:
2. EU institutions have an impact on Dutch evaluation culture in the policy area of CP
through the mechanism of mimetic behavior.
The third expectation helps to unravel the impact of EU institutions on Dutch evaluation
culture in the policy area of CP through possible normative pressures. These normative
pressures can take the form of professional training or professional networks wherein the EU
institutions can diffuse their norms and standards about policy evaluation to other Member
States concerning a given policy area. To examine whether this is taking place in the
Netherlands in the area of CP, the next hypothesis is formulated as follows:
3. EU institutions have an impact on Dutch evaluation culture in the policy area of CP
through the mechanism of normative behavior.
2.4. Overview of the expectations and the conceptual model
Scholars have not determined what the precise qualitative impact, thus referring to a concept
that covers qualitative experiences in addition to evaluation capacity, policy evaluation has
been on evaluation cultures in the EU (Jacob, Speer, & Furubo, 2015; Schwab, 2009). Figure
1 shows the conceptual model of the theoretical framework in this research to illustrate the
possible relationships between the explanatory variables (independent variables) and the
variables of response (dependent variables). The dependent variable is the national evaluation
culture within a certain policy area that is subject to EU obligations. The independent variables
are the adaptational pressures coming from the EU institutions, namely coercive behavior,
mimetic behavior, and normative behavior.
21
Figure 1: Conceptual model explaining the impact of EU institutions on Dutch evaluation culture