• No results found

Understanding the impact of the EU on Member States’ evaluation culture: A single-case study of the evaluation culture of the EU Cohesion Policy in the Netherlands

N/A
N/A
Protected

Academic year: 2021

Share "Understanding the impact of the EU on Member States’ evaluation culture: A single-case study of the evaluation culture of the EU Cohesion Policy in the Netherlands"

Copied!
105
0
0

Bezig met laden.... (Bekijk nu de volledige tekst)

Hele tekst

(1)

A.K. Viera

S1603590

Word count: 22760

Thesis Supervisor: Dr. V. Pattyn

Second reader: V.P. Karakasis MSc

A thesis submitted in partial fulfillment of the requirements for the degree of

Masters of Science in Public Administration

UNDERSTANDING THE IMPACT OF THE EU ON

MEMBER STATES’ EVALUATION CULTURE

(2)

1

Understanding the impact of the EU on Member States’

evaluation culture

A single-case study of the evaluation culture of the EU Cohesion Policy in the Netherlands

Anna Viera

a.k.viera@umail.leidenuniv.nl

May 2020

Master Thesis of the Public Administration program

Track: International and European Governance

Faculty of Governance and Global Affairs

Leiden University

Supervisor: Dr. Valérie Pattyn

2

nd

reader: V.P. Karakasis MSc

(3)

2

Index

List of abbreviations ... 3

List of tables and figures ... 4

Abstract ... 5

1. Introduction ... 6

Guide... 10

2. Theoretical framework ... 11

2.1. Definitions of and debates about policy evaluation in the European Union... 11

2.2. The analytical framework: Europeanization ... 16

2.4. Overview of the expectations and the conceptual model ... 20

3. Methods ... 22

3.1. Introduction ... 22

3.2. Research design ... 22

3.3. Data collection methods ... 23

3.4. Data analysis methods ... 26

3.5. Operationalization and procedure ... 27

3.6. Triangulation ... 36

4. Case study report ... 38

4.1. Introduction ... 38

4.2. Background information: Policy evaluation context of the EUCP in the Netherlands ... 38

4.3. Understanding Dutch evaluation culture ... 40

4.4. Analysis ... 50

4.5. Summary of the findings on all the expectations ... 55

5. Conclusion ... 56

5.1. Answer to the research question ... 56

5.2. Limitations ... 57

5.3. Relevance ... 59

Bibliography ... 61

Appendices ... 66

Appendix A: List of all the collected official EU documents ... 66

Appendix B: List of all the collected official Dutch documents ... 68

Appendix C: Interview questions (in Dutch) ... 70

Appendix D: Consent form for respondents ... 73

(4)

3

List of abbreviations

Commission

European Commission

EA

Economic Affairs and Climate

EAFRD

European Agricultural Fund for Rural Development

EMFF

European Maritime and Fisheries Fund

ERDF

European Regional Development Fund

ESF

European Social Fund

ESI

European Structural and Investment Fund

EU

European Union

EUCP

European Union Cohesion Policy

MA

Management Authority

NCA

Netherlands Court of Audit

NPM

New Public Management

OP

Operational Program

PA

Public Administration

PDSSD

Participation and Decentralized Social Services Department

SAE

Social Affairs and Employment

(5)

4

List of tables and figures

Tables

Table 1. Expectations of the research: the impact of different behavior of EU

institutions on the Dutch evaluation culture

Table 2. Operationalization of the dependent variable: Dutch evaluation culture

Table 3. Operationalization of the independent variables: Coercive behavior, mimetic

behavior, and normative behavior of EU institutions

Table 4: The four typologies of triangulation in social science research

Table 5: Summary of the findings on all expectations

Figures

Figure 1. Conceptual model explaining the impact of the EU on national evaluation

cultures

(6)

5

Abstract

European cohesion policy is one of the policy areas that is responsible for the largest

expenditures of the budget of the European Union (EU). This indicates the importance of

evaluation in this policy area by the EU institutions themselves and the EU Member States.

This thesis attempts to ascertain the impact of the EU institutions on Dutch evaluation culture

in the process of evaluating the EU Cohesion Policy since the 1990s. A single case study has

been carried out with the help of interviews and document analysis. In conversation with

policymakers and experts in the field of Dutch evaluation of EU policies, it was possible to

discuss what this Dutch evaluation culture entails and how the EU has impacted it over the past

three decades.

The analysis in this thesis demonstrates how adaptational pressure of the EU institutions,

such as coercive or normative behavior, could impact developments in Dutch evaluation culture

concerning evaluations of the EU Cohesion Policy. This research illustrates how European

integration theories, such as Europeanization, can provide academics with useful

understandings of how EU policy evaluation is carried out by a Member State. This research

offers the insight that improvements in the process of policy evaluation changed the Dutch

government into an entity with strong data collection methods and blossoming evaluators, in

which values such as legality and effectiveness of conducting evaluations are upheld.

Recommendations for further research are to perform research with other

complementary quantitative research techniques on the same sort of data and also to perform

this research on less developed EU countries, such as Poland, to determine whether these

countries show other effects in terms of the impact of the EU on the national evaluation culture.

Keywords: evaluation culture, policy evaluation, Netherlands, evaluation capacity, EU

Cohesion Policy, Europeanization, coercive, mimetic, normative

(7)

6

1. Introduction

Currently, the use of evaluations to review programs and policies is strongly integrated in

European governmental systems and influential institutions (Furubo & Stame, 2019; Jordan &

Liefferink, 2004). Some of these institutions direct the way evaluation develops globally, such

as the European Union (EU) and the World Bank (Furubo, Sandahl, & Rist, 2002, p. 5). For

example, the World Bank Group claims that evaluations could stimulate the use of evidence

from these evaluations to inform decisions in parliaments (Heider, 2014). Speaking of

evaluation, it is necessary to explain what this concept entails when discussing policymaking.

What exactly is meant by “evaluation” in this thesis falls in line with the definition that is given

by Furubo, Sandahl, and Rist (2002, p. 3), which they cited from Chelimsky; namely, “program

evaluation is the application of systematic research methods to the assessment of program

design, implementation, and effectiveness” (1985, p. 7).

It is important to place the EU in the right perspective before discussing evaluation in

the EU further (Furubo, Sandahl, & Rist, 2002, p. 6). In this thesis the EU refers to the

supranational institutions rather than the Member States themselves (Furubo, Sandahl, & Rist,

2002). Therefore, different policy programs are implemented and evaluated in the Member

States. Activities concerning evaluation are executed in every country and may be incorporated

into the political systems of nation states (Furubo, Sandahl, & Rist, 2002, pp. 5-6). Examining

these evaluation activities can provide insight into the development or maturity level of certain

evaluation activities in, for example, European Member States. When evaluation activities have

matured extensively, this can be seen as a “mature evaluation culture” (Furubo, Sandahl, &

Rist, 2002, p. 5).

Evaluation culture develops differently across nation states. Evaluation and

policymaking seem inextricably connected to each other, but evaluation has not always held

this important position. Developments in evaluation practices brought about policy evaluations

in EU Member States in the 1960s and 1970s (Furubo, Sandahl, & Rist, 2002). Scholars have

argued that the United States was a catalyst for putting evaluation on the political agenda in the

current EU countries (Furubo, Sandahl, & Rist, 2002). Therefore, these countries’ interaction

with American evaluation culture made it possible for them to establish their national evaluation

practices through exposure to American public administration theories and social sciences

(Furubo, Sandahl, & Rist, 2002). This can be seen as the first wave of evolution arriving in

Europe that changed the culture of policy evaluations. According to Furubo and Stame (2019,

p. 10), the revolution in evaluation has taken off since then. The study of evaluation practice

(8)

7

itself began in the 1960s, and scholars such as Weiss (1972) emphasized the importance of

having more evaluators who have the skills and equipment to carry out these evaluations

(Furubo & Stame, 2019). This renewed interest in evaluations can be seen as the “quantitative

expansions of evaluation” (Furubo & Stame, 2019).

The second wave of change in evaluation culture in the European nation states was

stimulated when Anglo-Saxon countries introduced more public sector reforms during the

1980s (Stame, 2003). Countries such the United Kingdom (UK) started to reform their

governments by introducing New Public Management (NPM) methods in government

institutions and policy evaluations (Stame, 2003). The third wave of change in evaluation

culture occurred when countries were encouraged by external factors to change their evaluation

culture in the public sector (Stame, 2003). For example, the EU could apply external pressure

to change evaluation culture in EU countries (Stame, 2003). These three waves have influenced

the way evaluations are carried out by different Member States in the EU. These sequential

waves have formed the context of the evaluation practice in the EU countries over the past few

decades, during which the practice of national policy evaluations was also developing.

Currently, there is an “obligatory pattern of evaluation” that is anchored in the

regulations of EU institutions such as the European Commission (the Commission) (Schwab,

2009, p. 116). The policy evaluations in member states are susceptible to these EU regulations.

This pattern entails three types of evaluation in the EU: 1) having ex-ante evaluations while

drafting legislation, 2) midterm evaluations during the implementation of policy programs, and

3) ex-post evaluations after the implementation phase and after funding policy programs.

Schwab’s (2009) findings suggest that evaluations of nation states’ EU-related policy programs

could be a driving force for the impact that the supranational EU institutions have on national

policy evaluations. Since the 1990s, it has been required for nation states to perform at least one

of these three types of evaluations per program or instrument of EU-related policies, such as

the Structural Funds of the EU Cohesion Policy (Schwab, 2009). It is evident that the

“quantitative expansions of evaluation” that had already started in the 1960s were reinforced

because of these EU requirements for policy evaluations (Furubo & Stame, 2019).

At the moment, these EU policy evaluation requirements for Member States have

changed into conducting both impact assessments and evaluations to improve the policymaking

process in at EU level (European Commission, 2020). “Impact assessments” here means

collecting evidence, such as findings coming from EU evaluations, to check whether future

actions of EU institutions are justified or are the best manner to achieve the policy objectives

being pursued (European Commission, 2020). As a supplement to these evaluations, there are

(9)

8

also so-called fitness checks that address how different legislative tasks have helped to achieve

certain EU policy goals (European Commission, 2020).

The impact of these evaluations on EU-funded policies may have a qualitative impact on

the evaluation of national policies, but so far this subject is under-researched. For instance,

Schwab (2009) has posed questions that have added value to the discussion about the qualitative

impact of policy evaluation on the national evaluation culture of a Member State, such as

whether the experience gained from these evaluations “contribute[s] to the development of

evaluation capacities for national instruments” (Schwab, 2009, p. 116). Another important

question derived from Schwab’s research is whether the importance of evaluation is growing

in national policymaking. The discussions that these questions produce are focused on the

impact that European policies have on national evaluation culture (2009). All of the countries

that have established their national evaluation culture since the 1990s, such as Southern

European nations like Italy, were required to do so due to some kind of external pressure such

the establishments of the EU (Furubo, Sandahl, & Rist, 2002).

However, it is not clear what impact the EU has on the qualitative aspect of policy

evaluation in Member States, namely the national evaluation culture (Furubo & Stame, 2019).

Therefore, this research will examine how the EU has impacted the national evaluation culture

in one of the Member States to better understand this research puzzle. The Netherlands is one

of the countries that has endured strong public sector reforms in recent decades, and it is

typically defined as a country in which that policy evaluation was placed on the political agenda

due to second-wave stimuli, which may lead to interesting outcomes of this research (Furubo,

Sandahl, & Rist, 2002, p. 94). This second-wave stimuli forced countries to adopt evaluation

and integrate evaluation policy evaluation in public governance (Furubo & Stame, 2019). Some

evidence has already been found for the impact of EU-related policies on evaluation, even in

domestic regional policymaking, and this thesis aims to build upon this kind of research

(Schwab, 2009; Furubo & Stame, 2019).

This thesis’s main research goal is to find out whether the EU has impacted the evaluation

culture in the Netherlands and if so, how. To reach this goal, it is important to choose a policy

field in which the EU has imposed certain regulations that may have changed the national

evaluation culture of Member States as a case to study.

(10)

9

To perform this research, the following research question was formulated:

Has the European Union impacted the evaluation culture of Dutch policies at the

national level, and if so, how?

The term “Dutch policies” refers to policies at the national level in the Netherlands that are

susceptible to EU regulations. These Dutch policies have to be evaluated by the Dutch

authorities within the regulations and directives of the EU evaluation framework. Also, the EU

has created regulations that the Member States have to follow while performing these

EU-related policy evaluations. To find an answer to this research question, is it important to gain

clarity about the evaluation practices of the Member States. This research attempts to discover

the impact of the supranational EU institutions on the national culture of policy evaluation in

the Netherlands. A secondary goal is to gain more knowledge about the EU’s impact on the

Netherlands as a second-wave country in the third-wave era, wherein the EU is seen as an

external pressure for possible change in the national evaluation culture of Member States.

Seeing the Netherlands primarily as a second-wave country provides the ability to determine

what changed in the evaluation practice of this Member State in the third-wave era. In sum, this

research aims to explore how the evaluation culture at the national level (in this case, in the

Netherlands) is affected by the EU.

The starting point of this thesis is using the theoretical lens of Europeanization to

examine the behavior of EU institutions and explain developments in the evaluation cultures of

Member States. Europeanization is a central concept in European Union policymaking

scholarship (Menz, 2011). The analytical top-down approach of Europeanization is suitable for

explaining how it impacts policy evaluation at the national level through different adaptational

pressures such as coercive, mimetic, or normative behavior of institutions (Menz, 2011, p. 438;

Featherstone & Radaelli, 2003, p. 30; Dimaggio, 1999).

In empirical terms, this research examines the impact the EU has had on the evaluation

culture of the EU Cohesion Policy (CP) in one Member State, the Netherlands. This thesis does

so by examining the evaluation process of the CP, the Structural Funds, in the most recent

program periods from 2007-2013 and 2014-2020. The policy area selected for this research is

the CP in the EU, with a focus on its Structural Funds as a policy instrument of the EUCP

(European Union Cohesion Policy) in the different Member States. The EUCP was developed

shortly after the introduction of the European Regional Development Fund (ERDF) in the

mid-1970s (Manzella, 2009). Thus, this policy area has developed and been evaluated over the past

few decades, which makes it suitable for the purpose of this thesis.

(11)

10

The scientific relevance of this research becomes apparent in the following arguments.

First, it builds upon the work of different scholars such as Schwab (2009, p. 116) and his quest

to find the impact of EU policies on EU Member States. He wished to discover whether there

had been Europeanization of national evaluation cultures in the EU Member States. This thesis

makes it possible to examine whether his assumption can be confirmed empirically because it

explores different mechanisms, such as coercive or normative behavior, to explain the top-down

and vertical lines of influence between the EU and Member States.

Second, there is a gap in knowledge about the qualitative effects of the EU’s current

impact on national evaluation culture. Decades ago Weiss (1972) stated that there are indeed

quantitative effects because there is government funding for evaluations at the national level in

the public sector. She also stated, decades later, that governments provide funding for

evaluation with restrictions to strengthen the standards and quality of policy research. In this

quest to improve policy evaluation, is it strange that government institutions can endure

pressure to account for policy evaluation expenditures (Msila & Setlhako, 2013). This almost

amounts to political pressure against having a well-established evaluation program and the

budget to carry out evaluations (Msila & Setlhako, 2013). Scholars have also indicated that

there is more to be discovered about the EU’s qualitative impact on policy evaluation and have

theorized about how this impact can be explained (Furubo & Stame, 2019). This thesis aims to

help close the knowledge gap surrounding these notions of qualitative effects of policy

evaluation by performing research about the EU’s impact on the evaluation culture in policy

evaluation.

Finally, by examining the evaluation culture of a nation state such as the Netherlands, it

is possible for other scholars to discover how the national policy evaluation cultures of similar

second-wave countries, such as the UK, work under the policy evaluation regulations enacted

by the EU institutions. Specifically, this thesis explores how understanding the evaluation

cultures of nation states can help to improve evaluation practices, for example by prompting

discussions and debates that stimulate the national discourse about evaluation.

Guide

This thesis is divided into five parts. The first chapter introduces the main research topic. The

second chapter examines the theoretical assumptions about the main research question in this

thesis and has two sections. The first section begins with a short literature study about

Europeanization in the EU and reveals how this analytical approach towards European

integration helps to build an evaluation culture in Member States’ policies. The theoretical

(12)

11

framework of this research is then presented in the second section. In the third chapter the

research design is clarified, and in Chapter 4 the case study is introduced. The fifth chapter

draws conclusions about the research, elaborates on the limitations of the research, and provides

some recommendations for further research.

2. Theoretical framework

This chapter presents the variables that are important to examine in this analysis. The first

section will briefly explain the history of policy evaluation in the EU and will also present the

definitions of the most important concepts in this research. This first consists of explaining the

history and function of policy evaluation in the EU. When considering national evaluation

cultures, this first section also illustrates how the key aspects of the research are related to each

other, so that it becomes clear how they are useful in answering the following research question:

Has the European Union impacted the evaluation culture of Dutch policies at the national

level, and if so, how?

The second section describes the theoretical framework of this research and presents the

expectations of the relationships between the key concepts derived from the theoretical

approach of Europeanization. The last section summarizes the expectations and details the

conceptual model that is derived from the entire discussion in Chapter 2.

2.1. Definitions of and debates about policy evaluation in the European Union

History of policy evaluation in the European Union

In all three waves of developments in evaluation practice, it is apparent that certain stimuli (e.g.,

American evaluation cultures and theories in the 1960s and 1970s, NPM in the 1980s and

1990s, and external pressure from the EU since the 1990s) created change in evaluation practice

in the EU. However, it remains unclear how evaluation practice has changed due to the latter

wave of external pressures and whether the first and second waves have left their traces in

current evaluation practice. Before addressing the theoretical aspects and approach to examine

policy evaluation in this thesis, this section presents a brief look at the history of policy

evaluation in the EU. This section will illustrate and explain what evaluation practice looked

like in 2019 before examining possible theoretical explanations for the impact of the EU’s

current evaluation practice on Member States.

In the twenty years from 1960 to 1980, evaluation developed in a remarkable way

(Furubo, Sandahl, & Rist, 2002, p. 226; Furubo & Stame, 2019, p. 9). According to Furubo and

(13)

12

Stame (2019, p. 10), the revolution in evaluation took off during the period from 1960-1980

(Furubo & Stame, 2019). A sudden awareness and increased study of evaluation practice started

in the 1960s, and scholars such as Weiss emphasized the importance of having more evaluators

who have the skills and equipment to carry out these evaluations (Furubo & Stame, 2019). This

renewed interest in evaluation can be seen as the “quantitative expansions of evaluation”

(Furubo & Stame, 2019). Weiss’s research is a perfect example of the quantitative

consequences of the renewed interest in evaluation (Weiss, 1972; Furubo & Stame, 2019, p.

10). According to Furubo and Stame (2019), the value of this excess burden would equate to

about 140,000 US dollars today. Due to this quantitative effect of the renewed interest in

evaluation in the 1960s, scholars view this as the formative period of evaluation in the US.

The diffusion of evaluation in the US was an idea that was developed in the 1960s

(Furubo & Stame, 2019). In the US, the government introduced the idea of “shared

responsibility” to solve policy problems locally with conservative measures, and to tackle

policy issues with new solutions in the form of experiments (Stame N. , 2008). These shared

responsibilities were the source of the spread of program evaluation in the US in the 1960s

(Stame N. , 2008). Some countries that later joined the EU adopted these American ideas in the

1960s and 1970s. When the EU countries were developing their own national evaluation

cultures, they did not build upon their longstanding European culture of carrying out evaluations

(Furubo & Stame, 2019). Instead, EU countries adopted the evaluation practice from the US

and therefore creating disconnection in the way how EU countries performed policy evaluation

and the new way of doing it. This disconnect between the new way of performing evaluations

coming from the US and the more original, more conservative way of doing so in Europe is

now seen as some heritage of the US (Furubo & Stame, 2019). Among scholars, this US

influence is seen as the first wave of developing national evaluation practices in what is now

the EU. The influence of the US in the 1960s and 1970s created a pessimistic attitude among

current EU Member States towards the new evaluation practice due to government policies that

failed in these years (Furubo & Stame, 2019, p. 43). These government failures meant that new

problems arose from the need to understand new programs that were introduced to solve societal

issues in current EU countries, especially because they were new programs that were

implemented in very complex and diverse contexts (Stame N. , 2008). Therefore, more

discussions emerged in the 1960s and ‘70s about, for example, taking the political context into

account when evaluating EU government policy programs (Stame N. , 2008).

The second wave emerged in the realm of evaluation practice because of this pessimistic

stance (Furubo & Stame, 2019). NPM transformed the public sector in the 1980s and 1990s by

(14)

13

following more private sector management styles and being focused on results instead of

improving the evaluation procedures (Furubo & Stame, 2019, p. 12). In other words, NPM is

more focused on managerial administration in making and evaluating policies and less focused

on the politics and bureaucratic administration that come with it (Furubo & Stame, 2019).

Examples of countries that introduced these NPM reforms were mainly those that had

Anglo-Saxon traditions, such as Australia, the UK, the Netherlands, New Zealand, and the

Scandinavian countries (Stame, 2003). Hood’s (1991) well-known article “A Public

Management for All Seasons?” prescribes the principles of NPM. These principles are

hands-on professional management in the public sector; explicit standards and measures of performance; greater emphasis on output control; shift to disaggregation of units in the public sector; shift to greater competition in the public sector; stress on private sector styles of management practice; and stress on greater discipline and parsimony in resource use. (Furubo & Stame, 2019, p. 43; Hood, 1991)

Of course, there are other scholars who have discussed which values prevail in policymaking

or in public administration itself.

The third wave began in the 1990s and was stimulated by the formation of the EU. In

fact, the foundations of the EU were established in the 1950s under the European Coal and Steel

Community/European Economic Community (European Union, 2019). This notwithstanding,

the EU as we know it was established in 1993 and created many changes, such as the completion

of the internal market and the Schengen agreement (European Union, 2019). In this wave, the

evaluation practice in the EU developed internationally in the form of networks, and the

communication between these networks became more important in the development of

evaluation practice in the EU.

Evaluation has become important to the EU in the present day. However, the Member

States have certain obligations to the EU regarding evaluation. Member States are obligated to

set up at least one ex-ante, midterm, or ex-post evaluation for EU policies (Schwab, 2009). One

policy to which this applies in Member States is the CP, which has better regulation guidelines

to assess employment, its social impact, and its funds (Schwab, 2009; European Commission,

2019). These types of evaluations provide learning experiences and possibilities to further

develop evaluation in the EU and its Member States (Schwab, 2009).

Evaluation culture, evaluation capacity, and institutions

Evaluation is such an integrated process in policymaking that it is almost taken for granted.

According to Furubo and Stame (2019, p. 14), there are multiple functions of policy evaluation

that must be clarified in order to understand the practice. The term “evaluation” is used in a

(15)

14

broad sense throughout the whole discussion in this thesis. Using this broad sense of this term

makes it possible to use this word throughout the thesis as some activity that secures

improvement of public policies without constantly having a predetermined position about what

evaluation exactly is in every sentence (Furubo & Stame, 2019, p. xiv).

Policy evaluation has multiple functions that have to be expounded on before diving

into the concepts of evaluation culture, evaluation capacity, and institutions. Evaluation has

received more attention by public sector policy makers and evaluators because of its ability to

increase effectiveness and efficiency in government expenditure (Rist, 1990). A large part of

the gross national product is used for government expenditures in most countries, and therefore

is it important to account for the funds coming from public funding (Rist, 1990). Another

function is strengthening government administration in order to help society. To do so, is it

necessary to have a systematic way of evaluating government with clear guidelines and enough

information about how to perform evaluations. Evaluation is also a useful tool for the feedback

process in government administration.

Throughout the years, there have been differences between the EU countries in their

evaluation cultures. According to Bachtler and Wren (2006), these differences are visible in

the evaluation capacities of the Member States. Evaluation capacity, according

to Bachtler and Wren, refers to “the availability of qualified and experienced evaluators, and

the presence of appropriate knowledge and skills among evaluators and evaluation

commissioners, and the institutional frameworks to manage evaluation and promote evaluation

practices” (2006, p. 149). Northern Europe, with countries such as the Netherlands, Germany,

and the Scandinavian countries, has strong national policies that have evolved have evolved to

become more effective over the past few decades (Bachtler & Wren, 2006). Southern Europe,

with countries such as Spain and Italy, has not been as successful in its development of national

evaluation cultures (Bachtler & Wren, 2006). In this thesis, the evaluation culture of a policy

domain is considered to be the more qualitative side of the evaluation practice, which falls in

line with Speer’s definition of it as namely “a mainstream preference for specific evaluation

practices, approaches, and systems” (Speer, 2012, p. 66). In this respect, it is important to keep

in mind that evaluations integrate certain values and orientations that evaluators or institutions

that commission evaluations, such as the European institutions, may have.

Schwab used the concepts of evaluation culture and evaluation capacity the two

concepts interchangeably and explains these terms as: “the ability of a political system to use

evaluation—and its affinity to do so” (2009, p. 117). In fact, has evaluation capacity often been

defined by other scholars in a narrower way. For instance, it can be the ability to carry out

(16)

15

evaluations in a more effective manner (Milstein & Cotton, 2000). Another narrower definition

of evaluation capacity is the scope of an organization to have resources or the motivation to

carry out, analyze, and deploy evaluations (Preskill & Boyle, 2008). However, this does not

mean that the concept of evaluation culture is difficult to use for research. If researchers are

interested in a concept that covers qualitative experiences in addition to evaluation capacity,

such as the “the role of ideas and attitudes” (Schwab, 2009, p. 117) or the norms and practices

in the evaluation practice of public sector programs, then evaluation culture serves those goals.

For this research, both concepts—evaluation capacity and evaluation culture—remain distinct

because this allows for a better understanding of how to separate certain aspects of evaluation

practice in the empirical world. In this respect, evaluation capacity is seen as the more

quantitative side of evaluation and is interlinked with evaluation culture. Some aspects of De

Peuter and Pattyn (2008) are used to explain the interlinkages between evaluation capacity and

evaluation cultures.

A third element that is helpful in explaining evaluation culture in empirical research is

the definition of institutions. After all, this research is focused on explaining the impact of EU

institutions on the evaluation cultures in the policy domains that are subject to their regulations.

Borrowing from the perspective of historical institutionalism, institutions are defined as “the

formal or informal procedures, routines, norms and conventions embedded in the organizational

structure of the polity or political economy” (Hall & Taylor, 1996, p. 6). This perspective views

institutions as political entities that were set on certain trajectories in the past. Due to their

regulations or decisions, these institutions have to face certain consequences. In line with other

historical institutionalists, this research sees institutions as “organizations and the rules or

conventions promulgated by formal organization” (Hall & Taylor, 1996, p. 7). This perspective

is suitable for explaining how the EU institutions, such as the European Commission or its EU

agencies, promulgate their rules and regulations to other institutions such as Member States.

This view also allows one to see regulation or policies as rules or certain conventions that the

formal EU institutions are trying to uphold. In this respect, the institutional arrangements and

the “supply of evaluators” are mapped out in certain policy sectors (Speer, 2012, p. 70).

Furthermore, these institutions create the necessary data for evaluation by at once pursuing their

organizational goals and “creating data infrastructures that support certain evaluation designs”

(Speer, 2012, p. 70). In the following paragraph, an analytical perspective is explained that will

help to formulate expectations about how the EU evaluation practice and institutions may have

an influence on the national evaluation culture of a Member State.

(17)

16

2.2. The analytical framework: Europeanization

Definition

The theoretical approach of Europeanization falls under the denominator of European

integration theories and is important in order to answer the research question. Europeanization

attempts to explain the system of how European integration, as in the EU institutions on a

supranational level, impacts the Member States (McGowan, 2007, p. 12; Featherstone &

Radaelli, 2003).

Scholars have debated for decades about how Europeanization can be conceptualized in

research and used as a conceptual framework (Featherstone & Radaelli, 2003, pp. 6-12).

However, Europeanization is not always used as the only pathway to arrive at the final answer

about EU-Member State dynamics. This analytical perspective is often used in combination

with other theoretical frameworks such as new institutionalism, multilevel governance, and

policy networks (Featherstone & Radaelli, 2003). This thesis only focuses on the theoretical

framework of Europeanization due to its applicability for investigating the impact of a

supranational institution such as the EU on the policies of Member States. As a

conceptualization of Europeanization, this thesis will uphold the following definition of

Radaelli and Featherstone:

processes of (a) construction (b) diffusion and (c) institutionalization of formal and informal rules, procedures, policy paradigms, styles, ‘ways of doing things’, and shared beliefs and norms which are first defined and consolidated in the EU policy process and then incorporated in the logic of domestic (national and subnational) discourse, political structures, and public policies (Featherstone & Radaelli, 2003, p. 17).

This analytical Europeanization perspective within European integration theories is

suitable because it is able to answer questions related to “the role of domestic institutions in the

process of adaptation to Europe” (Featherstone & Radaelli, 2003, p. 33). The kind of research

that is designed using Europeanization as an analytical approach always has the condition of

strengthening or limiting Member States’ space to make their own decisions. By

using Europeanization, the EU as a supranational institution can therefore be understood as a

venue involving multiple processes, policies, bargains, diffusions, and interactions with

national-level actors (Featherstone & Radaelli, 2003). Second, this perspective can be used for

organizations and individual actors even though it does not explicitly mention

them (Featherstone & Radaelli, 2003). It does so by explaining organizational behavior and

individual behavior and covering the interests of scholars by taking a closer look at the “political

structure, public policy, [and] identities” (Featherstone & Radaelli, 2003, p. 30).

(18)

17

However, there are other theories of European integration that seem suitable for this

research. For instance, one of these theories is neofunctionalism. Neofunctionalism is

applicable because of its ability to use spillovers to explain how the EU institutions could

influence policies at national level via spillovers (Tranholm-Mikkelsen, 1991; Niemann, 2017).

When taking a closer look at neofunctionalism, this theory is not suitable to answer the main

research question because this research is only focused on one specific policy area. In order to

use neofunctionalism properly, it would be necessary to understand the EU’s impact on the

evaluation culture in at least two policy areas in one Member State. By doing so, it would be

possible to explain the dynamics between these policy areas in terms of spillover effects.

In general, there are two directions of Europeanization, namely horizontal and vertical.

Vertical Europeanization refers to two different levels: the EU level, where policy is

established, and the national level where this policy has to be metabolized by Member States

(Featherstone & Radaelli, 2003, p. 41). In this model, Member States are pressured to adopt

policy that is made at the EU level. By contrast, in horizontal Europeanization there is decidedly

no pressure coming from the EU level to conform to a certain policy that is made at this

supranational level (Featherstone & Radaelli, 2003, p. 41). Due the fact that this thesis is

interested in theories that can explain the vertical rather than the horizontal process of

Europeanization, this vertical process of Europeanization is the basis upon which the theoretical

framework has been designed. In the quest to conduct an institutional analysis of the impact of

the EU on one of its Member States, the work of Featherstone and Radaelli (2003, p. 41) has

guided the design of the theoretical framework. This thesis can be placed under other research

that is interested in finding answers through the vertical process of Europeanization.

According to Europeanization and new institutionalism, adaptational pressures are

explained by the mechanisms of coercion and mimetic and normative behavior (DiMaggio &

Powell, 1983). This way of explaining how these EU institutions influence Member States looks

very similar to organizational analysis from the perspective of new institutionalism (DiMaggio

& Powell, 1983; Featherstone & Radaelli, 2003). Contemporary institutional theory is based on

new institutionalism but also incorporates certain aspects of older institutional economics from

authors such as DiMaggio and Powell (1983; 2012). These theories explain the impact of

institutional pressures on public sector organizations (DiMaggio & Powell, 1983; 1999;

Carpenter & Feroz, 2001). It promotes the idea that these pressures coming from the external

environment of public institutions in the form of “coercive legislation, professional norms, and

mimetic examples” may impact the practices inside a certain public sector organization

(Keerasuntonpong, 2018, p. 1171). Featherstone and Radaelli (2003, p. 41) provide a lens

(19)

18

through which to examine the mechanisms of coercion, mimetic and normative pressures to

explain the impact of the EU on a Member State. This becomes possible by seeing this impact

as a process of institutional change that occurs through these three mechanisms.

The first mechanism of adaptational pressure is coercion, which takes place when, for

example, institution A is dependent upon, is forced by, or is somehow culturally expected to

change because of institution B (Greenwood & Meyer, 2008). For example, because of coercive

behavior by EU institutions, Member States are obliged to carry out regulatory arrangements

before certain EU directive or regulatory deadlines. Thus, scrutinizing or reviewing the actions

of public officers could improve the accountability of the public sector in a Member State within

strictly coercive legislation (Keerasuntonpong, 2018). This behavior is very forceful top-down

steering by the EU institutions.

The second mechanism is mimetic behavior. This mimetic behavior between institutions

comes mainly from the institutions’ uncertainty, due to which they copy their role models that

appear more rational or advanced (Greenwood & Meyer, 2008). This insecurity about possible

solutions stimulates organizations to imitate successful examples from other organizations

(Keerasuntonpong, 2018). This amounts to EU countries adjusting to EU rules and legislation

to obtain greater economic, social, or other desired benefits, and other Member States join in to

have the same benefits (Featherstone & Radaelli, 2003). In this sense, institutional change

occurs with one institution because the institution itself or its entrepreneurs are imitating the

successful behavior of other institutions because this behavior seems attractive in solving their

own institutional problems (Beckert, 2010).

A third mechanism is normative pressures, which depends on how attractive one

institutional model appears for another institution. This is closely linked with the socialization

processes of institutions (Beckert, 2010; DiMaggio & Powell, 1983). When public sector

institutions fear losing their reputation or legitimacy, bad publicity could arise if they do not

address the criticism of the principal EU institutions to which they are subject. Professional

training and networks reinforce the mechanism of normative pressures coming from one

institution to another. Professional training influences the “cognitive and normative frames that

shape their perspectives on regulative goals and the likely means to achieve them” (Beckert,

2010, p. 156). Socialization creates the routines and institutionalization of practices in a certain

profession, policy area, or of an institution itself. Through professional networks, it is possible

for institutions to diffuse their norms and standards across boundaries, as in this case from the

EU level to the national level, and therefore create similar ideas about how to tackle regulative

(20)

19

problems or how to evaluate certain policy areas and aspects thereof (Beckert, 2010; DiMaggio

& Powell, 1983, p. 152).

Expectations

Radaelli’s (2003, p. 13) definition of Europeanization is central in this thesis. Radaelli (2003)

claims that there are three processes—construction, diffusion, and institutionalization—that are

integrated into the policymaking process at the EU level. These processes are then diffused to

the public policies, politics, and discourse at the (sub)national level. For example, the process

of institutionalizing certain formal regulations at the EU level can affect Member States’ public

policies. This thesis argues that the way the EU affects national policies could be through the

three causal mechanisms of diffusion.

Therefore, the EU institutions can have an impact on national politics, policies, or

discourse through diffusion. In EU policymaking, it is possible to speak of policy diffusion

through aspects that originate from sociological institutionalism (DiMaggio & Powell,

1983). These aspects refer to certain behavior or pressure of public institutions. National public

policies can change through “the mechanisms of coercion, mimetism, and normative

pressures” (DiMaggio & Powell, 1983; Featherstone & Radaelli, 2003, p. 41). In this way, it

becomes possible to see the similarity between the European and national evaluation cultures

because the national evaluation practice experiences certain coercive, mimetic, or normative

pressures from the European evaluation practice. For example, formal EU institutions can create

certain coercive pressures by setting up regulations that will change the way evaluations are

carried out. Based on the theoretical lens of Europeanization, this research aims to understand

whether and how EU institutions may have an impact on the Dutch evaluation culture through

diffusion. This means that this research explores in detail whether there are adaptational

pressures such as mimetic, coercive, or normative pressures coming from EU institutions on

Dutch evaluation culture. Three expectations are formulated and will guide this research to

better understand how the influence of the EU affects the evaluation culture of the Netherlands.

The first expectation entails explaining the impact of the EU on Dutch evaluation culture

by the mechanism of coercion. Via coercion, the formal and informal pressures coming from a

supranational EU institution on Dutch national institutions create a more similar evaluation

culture between these two levels in one policy area. This could happen because how a certain

policy area is evaluated in the Netherlands could be dependent on how the EU would like to

have its Member States evaluate this specific policy area. The first expectation is formulated as

follows:

(21)

20

1. EU institutions have an impact on Dutch evaluation culture in the policy area of CP

through the mechanism of coercive behavior.

The second expectation refers to mimetic behavior as an explanation for the impact of

the EU on Dutch evaluation culture via the mechanism of mimetism. As mentioned earlier,

uncertainty is a very important element in explaining this impact through mimetism. If certain

EU institutions evaluate a certain policy area using certain rules or techniques, this EU-level

evaluation culture become very attractive for Member States. Also, if EU institutions follow

certain advanced institutional evaluation models, it may become convenient for Member States

to copy this evaluation model in their own national evaluation cultures. To examine whether

this is happening in the Netherlands in the area of CP, the next hypothesis is formulated as

follows:

2. EU institutions have an impact on Dutch evaluation culture in the policy area of CP

through the mechanism of mimetic behavior.

The third expectation helps to unravel the impact of EU institutions on Dutch evaluation

culture in the policy area of CP through possible normative pressures. These normative

pressures can take the form of professional training or professional networks wherein the EU

institutions can diffuse their norms and standards about policy evaluation to other Member

States concerning a given policy area. To examine whether this is taking place in the

Netherlands in the area of CP, the next hypothesis is formulated as follows:

3. EU institutions have an impact on Dutch evaluation culture in the policy area of CP

through the mechanism of normative behavior.

2.4. Overview of the expectations and the conceptual model

Scholars have not determined what the precise qualitative impact, thus referring to a concept

that covers qualitative experiences in addition to evaluation capacity, policy evaluation has

been on evaluation cultures in the EU (Jacob, Speer, & Furubo, 2015; Schwab, 2009). Figure

1 shows the conceptual model of the theoretical framework in this research to illustrate the

possible relationships between the explanatory variables (independent variables) and the

variables of response (dependent variables). The dependent variable is the national evaluation

culture within a certain policy area that is subject to EU obligations. The independent variables

are the adaptational pressures coming from the EU institutions, namely coercive behavior,

mimetic behavior, and normative behavior.

(22)

21

Figure 1: Conceptual model explaining the impact of EU institutions on Dutch evaluation culture

Category Nr. Expectation

Coercive

behavior

1

EU institutions will have an impact on the Dutch evaluation culture in

the policy area of CP through the mechanism of coercive behavior.

Mimetic

behavior

2

EU institutions will have an impact on the Dutch evaluation culture in

the policy area of CP through the mechanism of mimetic behavior.

Normative

behavior

3

EU institutions will have an impact on the Dutch evaluation culture in

the policy area of CP through the mechanism of normative behavior.

(23)

22

3. Methods

3.1. Introduction

This chapter explains the research design, data collection, and data analysis methods, and how

the different variables as presented in Figure 1 in Chapter 2 were measured. To perform this

research, the following question was formulated:

Whether and how has the European Union impacted the evaluation culture of Dutch

policies at national level?

By means of a qualitative research design with case study research and hermeneutic

interpretation techniques, this thesis attempts to formulate an answer to this question.

Considering the research question, it is important to select a case study of a policy which

requires EU funds. It is also important to choose a case in which an EU policy has been around

during the waves of evaluation reforms in the EU. For this thesis, a single case study has been

chosen: Dutch CP.

3.2. Research design

The research design used in this thesis is qualitative research. The reason for choosing this

research design is because it allows one to formulate an answer by collecting data that is

strongly related to the key concepts in the theoretical framework and the main policy area of

this research. The research methodology consists of an explanatory case study research design.

The case selection in this thesis comprises the policy instruments of the EUCP, namely the

Structural Funds in the Netherlands. The form of the study design is a single case study.

A single-case study design is a study based on research analysis of one case instead of

multiple cases (Toshkov, 2016). This study allows one not only to focus on the link between

the independent and dependent variables, but also creates the opportunity to explore other

expectations and mechanisms of a particular theory in the research. Similar to detective work,

the explanatory case study research in this research focuses on finding empirical evidence for a

very small number of expectations (Toshkov, 2016). The added value of this type of research

primarily comes from the strong connection between the empirical case that is chosen by the

researcher and its applicability to the theoretical basis of the research. Theory is an important

determinant of the quality of the research and also guides this kind of research. Using a

well-established theory, such as the theory of Europeanization, as the fundament of this thesis is

useful in answering the main question of this research.

(24)

23

This thesis follows the steps of the case study research design method as detailed by

McNabb (2018, p. 287). These steps consists of the following elements: 1) framing the case, 2)

operationalizing relevant themes, 3) selecting and defining case units for data analysis, 4)

gathering case data with qualitative data collection methods, 5) analyzing and interpreting

collected data by organizing it around key concepts and identifying key constructs upon which

to base interpretation, 6) developing generalizations from data for interpretation, 7) testing

alternative interpretations, and finally 8) forming and/or refining the theory based on the case

and presenting a comprehensive report of the case (McNabb, 2018, p. 286).

A way of considering the data in this single-case study research is hermeneutics, which

allows one to understand and interpret the meaning of qualitative data that has been collected

(McNabb, 2018, p. 365). The emphasis in hermeneutics lies on the multiple ways of

understanding the information, and for this research is it preferable to also take other meanings

and understandings into account while analyzing the data (McNabb, 2018). Taking other

meanings into account can take certain information in the right context or perspective while

analyzing data. This way of interpretation depends heavily upon critical thinking. To structure

the analysis, the researcher chose to follow the hermeneutic circle, because an “understanding

of the whole text occurs as individual parts of the communication are analyzed and understood,

thereby contributing new knowledge to an understanding of the whole” (McNabb, 2018, p.

373).

In this research is qualitative data was gathered, coded, and conceptually ordered

(McNabb, 2018, p. 94). The five-step procedure for case study analysis, as described by

McNabb, guided the analysis in combination with interpretation of the gathered data through

hermeneutic analysis and using computer assisted tools such as Atlas.ti (2018, pp. 94, 291). In

later sections of this thesis, the empirical findings are explained and the expectations formulated

in the theoretical framework in Chapter 2 are confirmed or rejected. The research analysis and

results are reported in Chapter 4, “Case study report.”

3.3. Data collection methods

This research aims to use documents related to the evaluation of the EUCP in the Netherlands

and interviews with experts in this policy field to understand the impact of the EU on Dutch

evaluation culture. The qualitative data collection methods are document analysis and

interviews (McNabb, 2018, p. 290). The documents collection procedures and analysis are

clarified in Section 3.5 of this chapter, “Operationalization and procedure.” This paragraph

explains the procedure in respect to performing semi-structured interviews. The interviews with

(25)

24

experts and policymakers in the field were performed in the attempt to triangulate the data. The

procedure for performing the interviews was as follows: 1) sending consent forms and having

them signed, scanned, and sent back via email before the interviews as presented in Appendix

D; 2) performing in-depth semi-structured interviews, which consisted of questions about the

main themes in the thesis as presented in Appendix C; and 3) transcribing the interviews.

The target group for this research was policymakers and evaluators in the field of EUCP

in the Netherlands. This target group was chosen because it consists of the people who indeed

carry out these evaluations and experience the impact of the EU in performing their work. By

contacting them, it became possible to truly understand the dynamics of policymakers and

experts between the EU level and national level concerning EUCP.

Contacting important actors, such as policymakers at the Ministry of Social Affairs and

Employment and Dutch evaluators of the EUCP, went very well thanks to snowball sampling.

The snowball effect means that respondent 1 recommended contacting respondents 2 and 3, and

therefore allowing the researcher access to other important respondents. By contacting the

respondents through email and telephone, it was possible to plan four face-to-face interviews

and a short conversation over the phone with the fifth respondent. In contacting these

respondents, extra data was gathered from these respondents that could only have been accessed

with their help. These documents, such as transcripts of the interviews, were also used as extra

material for the thesis analysis in Chapter 4. The transcripts of all the interviews can be found

in Appendix E.

However, this method of collecting data is not as practical as it seems for the validity

and reliability of this research. Allowing interviewees to recommend other potential

respondents opens the door for biased information. Even if the information that is gathered is

illuminating in answering the research question, the risk remains that the research has not

questioned other possible respondents because of the potential biases of the first respondents

contacted. It is possible to refine the results using the insights of the theoretical perspectives,

but this adjustment through theory is only helpful to some extent (Toshkov, 2016). Due to time

limits and a lack of responses from other potential interviewees, the data collection of this thesis

is limited in terms of its validity and reliability.

This does not mean either that the data gathered was not helpful or that the respondents

offered a comprehensive understanding of the main topic. This research provides preliminary

answers to the main research question and opportunities for further research about evaluation

culture. Despite the convenience, self-selected respondents do not always provide “valid and

reliable inference about the” target group (Toshkov, 2016, p. 134).

(26)

25

To clarify the background and expertise of each respondent, their functions in their

organizations and areas of expertise are briefly explained below. Respondent

1

is

a

coordinating policy advisor at the Department of Innovation and Knowledge of the Directorate

of Innovation and Knowledge of the Ministry of Economic Affairs and Climate (EA).

Respondent 1 is responsible for the coordination of the policy program and evaluation of one

of the largest Structural Funds of the EUCP in the Netherlands, namely the ERDF. The Ministry

of EA is responsible for the ERDF, but not for the European Social Fund (ESF) which also is a

part of the EUCP in the Netherlands. The Ministry of Social Affairs and Employment (SAE) is

responsible for the ESF, but the Ministry of EA is only responsible for the ERDF.

Respondent 2 is a senior knowledge and research advisor in the Directorate

Implementation of the Ministry of SAE of the Netherlands. Since 2009, the respondent has been

involved in evaluations within the framework of the ESF as research coordinator, knowledge

and research advisor, and project manager for ESF evaluation from the principal side (ESF

Management Authority). Respondent 3 investigates and monitors evaluations of the ESF,

including implementation of the ESF on the process side and the effect side. Respondent 2 and

his colleagues manages the agency responsible for the ESF, which is the Agency of Social

Affairs and Employment, and determine the policy (what the ESF will look like), and which

investigations should be done. This Agency is distinct from the Ministry of SAE as mentioned

above.

Respondent 3 is senior policy advisor and deputy head of the Participation and

Decentralized Social Services Department (PDSSD) of the Ministry of SAE of the Netherlands.

This respondent was responsible for the policy of the ESF Operational Program 2007-2013 and

2014-2020. The respondent set up the 2014-2020 ESF program in the Netherlands and is also

responsible for the program. Respondent 3 also has knowledge about the 2000-2006 program

of the ESF, is involved in the implementation of the ESF program, and has much expertise

about the ESF program over the 2007-2013 program period. The respondent is strongly

involved with coordination of the ESF in the Ministry of SAE and is always informed of new

developments around the ESF.

Respondent 4 is a social domain market manager at a research agency in the

Netherlands. In practice, respondent 4 leads a great deal of research, mainly in the field of the

labor market and education. There is also a link with the ESF. Respondent 4 has more than

thirty years of experience in evaluation studies in the social domain, not only in the Netherlands

but also for the European Union and is a member of the European evaluation network. Together

with other researchers he performed a study in the Netherlands in the 1980s into the evaluation

(27)

26

culture at Dutch ministry departments, and later set up the policy research bureau. The

respondent has been researching the ESF since the 1992-1999 ESF programming period. He

became involved in monitoring ESF projects while working at the employment office in

Groningen in 1986. In the period from 2000-2006, he worked closely with European

Commission (Commission) to carry out the mid-term evaluation of the ESF in the Netherlands.

He also executed the same ex-ante evaluations for the new Member States, such as for Romania

in the 2004-2006 period and Hungary for the 2007-2013 program period and simultaneously

carried out the ex-ante evaluation for the Netherlands in the program period 2007-2013.

Afterwards, the respondent also contributed to the design of the ESF program for 2021-2027 at

the European level in collaboration with colleagues from other research agencies. This may

contribute to how specific parts of the ESF program are arranged differently at the national

level.

Respondent 5 works at the country desk of the EUCP Directorate-General (DG)

Employment, Social Affairs and Inclusion of the European Commission. The respondent is a

team leader for the Netherlands at the Country Desk of the European Commission in Brussels

and is an important user of all the ESF evaluation studies performed by the Ministry of SAE.

The authority and knowledge of all these respondents helped to analyze the documents and to

draw conclusions about the theoretical expectations by comparing the results of the document

analysis with those coming from the interviews.

3.4. Data analysis methods

When analyzing the data, it is important to follow certain phases in the case study analysis. In

order to perform qualitative research analysis, the data was analyzed with computer-assisted

tools. In this thesis, these computer-assisted routines with prepackaged software were

performed with Atlas.ti. This computer tool helps to code and categorize long texts in the

reports or documents and also helps to structure all the data (Yin, 2009, pp. 127-128). It is not

a substitute for performing qualitative data analysis oneself, but in fact helps maintain structure

and easily explain relationships between indicators and eventually between the independent and

dependent variables (Yin, 2009).

The procedure for the data analysis of the case study as follows: 1) “grouping the data

according to key concepts,” 2) “identifying bases for interpretation,” 3) developing

generalizations from the data, 4) testing other interpretations or explanations, and 5) comparing

findings or generalizations from the case study with the expectations in the theoretical

Referenties

GERELATEERDE DOCUMENTEN

This is in great contrast to the CCPA (which is elaborated on later), as in the case of the CCPA, the regulations only apply to organizations with a certain number of consumers,

This relationship, between the insured and the insurer, although the above is very basic to insurance law, is complicated by the principles involved in

The most significant changes in the Maasai culture are the dependency on tourism since the Maasai earn their income with it (see paragraph 6.1), the change in

STePS scholars are highly engaged in the international communities of STIS, through numerous collaborative research projects, multiple publication efforts, and active contribution

These reports comprise, firstly, a report of the Japanese government to the International Atomic Energy Agency (Government of Japan 2011), secondly, a report from the

In this paper the scores of the member states on compliance with the EU guidelines as given by the European Commission in the policy document national Roma Integration Strategies:

requirement but that of nationality of a Member State to claim citizenship rights under the Treaty, implying that it is not necessary to have exercised the right to move. In

However, despite this paper does not find significant effects of corporate governance variables of EU-targets on the acquisition announcement abnormal returns, this paper does