Increasing knowledge utilization
in the Netherlands:
individual efforts or structural
changes?
Master thesis educational sciences University of Amsterdam Sanne Kruijer, 10017453 Thesis supervisor: mw. dr. H. Kosar‐Altinyelken Second reader: mw. prof. dr. M.L.L. Volman Amsterdam, 23th of August 2015Increasing knowledge utilization in the Netherlands:
individual efforts or structural changes?
Master thesis educational sciences University of Amsterdam Sanne Kruijer, 10017453
Thesis supervisor: mw. dr. H. Kosar-Altinyelken Second reader: mw. prof. dr. M.L.L. Volman Amsterdam, 23th of August 2015
Table of contents Abstract ... 1 Acknowledgements ... 2 List of Acronyms ... 3 Chapter 1. Introduction ... 4 1.1 Problem statement ... 4
1.2 Aim and focus ... 5
1.3 Research questions ... 5
1.4 Scientific relevance ... 6
1.5 Societal relevance ... 7
1.6 Thesis outline ... 7
Chapter 2. Theoretical framework ... 8
2.1 Defining educational research, practice and policy ... 8
2.2 Defining knowledge utilization ... 9
2.2.1 Linear model ... 9
2.2.2 The relationship model ... 10
2.2.3 The systems model ... 11
2.3 The importance of knowledge utilization ... 12
2.4 Knowledge utilization activities ... 13
2.5 Challenges for knowledge utilization ... 15
2.6 Strategies to improve knowledge utilization ... 16
Chapter 3. The Dutch educational research context ... 18
3.1 A wide variety of stakeholders ... 18
3.2 The Dutch research funding system ... 19
3.3 The establishment of the NRO ... 20
3.4 Organisational structure of the NRO ... 22
3.4.1 Formulating research agenda’s ... 22
3.4.2 Allocating research budgets ... 23
Chapter 4. Research methodology ... 25
4.1 Research design ... 25
4.2.1 NRO documents ... 26
4.2.2 Research proposals and evaluations ... 26
4.2.3 Semi-structured interviews ... 27
4.3 Research sample ... 28
4.4 Data analysis ... 30
4.4.1 Document analysis ... 31
4.4.2 Policy documents and interview analysis ... 31
4.5 Validity of the research project ... 32
4.6 Limitations of the research project ... 33
4.7 Ethical considerations ... 34
4.8 Activities for knowledge utilization ... 34
Chapter 5. Approaches towards knowledge utilization ... 35
5.1 Definitions of knowledge utilization ... 35
5.1.1 Definitions by the NRO ... 35
5.1.2 Definitions by educational researchers ... 37
5.1.3 Definitions by policymakers ... 38
5.1.4 Definitions by practitioners ... 38
5.2 Importance of knowledge utilization ... 40
5.2.1 Perspectives within the NRO ... 40
5.2.2 Perspectives of educational researchers ... 42
5.2.3 Perspectives of policymakers ... 44
5.2.4 Perspectives of practitioners ... 44
5.3 Activities to increase knowledge utilization ... 46
5.3.1 Activities by the NRO ... 46
5.3.1.1 Facilitating the active involvement of practitioners and policymakers ... 46
5.3.1.2 Evaluating and financing different educational research projects... 47
5.3.1.3 Knowledge utilization during research projects ... 49
5.3.1.4 Increasing the availability of research knowledge ... 50
5.3.1.5 Network creation ... 51
5.3.1.6 Increasing knowledge about knowledge utilization ... 52
5.3.2 Activities by researchers ... 52
5.3.2.1 Traditional and innovative dissemination activities ... 52
5.3.2.2 Proposal evaluation ... 53
5.3.2.4 Extending existing networks directly ... 55
5.3.3 Activities by policymakers ... 56
5.3.4 Activities by practitioners ... 56
Chapter 6. Challenges and possible strategies ... 58
6.1 Challenges for knowledge utilization ... 58
6.1.1 The complexity of educational research knowledge ... 58
6.1.2 The fragmentized character of the Dutch educational domain ... 59
6.1.3 Language use, timing and other practical challenges ... 60
6.1.4 Fundamental differences between communities ... 61
6.1.5 Conflicting or a lack of incentives and cultures ... 62
6.1.6 Knowledge utilization by whom? ... 63
6.1.7 The complexity of the improvement of educational practices ... 64
6.2 Strategies to enhance knowledge utilization ... 66
6.2.1 Suggestions that relate to the role of the NRO ... 66
6.2.1.1 Structuring educational research knowledge ... 66
6.2.1.2 Strengthening the coordination function of the NRO ... 67
6.2.1.3 Investing in structural networks ... 68
6.2.1.4 Increase knowledge utilization in educational practice ... 68
6.2.1.5 The need for experimentation and reflection ... 69
6.2.2 Suggestions relating to the role of educational researchers ... 70
6.2.3 Suggestions relating to the role of policymakers ... 71
6.2.4 Suggestions relating to the role of practitioners ... 71
Chapter 7. Conclusion and discussion ... 74
7.1 The approaches and perspectives of different stakeholders ... 74
7.2 Overall conclusion ... 80
7.3 Theoretical discussion and implications ... 81
7.4 Recommendations for research, policy and practice ... 83
7.5 Recommendations for future research ... 85
References ... 87
Appendix 1 Interview guideline NRO employees ... 1
Appendix 2 Interview guideline researchers, practitioners, policymakers ... 2
1
Abstract
Debates about the limited impact of research on educational practice and policy have resulted in a long list of (inter)national efforts to establish connections between research, practice and policy. In the Netherlands, a central research coordinating body: the Netherlands Initiative for Educational Research (Dutch acronym: NRO), was established in 2012 to bridge the gap between research, practice and policy. The aim of this study is to understand the complex mechanisms underlying this gap and to gain insight in possible strategies to bridge it by addressing and comparing the approaches, experienced challenges and suggested strategies of multiple stakeholders with regard to knowledge utilization. To achieve this, data were
collected from NRO policy documents, research proposals, proposal evaluation reports and 46 semi structured interviews with NRO policy officers, researchers, practitioners and
policymakers. The results show that the gap between research, practice and policy is complex and differentiated, because of a wide variety of possible definitions of educational research, practice and the relation between both. The findings also show that efforts to establish connections and the eventual impact of these connections on educational improvement are highly depended on personal, social and structural factors that lay beyond the reach of individual researchers, policymakers, practitioners and the NRO. Therefore, respondents suggested that system-wide changes are necessary to bridge the gap. These changes include developing structural networks within and between research and practitioners organisations in the broad educational domain, implementing knowledge utilization incentives in these institutions and investing in knowledge infrastructures to improve the availability and exchange of both practitioners and research knowledge.
Key words: knowledge utilization, research gap, the Netherlands Initiative for Educational
Research, the Netherlands, knowledge exchange, knowledge mobilisation, knowledge dissemination, professionalization, educational improvement
2
Acknowledgements
In 2014 I started an internship at a recently established educational research cluster of a municipality in the Netherlands. My task was to formulate policy advices on the basis of recent research insights to inform politicians, policy advisors and policymakers about educational issues. While I had a great time during this internship, I often struggled with the fact that I had to give advice by describing my findings in a short review. The lack of interaction with decision makers during this process left me insecure about the level of influence I could have. This led me to the formulation of the research topic of this master thesis: the often discussed gap between educational research, practice and policy. After writing this thesis, I realized I experienced first-hand some of the core issues with regard to knowledge utilization. The parallel between my own experiences and the experiences of my respondents is prominent.
I want to thank my respondents for their willingness to participate in this research project besides their busy schedules, for their enthusiasm during the interviews and for their hospitality to invite me to a whole lot of different institutes in the educational domain throughout the Netherlands. This gave me the opportunity to meet very different people, to get to know different relevant institutions in the Dutch educational domain and to use my student public transport chip card to the fullest during my last months of studying. I also want to thank my colleagues at the NRO and my supervisor for their enthusiasm and support during the process of finishing this thesis. I hope my thesis shows the richness of my experiences in the last few months.
3
List of Acronyms
BOPO - Dutch programme for Policy-oriented research
EAPRIL - European Association for Practitioner Research on Improving Learning ECBO - Expertise Centre for Vocational Education
EU - European Union
EZ - Economic Affairs
KNAW - The Royal Netherlands Academy for Arts and Sciences LPG - National Pedagogical Centres
MBO - Vocational Education
NRO - The Netherlands Initiative for Educational Research NSO - Dutch School for Educational Management
NWO - The Netherlands Organisation for Scientific Research OPRO - Dutch Programme Council for Overarching research
PO - Primary Education
PPO - Dutch Programme Council for Practice-oriented research ProBo - Dutch Programme Council for Policy-oriented research PROO1 - Dutch Programme for Educational research
PROO2 - Dutch Programme for Fundamental research
SLOA - National Funds for Supportive Educational Activities OCW - Education, Culture and Science
VO - Secondary Education
1
The Programme Council for Educational Research of the NWO before the establishment of the NRO in 2014.
2
The follow-up programme of the PROO¹ since the operational start of the NRO on the 1th of January 2014.
4
Chapter 1.
Introduction
The future of educational research and its (limited) impact on educational practice and policy has gained increased public interest in several Western countries (Broekkamp & Van Hout-Wolters, 2007; Levin, 2013, 2004; Vanderlinde & Van Braak, 2010). In the Netherlands, this resulted in the establishment of a central research coordinating body: the Netherlands
Initiative for Educational Research (Dutch acronym: NRO). The central aim of the initiative is to improve educational quality by strengthening the Dutch educational knowledge
infrastructure (Covenant Netherlands Initiative for Educational Research, 2012). To achieve this, the NRO aims to coordinate the Dutch educational research domain and to reduce the gap between educational research, practice and policy (Netherlands Organisation for Scientific Research, 2015).
1.1 Problem statement
There are a number of different arguments to explain the importance of strengthening the relation between research, practice and policy (Blake & Ottoson, 2009; Hoppe, 2002; Levin, 2013; Landry, Amara & Lamari, 2001). Research could provide evidence to policymakers about what (not) to do and could therefore prevent highly ideological educational policies with unintended negative consequences (Hoppe, 2002). Research could also inspire educational practitioners to change ‘common sense routines’ into more effective and
innovative practices, which leads to the improvement of educational practice (Levin, 2013). Another argument is that public access to valid research knowledge is important, because it forms the driver of our modern days knowledge economies and information societies (Nutley, Morton, Jung & Boaz, 2010). In general, there seems to be public consensus about the
inherent value of research for policy and practice (Blake & Ottoson, 2009).
The establishment of an initiative to turn this potential into impact of research on policy and practice and vice versa is not distinctive for the Netherlands, nor for the educational domain, nor for the current decennium (Broekkamp & Van Hout-Wolters, 2006; Levin, 2013; 2004; Vanderlinde & Van Braak, 2010; Nutley e.a., 2010). Many (inter)national efforts have been undertaken, like the establishment of a research impact network in Europe and sector specific knowledge centres, Academic Schools and the NRO in the Netherlands. Still, these efforts often end up in disappointment about the eventual impact of research on practice and policy (Broekkamp & Van Hout-Wolters, 2007; Levin, 2013; Van der Linde & Van Braak, 2010).
5
The list with explanations for this disappointment is long and focuses both on shortcomings of researchers as well as on practitioners and policymakers: educational research is of poor quality, does not address practical issues, does not lead to useful
guidelines, researchers just do research for themselves or practitioners and policymakers have no time for, no interest in and no access to research (Hoppe, 2002; Landry et al., 2001; Levin, 2004, 2013). The gap could also be declared by a lack of knowledge utilization incentives for researchers and practitioners and a lack of communication possibilities between these
stakeholders (Levin, 2013). Each causal explanation leads to different strategies to strengthen the relation further. Still, none of the proposed strategies seems comprehensive to explain and bridge the gap (Broekkamp & Van Hout-Wolters, 2007; Levin, 2013; Vanderlinde & Van Braak, 2010). Some even suggest that measures to improve knowledge utilization might itself be based on assumptions, rather than research evidence (Levin, 2013; Vanderlinde & Van Braak, 2010). A comprehensive and realistic understanding of the gap is needed to overcome the problem of high efforts and low impact, and to improve the advantages of research for practice and policy that’s been aimed at for so many years (Levin, 2013; Vanderlinde & Van Braak, 2010).
1.2 Aim and focus
The aim of this qualitative research project is to understand the complex processes underlying the gap between research, practice and policy and to gain insight in suggested strategies to bridge it by addressing and comparing the perspectives of multiple stakeholders. A
comparison between different stakeholders leads to a more realistic understanding of the gap (Broekkamp & Van Hout-Wolters, 2007). The study focuses on the perspectives of the recently established research financier NRO and its network of researchers, policymakers and practitioners. Because the NRO uses the term knowledge utilization to refer to efforts to bridge the gap, the project is centred around this concept.
1.3 Research questions
The central question of this research project is:
How does the approach of the NRO towards knowledge utilization relate to the approaches of educational researchers, practitioners and policymakers and what strategies can be used by these parties to maximize knowledge utilization in the Netherlands?
6
To answer this question, five sub questions will be addressed. Question one, two and three aim to grasp conceptual difficulties in debates about knowledge utilization, while question four and five give insight in experienced challenges and possible strategies.
1. How is knowledge utilization defined by the NRO, educational researchers, policymakers and practitioners?
2. In what ways is knowledge utilization viewed as important by the NRO, educational researchers, policymakers and practitioners?
3. What activities do the NRO, educational researchers, policymakers and practitioners undertake to maximise knowledge utilization?
4. What challenges do the NRO, educational researchers, policymakers and practitioners experience with regard to knowledge utilization?
5. What strategies are proposed by the NRO, educational researchers, policymakers and practitioners to improve knowledge utilization?
In order to address these questions, data were collected from NRO policy documents, research proposals, proposal evaluation reports and 46 semi-structured interviews with diverse
stakeholders, such as NRO policy officers, policy makers of the ministry of Education, Culture and Science (Dutch acronym: OCW), professors, policy advisors, school leaders and teachers.
1.4 Scientific relevance
Despite growing databases of educational research results, increased public interest in knowledge utilization and the establishment of different initiatives, empirical studies on how and why educational research is used by diverse stakeholders are rare (Broekkamp & Van Hout-Wolters, 2007; Levin, 2013; Vanderlinde & Van Braak, 2010). Empirical studies that do exist often focus on one theoretical approach to explain knowledge utilization, address the perspectives of one stakeholder group and focus on challenges rather than existing
connections. The influence of research financiers as the NRO has only recently been recognized and is not yet examined empirically (Broekkamp & Van Hout-Wolters, 2007; Levin, 2013). These shortcomings lead to one sided analyses that fail to grasp the complexity of knowledge utilization (Broekkamp & Van Hout-Wolters, 2007; Vanderlinde & Van Braak, 2010). Some even suggest that the gap is exaggerated in public debates, because of rhetorical or political reasons (Vanderlinde & Van Braak, 2010; Pieters & Jochems, 2003). This study
7
aims to contribute to the quality of academic debates about the topic, by including multiple theoretical perspectives, multiple stakeholders and both existing connections as well as challenges. The results of this study are likely to contribute to an evidence informed debate about knowledge utilization. This is important because: ‘the irony has been noted more than once that the debate over the use of research is itself not well informed by research’ (Levin, 2013: 4).
1.5 Societal relevance
In general, the public discourse on knowledge utilization is dominated by criticism and scepticism about the limited impact of research on society (Levin, 2013). Different authors have explained this by a lack of empirical research and a one-sided focus on the perspectives of researchers, rather than practitioners and policymakers (Vanderlinde & Van Braak, 2010; Pieters & Jochems, 2003). Besides this scepticism, the ongoing efforts indicate that a public consensus exists about the need and possibility to strengthen connections between research, practice and policy (Broekkamp & Van Hout-Wolters, 2007). This empirical study
contributes to a realistic understanding of perspectives on the gap and possible strategies to bridge it, which are embedded in the Dutch context. This way, the knowledge infrastructure of the Netherlands can be strengthened further to improve Dutch educational development. The openness of NRO policy officers to the results of the research project and the enthusiasm of the respondents about the topic increases the chance of application of the research results. Or, in the words of one of the NRO policy officers: ‘Please be critical and tell me if I’m wrong saying knowledge utilization and data collection can occur at the same time’ (NRO, start meeting, 10th of December 2014).
1.6 Thesis outline
In the next chapter, the theoretical basis of this research project will be described further to gain insight into different strands within the academic debate. After this, the characteristics of the Dutch education context will be related to the theory leading to a more comprehensive understanding of the context of this research project. The research methods and processes will be described in chapter 4. Chapter 5 and 6 will explain the results of this study. Chapter 5 focuses on the approaches of different stakeholders (sub question one, two and three). Chapter 6 focuses on experienced challenges and suggested strategies. The master thesis will end with an overview of the main findings in relation to the theoretical framework, implications for further research and recommendations for research, practice and policy.
8
Chapter 2.
Theoretical framework
Ever since the theory of Weiss (1979), the literature about knowledge utilization expanded, which led to new perspectives on the relation between research, practice and policy (Nutley e.a., 2010). This chapter addresses the three main theoretical models to explain knowledge utilization. These models form the basis for a theoretical exploration of perspectives on the importance of knowledge utilization, activities, challenges and strategies. The theories will be illustrated with findings from explorative research projects in the Netherlands and Belgium.
2.1 Defining educational research, practice and policy
To understand the gap between research, practice and policy, a closer look at the meaning of these terms is necessary. Educational research is described as a ‘contested concept’ (Levin, 2013). This term refers to a lack of consensus about appropriate aims, methods and
understandings of educational research within and outside the academic community (Vanderlinde & Van Braak, 2010). Educational research could for example be defined in according to the object of study: educational practices, but could also be defined in according to its goals: the improvement of educational processes and outcomes. This division is also described as research on and research for education (Biesta, 2007. In: Vanderlinde & Van Braak, 2010: 299). Each definition implicates different perspectives on the importance of a relation between educational research, practice and policy. In this project educational research is defined broadly as ‘the structures, processes, products and persons that are part of the systematic development of knowledge of and knowledge for education3’ (Broekkamp & Van Hout-Wolters, 2007: 205). Educational practice is defined as: ‘structures, processes, products and persons that are directly involved in educational institutions, the determination of local and central educational policies, and the development of educational tools’ (Broekkamp & Van Hout-Wolters, 2007: 205). According to this definition, educational policy forms an aspect of educational practice4. Because of this wide variety of aspects of research and practice, several authors have suggested that the gap is much more differentiated than is currently assumed (Broekkamp & Van Hout-Wolters, 2007; Levin, 2013; Vanderlinde & Van
3
The term ‘knowledge for’ was added to the original definition of Broekkamp & Van Hout-Wolters (2007) to guarantee a broad definition, which includes the perspectives of different respondents.
4
When theories and research findings are specifically related to educational policy and not to educational practice in general, this will be highlighted in the text.
9
Braak, 2010). In the next section, different perspectives on the relation between research and practice will be described.
2.2 Defining knowledge utilization
In general, three theoretical models can be subdivided to explain the relation between research, practice and policy: linear models, relationship models and system models (Nutley e.a., 2010). Each model implicates different terms to refer to knowledge utilization.
2.2.1 Linear model
The linear model implicates a one directional relationship between research and practice, in which producers of research knowledge (e.g. professors) or research products (e.g. research reports) influence decisions of educational practitioners (e.g. policymakers) by providing research evidence. This one directional relation is often described as knowledge transfer or knowledge dissemination (Nutley e.a., 2010). The theory of Weiss (1979) forms one of the earliest linear models and is based on a division between instrumental, conceptual and symbolic research use:
Instrumental use refers to cases where knowledge of a single study directly leads to practical decisions. From this point of view knowledge utilization can be defined as ‘the direct use of scientific knowledge of a single study by policymakers and
practitioners to make particular decisions that would not have been made otherwise’ (Landry et al., 2001: 397).
Conceptual use implicates a less direct relationship between research results and decisions. From this perspective knowledge utilization refers to cases where a single study results in new ideas, theories and hypotheses leading to new interpretations about issues surrounding decision making contexts without direct attributable actions taken (Landry et al., 2001).
Symbolic or political knowledge utilization refers to ‘cases where practitioners and decision makers use knowledge to legitimate existing views’ (Landry et al., 2001: 398). In according to this view scientific knowledge does not influence practical thinking or decisions, but is only used to reinforce existing views.
While this model is often criticized for being naïve, it still forms a basis for more recent theories (Landry et al., 2001; Levin, 2013). Biesta (2007) for example, subdivides a technical role of research, which is closely related to instrumental use, from a cultural role, which focuses on a change of perspectives of practitioners, rather than a change of actions.
10
The knowledge utilization ladder is one of these more recent theories, which refers to different cumulative stages of knowledge utilization (Figure 1). This model implicates there’s no knowledge utilization when research results are not transmitted, while there’s a high degree of utilization when results are applied in contexts of practitioners and policymakers. According to this model, research knowledge has diverse influences on practitioners before its application.
Figure 1: Scales of knowledge utilization (Landry et al., 2001: 399).
While this theory is often cited, the perspective is also criticized for being primitive, because it includes the idea that educational research knowledge which is produced in one context can directly inform practitioners in another context about what to do (Landry et al., 2001; Levin, 2013).
2.2.2 The relationship model
During the nineties new perspectives on knowledge utilization emerged, leading to a focus on relationships and interpersonal communication between actors of different domains. This model is based on the idea that knowledge utilization does not occur by transferring
knowledge from one context to another, but occurs continuously and two-directionally during cooperation and interaction between different people. The connection between research and practice, according to this perspective, is often described as knowledge exchange, circular knowledge use or interaction (Nutley e.a., 2010). The theory is based on the idea that research
11
knowledge cannot be applied directly, but is mediated through interaction and personal
experiences (Levin, 2013). The model focuses on the importance of contextualised knowledge that is produced in cooperation between different stakeholders, either directly during
cooperative development of research products or indirectly during discussions about research themes (Nutley e.a., 2010).
2.2.3 The systems model
The systems model focuses on producer -, user - and mediation contexts instead of a relation between individual research products or researchers and individual practitioners (Figure 2). The production context refers to universities or other institutions that produce educational research knowledge. Research financiers as the NRO form an important aspect of the
production context, because research criteria could improve the relevance of research projects for practitioners (Broekkamp & Van Hout-Wolters, 2007). The user context refers to an educational practice setting where research insights can be applied. The mediation context refers to individuals and organisations who attempt to connect research to educational practice, such as media or educational advisory bureaus (Levin, 2013).
12
Speaking of contexts instead of individuals creates the possibility to subscribe different roles to one individual or organisation and to explain multidirectional relations between research, practice and policy. A professor who teaches in university could, according to this model, be producer, user and mediator at the same time. The model also indicates that a school could influence research topics of universities through an article in a newspaper (Levin, 2013).
Secondly, the model is based on the understanding that structural (e.g. legislations, interest groups and organisations) and social (e.g. cultures, ideologies, social developments and emerging technologies in society) characteristics of the context of an individual have a bigger impact on knowledge utilization than individual backgrounds and dispositions (Levin, 2013). In other words, interventions of a school might be more influenced by opinions of colleagues, parents and students and financial restrictions, than research insights about the effects of this intervention. The influence of research on practice contexts and vice versa is often described as knowledge integration and knowledge mobilisation (Nutley e.a., 2010).
2.3 The importance of knowledge utilization
Discussions about the relation between research, practice and policy can be traced back to the dawn of science and are not unique for the educational domain (Levin, 2013). Still, in the last few decades different waves of enthusiasm about the potential of research for the
improvement of society increased the intensity of debates about the topic (Nutley e.a., 2010; Levin, 2013). The growing attention for New Public Management in the nineties, caused discussions about the value of knowledge creation and knowledge use for economic growth and development. One argument that can be related to this period is that research should lead to societal development, because of the need for an adequate return for the investment of tax money. The last decade, the discourse has shifted from the need to apply research knowledge to society, towards the need to use research knowledge in public organisations. This can be explained by the importance of knowledge based decisions in modern day knowledge societies (Nutley e.a., 2010). These two focuses are also described as the need for research application and the need for the production of useful research knowledge (Pieters & Jochems, 2003).
Besides these societal trends, individual perspectives on the importance of knowledge utilization may vary widely. People who define knowledge utilization in according to its goal: the improvement of practice, might think knowledge utilization is more important than people who define it in according to its study object (Vanderlinde & Van Braak, 2010). Some state that the values and aims of research and practice are incompatible and can and should
13
therefore not be interfered with one another (Caplan, 1979). The validity and relevance of research projects, rather than the possibility to improve practices, should be the main
consideration of researchers. Scepticism about the importance of knowledge utilization is also explained by stating that too much focus on educational practice leads to utilitarian research agenda’s (Hammersley, 2003, 2005). Others suggest that too much focus on knowledge utilization in public organisations might result in the devaluation of practical expertise. In addition to these perspectives on the importance of knowledge utilization, some authors question what form of knowledge utilization is realistic. Expecting teachers to be up to date about the most recent scientific findings might be unrealistic, because teaching relates to a wide variety of disciplines, which results in the need to read an unrealistic amount of research reports. Discussions about the importance of knowledge utilization might therefore overlap with discussions about appropriate forms of establishing a connection (Levin, 2013).
Within the Dutch context, an exploratory study showed that the majority of educational researchers in the Netherlands described the improvement of educational practices as their core motive. Still, opinions between researchers differed (Vanderlinde & Van Braak, 2012). A Belgian study showed that teachers tend to be more sceptical about the value of educational research than school leaders, because teachers evaluate the value of research knowledge according to its direct relevance for the classroom (Vanderlinde & Van Braak, 2010). This is confirmed by a literature study about practitioners’ research projects, which highlights that teachers focus on the professionalization and the improvement of educational practices during research projects, rather than contributing to scientific theories (Zwart, Smit & Admiraal, 2015).
2.4 Knowledge utilization activities
Knowledge utilization activities are defined as actions by individuals and/or organisations to strengthen the relation between research and practice. In the scientific literature, knowledge utilization activities are categorized on the basis of four models: the research development diffusion model (RDD), the evidence-based practice model (EBP), the knowledge
communities model (KC) and boundary-crossing practices model (BCP). The first two can be related to linear perspectives on knowledge utilization, while the other two relate to the relation and system models.
14
2.4.1 Activities based on linear models
The RDD model focuses on activities to adjust research products to the needs of potential users and to diffuse these products to potential users. According to this model, knowledge utilization is a matter of ‘effective packaging’ (Nutley e.a., 2010: 135), which could be conducted by researchers or mediators (Broekkamp & Van Hout-Wolters, 2007). RDD activities relate to publishing adjusted research summaries in professional journals or presenting research findings at teacher conferences (Levin, 2013). This model focuses on research application. The establishment of sector specific knowledge dissemination institutions in the Netherlands is also based on this perspective (Bulterman-Bos, 2011).
The EBP model focuses activities by practitioners. According to this model, practitioners use methods and interventions that have been scientifically proved to be effective (Broekkamp & Van Hout-Wolters, 2007). An example of an EBP activity is a director of a school who chooses a bullying intervention on the basis of effect studies. The EBP model focuses on research use, rather than research application (Nutley e.a., 2010) and is closely related to the technical role of research (Biesta, 2007).
2.4.2 Activities based on relationship models
The KC model focuses on the need to establish heterogenic professional networks to learn from each other’s expertise and to develop knowledge in cooperation with one another (Broekkamp & Van Hout-Wolters, 2007: 210). Professional networks could consist of researchers and practitioners, but also of practitioners of different institutions. A KC activity is the establishment of structural partnerships between researchers and practitioners like the founding of Academic Schools in the Netherlands. This term refers to partnerships between PO and VO institutions and teacher education institutions that combine research projects at the local school level with in-service teacher training programmes. A study in the Netherlands showed that these initiatives are perceived as an important development for the improvement of knowledge utilization in the Netherlands (Broekkamp & Van Hout-Wolters, 2007).
Another study showed that Dutch researchers spend seventy percent of their time on applied and practice-oriented research projects. The study also showed that the majority of the Dutch researchers has a positive attitude towards cooperation with practitioners (Vanderlinde & Van Braak, 2012).
15
2.4.3Activities based on system models
While the prior activities focus on the improvement of information flows and (structural) relations, the third model implicates the need for structural and social changes in the context of individuals. In other words: ‘to know is not enough’ (Levin, 2013). The system model suggests that knowledge utilization can only be improved of it’s embedded in the priorities, cultures and incentives of the context of individuals (Levin, 2013). Examples of these activities are allocating knowledge utilization funds to researchers after research projects are finished or lobbying at the ministry of OCW to improve access to scientific databanks for teachers.
The BCP model also relates to the system approach and focuses on the need for boundary crossing. An activity that related to this model is the establishment of PhD. funds for teachers in the Netherlands. Another possibility could be to stimulate researchers to work part-time as researchers and part-time as a teacher (Levin, 2013).
2.5 Challenges for knowledge utilization
In this research project a challenge is defined as a difficulty in the process of strengthening the relation between research, practice and policy. Both scientific articles and public debates about the topic have addressed long lists of challenges that relate to shortcomings of
researchers, practitioners and the structures between them.
One of the most often cited challenges is the lack of conclusive and practical results of educational research. This could be explained by low investments in educational research in many Western countries, which results in small scale studies, a lack of replicative studies and a lack of hard evidence (Broekkamp & Van Hout-Wolters, 2007). The lack of strong evidence could also be declared by a focus on dissemination of single studies, rather than strong bodies of knowledge (Levin, 2013). Another explanation is that researchers have to deal with a lack of incentives to cooperate with practitioners to improve the level of practical results (Levin, 2013).
With regard to educational practices the main challenges refer to a lack of interest and a lack of (appropriate) research use by practitioners (Broekkamp & Van Hout-Wolters, 2007). Explanations relate to the fact that almost everyone in Europe went to school, which increases the influence of personal opinions about education on educational practice, rather than
scientific expertise. Another explanation is the lack of time for research use by teachers or the limited public access to scientific journals. The ‘information overload’ on the internet could also be an important challenge for finding scientific research knowledge. If these suggestions
16
are also applicable to the Dutch context, an important challenge for the NRO might be that improving knowledge utilization implicates changing communication channels and
institutional structures, rather than adjusting research products or facilitating cooperation (Levin, 2013).
A study in the Netherlands showed that different stakeholders (e.g. mediators, school leaders, teachers) expressed similar challenges with regard to knowledge utilization. The study showed that critical statements varied more within, than between stakeholder groups (Broekkamp & Van Hout-Wolters, 2007). For example, researchers, as well as teachers, policymakers and teacher trainers agreed upon the statement that the limited possibilities for equal collaboration and the limited time for research use by practitioners are important
challenges in the Dutch context (Broekkamp & Van Hout-Wolters, 2007). A study in Belgium exemplified that researchers tend to publish their findings through traditional communication channels like professional journals, while only a limited amount of teachers read these
journals (Van der Linde & Van Braak, 2010). An explorative study in the Netherlands showed that the level of knowledge utilization during activities at Dutch educational congresses is limited, because the activities focus on knowledge dissemination by researchers, rather than interaction between researchers and practitioners (De Vries & Pieters, 2007).
Finally, the literature shows that knowledge utilization with regard to educational policy might be easier than knowledge utilization with regard to educational institutions. Knowledge utilization in educational policy is often directed towards changing policy documents, which could be achieved by a limited amount of people and spread across a country relatively easy (Levin, 2013). Knowledge utilization in educational institutions is directed toward the improvement of educational practices, which implicates the need to change behaviours of a wide variety of people in a wide variety of educational institutions. Improving practices could therefore be a bigger challenge than changing policy texts (Levin, 2013).
2.6 Strategies to improve knowledge utilization
Finally, the study addresses suggested strategies for strengthening the relation between research, practice and policy. A strategy is defined as a coherent set of activities to enhance knowledge utilization, which is ideally based on a combination of different perspectives (e.g. researchers, practitioners and policymakers) and a complementary understanding of different knowledge utilization models (Vanderlinde & Van Braak, 2010). A knowledge utilization strategy could be to improve the use of innovative online dissemination strategies (e.g.
17
twitter) to increase the availability of summarized research knowledge (linear model), the chances for interaction between different stakeholders (relation model) and the creation of overarching communication structures across boundaries of organisations (system model).
A study in the Netherlands showed that stakeholders expressed the need for a coherent strategy that is based on the complementary advantages of all four solution models. To illustrate this, stakeholders referred to the importance of ‘Academic Schools’ in the Netherlands. Respondents values Academic Schools for its potential to: initiate research projects at local school levels (BCP), base school actions on existing research knowledge (EBP), improve equal and complementary cooperation between researchers and practitioners (KC) and translate practitioners’ findings into fundamental and practice-oriented research knowledge (RDD) (Broekkamp & Van Hout-Wolters, 2007). Another Dutch study showed the importance of the implementation of research findings in structural activities like teacher education courses, rather than disseminating it through traditional communication channels. Secondly, the authors highlighted the importance of strengthening pro-active attitudes of practitioners for bridging the gap between research and practice. The development of a database for good practices of educational practitioners to inspire researchers and the
establishment of conferences in which practitioners present and researchers respond could be strategies for enhancing knowledge utilization (Pieters & Jochems, 2003; De Vries & Pieters, 2007).
18
Chapter 3.
The Dutch educational research context
Besides global discourses on and international interest in the relation between research, practice and policy, the ways in which the concepts knowledge production and knowledge utilization are interpreted and institutionalized are highly depend on national contexts (Nutley e.a., 2010). This chapter provides insight into the context of this research project by clarifying Dutch stakeholders, the educational research funding system, the role of the NWO and the establishment and organisational structure of the NRO.
3.1 A wide variety of stakeholders
The Dutch educational domain is often characterised by its wide variety of relatively autonomous actors, which results in a multi-level, horizontally spread governance system. The wide variety of relatively autonomous actors might create challenges for governing the system (Waslander, 2010). The ministry of OCW formulates national educational policies that form a framework for Dutch schools and can be held accountable for educational outcomes by citizens and politicians (Waslander, 2010). Municipalities are formally responsible for
educational accommodation, school leavers and pre-school education. The Dutch Inspectorate of Education reviews the quality of schools using national guidelines and implicates measures if these guidelines are not met (Inspectorate of Education, 2015). The council for Primary Education (Dutch acronym: PO), Secondary Education (Dutch acronym: VO) and Vocational Education (Dutch acronym: MBO) represent school boards of different sectors and initiate sector specific programmes to stimulate educational quality. Besides these influences Dutch schools operate relatively autonomously and are formally held responsible for their own educational quality.
The Dutch context also exists of a wide variety of different research institutions. While research knowledge is mostly produced in universities, other institutions also produce knowledge on and knowledge for education, like: institutions for higher education (e.g. teacher education institutions), commercial research bureaus, test development institutes and the recently established academic teacher education institutions (Nutley e.a., 2010;
Waslander, 2010). Diverse knowledge institutions, like the Expertise Centre for Vocational Education (Dutch acronym: ECBO), support the educational infrastructure by translating research knowledge into useful knowledge for different stakeholders through methods, journals and organisational networks. Policy advisory councils, like the Education Council of the Netherlands, create links between research and policy by formulating policy advises for
19
the national government on the basis of scientific reports (Nutley e.a., 2010). Because of this wide variety of different stakeholders with different responsibilities the Dutch educational domain is often characterised as a ‘patchwork’. Therefore, the NRO is not the only
organisation that tries to influence educational research, practice and policy. In the next chapter, the position of the NRO as a subdivision of the NWO will be exemplified further by describing the educational research funding system.
3.2 The Dutch research funding system
The Dutch educational research funding system is divided into three flows of funding.
The first flow of funding consists of funds for universities, which are directly allocated by the ministry of OCW and the ministry of Economic Affairs (Dutch acronym: EZ). These 'lump sums' enable universities to set up stable, long term infrastructures, to provide education and to facilitate research projects. Because of shifts in budgets from the first to the second flow (e.g. Innovation Research Incentives Scheme) to improve research quality and increasing student numbers, universities are experiencing increased financial pressures, which results in a higher dependence on second flow funding (Interdepartmental Policy Research, 2014).
The second flow of funding consists of government funds, which are allocated by the Royal Netherlands Academy of Arts and Sciences (Dutch acronym: KNAW) and the
Netherlands Organisation for Scientific Research (Dutch acronym: NWO) to specific research institutions, research projects, research facilities or researchers. Both the KNAW and the NWO are independent directive bodies with legally established missions and tasks under the responsibility of the ministry of OCW. The KNAW forms a forum for excellent researchers from all disciplines, governances different research institutions and operates as a policy advisory board, which is specialised in governmental policies for science. The central task of the NWO is to allocate research funds to Dutch research institutions or individual researchers in a wide variety of domains (e.g. sea research, brain and cognition research, educational research). While some funds are directly allocated to aligned research institutes (figure 3), most funds are allocated by competition to improve the democratic value of allocation
processes and the quality of research projects. Different research institutions (e.g. universities, commercial research bureaus, aligned research institutes) can apply for research budgets from the NWO. Besides allocating research funds, the NWO formulates research agenda’s,
influences national and European science policies, facilitates access to research information and stimulates international and national network creation (NWO, 2015).
20
Figure 3: NWO branches and institutions (NWO, 2015)
The third flow of funding consists of all other public and private funds like research budgets from the European Union (EU), the ministry of OCW, the Council for Education,
municipalities, school boards and companies (Interdepartmental Policy Research, 2014). These budgets can be allocated through competition or by direct allocation.
While the majority of the publications highlights the existence of three flows of funding, some policy reports complement the model with a fourth and fifth flow of funding. These flows are development funds for educational practices, like the National Funding for Supportive Educational Activities (Dutch acronym: SLOA funds) and PhD. and master funds for teachers (Educational Council, 2003).
3.3 The establishment of the NRO
Before the establishment of the NRO in 2012, two policy advisory reports concluded that the Dutch educational research system was highly fragmentized. This fragmentation leads to low impact of important scientific developments on educational research programmes, educational
21
development and educational innovation (Commission National Plan Future of Educational Science, 2011; Coonen & Nijssen, 2011). The fragmentation was partly explained by the fact that educational research knowledge was produced in a wide variety of different institutions and financed by different flows of funding. Universities conducted research projects with funds from the first flow, while the NWO allocated research budgets through their
fundamental research programme for Educational Research (Dutch acronym: PROO) and the programme for Policy-oriented research (Dutch acronym: BOPO). Budgets for practice-oriented research were directly allocated by the ministry of OCW to specific research
institutions, like the SLOA funds to the National Pedagogical Centres (Dutch acronym: LPG) (NWO, 2015).
To overcome this fragmentation and improve the impact of research on educational quality and development, the commissions advised the ministry of OCW to establish a central coordinating body for educational research, to invest in the professionalization of educational practitioners and to formulate coherent and interdisciplinary plans per educational sector (Commission National Plan Future Educational Sciences, 2011; Coonen & Nijssen, 2011). After these advices, the NRO was established in 2012. The budget of the organisation, structurally fifteen million euros per year, is composed of funds from the prior research programmes of the NWO (e.g. the PROO and BOPO), research budgets from the ministry of OCW and the ministry of EZ and practice-oriented funds (prior SLOA funds). The budgets were derived from the second, third and fourth flow of funding.
The NRO became operational on the first of January 2014. The central aim of the organisation is to reduce the gap between educational research, practice and policy (Figure 4). To achieve this ambition the NRO coordinates educational research funding, translates
scientific, practical and policy questions into long term educational research agenda’s and facilitates knowledge utilization (NWO, 2015).
22
3.4 Organisational structure of the NRO 3.4.1 Formulating research agenda’s
The NRO consists of a central steering group, four Programme Councils and the operational bureau. The steering group formulates the long term vision, strategy and financial plan of the NRO, formally approves all granted research funds and selects the members of the four Program Councils for fundamental, policy-oriented, practice-oriented and overarching research. The group exists of seven representatives from different educational domains / sectors: one representative from the policy domain, a teacher organisation, the higher educational sector, the vocational sector, the secondary sector and two representatives from the research sector.
The NRO is subdivided into four research programme councils that formulate research agenda’s and decide about the allocation of research funds. The fundamental research
programme is a follow-up programme of the Programme for Educational Research, which started under the flag of the NWO. Fundamental research is defined as ground-breaking research, driven by curiosity. The aim of this programme is to stimulate theoretical and scientific innovation, to increase the practical significance of educational research and to increase the impact of research on the Dutch educational research infrastructure. The Programme Council for Fundamental Research (Dutch acronym: PROO5) consists of six representatives from different universities, one employee of the ministry of OCW and two representatives of the PO and VO.
The earlier established programme for Policy-oriented Research (Dutch acronym: BOPO), which started under the flag of the NWO, forms the basis for the current programme for policy-oriented research. Policy-oriented research is directed towards creating a scientific basis for policy questions about (educational) developments in society. The research
programme is developed in cooperation with the ministry of OCW. The Programme Council for Policy-oriented Research (Dutch acronym: ProBo) consists of three members from universities, one policy-member from the ministry of OCW, one policy member from the Dutch Inspection for Education and two directors of school organisations.
The programme for practice-oriented research has been fully established after the start of the NRO and does not have an historic predecessor within the NWO. Practice-oriented research projects have to be conducted and completed in cooperation between educational
5
This acronym is similar to the earlier established research programme for Educational Research of the NWO. In the next chapters, PROO refers to the current programme for fundamental research of the NRO.
23
researchers and practitioners. Therefore, only consortia (which consist of at least one educational practitioner, one researcher and optionally other educational professionals) can apply for research funds. The aim of this programme is to finance research projects that lead to useful research outcomes that can be used directly in educational practice. Practice-oriented research projects should also contribute to school development, the professionalization of practitioners and the development of a knowledge base for educational practice. The
Programme Council for Practice-oriented Research (Dutch acronym: PPO) consists of three researchers from different universities and teacher education institutes, three representatives of PO, VO and MBO and one member from the ministry of OCW.
In 2014, the NRO initiated the Programme Council for Overarching research (Dutch acronym: OPRO). The main goal of this council is to finance research programmes that overarch or intersect with fundamental, policy-oriented or practice-oriented research projects. The OPRO consists of six members of the other research councils (NRO, 2015). The ambition of the NRO is to develop one research programme in which the current programmes are combined.
3.4.2 Allocating research budgets
The daily operational tasks of the NRO with regard to the coordination and allocation processes are conducted by 20 staff members of the NRO bureau. The main steps of the allocation processes can be found in figure 5.
Figure 5: the allocation process of the NRO
Several times a year researchers can submit proposals to different ‘calls for research
24
NRO bureau and other stakeholders like the ministry of OCW. Individual experts, who are selected on the basis of their function and expertise, will review each proposal individually. While the evaluation criteria differ per call, each proposal is at least evaluated on the basis of scientific quality, the quality of the researcher(s) and the societal relevance of the project. A proposal can only be granted if each criterion is evaluated as ´good´. After the individual reviews, the experts meet to discuss the proposals and to develop a priority list. The associated programme council will uses this priority list to make a decision about the allocation of research funds.
While the allocation process forms the primary task of the NRO, the NRO also has the task to coordinate the Dutch educational research domain and to disseminate research knowledge to educational practice and policy (Covenant Netherlands Initiative for Educational Research, 2012).
25
Chapter 4.
Research methodology
The aim of this research project is not to measure the current level of knowledge utilization in the Netherlands, but to gain a realistic and in-depth understanding of the complex
mechanisms underlying the relation between research, practice and policy and possible strategies to bridge the gap between these domains. A qualitative research strategy fits to the aim of this research (Bryman, 2008). In the following sections, the research methodology will be explained by introducing the research design, methods, instruments, sample and methods for data analysis. Also, the validity of the research project, its limitations and ethical
considerations will be discussed. Because this project is about knowledge utilization, the chapter ends with an overview of knowledge utilization activities by the researcher.
4.1 Research design
Because knowledge utilization is such a complex process, the goal of this research is not to undo social reality of its complexity to find general patterns, but to maintain it as much as possible. This can be achieved by conducting a single case study (Bryman, 2008). The case study focuses on knowledge utilization in the Netherlands with specific emphasises on the NRO and its network of researchers, policymakers and practitioners. The reasons for the establishment of the NRO and the dependence of researchers from NRO funds legitimize this choice. The project also includes aspects of a comparative research design to examine the complexity of knowledge utilization extensively. The study highlights similarities and differences between different persons, research projects, and research councils and between the NRO and other stakeholders.
4.2 Methods and instruments
While 46 semi-structured interviews form the fundament of this research project, the project started with a document analysis of NRO policy documents, research proposals and research evaluations. This way the researcher could examine the way in which knowledge utilization is institutionalized in the policy guidelines, evaluation criteria and evaluation procedures of the NRO as well as in the proposals of researchers. This created the possibility to compare propositions on paper without a risk of reactivity with opinions that were derived during the interviews.
26
4.2.1 NRO documents
Since this research was conducted during an internship at the NRO, the researcher had access to all NRO documents that were available for regular employees. In consultation with NRO policy officers all relevant documents that gave insight in the approach of knowledge utilization within the NRO were selected: three handbooks about knowledge utilization for practice-oriented, policy-oriented and fundamental research projects, an internal document about the current state of knowledge utilization activities by the NRO and the NRO policy guidelines for knowledge utilization (2015-2018). To gain more understanding of different approaches within the NRO, one research call from each research council was selected in deliberation with the programme secretaries. The criteria for selection were both theoretical and practical. Research calls had to be initiated and research funds had to be granted under the flag of the NRO to make sure the analysis gave insight into the approach of the NRO, and not the NWO. OPRO documents were not included in this project, because the OPRO combines all three types of research projects. The final selection was based on practical considerations by the NRO: the amount of meetings between NRO policy officers and researchers during the research period to limit the workload for researchers. Figure 6 includes an overview of the documents that formed the starting point for this project
Documents Programme N
Policy guidelines for knowledge utilization - 1 Current state of the activities of the NRO - 1 Knowledge utilization handbooks PROO = 1 3
ProBo = 1
PPO = 1
Research calls PROO = 1 3
ProBo = 1
PPO = 1
Total 8
Figure 6: NRO documents
4.2.2 Research proposals and evaluations
To gain insight in the evaluation procedure of the NRO, proposals with an excellent score on knowledge utilization and an equal amount of proposals with the lowest scores were selected
27
from the pre-selected rounds (Section 4.2.1, explanation of selection of research calls). Because of the great amount of proposals within the PPO round (123) 6 proposals with the highest and 6 proposals with the lowest scores were selected. Besides these proposals the proposal evaluation reports were also selected. Figure 7 includes an overview of the amount of proposals and evaluation reports.
Documents Programme Score N
Research proposals PROO = 12 Approved = 6 12 and proposal evaluation
reports Disapproved = 6 ProBo = 8 Approved = 4 8 Disapproved = 4 PPO = 12 Approved = 6 12 Disapproved = 6 Total 326
Figure 7: Proposals and proposal evaluation reports
4.2.3 Semi-structured interviews
The information from the documents was complemented with 46 semi-structured interviews with different stakeholders. Semi structured interviews inherit a certain level of flexibility for both respondents and the researcher. Both parties can examine the objective of study broadly by creating the possibility to formulate diverse answers and to respond with questions to it (Bryman, 2008). The interviews started with an examination of the definitions, importance and current activities of/for knowledge utilization by the respondent. By focusing on existing activities the project takes a unique approach in comparison to the majority of studies that focus on challenges, rather than connections (Broekkamp & Van Hout-Wolters, 2007). The interviews ended with an examination of experienced dilemma’s, existing initiatives and possible strategies in the perspective of the respondent (Appendix 1 and 2). The interview questions were send to the respondents before the interview. This decision was made after three interviews in which one researcher stated: ‘You are asking me very fundamental questions. I could have thought about it more accurately if I read the questions before
6
28
participating in this interview’ (Researcher 5). The duration of the interviews varied between a half an hour and two hours. On average the interviews took one hour. While the majority of the interviews were conducted face to face on a location chosen by the respondent (e.g. offices), about 26 percent of the interviews were conducted by telephone, because of the preference of the respondents, travel distances or time schemes. Eighty percent of the total amount of interviews was fully recorded. During the other interviews notes were taken, because of technical issues or preferences of the respondent.
4.3 Research sample
In total, 46 interviews were conducted with different stakeholders (Figure 8). Respondents were selected on the basis of their function and participation in competitions rounds. Since the NRO is a small organisation, all relevant policy officers were interviewed (n=7). The
internship coordinator helped the researcher to select all relevant policy officers.
To gain insight in the approach of educational researchers, 22 interviews with researchers from policy-oriented, practice-oriented and fundamental research projects were conducted. The selection of researchers was based on different steps. Firstly, the respondents were
selected from three pre-selected grant rounds (see section 4.2.1: NRO documents). Because of ethical considerations only researchers with an approved proposal were contacted. For the approved policy-oriented and fundamental research projects all main applicants (n=11) were asked to participate in an interview. Because of the amount of approved practice-oriented research projects, the main applicants with the highest and lowest scores on the criteria for knowledge utilization were invited for an interview. If the main applicants did not respond after several notification e-mails and phone calls, the second applicant was invited. Because two main applicants stated the whole research consortium/group was not interested in participating in the research project, the respondents were selected from fifteen different research projects. This high response rate could be explained, because of the possibility to write an mail in the name of the NRO and because of the notification in the invitation e-mails to participate in an interview to have the possibility to influence NRO policy. After inviting the first or second applicant for an interview, the other respondents were selected on the basis of their functions (e.g. policymakers, teachers, school leaders, managers, professors or researchers from commercial organisations). By selecting different types of functions the researcher aimed to gain insight in a wide variety of perspectives.
Thirdly, different policymakers from the ministry of OCW were interviewed. Even though policymaking is officially included in the definition of educational practice, the
29
organisation of the NRO (policy-oriented research versus practice-oriented research)
legitimizes the choice to separate policymakers from other practitioners. Policymakers were selected on the basis of their involvement in the selected ProBo granting round in consultation with the programme secretary of the ProBo. In total, 9 employees of the ministry of OCW were interviewed from diverse departments (e.g. Department of knowledge, primary education, secondary education, vocational education and higher education).
The total amount of interviews is based on the level of theoretical saturation that is needed for an explorative research project. A full understanding of different perspectives per function or per project could take as long as a PhD project.
30
Stakeholder7 Council N Institution N Function N Sample
size
NRO - - Director 1 7
Knowledge utilization officer 1 Publicity officer 1
Policy officer 4
Researcher PROO 7 University 7 Professor 5 22 Researcher / teacher 2 ProBo 9 University 7 Professor 2
Coordinator 1
Researcher / teacher 4 Institute for
higher education
1 Lector 1
Research institute 1 Researcher 1 PPO 6 University 3 Professor 1 Researcher / teacher 2 Institute for test
development
1 Researcher 1 Expertise centre 1 Researcher 1 Educational
advisory bureau
1 Consultant 1 Practitioner PPO 8 VO institute 5 Teacher 2 8 Director / manager 2 Research coordinator 1 MBO institute 3 Director / manager 2
Policymaker 1
Policymaker ProBo 9 Ministry of OCW 9 Policymaker / advisor 6 9 Research coordinator 2
Project manager 1
Total 46
Figure 8: characteristics of the research sample
4.4 Data analysis
The basis for the data analysis is grounded theory. Grounded theory can be defined as ‘theory that was derived from data, systematically gathered and analysed through the research process’. (Strauss & Corbin, 1988. In: Bryman, 2008: 541). Using this method means the
7
Categories are based on the main function of respondents. Some respondents fulfil more than one function (e.g. Policymaker at the ministry and teacher in a teacher education institution).
31
process of data collection, data analysis and theoretical development are highly intertwined and therefore iterative. While the definition focuses on generating theory out of data, in reality grounded theory often generates concepts (Bryman, 2008). While theoretical labels have given a focus at the start of the data analysis, these were changed and complemented step by step during the research process to maintain the complexity of knowledge utilization. The choice for the method is based on the notification that many theories about knowledge utilization exist, but none of these theories seems comprehensive (Levin, 2013; Van Braak & Vanderlinde, 2010).
4.4.1 Document analysis
To gain insight in the evaluation processes of the NRO the sections for knowledge utilization and dissemination in the approved and disapproved proposals were compared. First the activities in the proposals were clustered on the basis of theoretical concepts: traditional, innovative, network (internal and external) and institutional activities. On the basis of the data and the interviews, in which respondents exemplified the importance of a difference between passive and interactive activities and different users the categories were extended to:
Traditional (scientific and non-scientific), interactive meetings, online (publishing and (inter)active), practical outcomes, training / courses and network (internal and external) activities. After structuring the data, the researcher compared the amount and the character of activities in proposals with a high and a low score on knowledge utilization. Complementary to this analysis, the arguments by the evaluation commission were clustered and compared with the findings of the researcher. Also, the researcher developed and compared overviews of ‘average’ proposed activities per research programme. Finally, the researcher compared statements derived from the interviews with the proposed activities in proposals.
4.4.2 Policy documents and interview analysis
The 46 interviews with different stakeholders are the centre of this research project. All interviews that were recorded were fully transcribed. The NRO documents, transcribed interviews and interview notes were inserted in the programme Atlas TI as soon as they were collected. The data analysis started from the moment the first documents were collected and the first interviews were transcribed. Firstly, the data were coded on the basis of concepts from the research questions: definitions, importance, activities, challenges and strategies and different stakeholders: the NRO, researchers, practitioners, policymakers and others.
