Developing a Performance Measurement System in the
public sector
A multiple case study at municipalities in the Netherlands about the design, implementation and use of a Performance Measurement System used for monitoring social community teams
Master thesis, MSc BA Organizational & Management Control University of Groningen, Faculty of Economics & Business
Thesis supervisor: H.J. van Elten Co-assessor: S. Tillema
Word count (including references and appendices): 12.953
ABSTRACT
A qualitative multiple research design was used to examine municipalities in the Netherlands that are developing a Performance Measurement System to measure the performance of social community teams. Social community teams are widely introduced as a response to the decentralization of the social domain from government to municipalities (1-1-2015), to guarantee a more efficient way of working. These mandated changes stimulate Dutch municipalities to control in a more result-oriented way. However, municipalities do not have a comprehensive, functioning, Performance Measurement System yet and are still developing and preparing. They adopted a result-oriented control style and are aware of the need to measure performance, but there seems to be a gap between intention and use.
___________________________________________________________________________
PREFACE
This thesis is the final step to receive my master’s degree in Business Administration: Organizational & Management control. Before starting at the Faculty of Economics & Business, I already knew my interest lays in the public sector, especially health care. Unfortunately little attention is paid to the public sector in the bachelor study program. Therefore, I am grateful for the opportunity to write my master thesis about a topic of my interest and learn more about public sector management.
Writing this thesis would not have been possible without the help of several people. First of all, I would like to thank Machteld Koelewijn (Partner Sociaal Domein, BMC Advies) for the opportunity for this research and her support. I would also like to thank my thesis supervisor Hilco van Elten for his professional feedback and help. Next to that, I am grateful to all the involved respondents for providing me a lot of useful information and for their collaboration in this research.
I hope you enjoy reading this thesis.
Tessa Bergsma
EXECUTIVE SUMMARY (DUTCH)
In Nederland zijn de zorgkosten als percentage van het Bruto Binnenlands Product (BBP) de afgelopen jaren flink gestegen (OECD, 2014). Het beheersen van deze kosten is een belangrijk aspect van het overheidsbeleid. Vanaf 1 januari 2015 vindt er een decentralisatie plaats van overheidstaken naar de gemeenten, die vanaf dan verantwoordelijk zijn voor een groot deel van het sociale domein. Zij moeten, met minder budget, zorgen voor een effectievere en efficiëntere aanpak in het sociale domein (Sok et.al., 2013). Om dit te garanderen, implementeren veel gemeenten sociale wijkteams. Sociale wijkteams kunnen er voor zorgen dat er integraler wordt gewerkt en er meer gebruik wordt gemaakt van de eigen kracht van burgers. Zo kan (dure) specialistische zorg worden voorkomen.
Omdat gemeenten willen zien hoe het gaat en of kosten in de hand worden gehouden, is het nodig dat prestaties worden gemeten (Rutten, Sterrenberg & Vos, 2014). Echter, het meten van prestaties en het ontwikkelen van een prestatiemeetsysteem (PMS) is niet makkelijk in de publieke sector (Kanter & Summers, 1987; van Thiel & Leeuw, 2000; Bouckaert & Halligan, 2006). Doelen zijn vaak moeilijk te meten en er moet rekening worden gehouden met vele belangen. Uit de literatuur blijkt dat een PMS in de publieke sector daarom zorgvuldig ontworpen en geïmplementeerd moet worden (Bouckaert & Peters, 2002; DeGroff et al., 2010; Dan & Pollitt, 2014). Het proces van het ontwikkelen van een PMS in de publieke sector is echter onduidelijk en er is niet veel onderzoek naar gedaan. Het doel van dit onderzoek is daarom een omvangrijke beschrijving van het ontwikkelen van een PMS bij gemeenten te geven. De onderzoeksvraag is: “hoe ontwerpen, implementeren en gebruiken gemeenten in Nederland prestatiemeetsystemen om de prestaties van sociale wijkteams te meten?”
zijn het meest relevant, omdat het aansluit op de doelen van de sociale wijkteams. Resultaten kunnen echter beïnvloed worden door andere factoren buiten de organisatie en het is daarom lastig om een oorzaak-effect relatie aan te kunnen tonen (Bruijn, 2002; Schalock & Bonham, 2003; Poister, 2003). Daarom moeten resultaten goed geïnterpreteerd worden. Daarnaast is het goed om een hoge variëteit van indicatoren te gebruiken, gebaseerd op zowel financiële als niet-financiële indicatoren. Zo worden belangen van verschillende stakeholders meegenomen en een compleet beeld van de resultaten gegeven (Brignall & Modell, 2000). Bij het ontwikkelen van de indicatoren is het verder belangrijk dat stakeholders betrokken worden. Door het betrekken van stakeholders, kunnen belangrijke indicatoren worden bepaald (Bruijn & Helden, 2006; van Thiel, 2009; DeGroff et al, 2010; Jaaskelainen & Sillanpaa, 2012)
.
Tenslotte kan het PMS het beste op een verkennende manier worden gebruikt, omdat op die manier een leerproces kan plaats vinden en het beleid kan worden aangepast. Door te leren, experimenteren en aan te passen, kunnen oorzaak-effect relaties ontdekt worden en wordt inzicht gegeven in de effectiviteit van het proces (Speklé & Verbeeten, 2009).Na het literatuuronderzoek is een meervoudige case study uitgevoerd. Er zijn interviews afgenomen bij 11 gemeenten in Noord-Nederland. Uit analyse van de 124 pagina’s uitgewerkte interviews blijkt dat geen van de 11 gemeenten al een functionerend, compleet PMS heeft. De meeste gemeenten zijn het systeem aan het ontwikkelen en slechts één van de zeven gemeenten die al een tijd met sociale wijkteams werken, is de prestaties van dit team aan het meten.
blijkt geen relatie te hebben met een hoge variëteit van indicatoren bij gemeenten. Wellicht is deze variëteit nog niet hoog, omdat er nog niet daadwerkelijk gewerkt wordt met een PMS.
Met betrekking tot de implementatie zijn er zes gemeenten die direct stakeholders betrekken bij het ontwerpen van de indicatoren en twee die helemaal niemand erbij betrekken. Uit de analyse blijkt dat er een positief verband is tussen het betrekken van stakeholders en het ontwerp van de indicatoren.
TABLE OF CONTENTS
1. INTRODUCTION ... 8
2. LITERATURE REVIEW ... 10
2.1MANAGEMENT CONTROL ... 10
2.2PERFORMANCE MEASUREMENT ... 10
2.3DEVELOPMENT OF A PERFORMANCE MEASUREMENT SYSTEM IN THE PUBLIC SECTOR ... 11
2.3.1 Design ... 12 2.3.2 Implementation ... 16 2.3.3 Use ... 16 2.4CONCEPTUAL MODEL ... 17 3. METHODOLOGY ... 19 3.1RESEARCH METHOD ... 19 3.2RESEARCH PLAN ... 19 3.3CASE SELECTION ... 20 3.4DATA COLLECTION ... 20 3.5DATA ANALYSIS ... 21
3.6VALIDITY AND RELIABILITY ... 21
3.7BACKGROUND CASE STUDY ... 21
4. RESULTS ... 23 4.1CONCEPTS ... 24 4.1.1 Design ... 24 4.1.2 Implementation ... 28 4.1.3 Use ... 30 4.1.4 Effectiveness ... 31
4.2RELATIONSHIPS BETWEEN CONCEPTS ... 32
4.2.1 Design – effectiveness ... 32
4.2.2 Implementation – effectiveness ... 33
4.2.3 Use – effectiveness ... 33
5. CONCLUSION, LIMITATIONS AND FUTURE RESEARCH ... 34
5.1CONCLUSION ... 34
5.2LIMITATIONS ... 35
5.3FUTURE RESEARCH ... 35
REFERENCES ... 36
APPENDIX A – CASE STUDY PROTOCOL (NEDERLANDS) ... 40
APPENDIX B – OVERVIEW INTERVIEWS ... 44
1. INTRODUCTION
Over the last years, the Netherlands has witnessed a sharp increase in healthcare costs as a percentage of the Gross Domestic Product (OECD, 2014). Cost-effectiveness needs to increase (OECD, 2013a) and controlling the healthcare costs is an important policy issue (de la Maisonneuve & Martins, 2013). As a response to this, a comprehensive decentralization of tasks and powers from the government to municipalities will be implemented in January 2015 (CPB, 2013). Municipalities will be responsible for a great part of the social domain. They need to assure a more efficient and cheaper way of working, with a reduced budget (Sok et.al., 2013; Nijendaal, 2014). To guarantee this, social community teams are widely introduced in which different social professionals work interdisciplinary together to provide services to citizens in a specific district (Arum & Lub, 2014). The fundamental idea is to help people with problems in an early stage, to prevent more intensive care.
Managers of municipalities are held accountable for the new processes and to make sure things are going well and finances remain manageable, there is increased demand for performance information and control (Rutten, Sterrenberg & Vos, 2014). Therefore, private sector techniques to measure performance are introduced in the public sector (Thiel & Leeuw, 2002). This reform can be labeled under the development called New Public Management (NPM), which denotes policies that since the 1980’s aimed to modernize the public sector by improving its efficiency and effectiveness (Brignall & Modell, 2000). Decentralized governments and performance measurement are part of the NPM philosophy. Since 2000, this development has shift to the term New Public Governance (NPG). NPG argues that the goal of government is not just to improve efficiency and effectiveness in the public sector, but also to promote the larger common good (Bao et.al., 2012). Literature provides mixed evidence of whether this shift is evident in practice (Wiesel & Modell, 2014) and therefore the reform in general will be called NPM in this thesis.
implementation of a performance measurement system (PMS) in which all conflicting interests of stakeholders in the public sector are represented, is difficult (Brignall en Modell, 2000; van Thiel en Leeuw, 2002; Chenhall, 2003). On the contrary, there is also research indicating that it is possible to measure performance in the public sector1. Recently, several authors claim that the development of PMS’s in the public sector is indeed a difficult task but that challenges can be overcome by careful design and implementation (Bouckaert & Peters, 2002; DeGroff et al., 2010; Dan & Pollitt, 2014). However, the process by which a PMS is developed is obscure (Modell, 2009) and the number of researchers investigating the design and implementation process in more detail is very small, despite the increased interest in using PMS’s (Kollberg et. al., 2005). This study therefore aims to provide a more comprehensive understanding of how municipalities in the Netherlands react to the increased demand for performance information and how they develop a PMS. It is proposed that the development of a PMS consists of three factors: design, implementation and use (Bourne et.al., 2000). This leads to the main research question: In which way do municipalities in the Netherlands design, implement and use performance measurement systems to measure the performance of social community teams?
The decentralization and introduction of social community teams provide a unique context to analyze the development of a PMS. Next to the contribution to theory by providing a comprehensive understanding of the development of a PMS, this research also contributes to the empirical research field of the coming decentralization because most literature in this field is theoretical and empirical research is scarce (CPB, 2013). Next to that, this research can have implications for practice. While there is evidence of successful implementation of PMS’s in the public sector, today too many projects still fail and the development work is wasted (Jaaskelainen & Sillanpaa, 2012).
This thesis proceeds as follows. The next chapter discusses relevant literature regarding management control, performance measurement and the development of a PMS in the public sector. Chapter three describes the employed methodology. Chapter four represents and discusses the findings of the research and this thesis concludes in the fifth chapter.
1
Literature review on performance measurement and management prepared by the center for business performance. Cranfield school of management
2. LITERATURE REVIEW
This background section starts with a general description of management control and performance measurement, followed by theory about the development of a PMS in the public sector. Literature about designing, implementing and using a PMS in the public sector is reviewed. Several theoretical propositions are developed in this literature review to direct attention to subjects that need to be examined in this study (Yin, 2014).
2.1 Management control
Organizations use evaluation and control to obtain cooperation among individuals with different objectives (Ouchi, 1979). Management control “includes all the devices or systems managers use to ensure that the behaviors and decisions of their employees are consistent with the organization’s objectives and strategies” (Malmi & Brown, 2008). The systems themselves are called management control systems (MCS). The purpose of a MCS is to provide information that is useful for decision-making, planning and evaluation (Merchant & Otley, 2006).
Result controls focus the attention of employees on desired results, by holding them accountable for their performance (Ouchi, 1979). Result based control systems have much in common with the NPM’s result-based control structure. However, while the NPM philosophy suggests that a result-based control style will benefit public organizations, most theories about management control argue that this form of control can only be effective in very specific situations (Speklé & Verbeeten, 2009). It is argued that result control will only be effective when the goals can be clearly specified in advance, when results are aligned with objectives and can be adequately measured and when employees really have an influence on the desired results (Speklé & Verbeeten, 2009). These factors are not so obvious in the public sector, as will be explained in paragraph 2.3.
2.2 Performance measurement
doing. Marr et. al (2003) present three general reasons why organizations use performance measurement: “to implement and validate a strategy, to influence employees’ behavior and to report externally on performance.”
A PMS can be defined as “the set of metrics used to quantify both the efficiency and effectiveness of actions” (Neely et.al., 2005). All PMS’s consist of a number of individual performance indicators. As an important component of management control, the PMS should help employees to act in line with the achievement of objectives (Speklé & Verbeeten, 2009). The PMS enables to measure results and this may result in steering the behavior of employees.
2.3 Development of a performance measurement system in the public sector
The increased attention towards performance measurement in the public sector coincides with public reforms (van Thiel & Leeuw, 2002). One fundamental element of reforms in the public sector is decentralization of decision-making. Public sector managers are now more accountable for their decisions. To see how these managers or newly established units, like a social community team, execute their tasks, new information and specifying performance is necessary (Jansen, 2004; Cavalluzzo & Ittner, 2004). Therefore, private sector techniques to measure performance are introduced in the public sector (van Thiel & Leeuw, 2002). The need to measure results is important to focus on performance improvement. These changes are fed by a strong belief in the measurability of performance in the public sector. However, critics of NPM argue that differences between public and private organizations are so great that business practices should not be transferred to the public sector. In fact, it remains an important monitoring and evaluation strategy (DeGroff et.al., 2010).
giving correct insight in the effects and performance of social community teams which results in steering and improving performance.
The next step is to regard theory about how to carefully design, implement and use a PMS in the public sector. The following research framework is developed to illustrate the factors that influence the effectiveness of a PMS. In the following subsections, these factors will be elaborated.
Figure 1: Research framework
2.3.1 Design
One of the three factors in the development of a PMS is the design of the system. Designing a PMS is about determining appropriate indicators to measure performance. Indicators can relate to the transformation process in non-profit organizations, which is displayed in figure 2. Management control in traditional public organizations is mostly input-oriented (Jansen, 2004). Inputs are the resources that are necessary to perform a task and efficiency can be defined as the rate at which inputs are transferred into outputs. The adoption of NPM implies that the emphasis in management control of governmental organizations needs to change from a focus on information concerning input, to information on other elements of the transformation process (Jansen, 2004). Especially effectiveness is becoming important.
Figure 2: The transformation process in non-profit organizations
Because effectiveness is becoming important, there has been a move in the public sector towards output and outcome measures. Output is defined as the product or service provided by the organization. Examples are the amount of judgments of a judge, the number of patients treated by a doctor or the number of graduates of a school. Outcomes are the desired effects of the outputs of the organization (Propper & Wilson, 2003). Effectiveness is defined as how well the output achieves the desired outcome. It can be stated that the public sector is becoming result oriented. Result orientation is defined as the intention of decision makers to focus on outputs and outcomes (Hengel, Budding & Groot, 2014).
Designing indicators, according to Bruijn (2002) and van Thiel (2009), should only be done at the level where they are actually realized: the output of organizations. However, Jansen (2004) argues that as a consequence of the absence of a market for the products of public organizations, it can be very difficult to identify ‘products’ and therefore complicated to define output-related indicators. The strength of social community teams is that they work demand-oriented and flexible. How much time they actually spent on one case and what kind of work they perform is hard to plan in advance (Rutten, Sterrenberg & Vos, 2014). Next to that, output indicators are not related to program goals and social effects, and therefore it can be argued that outcome measures are more relevant. According to Mainz (2003), outcome indicators can detect problems in the implementation of processes of care and he also argues that outcome data are most useful over long periods of time.
safety, health and knowledge (van Thiel, 2009). Because of the complexity of many public health problems, explaining causal relationships between program inputs, activities and these abstract outcomes while also taking contextual factors in account is challenging (Bruijn, 2002; Schalock & Bonham, 2003; Poister, 2003). In the private sector, the focus is on profit and value which makes it easier to set targets (van Thiel and Leeuw, 2002; Propper and Wilson, 2003). While in the public sector, one can never measure what the influence of the army is on national safety, what the contribution of the police is on safety on the street or how a university influences the functioning of their graduates on the labor market (Bruijn, 2002). Therefore, performance measures are often incomplete and this can lead to distortion. This means that you measure ‘A’ (for example the number of fines) while you want to measure ‘B’ (for example safety). Secondly, many other factors and other organizations can influence outcomes. As displayed in figure 2, outcome is located outside the organization. This may be referred to as noise. Because public sector goals are rarely attributable to one organization, one should be careful when trying to develop measures that represent complex social phenomena with many interacting factors (Perrin, 1998). Safety for example is an outcome of efforts of the army, justice and citizens. The problem with noisy measures is that they decrease the usefulness for monitoring and motivational purposes because they are less informative about performance and they put risk on professionals if compensation is influenced by the measured performance (Speklé & Verbeeten, 2009). Next to that, public health outcomes often take years to realize (Bruijn, 2002). The fact that outcomes may take years to achieve, challenges an important principal of performance measurement, namely providing timely and useful data for decision-making. As a result, the use of long-term outcomes as performance indicators can cause problems for many public health programs, especially in the context of results-based management (DeGroff et. al, 2010). For other public health programs, although their outcomes may be measurable, they relate to prevention of specific behaviors or incidents. For such programs, it is also a challenge to identify meaningful and useful outcome measures. The success of prevention, which is also a goal of social community teams by preventing specialist care, is namely hard to determine (Rutten, Sterrenberg & Vos, 2014).
PMS’s can only work if causal relationships are understood and when outcomes are related to program efforts. For a good judgment and interpretation of outcome, it can also be useful to look at the reality and stories. Additional information can lead to a solid basis to discuss and assess performance and to a more realistic and fair assessment (van Thiel, 2009; Rutten, Sterrenberg & Vos, 2014). Mainz (2003) refers to this as risk adjustment, which means that it is important to adjust outcome measures for factors outside your influence that might contribute to the outcome.
The first proposition is as follows:
Proposition 1: Using outcome measures increases the effectiveness of the PMS
to the fact that employees will organize their work around reaching indicators. Aspects that are not measured, will receive less attention. Purchasers are interested in the quality, quantity and costs of services. If they are not satisfied, they may switch to another supplier. To comply with these different stakeholder interests and give a complete picture of performance, a PMS needs to be balanced and measure diverse aspects (van Thiel, 2009). Having a diverse set of performance indicators is defined as measurement variety. This leads to the second proposition:
Proposition 2: Measurement variety increases the effectiveness of the PMS
2.3.2 Implementation
A second development factor that can influence the effectiveness of the PMS, is the implementation phase: the realization of the design. One of the main reasons for the failure of the development of a PMS is sole emphasis on the top level of an organization (Lawton et. al., 2000; Kaplan, 2001). Lawton et. al. (2000) found that employees feel distanced from performance indicators which are designed without their contribution. Involving key stakeholders by communication and interaction is therefore important (Zimmerman, 1995; Bruijn & Helden, 2006). It can help to determine the goal of performance indicators, set priorities and assure a meaningful set of indicators (Bruijn & Helden, 2006; van Thiel, 2009; DeGroff et al, 2010; Jaaskelainen & Sillanpaa, 2012). This leads to the third proposition:
Proposition 3: Involving stakeholders increases the effectiveness of the PMS
2.3.3 Use
The last factor integrated in above research framework that can influence PMS effectiveness, is the use of the system. Public sector managers need to consider how they are going to use performance information.
control structure with well-defined responsibilities and targets to guide behavior. Without such goals, result-oriented behavior cannot be expected. A more exploratory use of a PMS does not fit with the NPM idea because it involves learning, experimentation and adaptation to new insights. However, findings of Speklé & Verbeeten (2009) suggest that the use of performance indicators for incentive purposes may decrease performance and the exploratory use of performance measures may enhance performance. It is common that public sector organizations do not have control over the causes that affect their goals and therefore using performance targets in an incentive-oriented way may have dysfunctional consequences, decreases employee’s willingness to engage in experimentation and does not lead to learning effects. The emphasis on an exploratory use of measures provides public managers with the opportunity to communicate goals clearly, evaluate the goals and policy and explain why policies may need to be reconsidered. With an exploratory use, the organization needs to discuss about the performance indicators. The focus is on experimentation, learning and adaptation and not on meeting targets. In this way, performance measurement may help to discover cause-and-effect relationships and the information may be used to give insight into the effectiveness of alternative program options. This is a suitable way to deal with the complexity of the public sector. This contradicts the NPM philosophy, which suggests that result accountability and incentives benefits the public sector. Therefore, Speklé & Verbeeten (2009) argue that policy makers in the public sector need to broaden their view beyond the NPM philosophy. This leads to the fourth proposition:
Proposition 4: Using performance information in an exploratory way increases the
effectiveness of the PMS
2.4 Conceptual model
Previous propositions that are based on literature lead to the conceptual model showed in figure 3. It is an extended version of figure 2 and displays which assumptions categorized by design, implementation and use are important to develop an effective PMS.
Figure 3: Conceptual model
Design
3. METHODOLOGY
This chapter elaborates on the methodology of the research. The first paragraph explains the research method. Then the research plan, case selection, data collection, data analysis methods and the reliability and validity of this research are discussed. The chapter ends with a description of the cases. By elaborating on the research methodology, it is possible to conduct this research properly.
3.1 Research method
The purpose of this research is to provide a more comprehensive understanding of the development of a PMS in the public sector. The main research question that will be answered is: “In which way do municipalities in the Netherlands design and implement performance measurement systems to measure the performance of social community teams?” The emphasis is on description and explanation and therefore a qualitative method is most appropriate. The method that is used for this research is a multiple case study research. A case study is “an empirical inquiry that investigates a contemporary phenomenon (a case) in depth and within its real world context, especially when the boundaries between phenomenon and context may not be clearly evident” (Yin, 2014). A case study is a suitable approach when the research aims to understand the dynamics present in single cases (Eisenhardt, 1989). The cases give the opportunity to give some empirical insight about theoretical principles (Eisenhardt, 1989; Yin, 2014).
3.2 Research plan
with the topic, a pilot interview is conducted with M. Koelewijn (partner social domain, BMC) in the early stage of the study.
3.3 Case selection
The scope of this research is limited to municipalities in the north of the Netherlands (Groningen, Friesland, Drenthe, Overijssel). To obtain a clear picture of municipalities, a variety of cases were selected. We selected municipalities with a variety in the stage of implementation and development of social community teams and variety in control. Next to that, both municipalities that make use of consultancy firms and municipalities that do not were selected. Consultancy firms are more familiar with private sector techniques like performance measurement and therefore this can cause differences in developing a PMS. The cases also represent a mix of small (<40.000 inhabitants), medium-large (>40.000 - 80.000 inhabitants) and large (>80.000 inhabitants) municipalities.
3.4 Data collection
In-depth interviews are the primary data source in this research. As Yin (2014) argues, “interviews are an essential source of evidence because most case studies are about human affairs or actions and well informed interviewees can provide important insights into such affairs or actions”. The interviews are semi-structured, which means that pre-determined questions form the basis of the interview. The questionnaire is based on the literature review. In total 11 in-depth interviews were conducted in november and december 2014 among a variety of municipalities with the person who is responsible for the control of social community teams. To prepare for the interviews, documents about the set up and organization of social community teams at the municipalities were read. Appendix B gives an overview of the interviews. Interviews lasted from 25 minutes to 57 minutes and all interviews are taped and transcribed. Transcribed interviews were sent to the interviewees for approval.
3.5 Data analysis
In total 124 pages of transcribed interviews were collected. To analyze this data systematically, a matrix is developed with categories related to the concepts design, implementation, use and effectiveness of a PMS. Also a matrix with relationships between design, implementation and use as dependent variables and effectiveness as the independent variable is developed to assess whether the data supports the propositions. Relevant quotes and information were marked and selected as input for the matrices. Transcribed interviews were re-read several times. To interpret the findings, ‘+’ and ‘-‘ signs were used to indicate the presence or absence of certain characteristics. Tables are generated to interpret findings related to the different concepts. Similarities or contradictions between respondents about a certain category were analyzed.
3.6 Validity and reliability
The quality of a research can be judged by four criteria: construct validity, internal validity, external validity and reliability (Yin, 2009). The first criteria, construct validity, refers to the degree to which an instrument measures what it claims to be measuring. To ensure construct validity, interview questions and concepts are linked to the literature review (Rowley, 2002). Furthermore, multiple sources of evidence can increase construct validity. Interviews are the main source of data in this research. Unfortunately, time constraints limited us to use more sources of data. Secondly, internal validity refers to the extent to which causal conclusions based on the study are true. Internal validity is sought by looking for patterns and rival explanations or possibilities when analyzing the data. External validity refers to the analytical generalizability of results and is sought by using multiple cases and replication logic to support the propositions. A limitation of this research regarding external validity is that this research is limited to municipalities. Results are not generalizable to other public sector organizations. The last criterion, reliability, demonstrates that a study can be repeated by using the same procedures to reach the same findings. The goal is to minimize errors and biases in a study. To ensure this, procedures are clearly and stepwise elaborated and made transparent.
3.7 Background case study
4. RESULTS
The results of the analysis of the development of a PMS at municipalities will be presented in this chapter. A general description of all the cases can be found in appendix C. Table 2 gives an overview of the results. Following, the concepts regarding the design, implementation, use and effectiveness of a PMS will be analyzed independently. Next to that, links between the different concepts and the effectiveness of the PMS are analyzed to assess the propositions. # Working with complete PMS? Stage of PMS Meas-uring? List with indica-tors? Focus PMS: outcome or output? Measure-ment variety? Stakeholder involvement design indicators (Future) use of the PMS 1 No Monitoring
developed Yes Yes Outcome Medium Some directly involved Exploratory
2 No In
development
No Yes Outcome High Indirectly Exploratory
3 No No monitoring
developed No First draft Both Low involvement No Exploratory
4 No Monitoring developed
No Yes Both High All stakeholders directly involved Exploratory 5 No In development
* No Outcome High Some directly involved
Exploratory
6 No In
development No No Outcome Low Some directly involved Exploratory
7 No In
development * No Both Medium Indirectly Exploratory
8 No No monitoring
developed * No Both Low Indirectly Exploratory
9 No In
development
* Yes Output High All stakeholders
directly involved
Exploratory
10 No In
development No Yes Both High stakeholders All directly involved
Exploratory
11 No No monitoring first year
No Yes Output Medium No involvement
Exploratory
An important first insight is that none of the municipalities has already set up a functioning, complete, PMS. Most of the municipalities are still developing and building their monitor system. Some do not monitor on purpose the first year and two municipalities developed a monitor to measure performance of the social community teams. Solely one of the seven municipalities that are already working with social community teams is measuring performance of the team. All municipalities adopted a result oriented control style by focusing on output and outcome measures. Involvement of stakeholders when designing the indicators varies a lot between municipalities. All municipalities are going to use performance information in an exploratory way.
4.1 Concepts
The design, implementation, use and effectiveness of the PMS’s at municipalities will be analyzed in this paragraph.
4.1.1 Design
Table 3: design of performance indicators # List with performance indicators? Orientation Risk awareness/ adjustment Measuring costs? Measuring quality? Measurement variety? Measuring social effects? Examples indicators
1 Yes Outcome High Yes Yes Medium Yes Decreased number of professionals,
self-reliance
2 Yes Outcome Low Yes Yes High Yes Self-reliance, number reported
domestic violence, number of people using drugs, number of ‘maatwerk voorzieningen’, customer satisfaction,
number of indications, use of community facilities
3 First draft Output Low Yes Yes Low No Execution costs, number of
indications & 6 indicators related to client satisfaction
4 Yes Both High Yes Yes High Yes Self-reliance, nature and development
expenditures, customer satisfaction
5 No Outcome High Yes Yes High Yes Self-reliance, costs, number of people
receiving care, loneliness
6 No Outcome High Yes Yes Low Yes Number of people in debt, safety,
loneliness, health, return alcohol use, number of requests, number of
complaints
7 No Both Low Yes Yes Medium Yes
8 No Both High Yes Yes Low Yes
9 Yes Output High Yes Yes High Yes Self-reliance, number of requests,
people in debt, number of clients and in which categories
10 Yes Both High Yes Yes High Yes Self-reliance, number of applications
solved by own network, client satisfaction
11 Yes Output Low Yes Yes Medium Yes Self-reliance, decreased number of
meaning that indicators are related to output and outcome. This is in line with the findings of Propper & Wilson (2003) and Jansen (2004) that management control of governmental organizations changes from a focus on input to output and outcome indicators, because of the NPM movement. It can be argued that the decentralization of the Dutch government stimulated municipalities to control in a more result oriented way.
Most of the municipalities argue that outcome is the most important thing to measure because outcomes are related to the goals of social community teams. However, literature showed that it is difficult to develop appropriate outcome measures, because public goals are focused on abstract outcomes and contextual factors can also influence these outcomes. The column ‘risk awareness’ in table 3 indicates if the respondent mentioned risks and challenges of measuring outcome. Most (n=7) of the municipalities are highly aware of this challenge. One respondent explains:
“There are so many more factors than the things that we are doing that influences outcome.”
Another respondent confirms this, and stresses that although it is difficult, you still have to try to measure it:
“It is questionable if there is a direct link between your actions and the result you are measuring. If there is a causal relationship, is hard to prove, but you have to assume that it’s true.”
Some municipalities put it differently, arguing that the focus will be on output first, because it takes time to figure out how they can measure outcome:
“In the beginning we will look at numbers. Money. And eventually you want to look at outcomes.”
“Output is something that is measurable now. I would be happy if we can measure the necessary things at output level.”
output related things like the amount of clients, number of indications, decreased amount of professionals and throughput time. Out of the three municipalities that focus on output, two will also measure social effects. Therefore, a focus on output does not mean that outcome (goal of the social community team) is not measured at all.
Seven municipalities are highly aware of the risks when measuring outcome. Out of the four municipalities that did not mention something about risk adjustment, two focus on output, one on outcome and one on both. Therefore, there seems to be no link with the orientation of measures and risk awareness. The results show that municipalities are aware of the challenges when measuring outcome because of the uncertainty about causal relationships between input and outcome. Municipalities try to understand these relationships in different ways. One municipality mentioned that they just assume a causal relationship to be true. Some try to interpret outcome by looking at individual cases. However, others argue that the overall picture is more important than individual cases, because in that way you can see the results in a specific area where a social community team is active. Another aspect to interpret outcome that is frequently mentioned, is asking people for stories. This is in line with the argument of van Thiel (2009), that additional information is needed to interpret performance. She argued that when numbers and stories are combined, it could lead to discussions and to a more realistic and fair assessment.
Speklé & Verbeeten (2009) argued that result control can only be effective when goals are specified in advance, when results are aligned with objectives and can be adequately measured and when employees really have an influence on the desired results. Municipalities adopted a result oriented control style and indicators are aligned to objectives. However, they indicate that measurement is difficult and the influence of employees on results in unclear. An explanation can be that municipalities are in the beginning of measuring performance, perhaps in the future they better know how to measure performance and link it to the work of social community teams. Another explanation can be that it is too early to see differences in outcome, because outcomes can take years to realize (Bruijn, 2002).
“We are going to focus on qualitative monitoring next to numerical information. Costs are only one aspect of monitoring.”
However, there is no link between measuring both aspects and measurement variety. High measurement variety means that qualitative and quantitative aspects are measured related to a variety of aspects. All municipalities measure costs as well as quality, but only five are interpreted as high measurement variety. There is also no relationship between measurement variety and the orientation of the indicators. It needs to be recognized that not all municipalities developed a complete list of indicators, which may explain why measurement variety is not high yet because they simply do not know which indicators they will use. 4.1.2 Implementation
Another stage in the development of the PMS is the implementation stage, which refers to the involvement of stakeholders when designing indicators. Table 4 shows an overview of the stakeholders that are directly and indirectly involved and stakeholders that are not involved at all. Three columns of table 3 are inserted, to compare involvement with design.
# Directly involved Indirectly involved Not involved List with
indicators Orien-tation Measu-rement variety
1 Team members Citizens, care
organizations Yes Outcome Medium
2 Partners Team members Yes Outcome High 3 Nobody involved First draft Output Low 4 Citizens, team members, care
organizations, controller
Yes Both High
5 Care organizations, controller No Outcome High 6 Manager team, municipality members Care
organizations
No Outcome Low
7 Citizens, care
organizations No Both Medium
8 Team members, team leaders and managers,
citizens
No Both Low
9 Financial employees, team leaders,
project leader, citizens (clients) Yes Output High
10 Citizens (clients), care organizations, team members, project team and
coordinators, NHL Leeuwarden
Yes Both High
Only a small majority (n=6) of municipalities directly involve stakeholders in designing indicators. One respondent explained the importance of involvement:
“The things we will steer on are created together. We got compliments from care organizations for our transparency and their opportunity to give advice about things we are going to measure.
Everyone is behind it, in that way you create your support.”
Theory stressed that involving key stakeholders can help to determine purposes, set goals and assure a meaningful set of measures (de Bruijn & Helden, 2006; van Thiel, 2009; DeGroff et al, 2010; Jaaskelainen & Sillanpaa, 2012). In this paper it is argued that ‘a meaningful set of measures’ are measures related to outcome (eventual in combination with output measures). There seems to be a link with the involvement of stakeholders and the orientation of indicators. Three out of the six municipalities that directly involve stakeholders focus on outcome, two on both outcome and output and one on output.
Vakkuru & Meklin (2006) argued that PMS’s in the public sector are often designed by individuals instead of their actual users. There are only two cases that involved nobody in discussing indicators. In both cases it was one person, the respondent, which developed the indicators. One of those two mentioned that the indicators were a suggestion and that team members can say something about it in a later stage. It is interesting to see that these two cases both focus on output instead of outcome. This, in combination with the fact that municipalities that directly involve stakeholders focus more on outcome (eventual in combination with output), provides support for a relationship between involving stakeholders and measuring on outcomes. This indicates a positive link between implementation and design.
Another striking point is that two out of the three municipalities that indirectly involve stakeholders, will work with social community teams in the form of a foundation. It seems that this leads to indirectly involving stakeholders instead of direct involvement.
4.1.3 Use
The last factor that plays a role in the development of a PMS is the use of the system. Public sector managers consider how they are going to use performance information. It is notable that most municipalities mention how they are going to use performance information instead of how they use it already. This also suggests that municipalities are not working with performance measurement yet.
Although most of the municipalities adopted a result oriented control style by using outcome and output indicators, results show that none of them is going to use performance measurement for incentives and reaching targets. Respondents explain:
“It is more evaluating, you know, these kind of processes are hard to link to performance.” “I really think you need to use it to make adjustments.”
4.1.4 Effectiveness
Although municipalities are aware of the fact that they have to develop a PMS and adapt a more result oriented control style, they do not use monitoring yet and most PMS’s are in development and not ready for use. Because active adoption of PMS’s is not yet realized, it is difficult to say something about effectiveness of a system. In the literature section, effectiveness is defined as giving correct insight in the effects and performance of social community teams, which results in steering and improving performance. Respondents mentioned:
“We have a long way to go.”
“It is too early to say something about the quality of monitoring.”
Still, most respondents are aware of the fact that they need a well-functioning system in the near future to monitor performance. Therefore, it can be argued that there is a gap between intention and use. It is explained by several respondents why they are not working with a PMS yet:
“I would be happy if people first start working and knowing what they are doing. That is the first step. Then the measuring part comes. You know, it is not an insurance you take, or the number of phone
calls in a call center. I think that is of minor interest.” “I do not think we were developed enough to start measuring.”
4.2 Relationships between concepts
In the previous paragraph, the results related to the different concepts are elaborated. In this paragraph links between the independent factors design, implementation, use and the dependent factor effectiveness of the PMS will be analyzed. The propositions in the literature review of this research form, enhanced by the findings in paragraph 4.1 and 4.2, the groundwork for analytical generalization. If two or more cases show support for a proposition, replication can be claimed. The higher the number of cases that claim replication, the more support for a theory (Rowley, 2002). However, it is difficult to say something about the effectiveness of PMS’s because they are in development and not used yet. Therefore it has to be recognized that it is also difficult to claim convincing support for relationships between concepts and effectiveness of the PMS.
4.2.1 Design – effectiveness
Regarding a link between design and effectiveness, two propositions were developed in the literature section. The first one stated that using outcome indicators increases the effectiveness of the PMS. Two respondents mentioned something about the link between outcome indicators and the effectiveness of their PMS:
“I think our focus is on output now and we make inadequate use of what we can retrieve from the system. Outcome is a bit too far away.”
“I think we do not have a perfect monitoring, so we have to seek how we can monitor on outcome instead of output.”
This indicates that because the focus is not on outcome yet, the effectiveness of their monitoring is not high and thus supports the first proposition.
There are also two respondents that explained that performance could increase because of cost awareness. They argue that professionals did not know how much they cost and are now more aware of this, which makes them wonder if it is responsible what they do. However, another respondent mentioned the downside of a focus on costs on monitoring:
This indicates that seeing the bigger picture is more important and steering on incidents is not necessary. This creates space for professionals to perform their jobs. It increases support for the first proposition, because outcome is what matters. However, convincing support for the proposition cannot be claimed, because differences between municipalities that focus on either outcome or output and the effectiveness of their system cannot be seen yet.
The second proposition regarding design stated that measurement variety increases the effectiveness of the PMS. No explicit things are found at this moment to support this proposition. However, neither anything is found that could claim that this proposition is not true. An explanation is that it is too early to say something about this relationship, because of the fact that PMS’s are not functioning yet. Because performance is not measured already, missing indicators to give a broad insight in performance may not be recognized yet.
4.2.2 Implementation – effectiveness
Regarding the implementation of the PMS, the proposition stated in literature mentioned that involvement of stakeholders increases the effectiveness of the PMS. No specific information is found to support this relation. However, in paragraph 4.1.2. it is found that stakeholder involvement positively influences the orientation of indicators and measurement variety. Therefore, support is found for an indirect relationship between implementation and effectiveness.
4.2.3 Use – effectiveness
A majority of the respondents mentioned that an exploratory use gives insight and room for adjustment, which increases performance. To illustrate, one respondent said:
“You gain insight and when you have insight then you can change.”
This supports the fourth proposition that an exploratory use of the PMS can increase effectiveness. This support is further increased by a respondent that argued conversely:
“We choose to not give performance targets, because it is difficult to quantify and hard to see what it means.”
5. CONCLUSION, LIMITATIONS AND FUTURE RESEARCH
5.1 Conclusion
The goal of this research was to provide a comprehensive understanding of how municipalities, that will be responsible for a great part of the social domain because of government decentralization, react to the increased demand for performance information and how they develop a PMS.This study tried to answer the research question how municipalities in the Netherlands design, implement and use PMS’s to measure the performance of social community teams. It was found that the decentralization of the Dutch government stimulates municipalities to measure performance and become result-oriented. Municipalities are busy developing and implementing PMS’s and are aware of factors that can possibly lead to an effective PMS. They become result-oriented and focus on outcome indicators, which according to literature are most relevant because they are aligned with program goals (Speklé & Verbeeten, 2009). Next to that, to comply with different stakeholder interests and give a complete picture of performance (van Thiel, 2009), municipalities will use a diverse set of performance indicators related to costs and quality. It seems that there is a positive link between the involvement of stakeholders in designing indicators, because municipalities that directly involved stakeholders designed outcome indicators and are interpreted as having high measurement variety. However, not all municipalities already developed indicators and use a functioning, complete PMS. Some municipalities are further with their PMS development but there seems to be no link between development and the use of the PMS in practice. All municipalities are going to use the PMS in an exploratory way, to gain insight. To achieve the goal of PMS’s to improve performance, systems need to be further developed and taken into use. The fact that this research was done before the actual decentralization may explain why the PMS’s are not ready for use.
5.2 Limitations
The employed methodology of a multiple case study allowed to gain an in-depth understanding of the development of a PMS in the public sector. Despite the fact that measures were taken to increase reliability and validity, this study also has a few limitations that need to be recognized. First of all, the analysis and interpretation of case study data may be influenced by subjectivity. Second, the analysis is a snapshot of the current situation but it could be the case that things change rapidly because data is gathered before the actual decentralization (1-1-2015). Perhaps the development of a PMS is more a long-term development and may become more important after the decentralization, which make the results less reliable. Third, respondent of the second case mentioned that another employee of the municipality knows more about the indicators and performance measurement. This weakened the analysis of the second respondent.
5.3 Future research
Arum van S., Lub V. (2014). Wat gemeenten van sociale wijkteams verwachten. Beleidsonderzoek online.
Bao, G., Wang, X., Larsen, G.L., Morgen, D.F. (2012). Beyond New Public Governance: a value based global framework for performance measurement, governance and leadership. Administration & Society. 45(4), 443-467.
Bouckaert, G., Halligan, J. (2006). Performance and Performance Management. In B.G. Peters & J. Pierre (eds.), Handbook of Public Policy (p443-459). London, SAGE Publication Ltd.
Bouckaert G., Peters Guy B. (2002). Performance measurement and management: the achilles heel in administrative modernization. Public performance and management review. 25 (4), 359-362.
Bourne, M., Wilcox, M., Neely, A., Platts, K. Designing, implementing and updating performance measurement systems. International Journal of Operations & Production Management. 20(7), 754-771.
Brignall, S. & Modell, S. (2000). An institutional perspective on performance measurement and management in the ‘new public sector’. Management Accounting Research, 11 (5), 281- 506.
Bruijn, H. De (2002). Prestatiemeting: tussen politiek en professie. CDV (3).
Bruijn, H. de (2002), Managing Performance in the Public Sector ‘Output Steering in Public Organizations. About the Use of a Product Approach and a Process Approach’, Managerial Accounting Journal. 18(4), 303–12.
Bruijn H. de, Helden van J. (2006). A plea for dialogue driven performance-based management systems: evidence from the Dutch public sector. Financial accountability & management, 22(4), 405-423.
Cavalluzzo, K.S., Ittner, C.D. (2004). Implementing performance measurement innovations: evidence from government. Accounting, Organizations & Society 29. 243-267.
Chenhall R.H. (2003). Management control systems design within its organizational context: findings from contingency-based research and directions for the future. Accounting, Organizations and Society 28, 127–168.
Dan, S., & Pollitt, C. (2014) NPM can work. An optimistic review of the impact of new public management reforms in central and eastern Europe. Public management review.
DeGroff, A., Schooley, M., Chapel, T., Poster, T.H. (2010). Challenges and strategies in applying performance measurement to federal public health programs. Evaluation and program planning 33. 365-372
Eisenhardt, K.M. (1989). Building Theories From Case Study Research. Academy of Management Review. (14), 532-550.
Frederickson, D.G. (2003). Performance measurement and third party government: a study of the implementation of government performance and results act in five health agencies. Dissertation. Indiana University.
Hengel, H.G. van, Budding, G.T. & Groot, T.L.C.M. (2014). Loosely coupled results control in Dutch municipalities. Financial Accountability and Management, 30(1), 49-74. Hendriks, F., Tops, P. (2003). Local public management reforms in the Netherlands: fads,
fashions and winds of change. Public administration. 81(2), 301-323.
Jaaskelainen, A., Sillanpaa, V. (2012). Overcoming challenges in the implementation of performance measurement, case studies in public welfare services. International journal of public sector management. 26(6), 440-454.
Jansen, P. (2004). Performance Measurement in Governmental Organizations: A Contingent Approach To Measurement and Management Control. Managerial Finance, 30(8), 54-68.
Jarrar, Y. & Schiuma G. (2007). Measuring performance in the public sector: challenges and trends. Measuring Business Excellence. 11(4), 4-8.
Jennings, E.T., Haist, M.P. (2004) Putting performance measurement in context. The art of governance: analyzing management and administration. Washington DC: the national academies press.
Jordan, S., Messner M. (2012). Enabling control and the problem of incomplete performance indicators. Accounting, Organizations and society. 37, 544-564.
Kaplan, R.S. (2001). Strategic performance measurement and management in non-profit organizations. Non-profit management & Leadership. 11(3), 353-370.
Kickert, W.J.M., Klijn, E.H., & Koppenjan, J.F.M. (1997) Managing complex Networks: Strategies for the Public Sector. Thousand Oaks: SAGE Publications.
Kollberg, B., Elg, M., Lindmark, J. (2005). Design and implementation of a performance measurement system in Swedish health care services: a multiple case study of 6 development teams. Quality management in health care. 14(2), 95-111 Lawton, A., McKevitt ,D., Millar, M. (2000) Coping with ambiguity: reconciling external legitimacy and organizational implementation in performance measurement. Public money & management, 20:3, 13-20.
Mainz, J. (2003). Defining and classifying clinical indicators for quality improvement. International journal for quality in healthcare. 15(6), 523-530.
Maisonneuve, C., de la & Martins, J. O. (2013). Public spending on health and long-term care: a new set of projections. OECD Publishing.
Malina, M.A. & Selto, F.H. (2001). Communicating and controlling strategy: an empirical study of the effectiveness of the balanced scorecard. JMAR. 13, 47-76.
Malmi, T. & Brown, D.A. (2008). Management control systems as a package – opportunities, challenges and research directions. Management accounting research. 19, 287-300. Merchant, K. & Otley, D.T. (2006). A review of the literature on control and accountability.
Handbook of management accounting research. Elsevier. 785-802.
Modell, S. (2009). Institutional research on performance measurement and management in the public sector accounting literature: a review and assessment. Financial Accountability & Management. 25(3), 277-366.
Neely A., Gregory M., Platts K. (2005). Performance measurement system design. A literature review and research agenda. International journal of operations and production management. 25(12), 1228-1263.
Nijendaal van G.A. (2014). Drie decentralisaties in het sociale domein. Jaarboek overheidsfinanciën 2014.
OECD. (2013a). What Future for Health Spending?, OECD Economics Department Policy Notes, No. 19 June 2013. Retrieved from: www.oecd.org/economy/health-spending.pdf
OECD. (2014). OECD statistics. Retrieved from database: http://stats.oecd.org/.
Perrin, B. (1998). Effective use and misuse of performance measurement. American journal of evaluation. 19(3), 367-379.
Poister, T.H. (2003) Measuring performance in public and non profit organizations. San Francisco, CA: Jossey-Bass.
Propper, C., Wilson, D. (2003). The use and usefulness of performance measures in the public sector. Oxford review of economic policy. 19 (2), 250-267.
Rowley, J. (2002). Using case studies in research, Management Research News. 25(1), 16 – 27.
Rutten, W., Sterrenberg, L., Vos, W. (2014). Alleen cijfers meten is niet genoeg. Hoe uitvoerders, wethouders en gemeenteraden samen prestaties kunnen meten in het sociale domein. Platform 31.
Schalock, R.L., Bonham, G.S. (2003) Measuring outcomes and managing for results. Evaluation and program planning. 26, 229-235.
Sok, K., Bosch, A. van den, Goeptar, H., Sprinkhuizen, A. Scholte, M. (2013) Samenwerken in de wijk. Actuele analyse van sociale wijkteams. Utrecht: Movisie.
Speklé, R.F. & Verbeeten, F.H.M. (2009). The use of performance measurement systems in the public sector: effects on performance. Nyenrode Business Universiteit, 2009. Thiel van, S., & Leeuw, F. (2002). The performance paradox in the public sector. Public
Performance and Management Review, 25 (5), 267-81
Thiel van, S. (2009). Prestatiemeting in de publieke sector. Overheidsmanagement nummer 1, januari 2009.
Wiesel, F. & Modell S. (2014). From New Public Management to New Public Governance? Hybridization and implications for public sector consumerism. Financial accountability and management. 30(2), 175-205.
Wouters, M., Wilderom, C. (2008). Developing performance measurement systems as enabling formalization: a longitudinal field study of a logistics department. Accounting, organizations and society. 33(4-5), 488-516.
APPENDIX A – CASE STUDY PROTOCOL (NEDERLANDS)
Omdat dit onderzoek bestaat uit meerdere casussen en omdat twee onderzoekers bijdragen aan de database die door beide wordt gebruikt voor data analyse, is dit protocol ontwikkeld ter voorbereiding. Het protocol is in het Nederlands, aangezien de interviews in het Nederlands worden afgenomen.
Algemene afspraken
• Beide onderzoekers volgen dit protocol.
• Gemeentes worden telefonisch benaderd voor een interview en ontvangen vervolgens een document met informatie over het onderzoek per mail.
• Interviews worden opgenomen (indien de geïnterviewde hiermee akkoord gaat) en in detail uitgewerkt en gedeeld in de database.
• Doelen van dit onderzoek voor beide onderzoekers en voor BMC zijn voor beide duidelijk, zodat tijdens de interviews ingespeeld kan worden op informatie behoeftes. • Achteraf worden gemeentes bedankt en het uitgewerkte interview wordt opgestuurd
ter goedkeuring.
Het onderzoek
• Tijdens de interviews een rijk dialoog creëren. Niet vast blijven houden aan de opgestelde vragen, maar nieuwsgierig zijn en door vragen.
• De opgestelde vragen als checklist gebruiken, om ervoor te zorgen dat wel alles aan bod komt tijdens het gesprek.
• Goed de tijd bewaken, zodat het interview niet uitloopt en er tijd is voor het hele interviewprotocol.
• Proberen goed te luisteren en niet bevooroordeeld zijn door bepaalde ideeën en theorieën. Het kan in de praktijk heel anders zijn dan in theorie.
• Er worden enkel open vragen gesteld en geen leidende vragen, om een gesprek op gang te houden en zo eerlijk mogelijke antwoorden te krijgen.
• Om het voorgaande punt onder andere te garanderen, wordt een pilot interview gehouden met een medewerker van BMC. Zo wordt het gevoel voor de praktijk versterkt en kunnen we wennen aan vaktermen die worden gebruikt bij gemeentes.
Vooraf (maximaal 5 minuten)
- De persoon die wordt geïnterviewd bedanken voor zijn/haar tijd en medewerking aan het onderzoek.
- Aangeven dat eerst een introductie van mijzelf, het onderzoek en de opzet van het interview volgt voordat de interview vragen worden gesteld.
- Introductie: master student Organizational & Management Control aan de Rijksuniversiteit Groningen. Geïnteresseerd in de publieke sector.
- Afstudeer onderzoek (in samenwerking met een medestudent) naar sociale wijkteams. De bedoeling is dat de sociale wijkteams zorg efficiënter maken. Een belangrijke vraag hierbij is echter: hoe weet een gemeente nu of de sociale wijkteams doelmatig functioneren? Wij zijn met name geïnteresseerd in vragen omtrent de prestatiemonitoring van sociale wijkteams, vanuit het perspectief en informatiebehoefte van de gemeente. Prestatiemonitoring is heel breed en gaat hierbij in essentie om de vraag: hoe krijgt de gemeente een beeld van het reilen en zeilen van de sociale wijkteams? Wij onderzoeken het ontwerp, de implementatie en het gebruik van de prestatiemonitoring en vergelijken theorie met de praktijk.
- Aangeven dat er geen namen van individuen in de resultaten worden genoemd en dat het uitgewerkte interview ter goedkeuring per e-mail wordt opgestuurd.
- Toestemming vragen voor het opnemen van het interview. Deze opnames zullen alleen worden gebruikt door mij en medeonderzoeker.
- Helder aangeven dat het belangrijk is om alles aan bod te laten komen, en dat daarom de tijd in de gaten wordt gehouden. Vragen of het goed is dat ik hierin de leiding neem, en bijstuur waar nodig.
- Als de respondent meer informatie over het onderzoek vraagt, aangeven dat dat na afloop van het interview kan worden gegeven, maar dat het om methodologische redenen niet verstandig is om dat nu te doen.
Introductievragen (maximaal 5 minuten)
- Sinds wanneer werken jullie met sociale wijkteams?
- Kun u kort vertellen iets vertellen over de rol van sociale wijkteams in uw gemeente en de relatie tussen de wijkteams en de gemeente?
- Welke functies zijn vertegenwoordigd in het sociaal wijkteam?
Inleidende vraag naar hoofdvragen
- Wat is uw rol binnen de monitoring/aansturing van sociale wijkteams?
Hoofdvragen Ontwerp
- Hoe krijgt u een beeld van het functioneren van het sociale wijkteam? - Wordt er gewerkt met indicatoren?
- Ligt de nadruk op het meten van de output (bijvoorbeeld het aantal behandelde cases) of de outcome (bijvoorbeeld vermindering van eenzaamheid in de wijk)?
- Werkt u veelal met financiële of niet-financiële gegevens voor de aansturing van de sociale wijkteams? Kunt u hier voorbeelden van noemen?
Implementatie
- Hoe zijn de indicatoren van het sociale wijkteam tot stand gekomen?
Worden de doelen en de indicatoren gecommuniceerd aan de sociale wijkteams? Of worden wijkteams betrokken bij de ontwikkeling van de doelen en indicatoren?
- Hoe wordt omgegaan met externe belanghebbenden?
Hebben die ook invloed op de ‘prestatiemonitoring? (bijvoorbeeld: burgers, zorginstellingen). Wie zijn die stakeholders die betrokken zijn bij de invulling van de prestatiemeting?
Gebruik
- Wordt het werk van het sociale wijkteam geregistreerd?
- Hoe wordt er omgegaan met de informatie over de prestaties van sociale wijkteams (dat bijvoorbeeld uit het registratiesysteem komt)?
