A critical review of performance measurement in public sector organizations: the case of social community teams

(1)

Master’s Thesis

MSC Business Administration

Specialization Organizational & Management Control

A critical review of performance measurement in public

sector organizations: the case of social community teams

Abstract

This case study aims to provide more insight into performance measurement in the public sector. The public sector has fundamental differences in comparison with the private sector, which makes performance measurement in the public sector more difficult. This case study offers four strategies to overcome these difficulties in order to maximize the effectiveness of performance measurement in the public sector. These strategies conflict with the principles of New Public Management which ignores the limitations of public sector performance measurement. Interviews with public managers and policy advisors of Dutch municipalities were held to observe whether municipalities use these strategies for performance measurement of social community teams. The main finding of this case study is that performance measurement which takes place at municipalities is consistent with the strategies from literature. Municipalities use performance information mainly for exploratory purposes instead of command and control purposes which are emphasized by New Public Management.

Author: W. Leistra

Student number: 2003910 Supervisor: Dr. H.J. van Elten Co-assessor: Dr. S. Tillema Date: 20-01-2015

Number of words: 12.689

(2)

2

Preface

This is the final result of my Master’s Thesis. With this thesis, I finished the track Organizational & Management Control, which is a specialization of the master Business Administration at the University of Groningen. The courses of this specialization helped me to develop my academic skills and to gain a better insight into management control issues. With these skills, I was able to write this thesis which also discusses a management control issue, specifically the issue of performance measurement in the public sector.

I am satisfied with the final result and the way the research has been performed. In this research, I was able to observe how theoretical constructs are present in practice. I am enthusiastic about this way of conducting research, because it gives insight into practical issues, which I find more interesting than conducting theoretical research. I have experienced that the public sector is a supportive sector. The main reason for this is that we were able to find enough respondents in this period, while the municipalities were very busy with the creation and development of the social community teams.

I would not have been able to conduct this research without the support of several people. First of all, I would like to thank drs. M. Koelewijn of BMC for offering the opportunity to conduct a pilot interview. I am also very thankful to my supervisor dr. H.J. van Elten for his critical feedback and useful suggestions which have been valuable for the final result. I thereby want to thank my fellow student Tessa Bergsma for her cooperation in the collection of the data. This created the possibility to obtain a larger research sample, which is a useful contribution to this research. Furthermore, I thank the respondents of the municipalities for giving the opportunity to conduct interviews. Finally, I want to thank my family and friends for their support during the whole process.

Groningen, January 2014

(3)

3

5

1. Introduction

The Dutch government is currently reorganizing its healthcare system, because the costs of Dutch healthcare are increasing due to an ageing population. One of the reorganizations of the government is to make local municipalities responsible for: child welfare, employment and income, long-term sick and elderly at the first of January 2015. This decentralization project is meant to make the public sector more efficient, and budgets for healthcare will be reduced. The core of the new system is the creation of social community teams to combine different care tasks. A social community team will provide different forms of care to citizens in a specific neighborhood, and may for example consist of a nurse, an elderly advisor, and an employee of the Social Support Act. The way in which a social team will be controlled may vary by municipality and municipalities are currently experimenting with these teams to find out how they can be used effectively. Since there is still uncertainty about how municipalities are going to measure performance of these social teams, it is an interesting topic for this research.

(6)

6

Nevertheless, there is a lot of criticism on NPM in literature. Main reason for this is that according to NPM, private sector management techniques to measure performance should be used in the public sector, but there are fundamental differences between the public sector and its private counterpart (Thiel & Leeuw, 2004; Boyne, 2002; Ranson & Stewart, 1994). First of all, public organizations have to deal with multiple stakeholders who are interested in different types of performance. Furthermore, goals are more vague in the public sector, because policies are judged by the political system. In such a system, policies can change frequently and politicians can put pressure on lower government layers to achieve short-term results (Propper & Wilson, 2003). Moreover, the public sector focuses less on efficiency, because these organizations pay more attention to the fulfillment of social goals and face less competitive pressures. There is also a weak link between performance and rewards, because the public sector has difficulties in defining the nature of public services (Boyne, 2002). These special properties make it more difficult to measure performance in accordance with the principles of NPM.

De Bruijn & van Helden (2006) further argue that organizations that measure performance with the principles of NPM should have control systems which are based on output control and bureaucratic practices. However, the Dutch political system is characterized by a polder model, which is based on consensus. Implementing a NPM-type PMS may contradict with this political system, because a NPM-type PMS is characterized by monitoring and offers little room for discussion. Implementing a NPM-type PMS may lead to misunderstanding among employees. It should therefore be questioned whether the assumptions of NPM are appropriate for the monitoring of social community teams.

(7)

7

Which risks can arise when performance of social community teams is measured and are there possibilities to mitigate these risks?

The theoretical contribution of this research is a critical summary of PSPM which can be used to provide useful insights for practice. Management accounting is a research field with a clear link to practice, and especially in the public sector more practical insights are needed (Scapens, 2008; Malmi & Granlund, 2009; Rynes & Shapiro, 2005). Van Helden & Northcott (2010) further emphasize the importance for researchers to solve practical problems for governments, policy-makers and other accounting practitioners. Reason for this is that new advanced management accounting techniques such as NPM have been introduced to improve the functioning of the public sector (Hood, 1995; van Helden & Northcott, 2010; Olson et al., 1998). These new techniques need to be examined in practice to find out whether they effectively improve public sector performance. Social community teams give the opportunity to learn if the suggested problems from literature are present in practice. The practical contribution of this research is that it can give recommendations to municipalities about the way the social teams should be monitored. Furthermore, research in this particular field is scarce and municipalities are now in the position to revise their policies, because the teams are currently being tested.

This paper can be divided into several sections. The next section explains the relevant literature about PSPM. The third section explains the methods which are used to conduct this research. The fourth section gives the results, and the paper will conclude with a discussion and conclusion of the results, and some suggestions for further research.

2. Literature review

(8)

8

2.1 What is performance measurement?

De Waal and Kourtit (2013) found that organizations start using a PMS if a larger focus on control or strategy is necessary. This can be explained by the fact that there is a clear relationship between strategy and PM. Performance indicators are created to determine whether an organization its strategy delivers a satisfying level of performance. Performance indicators must therefore be consistent with the strategy of the organization. Otherwise these indicators do not have any predictive value (McMahan, 2005). McDougall et al. (1995) see performance measurement as an evolutionary process of: (1) understanding key performances, (2) obtaining accurate measures, (3) learning from finding, and (4) adjusting the measures.

Neely (2005) defines performance measurement (PM) as a process of quantifying the efficiency and effectiveness of action. A performance measurement system (PMS) is the set of metrics used for this quantification process (Neely, 2005). Efficiency and effectiveness are related in such a way that efficiency is the pursuit of perfection of a given approach, while effectiveness is the pursuit of the most economic approach (McDougall et al., 1995). A PMS can increase efficiency and effectiveness, because it makes management aware of its current performance. Thereafter, management is able to optimize its current performance to improve efficiency and adjust the current strategy to improve effectiveness.

2.1.1 Types of measures

Performance measures can be divided into different types. A distinction can be made between financial and non-financial measures. A financial measure is expressed in a monetary value and a non-financial measure is a quantitative measure of performance, which determines the level of a certain quality aspect, for instance. An alternative way to differentiate between measures is to pay attention to measures that relate to results, and measures that relate to the determinants of these results (Neely et al., 2000). Result measures which are based on past performance are called lagging indicators and measures which are based on the determinants of results are leading indicators (Brignall et al., 1991).

(9)

9

stakeholders. Process measures focus on the process itself. An example of such a measure is process duration (Propper & Wilson, 2003). This type of measure can be used as benchmarking tool or to improve process quality. Output measures give direct results of the process, such as the number of products and services created. Outcome measures describe the consequence of output. It answers the question whether customers are satisfied with a product or service (Neely et al., 2000).

2.1.2 Functions and advantages of a PMS

A PMS has different functions and can offer several benefits to an organization. Hansen & Van der Stede (2004) mention four different functions of performance measurements: (1) operational planning, (2) performance evaluation, (3) communication of goals, and (4) strategy formation. Speklé & Verbeeten (2009) describe three functions of performance measurements: (1) operational function by monitoring and planning, (2) reward function by providing incentives and rewards, and an (3) exploratory function for double loop learning, priority setting, and policy development.

The literature describes the three main advantages of a PMS:

A PMS brings transparency

PMSs play a vital role in the flow of information between departments because strategies, goals and directives are communicated to lower organizational levels (Nudurupati et al., 2011). This flow of information gives employees insight into products, costs, and activities that contribute to the output of the organization (de Bruijn, 2002). These insights might entail a discussion between employees, which can help to improve performance.

A PMS can increase productivity

(10)

10

A PMS can create more accountability

A PMS can create more accountability, because employees with a lot of autonomy can show performance results to stakeholders, since information about performance is quantified and can therefore be communicated (de Bruijn, 2002). In the same way, it can function as communication device to make results more transparent.

2.1.3 The relation between performance measurement and control

Organizations and the study of control are closely interrelated, and organizations cannot exist without some form of control (Otley & Berry, 1980). Ouchi (1977) defines control as an evolutionary process which is based on monitoring and evaluation of behavior or outputs. A PMS can be used as a management tool to achieve organizational control. If it is not possible to rationally evaluate the work of the organization, the use of behavior or output control is not an appropriate way to control the organization (Ouchi, 1979). In that case, clan control can be used as an alternative. This informal way of control can be used when organizational members share values, beliefs, and cultural norms (Ouchi, 1979).

Output and behavioral control are extreme opposites of control and organizations may choose a strategy that falls within these extremes. Current research pays more attention to management control systems (MCS) which is a broad term that encompasses management accounting systems and other ways of control such as clan controls (Chenhall, 2003). An individual control system can be a PMS that helps to align individual’s activities with organizational goals (Malmi & Brown, 2008). Malmi & Brown (2008) point out that controls should not be defined holistically as a single system, but instead as a package of different control systems. For example, clan control can be part of a MCS and other controls can be used to measure performance.

2.2 The relation between performance measurement and NPM

(11)

11

2.2.1 New Public Management

New Public Management (NPM) is a research stream formally conceptualized by Hood (1991). Hood (1991) describes seven doctrines that prescribe how the public sector should be organized and controlled according to NPM. NPM can be viewed as theory or as a set of concepts and practices. The ideas of NPM are spread around many different countries, and are seen as a route to modernization (Pollitt & Dan, 2011).

Batley and Larbi (2004) pointed out that there are two main perspectives of NPM. First of all, NPM can be seen as a ‘good managerial approach,’ which means that the approach is result-oriented and pays attention to efficiency (Holmes and Shand, 1995). Managerial improvement will be reached through decentralization, disaggregation and downsizing (Mongkol, 2011). NPM has also a strong focus on performance measurement and auditing (Minogue, 2001). The second perspective pays attention to market forces and competition by using management techniques from the private sector (Monkol, 2011). This perspective describes how the public sector can increase its efficiency by using market-like mechanisms to deliver public services, by means of for instance privatization and contracting (Mongkol, 2011).

(12)

12

2.2.2 New Public Governance

The ideas of NPM stem from the 1980s and since then a lot has been written about the assumptions of this concept. According to Osborne (2006), NPM is outdated nowadays and a current ‘paradigm shift’ is taking place in the literature, namely the replacement of NPM by NPG. NPG has a large focus on the complex nature of public services and can be seen as a response to NPM-thinking (Osborne, 2010). According to Klijn (2012), NPM focuses on the adoption of private sector techniques into the public sector, while NPG primarily focuses on public sector values. Furthermore, NPG pays attention to the collaborative outcomes of a network of public organizations, while NPM focuses on the outputs of the individual organization (Klijn, 2012). Collaborative outcomes are relevant goals for public organizations, because public goals are often fulfilled by a network of organizations. Another difference between NPM and NPG is the way how performance information is used. Since NPG pays attention to a public organization and its environment, performance information is mainly meant to create a discussion among the network partners to enhance business processes (Almquist et al., 2012). Therefore, performance information is not used in a ‘command and control’ manner, but to increase the performance of the network by having debates with network partners (Almquist et al., 2012). This view could be interesting for public sector organizations, however the concept of NPG is relatively new, and it is expected that researchers will further examine these assumptions.

2.3 Risks of performance measurement systems

This paragraph focuses on the negative effects of a PMS. First, the features that make the public sector differ from the private sector will be discussed. The next subparagraph will pay attention to the risks of PSPM. Finally, strategies will be offered that can be used to limit these risks and to maximize the effectiveness of the PMS.

2.3.1 Public sector characteristics

In this section the special characteristics of the public sector that make it more difficult to measure performance will be discussed.

(13)

13

1. Source of ownership

The first fundamental difference between the public sector and its private counterpart is the source of ownership (Rainey et al., 1976). Public organizations are largely funded by taxation and controlled by political forces, while their private counterparts are funded and controlled by shareholders, investors or managers (Boyne, 2002). A government operates in a public network of interdependent organizations that collaborate to achieve public goals (Boyne, 2002). Therefore, a public organization faces different types of stakeholders such as consumers, producers, tax payers, and service providers who are likely to conflict with each other, creating a larger demand for accountability (Boyne, 2002). Stakeholders should find an agreement on what kind of performance is relevant (Otley, 1999).

2. Public instability

Public organizations are working as open systems that face a lot of pressure from stakeholders. Political disturbances in higher government layers can put pressure on public services to achieve short-term results or to limit a budget (Boyne, 2002). Public managers have to take these pressures into account when they create policies.

3. Lack of competition

Public organizations work collaboratively to achieve public goals. This means that not every public sector organization faces competitive pressures (Boyne, 2002). This absence of competition often results in lower efficiency compared to the private sector.

4. Public sector goal creation difficulties

While private agencies focus on profits and efficiency, public agencies pay attention to the fulfillment of social goals such as equity and accountability (Propper & Wilson, 2003). These goals are more difficult to fulfill, because the public sector serves a large amount of stakeholders. Goals are also more vague in the public sector, because policies have to be approved and supported by politicians.

5. Incentives and rewards

(14)

14

2.3.2 The risks of measuring performance in the public sector

This subparagraph will discuss the risks of PM which can arise when PM is used in the public sector.

1. Risks of performance indicators

One of the risks of PM is that performance indicators can lose their value over time or do not measure what they intend to measure (Thiel & Leeuw, 2002). This problem arises because not everything can be quantified in a way which is desired and goals may change over time. Especially in public organization, goals may change over time, because these goals have to be approved by different councils that are participating in the political system. The public sector also faces quantification difficulties due to the nature of public services. This makes it more difficult to define precise indicators of performance (Burgess & Ratto, 2003). Thus, the design process of a PMS in the public sector can be time-consuming. However, a PMS has to be reviewed periodically to keep measures relevant. Otherwise it is a costly device that generates no value to the organization (Neely, 2000).

2. PM may create a limited scope of performance

Managers use output of a PMS for several purposes, such as rewarding, planning and evaluation. However, a PMS does not always give a fair view of performance, because not everything can be measured without distortion. If managers are not aware of this risk and they only pay attention to the output of a PMS to reward, plan or evaluate, they have a limited scope of performance which could lead to a wrong interpretation of results (de Bruijn, 2002).

3. PM can increase bureaucratic practices

When a PMS is implemented, organizations often adopt procedures to make processes measurable. For example, if an organization is interested in the throughput time of a process, employees should register when a process begins and management needs to create a procedure for this task. As a result, tasks become more target-oriented and managers start to focus on the optimization of production processes (Balk & Bouckaert, 1991). This could lead to a more bureaucratic organization with a loss of focus on innovation (de Bruijn, 2002). This could decrease employee motivation, since the tasks of employees become more standardized and they feel more pressure from top management.

(15)

15

4. PM can increase competition and reduce public responsibility

De Bruijn (2002) mentions the aspect of system responsibility which means that public sector organizations have to report their professional findings to other public organizations. PM can make public organizations more competitive in terms of performance, making them less willing to share their best practices (De Bruijn, 2002). This behavior is inconsistent with the collaborative and social character of the public sector.

5. PM can increase a focus on short-term results

Unstable political environments can increase pressure on public organizations to achieve quick results in the short run (Boyne, 2002). Especially when a political system is in crisis, politicians can put pressure on lower governments to show short-term results. This creates the risk that a focus on long term performance will be lost, which could give an incomplete view of performance (Balk & Bouckaert, 1991).

2.3.3 Strategies to reduce the risks of performance measurement

This paragraph will offer strategies which can be used to mitigate the risks of PSPM. These strategies are subdivided in factors of PMS design, PMS implementation and PMS use. It is proposed that if these strategies are followed, the potential risks will be limited, and subsequently the PMS will become more effective. This subparagraph will discuss the different strategies and explain how a strategy can reduce the potential risks of PSPM.

2.3.3.1 PMS design

(16)

16

The use of various measures makes the PMS a more reliable and useful management tool. It gives managers the opportunity to draw valid conclusions and reduces the risk that a limited scope of performance arises. Another advantage of adding different measures to the PMS is that it makes it possible to measure short-term and long-term performance, so that a view on long-term results will not be lost. This leads to the following proposition:

Proposition 1: Using a limited number of performance indicators reduces the effectiveness of a PMS.

When we speak of measurement variety, a distinction can be made between output and outcome measures. Output is a direct result of the organization, such as the number of patients that visited a doctor. Outcomes are the consequences of output; for instance the increase in health which could be enabled by visiting a doctor. In general, public managers are familiar with financial measures, but they find dealing with the concept of outcome more difficult. This is because outcome measures are influenced by a lot of different factors. For instance, there is no causal relationship between the number of doctor visits and the influence on health. This is because an improvement in health can be influenced by a lot of different factors such as a healthy lifestyle. If public managers are struggling with these measures, problems can arise when the PMS is designed (Arnaboldi & Azzone, 2010). This is because if public managers do not know the risks of outcome measures, they might interpret these measures in an incorrect way.

(17)

17

However, measuring and interpreting outcomes can be useful for municipalities to get an impression of the current quality of a neighborhood, but an outcome result should not be used to judge a specific policy. Thus, if managers of public organizations are aware of the risks of measuring and interpreting outcomes, the risk of gaining a limited view of performance will be prevented. This leads to the following proposition:

Proposition 2: If public managers are not aware of the risks of outcome measures, it may limit the effectiveness of a PMS.

2.3.3.2. PMS implementation

Before a PMS can be used, it needs to be implemented. De Bruijn and van Helden (2006) mention that if steering intentions of a PMS are stronger, more risks will arise which can make the PMS less effective. This can be explained by the fact that if an organizational change becomes larger, it can lead to more dissatisfaction and disruption among stakeholders. Therefore, agreements should be made between managers and professionals about the purposes of the measures before the PMS is implemented (de Bruijn, 2002). Making the purpose of a PMS explicit to stakeholders can prevent distrust and make it easier to find explanations for poor indicators of performance. Subsequently, cooperation between managers and professionals may stimulate organizational learning. As mentioned before, the public sector has to cope with a lot of different stakeholders that can conflict with each other, which makes stakeholder commitment even more important. Furthermore, stakeholders should be involved in the implementation process. Otherwise they may lose motivation, since they do not understand why a PMS will be used.

Stakeholder involvement can increase effectiveness of a PMS, because it can make the purposes of a measure more clear and it ensures that PMS designers view measures from different perspectives (de Bruijn & van Helden, 2006; van Thiel, 2009). Stakeholder involvement makes public organizations also more willing to share their best practices, which limits the risk of increased competition between public organizations.

(18)

18

2.3.3.3 PMS use

When a PMS is used, public managers can review results periodically and use them for different purposes. NPM focuses on a result-oriented control structure that gives employees clear assignments and responsibilities (Hood, 1991). Performance measurement systems which are created with the principles of NPM have specific goals or targets and a great emphasis on output measurement (Hood, 1991). Speklé & Verbeeten (2009) primarily distinguish between two types of PMSs. The first is a NPM-type PMS which focuses on monitoring and rewarding. The other type focuses on a more exploratory use of PM, which means that the PMS should be used for: double loop learning, strategy formation or experimentation and innovation (Speklé & Verbeeten, 2009). This second type of PMS has similarities with the principles of NPG, because both NPG and the exploratory PMS emphasize the importance to cooperate and discuss.

Speklé & Verbeeten (2009) argue that the way public sector organizations use their PMSs depends on the contractibility of performance. Contractibility of performance consists of goal clarity, outcome measurability, and knowledge of the transformation process. Speklé & Verbeeten (2009) found that an exploratory use of a PMS increases organizational effectiveness in public sector organizations. However, this increased effectiveness is unrelated to the level of contractibility. Furthermore, they found that the use of a PMS for incentive purposes reduces organizational effectiveness. Furthermore, the social community teams are only in the development stage, which increases the need for learning and experimentation.

Using an exploratory PMS will limit the risk that bureaucratic practices arise, because employees will not be judged based on PMS results. Furthermore, it helps to emphasize the importance of long-term results, because the PMS results are used for evaluation and development. Therefore, municipalities should use a PMS in an exploratory way in order to make effective use of it. This leads to the following proposition:

(19)

19

The conceptual model is presented in figure 1. It is proposed that if municipalities design, implement and use a PMS in line with the propositions, the effectiveness of the PMS will be maximized.

Figure 1. Conceptual model

The constructs which are presented in figure 1 have been translated into a questionnaire which is used to gather data from municipalities (appendix B). The next section is the methodology which enlightens how the data have been collected and analyzed.

3. Methodology

This chapter explains how the research has been conducted. It will pay attention to the research methods and the way data have been collected and analyzed. Furthermore, a case study background is provided which is meant to make the interpretation of the results easier.

3.1 Research method

(20)

20

currently experimenting with social community teams and the goal of this paper is to gain more understanding of the complexities of PM that municipalities are facing. The paper is also meant to discover relations that can be interesting for the public sector in general, because there are a lot of authors who wrote about the risks of PSPM. The research method which is used is a multiple case study, because data are collected in an inductive way to be able to give an extensive description of PM of social community teams (van Aken et al., 2012). Furthermore, Eisenhardt (1989) argues that case studies are particularly useful in new research areas. Since the social community teams are a relatively new concept, selecting the case study as research method seems to be an obvious choice.

3.2 Case study background

This section gives a plain description of the case study background. The Dutch government has three governmental layers: central government, provinces and municipalities. Every governmental layer has its own responsibilities and the relation between the layers is hierarchical. Municipalities have to carry out the tasks which are imposed by the central government and the province. However, they have sufficient freedom to fulfill these tasks and to create their own policies. The current decentralization gives the municipalities more responsibilities in the social domain. Starting from the first of January 2015, municipalities will have more responsibilities in the field of youth care, social employment and care for elderly. The municipalities receive a limited budget from the government to carry out these tasks. It is expected that municipalities will perform the tasks in a different way. The main idea is that municipalities will cooperate with social organizations and citizens to bring care closer to the citizens, but that municipalities retain control of the care services. To retain control, municipalities have to monitor the assisting organizations to make results visible. Municipalities have different roles when controlling the assisting organizations. Some of them chose for a collaborative way to deal with their care partners, while others decided to keep control in their own hands.

(21)

21

in their neighborhoods. In general, a social community team is a multidisciplinary collaboration of professionals and assistants with the aim to provide assistance to individuals, families or households in an integrated manner (Sommer & Pool, 2013). In such a team, social workers from various health organizations work together in order to make these teams function as an access point for different types of care. The intention of this way of working is to discover citizens with problems in an early stage and to provide an adequate solution. When care problems are found earlier, additional care can be prevented and healthcare costs can be saved. In this way, the social teams can reduce the total healthcare costs and still provide the care which the citizens require.

Sommer & Pool (2013) discern between several roles of social community teams: - Increasing the self-sufficiency of citizens;

- The connection of formal and informal forms of care; - The creation of a single integrated access point;

- The provision of assistance and services closer to citizens in their own neighborhood or district so that the discretionary power of citizens and their social network is deployed;

- A more efficient plan for multi-problem households, i.e. one household, one plan, one rescuer;

- A clear link between care, education, work, income, housing and welfare; - The pursuit of cost control (better care quality at lower cost).

Some municipalities decided to focus mainly on some of those roles, for example on multi-problems households. The expectations of social community teams are high. However, every municipality faces a limited budget and should therefore make the care more efficient.

3.3 Case description

(22)

22

(1998) argues that municipalities with a population smaller than 15,000 do in general not use advanced systems of planning and control, but they choose for a more informal way of control. It generally takes longer for these smaller municipalities before a PMS for the social teams will be implemented, because there is less pressure on the measurement of performances. These municipalities may for example consider monitoring the social teams only in informal ways in the beginning. Therefore, municipalities with less than 15,000 inhabitants were not selected for interviews.

Interviews were held with a mix of medium and large municipalities to find some evidence about the size and the way performance is measured. Municipalities are allowed to determine the way in which social teams are controlled by themselves. This could result in a broad variety in the ways performance is measured. Furthermore, a variety in the development of the social teams exists between municipalities. Some municipalities are already using the PMS, while others are still in the design phase. These variations in data tell more about how municipalities deal with performance measurement issues, giving the opportunity to discover more different relations in the available cases.

Table 1. Case description

3.4 Data collection

The data have been collected in cooperation with BMC Advies Amersfoort. BMC is a consulting firm in the public sector, which is interested in how municipalities are dealing with

Municipality Interviewer Number of residents

Borger-Odoorn Tessa Bergsma 25.632

Emmen Tessa Bergsma 107.943

Friese Meren Wietse Leistra 51.229

Groningen Wietse Leistra 197.823

Hoogeveen Tessa Bergsma 54.733

Ommen/Hardenberg Wietse Leistra O: 17.333 / H: 59.689 Ooststellingwerf Wietse Leistra 25.696

Steenwijkerland Wietse Leistra 43.338 Sudwest Fryslan Wietse Leistra 84.356

Tynaarlo Tessa Bergsma 32.543

(23)

23

performance measurement issues. BMC gave the opportunity to conduct a pilot interview with one of their senior consultants to test the interview protocol which was prepared in advance (appendix B).1 More than one researcher collected evidence. Therefore, a case study protocol is developed to ensure that all the relevant constructs were discussed during the interviews. Furthermore, the protocol functioned as guide during the interviews.

When the pilot interview was completed, the case study protocol was given a final check. Then, municipalities were contacted by telephone and email to find respondents who wanted to participate. The data have been collected in November 2014. In this period, appointments with managers and policy advisors of municipalities were planned to conduct the interviews. The collected data consist of primary data from semi-structured interviews and secondary data from analyzing documents. The respondents also provided policy documents about monitoring to support their statements. The interviews are semi-structured, because the goal of theory development is to gain a deeper understanding of different phenomena. In this way, it was possible to discover underlying thoughts about PM of the social community teams. The interviews also gave the possibility to learn more about the type of social team the municipalities were using. Before the interviews were conducted, documents about social teams were analyzed to get more familiar with how the social teams work in practice and how municipalities perform their control function. In this way, the analysis of secondary data worked in a complementary way to support the collected data from the interviews. The interviews lasted about 45 minutes. Every interview is recorded and transcribed. The transcriptions were sent back to the respondents with the question to check whether they agreed with the content.

When the interviews were conducted, most of the municipalities were at the beginning of the development of their PMS. Therefore, not all municipalities were able to explain whether the PMS for social community teams worked effectively. In some cases, municipalities did not decide what performance information was going to be measured yet. Therefore, respondents had to answer questions which were based on expectations. This could have harmed the empirical results, because if this research had been conducted several months later the answers could have been different.

(24)

24

3.5 Data analysis

This subsection explains how the data are analyzed. A case study database is created to structure the data. The data have been analyzed using a Format Matrix. With help of this matrix, relevant evidence for each proposition has been categorized, examined and judged. Furthermore, this matrix gives a clear description and overview of the data. The format matrix consists of two parts. The first table gives an overview of the results and the second table contains an overview of the relations between the different propositions and PMS effectiveness. Because the interviews were in Dutch, the results are also presented in this language to keep the answers valid and recognizable.

After the Format Matrix was completed, all possible relations and patterns were analyzed in an unbiased and objective manner (Rowley, 2002). Also citations were selected from this matrix to support the empirical findings. Because the research questions are adequately based on propositions and a clear overview of the data has been created, the research is supported by a Chain of Evidence (Yin, 2014). According to Yin (2014), the cases of a multiple-case study should be selected carefully, either that it generates similar results or predicts contrasting results. The empirical results of each municipality are therefore taken together to draw overall conclusions.

3.6 Controllability, reliability and validity

This subparagraph explains what has been done to conduct a valid research. Controllability, reliability and validity are necessary conditions to do valid research and to reach inter-subjective agreement (van Aken et al., 2012). Inter-inter-subjective agreement is the consensus between actors who deal with a research problem (van Aken et al., 2012).Controllability is a condition that needs to be fulfilled before validity and reliability can be evaluated (van Aken et al., 2012). Controllability means that the researcher has to show how he or she has performed the research (van Aken et al., 2012). Therefore, a detailed description of the way how the research is carried out is given in the previous paragraphs.

(25)

25

should generate similar results (van Aken et al., 2012). Reliability will be realized if potential biases are controlled. Insights from other researchers are used and the data are analyzed in a structured way, which adds to the reliability of the paper. This also makes the paper more standardized, which makes the research replicable for other researchers (van Aken et al., 2012). The data have been collected by two researchers and a detailed case study protocol guaranteed that the interviews were replicable. Instruments bias is controlled by using Triangulation, which means that interviews are conducted to obtain primary data, but also secondary data are analyzed (Yin, 2014). Respondent’s bias is controlled by interviewing multiple respondents. Finally, reliability of circumstances is limited by conducting interviews with members of different municipalities to experience a multitude of circumstances (van Aken et al., 2012).

To make the paper valid, the way results are generated should provide good reasons that research results are adequate (van Aken et al., 2012). Construct validity is reached when operational measures of the concepts are established correctly (Yin, 2014). In this research, attention has been paid to operationalize the concepts by using an interview protocol and conducting a pilot interview. In this way, the researchers were able to put the theoretical constructs into practice by asking valid questions during the actual interviews. External validity refers to generalizability of the case study (Yin, 2014). This research is not externally valid to all organizations in the public sector, because it focuses on PM of social community teams, which is developed by municipalities. However, other Dutch municipalities can use results of this paper to gain a deeper of the risks of PM of social teams. Furthermore, the goal of this case study is not to be generalizable to every public organization, but to discover relations and to gain a deeper understanding of this specific new phenomenon (van Aken et al., 2012). Besides, the generalizability of a thesis is not large in general, because only a limited number of cases are studied due to time limitations. This research could however be a good start for further research in this field.

4. Results

(26)

26

4.1 Results of propositions

This section discusses the empirical results of the interviews. A summary of the results can be found in table 2. This table represents an assessment of every construct which is shown in the conceptual model. The propositions are ranked from -- to ++, where -- means there is no fit between the empirical result and the proposition, and ++ means that the empirical finding is perfectly in line with the proposition.

Regrettably, nearly every municipality which has been interviewed was not able to say something about PMS effectiveness yet, because most of the municipalities were still working on the design of the PMS and did not obtain any results yet. Only one municipality already obtained some PM results, but these results were obtained with a very basic measurement tool. In general, the municipalities use the year 2015 as a transition year to complete the design and implementation of the PMS. This is because municipalities are especially working on the development of the social community teams themselves until the first of January 2015. The development of the PMS is a concern for thereafter. The methodology section already described that the selection of very small municipalities has been avoided, because it was expected that these municipalities did not have a working PMS yet. However, in practice it seems that the municipalities in the northern part of the Netherlands are still working on the development of their PMS as well.

Table 2. Results of propositions

(27)

27

4.1.1 PMS design

According to table 2, the empirical findings were in general in line with the first proposition. This is because every municipality uses different indicators to measure performance. However, some municipalities use more different measures than others. These differences between municipalities will be discussed in this section. A summary of the output and outcome indicators which are used to measure performance is given in table 3. This table gives a description of the variety in measures as well.

Table 3. Measurement variety

No. Output measures used

What kind of output is measured

Outcome measures used

What kind of outcome is measured

1. Yes - Operating costs

- Social costs and benefits

Yes - Self-sufficiency - Client satisfaction - Social costs and benefits

2. Yes - Operating costs Yes - Self-sufficiency

- Client satisfaction - Case analysis

3. Yes - Operating costs - Throughput

Yes - Client satisfaction

4. Yes - Operating costs

Yes - Self-sufficiency - Client satisfaction - Social costs and benefits

- Client satisfaction - Case analysis

6. Yes - Operating costs - Scenario analysis

7. Yes - Operating costs - Throughput

- Client satisfaction

- Client satisfaction - Employee satisfaction

10. Yes - Operating costs Yes - Self-sufficiency

- Client satisfaction - Baseline measure

11. Yes - Operating costs

(28)

28

Every municipality measures or intends to measure on costs and quality, because the city council is especially interested in these measures. In addition to these compulsory measures, some municipalities use the sufficiency matrix. This matrix gives insight into the self-sufficiency of citizens. This matrix is able to show the level of self-self-sufficiency for every citizen. When these matrices are taken together, it gives a picture of the self-sufficiency of a certain area which can be interpreted as an outcome result. However, municipalities are in general still struggling with this measurement tool, since it can be difficult to prove that a citizen has become more self-sufficient. This tool should not be used as a mechanism to judge social workers or care partners, because it is difficult to show that a possible increase of the self-sufficiency of citizens has been caused by the actions of the social team. Despite this fact, the matrix can often be easily subtracted from the registration system and it can tell municipalities something about the self-sufficiency of a neighborhood if data are registered in the right way.

"It is a tool to put the client in their own strength and not to judge the actions of your care partner.” (Project manager, municipality 10)

Another PM tool which is used in public sector organizations is the social costs and benefits analysis (SCBA). This tool tries to calculate the social costs and benefits of a policy. Some of the municipalities also use this instrument for the PM of social community teams. This measurement tool can also provide outcome measures. However, the SCBA makes forecasts which simplify practice. Therefore, these forecasts are not always reliable and it is often the case that in reality the outcome is different.

“The SCBA is an attempt to depict a complex practice in an easy manner. And you never know if it is right what it says. That is an assumption." (Policy advisor, municipality 1)

(29)

29

However, the problem is that municipalities are getting more care functions which were not their responsibility beforehand. Therefore, it is recommended to perform a baseline measure when the social teams are one year in use. Then, this measure can be compared with future results.

"The first year the monitoring is designed to gain insight into the number of redirections and client requests for different products. So we actually want to make a kind of baseline during this year." (Project manager, municipality 9)

A lot of municipalities stated that the story of citizens is an important aspect to focus on, because it gives the opportunity to learn more from client experiences than by simply providing a questionnaire. The story behind the numbers can be observed by doing a case analysis. Then, experiences of citizens are specified on paper. However, this method is very time consuming. In conclusion, the municipalities use qualitative and quantitative tools to measure performance, and the city council is especially interested in financial results while the public managers have more interest in the qualitative aspects such as experiences of citizens.

The second proposition is about output and outcome measures. In general, municipalities find outcomes more difficult to measure than outcome, but outcomes are the measures they are especially interested in. The reason for this is that outcomes can better describe the impacts on the society. Municipalities exist to deliver services to the society, thus if they are able to measure outcomes, they are able to get more insight in their social performance. Social community teams should not be judged on the basis of outcome results. Outcome results can however be useful to see which areas have more vulnerable households. For example, such insights can be used to adjust the composition of a social team in a specific area. As presented in table 2, every municipality received +/- or higher, because every respondent was at least able to give a relevant explanation of outcome measures.

(30)

30

If a municipality wants to know if the functioning of a social team improves social welfare in an area, a detailed inquiry of individual cases will be necessary to discover the social effect of a social team. A valid way to do so is to question every client that visits the team whether his or her well-being has been improved. When the results of every individual case are added together it could give a fair view of social performance.

"On things like self-sufficiency, you can attach all kinds of indicators to measure well in my opinion, but you will have to do that mainly by asking. And by adding all those results per client together, you can have the whole picture.” (Policy advisor, municipality 5)

However, a disadvantage of this method is that it is very time-consuming to question all the clients that visit a social team. Social workers are not responsible for this task, because their main task is to provide care and support. A researcher could be appointed to perform such kind of measurements, but most of the municipalities do not have sufficient budgets to make that possible. Municipalities should therefore try to discover which things can be measured easily to get an idea of outcome results. Care partners are cooperating with the municipality to develop the social teams. Municipalities may therefore consider developing an outcome measure in collaboration with these organizations, because they have their own databases and measurement methods already.

“With all the livability studies available at the different organizations, there is just a lot of information together.” (Policy advisor, municipality 2)

In conclusion, outcome measures can be helpful for municipalities to learn about the social welfare of a specific area. Municipalities should consider a way in which they are able to get an impression of outcomes which can be interpreted in a valid manner.

4.1.2 PMS implementation

(31)

31

Table 4. Stakeholders which are involved in PMS development

No. Stakeholders 1. - Municipal council

- Employees of social team

2. - Municipal council - Care partners

3. - No stakeholders which are involved yet

4. - Citizens - Care partners - Municipal council

- Employees of social team

5. - Citizens - Care partners - Municipal council

6. - Care partners

- Managers of social team - Municipal council

7. - Care partners - Citizens

8. - Care partners - Citizens

- Managers of social team

9. - Municipal council - Citizens

- Managers of social team

10. - Municipal council - Citizens

- Employees of social team - Care partners

11 - Municipal council

(32)

32

Most municipalities involve citizens indirectly by discussing performance indicators with the board of citizens or at meetings where citizens are invited. In addition, the municipalities have consultation with care partners which are involved in the development of the social team. This consultation generally takes place during meetings or sessions. Furthermore, Social workers are often involved in the creation of performance indicators. Some municipalities chose to involve only team managers and others decided to involve every team member. Involving team members can have several advantages. First of all, it can make team members more aware of the importance to register in a valid manner. Registration in a valid manner is important, because a lot of information from the registration system is used to measure performance. Furthermore, involving team members limits the chance that bureaucratic practices arise.

“We want to ensure that we are not going to create an entire bureaucratic organization, so we want to be very critical in what we ask from the social workers what they need to register." (Project manager, municipality 8)

The team members also have a different perspective on performance indicators than policy makers, since they are working on the process itself. It would therefore be possible that they have relevant ideas about what kind of performance should be measured.

"The social workers help us to shape the system, so the system is filled from the workplace." (Project manager, municipality 10)

Involving social workers can also make them more aware of the costs. In general, the respondents mentioned that the social workers have no idea of the costs of a particular treatment. The creation of social teams also comes with a budget reduction and when social workers are more aware of the costs of different treatments, they might be better in selecting a less expensive appropriate solution for the problems of a certain citizen.

(33)

33

Involving every stakeholder has also a risk. When every stakeholder brings in performance indicators that should be measured, the PMS may become too large, i.e. there are too many indicators that should be measured. Then, the registration becomes a very time-consuming activity. Policy advisors should therefore consider which management information is absolutely necessary in order to be accountable to the city council, because that is main reason why performance is measured in the first place.

"What I want is a little prevention that the social workers are spending hours per week on registering all indicators. So that's a bit of a tradeoff.” (Project manager, municipality 4)

In conclusion, stakeholder involvement is important, because it can increase motivation to register, limit bureaucratic practices, and create a more complete PMS. Policy makers should decide which stakeholders are involved and ensure that only relevant indicators are measured, because otherwise the social teams will face unnecessary registration pressure.

"The challenge is, I think, to develop a monitoring system that meets the demands of the city council, but also meets the wishes of the social workers so they do not have to register everything we wish." (Project manager, municipality 8)

4.1.3 PMS use

It was proposed that municipalities design a PMS for exploratory use, because the social teams are still in the development stage. Most of the municipalities do not have any monitoring results yet, because some municipalities are still in the development phase of the PMS. Although, most municipalities already thought about the interpretation of the monitoring results and mainly conclude that the results will be used in exploratory way. The goal of this approach is to develop a well-functioning social team in cooperation with care partners and team members.

“The mission of the members of the team until now was: develop a methodology. In the future the focus lies more on evaluating than rewarding.” (Project manager, municipality 1)

(34)

34

on performance information. Furthermore, when a social team uses more budget than the other teams, it does not have to mean that a social team is malfunctioning, because it might be that the citizens the team assisted required expensive support. Most municipalities also have team meetings where team assessment can be realized. The team members also monitor each other, because there is a strong emphasis on cooperation within the social teams.

"Assessment is not bad. That is, in itself fine, but that does not necessarily have to come from those monitoring results." (Project manager, municipality 8)

The provision of financial rewards is very uncommon in the public sector. However, it does not mean that the use of incentives cannot improve motivation of team members. Municipalities will focus on rewards that motivate the whole team. A possibility to reward is to give every social team their own budget. If there is remaining budget at the end of a period, social teams can decide how they will spend it. This may stimulate social workers to pay attention to the costs of the care they prescribe. However, the problem with this system is that not every neighborhood has the same care demands. Therefore, before such a system is implemented, a thorough scan of the care demand of each neighborhood should be made and compared to determine appropriate budgets. It would therefore be better to implement such a system after a period of experimentation. A suggestion to reward the social teams in the current situation is to give them a training which can improve the functioning of the workers. Such actions contribute to the team spirit of the workers and might improve the performance of the whole team.

"We celebrated a milestone with the aldermen, the social teams and other partners. We gave them a box of chocolates as a token of appreciation. The words are very important.” (Project manager, municipality 10)

(35)

35

4.2 Relations

This subparagraph pays attention to the relations between the different stages of PMS development and the effectiveness of a PMS.

4.2.1 Relation: PMS design – PMS effectiveness

Two propositions are developed which discuss the relationship between PMS design and PMS effectiveness. These are measurement variety and a distinction between output and outcome measures. Measuring qualitative aspects will be very helpful for municipalities in learning about outcome. Listening to stories of citizens can provide proper insights, but is very time consuming. Alternatives are questionnaires or meetings with citizens where the quality of the service is questioned. In summary, public organizations should find a balance between measures which are easy to obtain and measures which are useful. A variety of qualitative and quantitative measures will be necessary to get a complete picture of relevant output and outcomes.

"Monitoring is not only a kind of numerical system, but also storytelling, customer panels, those who have eyes and ears open in those areas." (Policy advisor, municipality 5)

Outcome results should be used to get a view of social welfare in a neighborhood, but these outcomes should not be assessed on goals which were formulated in advance, since there are many things that influence outcome results.

"Outcome is indeed the most important, and in our view you can do that by saying the unemployment decreased, apart from what you formulated in the beginning.” (Policy advisor, municipality 5)

(36)

36

4.2.2 Relation: PMS implementation – PMS effectiveness

Stakeholder involvement creates transparency and cooperation. However, a disadvantage of involving stakeholders is that there are more opinions which have to be taken into account. This can create long-lasting discussions and disagreement between stakeholders. Besides, municipalities have to retain control and involving too many stakeholders may lead to situations where that becomes too difficult. Municipalities should evaluate whether it has additional value to the PMS to involve a certain stakeholder. If every relevant stakeholder is involved in the right way, it will make the PMS more effective.

4.2.3 Relation: PMS use – PMS effectiveness

The social community teams experience a period of development, because it is a new way of working. Municipalities have to give social teams the opportunity to get used to this new way of working and monitoring results should be used for evaluation and development instead of assessment. This is especially the case for outcome results, because these will become more valuable when it is possible to compare results of different years. For example, if the quality of life of a specific neighborhood improves each year for a period of ten years, it is more reliable to draw conclusions than when the result of only one year is interpreted.

“What I find very difficult is that you measure long-term effects. You cannot measure in six months if your project is a success or not, but that is what politics and aldermen want." (Project manager, municipality 1)

When the social teams are in use for a while, municipalities may think of an incentive that can be used to stimulate social teams to improve their way of working. However, these incentives should not be based on production, because a high production does not mean that the social team provided the right amount of care.

(37)

37

An assessment of a team based on monitoring numbers is exceptional, because, for instance, there can be various reasons why a team uses a higher budget than other teams. The results are often reason for conversation instead of a direct assessment. For example, when certain costs are higher than expected, it gives the municipality a reason to discuss with the team manager how this could have happened. In the current stage it seems that using the PMS in an exploratory way adds to the effectiveness of a PMS. After a period of experimenting, municipalities may think of ways to reward and assess the social teams. However, monitoring results are especially meant to give a general overview of performance at macro level in order to fulfill the accountability obligations which are imposed by the council.

"Doing an assessment which is based on data that comes from the analysis without knowing the story behind it is really lethal." (Project manager, municipality 11)

5. Discussion

This section discusses the main findings of this research in the context of academic literature.

(38)

38

The second proposition discusses the risks of outcome measures. It was proposed that not being aware of the limitations of outcome measures could limit the effectiveness of a PMS. In line with the expectations, it seems that public managers are in general more interested in outcome than output. Furthermore, it seems that municipalities are aware of the limitations of outcome measures. The municipalities that were less aware of these risks mentioned that it was a responsibility of a colleague or a research agency which was specialized in the measurement of outcomes. Van Thiel (2009) and de Bruijn (2002) argue that a PMS should only measure the things which are actually realized by the organization. This view is limited for PM of social community teams, because a social community team is a public service which is developed to serve a large number of people. Therefore, measuring on output will not give a complete view of performance. Measures such as the self-sufficiency matrix are especially developed to give a view of outcome results, and the coming years are there to develop more of these measures. These measures are useful as long as municipalities realize that outcomes are achieved by a network of organizations and give an overview at macro level.

The third proposition is about stakeholder involvement. It was proposed that stakeholder involvement is important, because it limits the risk of distrust and the risk of increased competition between public organizations. Furthermore, stakeholder involvement may provide several insights from different perspectives to create a better functioning PMS (de Bruijn & van Helden, 2006). In practice it seems that the involvement of social workers has several advantages, such as: cost awareness, the generation of new PMS input, and creating support for PM. Public organizations have a focus on the fulfillment of social goals such as equity and accountability (Propper & Wilson, 2003). For social teams, it can be concluded that a PMS should be developed which meets the requirements of the council, but also takes into account the registration pressure which is faced by social workers. In this way, stakeholder satisfaction is observed as an important aspect, but involving too many stakeholders may lead to an overflowing PMS. Therefore, municipalities should carefully consider which stakeholders should be involved.