MEASURING SOCIAL IMPACT

(1)

R. D. Rinsma MA

MEASURING SOCIAL IMPACT

(2)

1

Measuring Social Impact

A study of policy measurement at two Dutch local governments guided by organizational theory

Robin Doris Rinsma MA

S1690574

Rivierenhof 106

9725HC Groningen

+31636319218

robinrinsma@hotmail.com

Supervisor dhr. J.W.J. De Kort

Co-assessor drs. B. Crom

Master thesis

RijksUniversiteit Groningen

Faculty of Economics and Business

MSc Business Administration – Organizational and Management Control

July 2014

(3)

2

Abstract

This thesis aims to answer the research question: to what extent can the measurement of social impact of government policies be improved based on knowledge of measurement systems developed in organizational theory?

Politicians measure the (social) impact of their policies in order to determine whether their current policies are working towards the set goal and to determine what course they should take for the future. The aim of this thesis is to identify possibilities to improve the way social impact of government policies is measured at two Dutch local governments. Organizational theory guides this search; knowledge of performance measurement systems and criteria for efficient measures described in organizational theory are used to assess policy measurement systems. Policy measurement systems were studied using a multi-case study approach. Difficulties in policy measurements are identified and an analysis is made of whether characteristics of performance measurement systems can help overcome these difficulties.

Based on the findings, it is argued that organizational theory can be used to improve the measurement of social impact of government policies in two ways. The first way is to use requirements of performance measurement systems to evaluate the currently used instruments. The second is to use existing performance measurement theories and studies to identify possibilities to improve current policy measurements based on positive characteristics of performance measurement systems. However, not all difficulties of social impact measurement might be addressed using this approach. This shows that the extent to which organizational theory can be used to improve measurement of social impact of government policies is substantial, but not perfect.

(4)

3

Introduction

Politicians measure the (social) impact of their policies in order to determine whether their current policies are working towards the set goal and to determine what course they should take for the future. The aim of a policy might be to improve main roads in a certain district, this will be done by investing a certain amount of money. There might only be a financial control; have the expenses stayed within the budgeted amount, then a good job has been done on improving the roads. Traditionally local governments have measured the impact of their policies in this way (Palmer, 1993; Atkinson & McCrindell, 1997; Kloot & Martin, 2000), this also counts for Dutch governments. In recent years Dutch policymakers have realized that this is a very limited way to measure the effectiveness of policies.1 For example, in the case of road improvement it would also be necessary to know the quality of the roads. Where it gets really complicated is when we try to measure the social impact of government policies. Science nor practice has found a satisfying solution to this problem. In this thesis I aim to identify possibilities to improve the way social impact of government policies is currently measured at two Dutch local governments, organizational theory guides this search.

To evaluate the effectiveness of policies, policy goals have to be formulated in an effective manner, this thesis however will focus on the measuring. Since the Netherlands have a democratic system where citizens directly vote for politicians to represent them in (local) governments, these citizens also have a need to monitor the effectiveness of policies of the installed government. Citizens need this information in order to make an informed decision about whether to (re)elect certain politicians, or not. Both citizens and politicians should know how much was spent on improving roads as well as the quality of these roads. However, measuring social impact is easier said than done. In this thesis the definition of social impact by Duncan and Jones (1976) is used: ‘social impact is a significant improvement or deterioration in people’s well-being or a significant change in any aspect of community concern’. The challenge is to determine how we can measure this social impact of policies in a meaningful manner.

In the organizational domain the need for the measurement of results has been recognized, both in practice and by science. Organizational theory shows numerous studies that develop, analyse and test all sorts of performance measurement systems. Over the years there has been a similar development in these measurement systems such as the one described above: there was a shift from purely financial controls to integrated measurements, which combine financial and non-financial measures (Yadav, Sushil & Sagar, 2013). However, the realization that traditional accounting measures might be insufficient came as early as 1950 for the French engineers who introduced the

1

(7)

6 tableau de bord; one of the first integrated measurements (Epstein & Manzoni, 1997). Only in the 1990s did this start to affect the bulk of organizations and organizational theorists (Yadav et al., 2013), which was predicted by Eccles in 1991: ‘within the next five years, every company will have to redesign how it measures its business performance’ (p. 131). Arguably the most important development in organizational performance measurement was the introduction of the balanced score card in 1992 by Kaplan and Norton. This widely known and applied measurement framework integrates financial and non-financial measures, such as quality, customer satisfaction, improvement activities and strategic orientation (Kaplan & Norton, 1992). Since 1992 numerous studies have been conducted testing the balanced score card, presenting new variations to it (e.g. business scorecard (Kanji & Sá, 2002), holistic scorecard (Sureshchandar & Leisten, 2004)) and introducing new measurement systems (e.g. sustainability performance measurement system (Searcy, 2011)).

Might local governments in the Netherlands gain from the knowledge about measurements that is readily available in organizational theory? I will use criteria for effective measures described in organizational theory to assess policy measurement systems. Weaknesses in policy measurements will become clear and an analysis is made of whether certain performance measurement systems can help overcome these weaknesses. Some studies have applied these performance measurement systems to assess actions of local governments, for example by Kloot and Martin (2000). These authors present a study into performance management at a local government in Australia using the balanced scorecard. They show that the focus of the local government is on the financial performance and only to a lesser extent on how the community views performance. These findings emphasize the problem discussed in this thesis. Kloot and Martin (2000) do not present a solution regarding how the government should organize a measurement of non-financial impact of its policies. In this thesis I look for difficulties in measuring social impact, but also for possible solutions. The aim of this study is to contribute to organizational theory by exploring the application range of performance measure systems by extending it to government policies. Furthermore this study contributes to practitioners by beginning to explore solutions to the problem of measuring social impact of government policies. The research question I will answer is the following: to what extent can the measurement of social impact of government policies be improved based on knowledge of measurement systems developed in organizational theory? To answer this question I have specified the following sub-questions:

1. What is social impact and social impact measurement?

(8)

7 4. How is (social) impact of government policies measured currently by local governments? 5. To what extent can the policy measurements be characterized as effective measurement

systems?

In the next chapter I lay down the theoretical foundations of this thesis by discussing and defining the concept of social impact measurement (sub-question one) and the concept of performance measurement systems (sub-question two and three) based on literature research. The second chapter is devoted to the methodology used in this thesis; a case study on the policy measures used by two local Dutch governments. In the third chapter I present policy measurement systems currently used at these governments (sub-question four). This presentation is guided by the characteristics of effective measurements and it is stated to what extent these characteristics are reflected in the policy measurement systems (sub-question five). This leads to an identification of difficulties in current policy measurements. In the analysis the empirical findings are compared with the knowledge of performance measurement systems to assess whether solutions can be found for the difficulties of measuring social impact of government policy, which leads to an answer on the research question of this thesis. Finally, a conclusion, discussion, including suggestions for further research, and reflection conclude this thesis.

Theoretical framework

In this theoretical framework I will set the foundations of this study by introducing and exploring the two central concepts; social impact measurement and performance measurement systems. To study how social impact can be measured effectively, the first step should be to define what it is you want to measure and to explore what is already known about social impact measurement. The second step in this thesis is to discuss the subject of performance measurement systems. This is a broadly studied subject in organizational theory and is related to policy measurement, since this might also be described as the measurement of the performance of government policies. The question posed in this thesis is whether knowledge of performance measurement might contribute to improvement of policy measurement. Therefore characteristics of effective measurements are also discussed in this theoretical framework.

Social impact

(9)

8 community concern’ (p.8). This definition will also be used in this thesis to describe the concept social impact. The definition specifies impact as a ‘significant change’, for better or for worse, and it specifies the social aspect as ‘people’s well-being’ or ‘community concern’. Well-being refers to the quality of life of people (Duncan & Jones, 1976). Change in people’s well-being therefore concerns direct impact of the private life, while a change in community concern might not affect people’s personal situation, but considers impact the is generally seen as important by community as a whole. The reference to ‘any aspect of community concern’ underlines how broad the concept of social impact is; social impact concerns not only the private life of people, but also everything that is of concern to the community, or society in general. As Cramer, Dietz and Johnston (1980) argue the boundaries of social impact are not clear in this definition. First, it is not clear what ‘people’ and how large a ‘community’ should be considered. There is no general solution for this, the specific boundaries differ per situation and in the case of social impact of policies the boundaries are generally determined by political jurisdiction. The second ambiguity has to do with the term ‘significant change’. What is significant depends on the specific case, but it is hard to determine what is important and what not. What seems not significant now might become important in the future (Cramer et al., 1980).

There are two kinds of social impact; a subjective and an objective kind, which of course also exist together. Objective social impacts are those ‘measured’ by an outsider, whether or not those impacts are of concern to those affected. Subjective social impacts are those perceived by the affected, whether or not an outsider can assess them (Cramer et al., 1980). Studies have found that subjective perceptions and objectively measured ‘reality’ are only loosely associated. Cramer et al. (1980) give two reasons for this: people do not always perceive reality ‘accurately’, furthermore people’s evaluation of reality differ based on personal characteristics and beliefs. The impact, or effect, is always brought about by an event (Dietz, 1987), which in this case is the introduction of a certain government policy.

(10)

9 or ‘effect’, situations have to be compared, therefore social impact is always relative to a initial situation. This is clearly stated in the following quote: ‘for me social effect is mainly the added value you create relative to a starting situation’. This ‘starting situation’ might be hard to determine. Furthermore, during the research it became clear that in the case that for example the safety in an area is rated ‘average’ and a year later is still rated ‘average’, this does not necessarily mean that there was no social impact. In this year the safety of all surrounding areas might have decreased and so would have the safety in the measured area, if it were not for certain policies in this area. Even though the measurements do not show change, the policy had impact.

Social impact measurement

Measurement can be broadly defined as acquiring and analysing information about an object of interest (Kerssens-van Drongelen & Cook, 1997). This study is about measuring the social impact of government policies. In a general sense the goal is to measure performance, that is the performance of policies. In organizational theory performance measurement is a topic that is frequently discussed, while a lot of authors do not give a definition of this concept, some definitions can be found (Franco-Santos et al., 2007). Kerssens-van Drongelen and Cook (1997) are one of the few to also specify ‘measurement’ in their definition of performance measurement: ‘the acquisition and analysis of information about the actual attainment of company objectives and plans and about factors that may influence plan realization’ (p.347). Even though their definition refers to ‘company objectives’, this can easily be replaced by policy objectives. The definition takes performance measurement further than measuring the direct attainment of objectives by including ‘factors that may influence plan realization’, thereby taking on a more future-oriented, long term approach. This is in line with the addition Sinclair and Zairi (1995) make to their definition of total quality-based performance measurement; performance measurement takes place ‘with a view to the continuous improvement of performance against organizational objectives’ (p. 50).

(11)

10 consequences. In accordance with Neely et al. (1995) I will refer to a performance measurement system as the set of metrics used to measure performance.

I refer to policy measurement systems as the set of metrics used to measure the social impact of government policies. These policy measurement systems are related to social impact assessment (SIA), which was defined by Burdge and Vanclay (1996) as ‘the process of assessing or estimating, in advance, the social consequences that are likely to follow from specific policy actions or project development, […], or provincial environmental policy legislation’. SIA research took a flight in 1969 when it was formalized in the United States in the National Environmental Policy Act (NEPA) (Burdge & Vanclay, 1996). NEPA required that any government policy likely to have any impact on the environment was first subjected to an assessment of the possible outcomes (Freudenburg, 1986). The act itself was concerned with environmental impact, but soon economic and social impact were also included. For many countries social impact assessment has become compulsory government action (Becker, 1997).

There are two broad approaches to SIA. The first is approach is to perform the minimum assessment required by legislation. The second approach is most popular in academics and is based on the idea that informed planning is possible through the agency of social impact assessment (Harvey, 2011). In the early days of NEPA legislation SIA was centred around technical judgment, since then academics have developed into areas that did not seem important in the 1970s, such as gender impact assessment, human rights, conflict-sensitive, cultural, livelihood, health and ecosystem impact assessment (Vanclay & Esteves, 2011). Throughout the years a vast base of literature has been developed, covering the topic in the most broad sense.

(12)

11 As was mentioned, social impact is a very broad concept. A returning problem in literature about social impact assessment is that it is hard to determine what social impact is and what not. Furthermore, who is considered ‘affected’ and who is not? Burdge and Vanclay (1996) state that since a social impact assessment is usually commissioned by a (local) government or developer, ‘the impacts that are considered […] are those that are politically or socially determined at the time the study is done. Many potential impacts are excluded from consideration because they may not be regarded as important at the time’ (p.74). To make the assessment more independent of the commissioning party the authors refer to the better use of the guidelines and principles for social impact assessment, a document developed to fit within the regulations in the U.S. at that time. The assessment in other words has to be reliable; independent of the particular characteristics of that study (Yin, 2009). Measurement systems of social impact should also be reliable and a central guideline would contribute to this by guiding what impacts should be considered and how the group of affected can be determined.

In the Netherlands there does not seem to be a central guideline, but a number of separate initiatives developing social impact measurement systems. This is illustrated by the rapport of Radar (2013) that lists twenty-two different instruments that are being used to measure social impact, of which most are actually based on ex ante estimation of the impact. In this thesis I identify and analyse the social impact measurement systems that are in use at two Dutch local governments. I will add an academic perspective by combining the findings with knowledge gathered by organizational theory concerning performance measurement systems.

Performance measurement systems

(13)

12 Yadav et al. (2013) categorise the performance measurement and management frameworks into five broad themes. I will discuss the themes briefly, for the complete overview see table 1. The first theme, ‘classical and dominant’, includes the systems that have been most popular in literature as well as most used by practitioners. These were the first frameworks to incorporate non-financial performance measures. The second group of systems is ‘holistic and integrated’, these have even further integrated non-financial measures to aligning performance with the future and individual performance with enterprise performance, and to integrate operational and strategic aspects into the framework. The third group of performance measurement systems consists of updates of the balanced scorecard approach. The fourth theme consists of context-specific frameworks, which discuss specific contexts of performance, such as time-based competition, economic value and shareholder value. Finally there are two recently developed performance measurement and management frameworks, which are relatively novel.

Table 1. Performance measurement and management systems overview. Based on Yadav et al. (2013).

(14)

13 models have proven to be dominant in both literature and practice (Yadav et al., 2013), this means that these frameworks have a lot of positive characteristics which might be applicable for policy measurement as well. Furthermore I will discuss the two recent models, these have not yet had the chance to become ‘classic and dominant’ and these address issues that are currently relevant, which might be important for policy measurement as well.

Balanced scorecard

The BSC provides a ‘balanced score’ due to the integration of financial and non-financial measures, and short- and long-term measures. The scorecard ‘scores’ business on four perspectives, the measures answer four questions; one for each perspective (Kaplan & Norton, 1992, p.72):

- How do customers see us? (customer perspective) - What must we excel at? (internal perspective)

- Can we continue to improve and create value? (innovation and learning perspective) - How do we look to shareholders? (financial perspective)

Since the BSC is presented as a one page report, managers can get a quick, but comprehensive view

Figure 1. BSC: translating vision and strategy – four perspectives. Source: Kaplan & Norton (1996).

(15)

14 list of critical indicators (Kaplan & Norton, 1992). Management uses the BSC ‘as a way to clarify, simplify, and then operationalize the vision at the top of the organization’ (Kaplan & Norton, 1992, p.73). The general guideline for each perspective is to translate organizational goals based on the vision and strategy of the company into specific measurements (Kaplan & Norton, 1992). If the strategy of the company is good, improvements in the first three perspectives should induce improvements in the relevant financial measures, according to Kaplan and Norton (1992).

Yadav et al. (2013) appreciate the introduction of the BSC for marking ‘a shift in performance measurement from an operational perspective to a strategic perspective’ (p.955). However, despite its popularity, studies have found many shortcomings, such as the static nature, lack of cause-effect relationships and clustering of performance measures (Yadav et al., 2013). As a response to these shortcomings several studies have presented new versions of the BSC methodology. Kanji’s scorecard (Kanji & Sa, 2002) broadens the perspectives to ‘delight the stakeholders’, ‘process excellence’ and ‘organizational learning’. Kanji and Sa claim that causal relations between the perspectives are problematic; they are more like interdependences. In their holistic scorecard, Sureshchandar and Leisten (2005) add to the financial, customer and business process perspective, the social, employee and intellectual capital perspective. Both updates addressing the critique that the BSC focussed too strongly on shareholders and customers, thereby forgetting other important stakeholders. Rampersad (2005) introduces a cycle of continuous learning and improvement to the BSC. Barnabe (2011) and Chytas, Glykas and Valiris (2011) have explored the interdependences further and integrated simulation techniques in order to test the feasibility of actions by showing future results. Addressing the critique that the BSC is too static, relative to the dynamic environment.

Performance pyramid

(16)

15

Figure 2. The performance pyramid. Source: Cross & Lynch (1989).

time’ and ‘cost’ (‘cost’ are later changed into ‘waste’, Lynch & Cross, 1991) The pyramid method ensures that the strategic direction is translated into department actions (Cross & Lynch, 1989). Lynch and Cross took a lead by developing a framework aimed at bridging the gap between the top level and operation floor. The performance pyramid and the SMART system are praised for integrating the corporate objectives with operational indicators, but it fails to provide a mechanism for identifying those key performance indicators (Ghalayini, Noble & Crowe, 1997). Some of the critiques of the BSC also apply to the performance pyramid; it too has not integrated the idea of continuous improvement and focusses mainly on the shareholder and customer perspectives.

The excellence model

In 1992 the business excellence model was established by the European Foundation for Quality Management (EFQM) to promote sustainable excellence and awareness of the importance of quality in European organizations. Each year the EFQM rewards a European company with the quality award based on assessment of the nine criteria that form the basis of the model. The EFQM is used in various ways by organizations, for example, as a tool for self-assessment, as a guide to identify areas for improvement and as a structure for the organization’s management system (Kim, Kumar & Murphy, 2009). The latest update of the model contains three elements: fundamental concepts of excellence, criteria and RADAR (EFQM, 2013), see figure 3.

(17)

16 for systematically improving (EFQM, 2013). The excellence model allows organizations to identify their strengths and areas for improvement. However, the model does not ‘offer any answers regarding how to improve upon an organization’s strengths or weaknesses’ (Kim et al., 2009, p.693). The specific application and results of the excellence model differ from organization to organization, however, it does provide a common language within and between organizations (EFQM, 2013).

Figure 3. The EFQM excellence model. From left to right:fundamental concepts of excellence, RADAR, criteria (bottom). Based on EFQM (2013).

Performance prism

(18)

17

Figure 4. The performance prism. Based on Neely, Adams & Crowe (2001).

The performance prism encourages managers to think through the key business issues and links between them by asking critical questions (Neely et al., 2001, p.6):

- Stakeholder satisfaction: Who are the important stakeholders in your organization and what do they want and need?

- Strategies: What are the strategies we require to ensure the wants and needs of our stakeholders are satisfied?

- Process: What are the processes we have to put in place in order to allow our strategies to be delivered?

- Capabilities: What are the capabilities we require to operate our processes? - Stakeholder contribution

Neely et al. (2001) argue that the starting point of other measurement systems, the vision, is wrong; instead it should be the stakeholder satisfaction. However, the relationship between organization and stakeholder should also involve the stakeholders contributing to the organization.

The performance prism, like the excellence model, is ‘not a prescriptive measurement framework. Instead, the Performance Prism is a framework – a tool – which can be used by management teams to influence their thinking about what the key questions are that they want to address’ (Neely et al., 2001, p.7). The advantage of this is that the application possibilities of the prism are very broad. The disadvantage, however, as Yadav et al. (2013) point out, is that the model gives only very limited guidance for determining and measuring the actual indicators.

Flexible strategy game-card

(19)

18 for. Yadav et al. (2013, p.960) state that the framework ‘needs empirical validation’, the same goes for the sustainability performance measurement system by Searcy (2011). However, the flexible strategy game-card has now been tested in practices in several studies (for example see Mittal & Jain (2012), and Bhawan & Marg (2012)) and a more elaborate theoretical foundation of the flexible strategy game-card has been published (Yadav & Sushil, 2014).

The flexible strategy game-card is an adapted version of the BSC. It categorizes the four original perspectives into two broad perspectives: the enterprise perspective covers the financial, internal process and learning and growth perspective, and the customer perspective. The enterprise perspective includes all relevant stakeholders. The customer perspective is discussed separately, since the customers are seen as the centerpoint for strategy development. Strategy formulation is based on the strategic factors covered by the perspectives. The enterprise perspective entails the SAPP factors; situation, actor, process and performance factors. The customer perspective entails generic as well as specific customer factors (Sushil, 2010).

Implementation of the flexible strategy game-card requires identifying and linking strategic factors. These factors are mapped on the strategy landscape, both for the existing and proposed levels, based on this an organization is enabled to define a strategic direction. Important is that the resulting performance of the taken actions is assessed, so that actions can be adapted via a feedback mechanism (Sushil, 2011).

Sustainability PMS

Searcy (2011) argues that there is a need for more research on the evolution of corporate sustainability performance measurement systems (PMS). Corporate sustainability programs address the ‘triple bottom line’ issues of economic, environmental and social performance. Therefore a sustainability PMS must ‘have a long-term focus’. No actual measurement system is presented, but a three-phase approach for the continuous updating of the existing sustainability PMS within an organization. The three phases are the following: planning for an assessment of the sustainability PMS, conducting an assessment and following up on the results of the assessment, see figure 5.

(20)

19

Concluding: performance measurement systems

In the performance measurement systems described above, the trends as indicated by Yadav et al. (2013) are reflected. The first major shift took place in the 1990s, when researchers realized that a financial perspective only is not sufficient to measure organizational performance. Measurement systems based on an integrated perspective were developed, including both financial and non-financial performance measures, and both lagging and leading performance indicators. In the second half of the 1990s and early 2000s continuous improvement, dynamics and a broader stakeholder perspective are important issues in the development of performance measurement systems. This leads to updates in the BSC approach, overcoming initial limitations. Finally, the most recent development in performance measurement is to include sustainability and simulation techniques.

Effective measurement

In their review of the ‘evolution of performance measurement systems’ Kennerley and Neely (2002) state that ‘at the heart of the processes, frameworks and criteria discussed, as with much that has been written on the subject of performance measurement, is the premise that measures and measurement systems must reflect the context to which they are applied’ (p. 1224). This is a first important condition for a measurement system to be useful; it has to fit the situation to which it is applied. In organizational theory a lot of requirements for ‘good’ performance measurement systems have been discussed. Before I describe these, I will give an overview of the most important requirements of measures that have been identified in literature about doing research in general: reliability, validity and controllability. These criteria will enable the assessment of policy measurements currently used by local governments.

(21)

20 Personnel economists Lazear and Gibbs (2009) identify five performance measure properties that determine a good measure to evaluate the performance of personnel: the measure’s risk profile, distortion, scope, match to job design and potential for manipulation. I will describe these properties based on Lazear and Gibbs (2009) and explain how these can be applied to policy performance measurement. The first property is the risk profile of the measurement, the general rule is that a good performance measure should exclude whatever is uncontrollable by the employee. However, besides uncontrollable risk, which is variation in performance that is beyond the employee’s ability to control, there is also controllable risk, which lies at least to some extent within control of the employee. For policy makers there might also be some controllable risks, these variations they can anticipate. Controllable risks should be included in the measurement, since anticipation of these risks will minimize the impact of the possible variations. Uncontrollable risks should be excluded, this can be done by choosing more narrow performance measurements.

The second property is distortion; too narrow measures might distort performance. The example Lazear and Gibbs provide concerns a custodian whose performance is measured only by the cleanness of the floors. The custodian is not motivated to perform his other tasks or to be cost conscious; he will only concentrate on cleaning floors. This property can be recognized in some important problems of traditional financial measures, which are criticized by numerous authors because they encourage focus on short term results, lack strategic focus, do not measure quality, responsiveness and flexibility, and encourage to minimize variances from standard rather than to seek improvements (Neely, 1999). In other words financial measures distort because they encourage to focus only on short term financial aspects of performance. To lower distortion the measurement should be more broad as to include all aspects of performance, however, this might bring along more (uncontrollable) risk. A possible way to deal with this third property of measurements, the scope, is to combine several measures of different dimensions of performance.

(22)

21 The final property that Lazear and Gibbs ascribe to performance measurements is the potential for manipulation. Manipulation arises when an employee uses his specific knowledge strategically to improve the outcome of the measurement, after the measurement instruments are chosen, ‘even when such actions do not improve firm value’ (p.246). Politicians might have an incentive to manipulate the outcomes of certain policy measures since positive results might lead to their re-election. In the case of personnel performance measurement Lazear and Gibbs suggest that the potential for manipulation can be limited by including a subjective performance measure in the form of an evaluation conducted by a direct supervisor. In this study I will research to what extent these properties, including the potential for manipulation, are present in current policy performance measurement systems. Here I will first review research that has already been conducted concerning the measurement of social impact of government policies.

Conceptual model

Based on this theoretical framework, I have formulated the conceptual model to guide this study, see figure 6.

Reliability Validity Controllability

Effectiveness of Policy Social impact policy measurement

Risk profile Distortion Scope Match to Potential for environment manipulation

Figure 6. Conceptual model

(23)

22 rapports on the measurements I have looked for two important sources of reliability; use of standardized research methods and use of multiple sources of evidence (Van Aken, Berends & Van der Bij, 2012). Validity, or the extent to which the measures measure what was intended to measure, is studied by asking informants to what extent they think the instrument measures the impact of the concerning policy. The rapports were analyzed to determine whether set goals were reached, for example, the monitor livability and safety defines livability as ‘the extent to which the living environment satisfies the needs of the residents’ (O&S Groningen, 2013, p.14), the question is whether the indicators cover this. To examine controllability of the studies it has to be determined if it is clear for the politicians how the measurements were conducted, this is asked during interviews and the rapports are analyzed to find if the researchers explain how the study was conducted. The risk profile of the measurement concerns the inclusion of risks. The informants were asked whether they think that all aspects within their influence are taken into account and whether indicators are included that they cannot or barely influence. Especially in the social domain this is a complicated matter since most municipalities subsidies external organizations to do the actual work, the question to what extent the local government remains responsible was discussed. Distortion was studied by asking informants whether the measurements focus too much on certain aspects. The archived recordings of the municipality were analyzed to determine whether there are aspects that are strongly emphasized by politicians. The scope of the measurement can be determined based on the risk profile and the potential for distortion; if a lot of uncontrollable risks are included the scope is broad, if the measures focus only on certain aspects the scope is narrow (Lazear & Gibbs, 2009). The characteristic of ‘match the environment’ is very broad, in the interviews the subject is firstly discussed in the broadest sense: to what extent does the informant think the measure instrument fits the policy domain. Second, the changes over time in the measurements are discussed, this is an important aspect of the match to the environment (Lazear & Gibbs, 2009). The changes in indicators are analyzed in the rapports from different years. Potential for manipulation can have many different forms, therefore the informants are asked if they think the instruments have potential for manipulation and if they could explain their answer. The study of the other material focuses on a specific potential for manipulation, which is expected to be of importance in this context; the interpretation of the results by politicians.

Methodology

(24)

23

Figure 7. Academic problem solving approach. Source: Van Aken, Berends & Van der Bij (2012).

The ‘business phenomenon’ that triggered this study is the fact that a lot of governments have difficulties measuring social impact. There are several players involved, most importantly the politicians in local governments and the researchers who conduct measurements for these local governments. I have conducted a multiple-case study at two local Dutch governments in combination with studying several independent research bureaus. The collected data was analysed and compared with the findings of a literature study of performance measurement systems. Based on this I will answer the main research question (to what extent can measuring social impact of government policies be improved by measurement systems developed in organizational theory?). An implementation of my conclusions is out of reach for this thesis.

In this section I shortly describe the methodology of a case study and argue why this approach is most appropriate for this thesis. After this the two governments are introduced and I explain how data was collected and analysed, considering reliability and validity. Finally I explain how the literature study was conducted.

Case study

Yin (2009) has formulated a comprehensive definition of case studies, which consists of two parts. The second part of this ‘technical’ definition concerns the manner in which data is collected and analysed, this will be discussed later in this section. The first part concerns the scope of a case study: ‘a case study is an empirical inquiry that investigates a contemporary phenomenon in depth and within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident’ (p.18). Measuring social impact concerns the contemporary, since the measurements consider the most recent data and events to assess the impact of current policies. This also shows that the context is very important and closely connected to the measurement systems, which requires an extensive and in-depth description of the phenomenon. This makes the case study method the most relevant method of research (Yin, 2009). Furthermore case studies are

(25)

24 especially appropriate in new topic areas, since theories are developed using this method, not tested (Eisenhardt, 1989). The nature of this research is exploratory; I make a start towards finding effective ways to measure social impact of policies.

I have chosen to conduct a multiple-case study based on two local Dutch governments and several independent research bureaus. This will give an even more in-depth insight into what measuring social impact currently entails and adds a new dimension to the analysis because the results of the case studies can be compared. The two local governments that are discussed in this study are the municipality of Groningen and the municipality of Rotterdam, with emphasis on Delfshaven, a part of the municipality with its own administration (however, with limited responsibilities).

The cases

Groningen is the capital city of the province of Groningen, which lies in the north of the Netherlands. Groningen is known as a ‘studentenstad’, or studentscity; one in every six inhabitants is a student. As of the start of this school year (September 2013) the University of Groningen, the ‘RijksUniversiteit Groningen’, or RuG, educates 29.407 of these students, and is therefore one of the largest universities in the Netherlands (RuG, 2014). The total number of inhabitants of the municipality Groningen at January 1st 2014 was 198,395 (O&S Groningen, 2014). Considering this Groningen is the largest city in the north of the Netherlands, however the cities in the ‘Randstad’, the industrial and metropolitan conurbation in the west of the Netherlands, are all larger. Rotterdam is located in this economically vital area. Rotterdam is the biggest city of the province of Zuid-Holland, in the west of the Netherlands. The port of Rotterdam is one of the largest in the world and because of this Rotterdam is also called the ‘Gateway to Europe’. The annual throughput of the port is about 450 million tons, with a total of 88,649 employees in 2011 (Port of Rotterdam, 2013). The municipality of Rotterdam has a total of 616,319 inhabitants, making Rotterdam the second largest city in the Netherlands (Rotterdam, 2014).

(26)

25

Data collection

The second part of Yin’s (2009) definition of case studies, concerning the data collection, is the following: ‘the case study inquiry copes with the technical distinctive situation in which there will be many more variables of interest than data points, and as one result relies on multiple sources of evidence, with data needing to converge in a triangulating fashion, and as another result benefits from prior development of theoretical propositions to guide data collection and analysis’ (p.18). Key in the collection of data in case studies are the multiple sources of evidence. I use multiple data collection methods, which strengthens the grounding of the theory, and at the same time my data collection methods are flexible and opportunistic (semi-open interviews, open for all data that is presented regarding the general topic of measuring social impact), which allows me to take full advantage of the case (Eisenhardt, 1989).

In March 2014 elections were held for the Dutch local governments, which made the timing of this study somewhat unfortunate. However, I have conducted an interview with one politician from the municipality of Groningen, who was also re-elected, and an interview with one politician who has worked for the municipality of Rotterdam and who is now working for the municipality of Deventer.2 Furthermore I have read and analysed the relevant documents of both municipalities and watched recordings of several meetings of the administrations. I have conducted an interview with a researcher at the Verwey-Jonker Instituut, unfortunately it was not possible to interview one of the researchers at the RadarAdvies, but there was the possibility of mailing questions. For both bureaus the relevant documentation was studied. The different sources of data are highly complementary. The most important advantage of using several sources of data is ‘the development of converging lines of inquiry, a process of triangulation and corrobation’ (Yin, 2009, p.115-116).

The reliability of this study can be viewed from three perspectives; that of the researcher, the instrument and the respondents (Van Aken, Berends & Van der Bij, 2012). Reliability of the study regarding the researcher means that the study should be independent of the researcher that has conducted the study. One way to make the study more reliable in this regard is the use of standardized research methods (Van Aken et al., 2012). In this section the basis of a case study protocol is described, which increases reliability (Yin, 2009). In advance of the interviews with the politicians I have chosen two measurement instruments based on recommendations by experts (in the case of Groningen the instruments were recommended by the secretary of the council for education and wellbeing, for Rotterdam the instruments were mentioned by a former member of the administration of Delfshaven). During the interviews I used the schedule in appendix A as a guideline. I have chosen to use ‘reflective’ interview techniques. The researcher does have a list of subjects that

2

(27)

26 will be discussed in the interview, but the exact questions vary with the situation and the informant in order to remain open for all relevant information (Hammersley & Atkinson, 2007). The interviews were more guided conversations rather than structured inquiries (Yin, 2009). The topics to be discussed are directly related to the findings of the theoretical framework; the headings in de interview guide show the part of the theoretical framework that the questions are based upon. The interviews are recorded, so that the interviews can be transcribed at a later moment.

Reliability with regard to the measurement methods means that the results should be independent of the instruments used (Van Aken et al., 2012). This is done by using multiple sources of evidence and triangulation of the data. Reliability regarding the respondents can be reached by selecting cases randomly and looking at all different points of view concerned (Van Aken et al., 2012). The municipalities were chosen because they are similar in some aspects (both large cities), which makes it possible to compare, while in other aspects they differ (for example, Delfshaven has a higher poverty and immigrant rate), which adds depth to the analysis. It is beyond the scope of this research to interview all groups involved with measuring social impact. However, the archived video registrations of meetings of city councils involve a large pool of politicians and the studies of independent research bureaus add a new perspective to the subject. The use of multiple sources of evidence increases the construct validity of this study. Furthermore the results of the various research methods are analysed and compared. Yin (2009) also introduces the concepts of internal and external validity, these will be discussed in the next section.

Data analysis

Internal validity concerns explaining phenomena in a credible way. This can be established by using several case study techniques: within-case analysis, cross-case pattern matching, explanation building and systematically relating concepts and findings (Riege, 2003). To establish internal validity the data in this thesis is analysed according to the grounded theory approach of Strauss and Corbin (2008). The analysis starts open and free, using open coding to identify concepts and categories. This is followed by axial coding to identify the links between concepts. Finally selective coding is used to refine the findings. The coding strategies differ from each other, but during analysis the processes occur mostly simultaneously (Strauss & Corbin, 2008).

(28)

27 instruments of the municipalities of Groningen and Rotterdam. The way governments measure the social impact of their policies differs, therefore it is hard to determine the extent to which this thesis applies to other (local) governments in the Netherlands, let alone beyond the Netherlands. This might be seen as a limitation of this thesis, however, it is not the goal of this thesis to formulate a universal theory. The goal is to explore the subject of measuring social impact of government policies, the case study approach seems most appropriate for this goal.

Policy measurement systems

This chapter discusses several instruments for measuring the impact of government policy that are currently available in Groningen and Rotterdam. First, some of these measurement systems are introduced, after that a more detailed discussion follows, considering policy measurement with regard to various elements described in the theoretical framework of this thesis. This chapter is based on the interviews and other materials, such as documents and archived videos, concerning measuring social impact of government policy.

The instruments

As mentioned, in advance of each of the interviews I selected two rapports that seemed fitting and interesting to discuss in the interview. I describe those here. However, in each interview it became clear that there are so many more instruments and measurements. There are some overall monitors, like the ‘stadsmonitor’ (city monitor) and the ‘gemeentelijke rekening’ (municipal account) in Groningen. Besides these, there are a lot of more specific measurements that for example focus on one domain, like the ‘sociale index’ (social index) in Rotterdam, which focusses on the social domain, or even more specific, like the ‘veiligheidsindex’ (safety index) in Rotterdam, concerning safety, or measurements that focus on one specific group within the population, such as ‘kinderen in tel’ (children in count), a measurement focussing on children developed among others by the Verwey-Jonker Instituut. The diversity of measurements is amplified by the various groups that perform the measurements; both the local government of Rotterdam and Groningen, like most of the larger municipalities, have their own specialised research group, but also hire independent research bureaus, which have their own ways of measuring, like the Verwey-Jonker Instituut and RadarAdvies, and also there are different bodies that keep their own records which local governments can use, like the ‘GGD’, the public health service.

(29)

28 research bureaus, these measurements are not free. The interviewed researcher is very aware of this and mentions an example of a project by a local government where the research on the project turned out to cost as much as the project itself. He questions the profitability of research like that, for him it depends largely on the quality of the existing administration. In a lot of cases data is already to a far extent available, which makes the impact measurement more feasible.

The instruments of the municipalities of Rotterdam and Groningen that I describe here, are used on an annual basis to monitor developments. For Groningen I was recommended the city monitor and the monitor livability and safety. The politician comments on these two measurement systems: ‘these are two instruments we have, but not the ultimate. At the same time, there is no ultimate instrument, but it is the sum that allows you to look at impact and effectiveness’. I will now shortly introduce several instruments and measurements separately, however in the discussion below, guided by the requirements of measurements discussed in the theory section, the instruments will not be discussed separately. In the interviews the measurement instruments were discussed together and in comparison with each other, a lot of other examples were also mentioned, because all these measurements are referred to by politicians and researchers to create the bigger picture.

Groningen

The rapports that were chosen before the interview with the politician of Groningen were the city monitor and the monitor livability and safety, these are described. Furthermore in the interview the municipal account was an important, returning topic, therefore this measurement will also be included in this short introduction.

(30)

29 The monitor livability and safety (O&S Groningen, 2013) is conducted every two years by the O&S Groningen, the research bureau connected to the local government. The rapport includes a paragraph shortly explaining the research methods, the sample and the extent to which the numbers are representative. The monitor was praised by professors mainly because of the insightfulness and the connections that are made. The rapport of 2012 starts with an overview of some developments in the city and some interdependencies are suggested; for example: ‘safety experience is connected to actual crimes, but also to hindrance and social and physical livability’ (O&S Groningen, p.14). These interdependences are supported by studies and are most of the times indeed visible in the numbers that are presented. However, it is stressed that that ‘in a lot of cases the interdependences are not on a one-on-one basis, because a lot of other factors play a role’ (O&S Groningen, 2013, p.14). Indeed can examples be found where for example residents of a neighbourhood feel unsafe, but crime reports are low. After this overview, safety and four themes of livability are discussed for the entire city: livability in general, physical livability, social livability and content with public goods. Finally detailed developments are described for each neighbourhood, illustrated by tables showing relevant numbers over the last ten years (2002, 2006, 2010, 2012) and the city average.

(31)

30

Rotterdam

The measurements which I have studied from the municipality of Rotterdam are the safety index and the social index. During the interview the new way of making public contracts introduced by the politician from Rotterdam was an important issue, therefore this will be shortly described too. The three topics are all interrelated. The social index ‘measures the social quality of Rotterdam in general and of the separate submunicipalities and neighbourhoods’ (Rotterdam, 2012, p.2). Rotterdam and the neighbourhoods are graded on social quality, this grade consist of the average of the grades for four themes: participating, social bond, capacities and living environment. Each of these themes is again subdivided in several subjects, which are graded and averaged. First the results of the city are described. Then the most important outcomes are compared with earlier results of measurements of the social index and comparison takes place between the neighbourhoods. Finally all results are shown for the submunicipalities and neighbourhoods within these. The scores are in the appendix, in the actual rapport the grades are discussed and shown qualified as ‘socially very weak’ (dark pink), ‘problem’ (pink), ‘vulnerable’ (grey), ‘socially adequate’ (light green) and ‘socially strong’ (dark green) (Rotterdam, 2012, p.4, see figure 8). The rapport includes an account of the reliability and validity of the results.

Figure 8. Social Index Delfshaven 2012. Source: Rotterdam (2012).

(32)

31 In the social domain governments work with partners to realize goals. In the submunicipality of Delfshaven, Rotterdam, the interviewed politician has changed the relationship with these partners. He started by writing out public contracts with criteria that the partners have to fulfil. Once matching partners are found they enter into a performance-contract with the local government. The performance targets are based on the criteria from the public contract. In these criteria the politician has made the connection between indicators from measurements like the social and safety index and policy. He explains: ‘They [the indexes] objectify the problem and furthermore address an outcome effect which can be directed at. If you see that a social situation the effect is of, among other things the effect is of the installed policies by organizations and government, […], then you know that you have to make agreements about this if you want to change it. […] It quantifies the problem’. This is what the politician calls ‘making that which is soft more hard’. He also explains how this differs from the previous method. If there was a problem, for example with disturbance from youth, a city council member would hire a youth worker. This, according to the interviewed politician, is not effective, since the city council member does not know the details and this should also not be expected from him. The social partners know the neighbourhoods from another perspective, they can react to problems more effectively. The local government hands the partners specified targets; ‘the what’, and the partners get to decide ‘the how’.

Research bureaus

(33)

32 The second rapport deals with the ‘social profitability analysis of social council members’ in Beverwijk (Hamdi, de Meere, Verheijen & Hermens, 2011). This is a pilot study of the instrument of the MRA, ‘maatschappelijke rendement analyse’ (social profit analysis). The instrument ‘is meant for municipalities that want to know what effect the instalment of (social) policies (might) have and what the specified cost and benefits are’ (Hamdi et al., 2011, p.23). The instrument is based on the MKBA, ‘maatschappelijke kosten en baten analyse’ (social cost benefit analysis), which is similar to social impact assessment. Cost and benefits of a policy are considered, not only the financial effects, but also non-financial effects. However, as the rapport follows from the rapport it is difficult to express these in financial terms, for this more effect-research is necessary. The outcome of implementing the policy is compared with the ‘zero alternative’, a situation without the policy, in this case without the social council members.

Finally, the researcher recommended the rapport on the evaluation of the neighbourhood teams in Utrecht (de Meere, Hamdi & Deuten, 2013), because the method of this measurement was so different. Before this research took place, another research bureau had performed social cost benefit analyses concerning the neighbourhood teams. The researcher explains: ‘We have said that you should not perform a social cost and benefit analysis; it is too much thinking from behind the desk […]. You don’t know what really happens. […] We have reviewed twenty individual social work routes; what is happening and what would have happened if the old way of working was continued. […] We have only looked at twenty cases, but we find almost identical conclusions to the fifty SCBAs’. This analysis is similar to the social profit analysis, in the sense that all cost and benefits are considered and expressed in monetary values. The difference is the level of analysis, this studies is concerned with individual cases, instead of the overall image.

(34)

33 support program and one later on in the process. Based on this, agreements can be made with residents concerning the improvement of their self-reliance.

Requirements

In the theory section eight requirements of measurement systems were introduced. The first three requirements are general for any kind of measurement; reliability, validity and controllability. The other five requirements are found in organizational theory, more specifically in the work of Lazear and Gibbs (2009), these are the following: risk profile, distortion, scope, match to the environment and potential for manipulation. The further description of the policy measurements will be guided by these requirements.

Reliability

The diversity of measurements allows for testing on reliability, since some topics are discussed in, for example, the city monitor, the monitor livability and safety, the municipal account, the health measurement of the GGD and the youth poll in Groningen. The politician from Groningen compares the findings of the measurements: ‘sometimes it is hard to compare because populations differ, I think, and different research methods. […] This sometimes leads to discrepancies, which makes you wonder what really is going on. In that case you can just ask the council how these numbers should be interpreted’. These variances in results between different measurements have various causes and can be explained by the relevant persons. Furthermore reliability of the measurements executed by the research groups connected to the local governments is strengthened by the use of standardised techniques and random selection of respondents. This is accounted for in the relevant documents (see O&S Groningen, 2013; Rotterdam, 2012; Rotterdam, 2012a).

Validity

(35)

34 In all interviews the distinction between the ‘objective’ and ‘subjective’ views of social impact were discussed. Objective and subjective is written between quotation marks, since not all interviewees agree with the terminology and stress that ‘objective’ numbers are not always objective and that there is a lot of truth in a subjective opinion. However the idea that there is that which is observed by an outsider and that which is perceived by the affected, and that these might differ, was noticeable in all interviews. This can be illustrated by the answer of the politician from Groningen to the question in what way she encounters measurement of social impact. First she explains there is ‘the measurement of social impact in the technical sense’, by this she means using measurements to consider whether set goals were accomplished. She describes that this is what the municipal account does; describing what goals were set and what goals were accomplished. She continues: ‘Further social impact of policies; really every day. You live in the city where you are part of the local council and besides that you get endless mails and phone calls of people asking “this has been decided, that means this for me, so can we talk?” That is also a kind of social impact that I experience daily’. These experiences of the residents of the municipality reflect the subjective aspect of social impact.

The measurements in Rotterdam and Groningen are based mostly on standardised multiple choice surveys. In these surveys people are questioned about their experience on certain issues, such as: to what extent do you feel unsafe in your neighbourhood? (Rotterdam, 2013) The indexes and monitors used in Rotterdam and Groningen do not include stories; no experiences of residents concerning their safety, but a number reflecting on average how safe people feel. These ‘subjective’ numbers are complemented by ‘objective’ numbers about aspects that can be observed, such as the amount of burglaries in a neighbourhood. These subjective and objective aspects do not always match. This was indeed the case for the latest results of the safety index in Rotterdam; there was a strong growth in the objective numbers, but not so much in the subjective numbers. As an article on the website of the municipality of Rotterdam states about the older neighbourhoods in the city: ‘striking is that especially in these neighbourhoods the residents do not experience the positive developments in the objective numbers that way’ (Rotterdam, 2014a, p.1). The new policy is aimed at improving the safety experience before the next safety index measurement, which will be incorporated in the ‘Wijkprofiel’, neighbourhood profile measurement. Concerning this the article states that ‘in advance of the new Neighbourhood profile major Aboutaleb will be talking with residents about the question what they, the police and the municipality can do to improve the safety experience’ (Rotterdam, 2014a, p.1). The major will be looking for the earlier mentioned experiences of residents.

(36)

35 effect on the safety experience’ (Rotterdam, 2014a, p.1). The interviewed researcher states that this discrepancy is observed more often, he has found two explanations: first feelings are influenced by what happens in the street, but also by other things such as what people see on television, second objective numbers might not tell the entire story, for example some crimes might stay unnoticed by the police. These possible explanations show that different domains are interconnected, for example employment and media are connected with feelings of safety. The politician from Groningen explains that interdependencies are becoming more and more important for the impact of policies: ‘the society is continuously becoming more complex, things are continuously becoming more connected and if I press a button here or install a policy there, this might also effect something else’. This is interdependency between different domains is not reflected in the measurements currently in use. The municipal account discusses effects separately per domain. However, the municipal account also discusses the financials for each domain. As the politician from Groningen states about the account: ‘it is the account on both content and finances, so in there are both elements of effectiveness and efficiency’. The indexes and monitors described do not incorporate finances. The methods used by research bureaus do consider both the extent to which the set goal was reached and the resources used to reach this goal. However, since most methods are assessments before the policy is installed, these measurements are based on assessments, rather than the actual outcomes.

In general it can be stated that the interviewees feel that the outcome, the social situation, is the effect of policy among other things, therefore most of the time there is ‘no one-to-one relationship’ between what an indicator measures and an installed policy. The measurements give a good idea of developments in society, but it cannot be said with certainty that an effect is indeed caused (solely) by the installed policy. The following quote of the politician from Groningen illustrates this view: ‘when you read this [the installed policies, actions taken] in connection with a certain indicator, you get a better idea of that connection between policy and outcome, without actually knowing whether that policy is the only factor influencing that outcome. You don’t know that’. The relationship between what a policy wants to accomplish and that which is measured by the indicators of the measurement instruments is further discussed in the paragraph on distortion.

Controllability

MEASURING SOCIAL IMPACT

R. D. Rinsma MA

MEASURING SOCIAL IMPACT

Measuring Social Impact

Robin Doris Rinsma MA

S1690574

Rivierenhof 106

9725HC Groningen

+31636319218

robinrinsma@hotmail.com

Supervisor dhr. J.W.J. De Kort

Co-assessor drs. B. Crom

Master thesis

RijksUniversiteit Groningen

Faculty of Economics and Business

MSc Business Administration – Organizational and Management Control

July 2014

Abstract

Contents

Introduction

Theoretical framework

Social impact

Social impact measurement

Performance measurement systems

Effective measurement

Conceptual model

Methodology

Case study

Policy measurement systems

The instruments

Requirements