Methods to measure Social Performance
How do the methods and their differences connect with user-characteristics?
Amsterdam, July 13th, 2008
M.E. Jansen
Rijksuniversiteit Groningen
Faculty of Economics and Business
Msc. International Business and ManagementMethods to measure Social Performance
How do the methods and their differences connect with user-characteristics?
July 2008 M.E. Jansen
S1342207
Rijksuniversiteit Groningen
Faculty of Economics and Business
Msc. International Business and ManagementPreface
This study is performed as a master thesis for the Master International Business and Management of the Faculty of Economics and Business in Groningen. It was performed as an assignment of Corporate Connect, an investment advisory company focused on alternative products and emerging countries. The aim of this study is to identify the differences between the methods to measure impact and the degree to which the methods fit the needs of their users.
In order to give an answer to this, two case studies were performed. The first case study was performed at Aqroinvest, a Microfinance Institute in Azerbaijan as an assignment of Oikocredit. The other study was performed for the North Star Foundation, a cooperation between TNT and the World Food Program.
A number of people were very helpful during this process and should receive special thanks. First of all, I would like to thank my supervisors at Corporate Connect, Bernadette Blom and Tanja Pelle. Furthermore, I would like to thank Oikocredit for their time and support. Also, I would like Aqroinvest, especially Tamilla Babayeva and Ayyub Ahmadov, for giving me such a warm welcome in Baku. Also the people from NSF, especially Esther Bosgra and Luke Disney, were very helpful during the whole process of performing the analysis. Finally, I want to thank my supervisors of the faculty of Economics and Business, dr. B.J.W. Pennink and prof. dr. L.J.R. Scholtens, for their patience and constructive advises.
Table of Contents
Preface ___________________________________________________________________ 4 Table of Contents ___________________________________________________________ 5 1. Introduction___________________________________________________________ 7 1.1 Terminology social performance measuring _________________________________ 7 1.2 Developments in social performance measurement ___________________________ 8 1.3 Research objective ____________________________________________________ 10 1.4 Outline______________________________________________________________ 11 1.5 Summary ____________________________________________________________ 11
2. Methodology _________________________________________________________ 12 2.1 Type of research ______________________________________________________ 12 2.2 The user groups_______________________________________________________ 12 2.3 Determining the important criteria for the user groups________________________ 14 2.4 Choice of methods included in the research ________________________________ 15 2.5 Cases _______________________________________________________________ 16 2.6 Framework for analysis _________________________________________________ 17 2.7 Limitations of the research ______________________________________________ 21 2.8 Summary ____________________________________________________________ 21
3. Requirements and needs user groups______________________________________ 22 3.1 The respondents ______________________________________________________ 22 3.2 The output of the questionnaires _________________________________________ 22 3.3 Similarities and differences______________________________________________ 25 3.4 Summary ____________________________________________________________ 27
4. Case Analyses_________________________________________________________ 28 4.1 Case 1 Aqroinvest _____________________________________________________ 28 4.2 Case 2: North Star Foundation ___________________________________________ 50 4.3 Summary ____________________________________________________________ 64
5. Evaluating the methods_________________________________________________ 65 5.1 Introduction _________________________________________________________ 65 5.2 Reliability____________________________________________________________ 65 5.3 Validity _____________________________________________________________ 66 5.4 Accuracy ____________________________________________________________ 67 5.5 Comparability ________________________________________________________ 67 5.6 Time needed _________________________________________________________ 68 5.7 Quantifiability ________________________________________________________ 68 5.8 Monetization_________________________________________________________ 68 5.9 Summary ____________________________________________________________ 68
Conclusion _______________________________________________________________ 73
References _______________________________________________________________ 77
1. Introduction
To determine the success of a project or investment, performance measurements are commonly used. Companies have been measuring performance for a long time. Performance measurement is defined by Baruch and Ramalho (2006) as “determining the magnitude of the effect or impact the organization has over whatever object one might want to analyse, being the employee, the employee’s family, the competition, the community, the market or the society at large”. When focusing on social performance, measuring is a lot less common. Performance measurement in organizations with a (partly) social mission is relatively new.
Social performance refers to “a systematic analysis of the effects of an organization on its communities of interest or stakeholders” (Quarter, Mook, and Richmond, 2003). The aim of measuring social performance is to get a clear understanding of the effects that activities and organizations have on society. The ultimate goal of the development of including social performance in measurement, is to create a wholly integrated accounting system that tracks, documents, and analyzes financial and social data, in order to give insight in the value proposition of capital investments and the returns generated by them (Emerson, 2003; Gowdy, Emerson and Gair, 1999.) But until now, standard methods or common agreement on this topic has not yet been accomplished.
Within this research, different methods to measure social performance will be analysed in depth. This introductory chapter will first discuss the important terminology in the field of social performance measurement. This is followed by an overview of the developments of social performance measurements. Furthermore, the research objective as well as the outline of the thesis will be given in this chapter.
1.1 Terminology social performance measuring
Since the topic of evaluating social performance is relatively new, it is important to create a common understanding of the terminology that is used. Within the subject of performance measurement, the aspects input, output, outcome and impact are important aspects. The figure below gives an outline of the relationship between the different aspects. A further explanation of the aspects is given below.
Figure 1: Important aspects in social performance measuring
By performing an activity, the input will be transformed into output, the result of the activity or investment. Based on Clark et al (2005), output is here defined as aspects that are a result of an organization’s operations, which are directly measurable.
The output of the activity or investment has an effect on the people where the activity or investment was focused on. These effects can be defined as the outcome. Again, the definition is based on Clark et al (2005), who define outcome as “specific changes in attitudes, behaviours, knowledge, skills, status, or level of functioning of the people where the activity or investment was focused on.” The difference between outcome and impact is the attribution. Since there are often a number of inputs that have an effect on an outcome, such as healthier people, less poor people, improved local employment etc., it can be difficult to determine the effect of one input. Attribution concerns with the effort to focus only on the effects that are a clear result of the activities that has taken place. In other words, impact tries to exclude the effects that are a result of external inputs. Finally, it is important to note that the result of an activity can be either a qualitative, quantitative or a monetized outcome.
The example below can help in understanding the difference between the aspects outlined above.
A Microfinance Institution (MFI) can be considered as an organization with a both a financial and social objective. The goal of an MFI is to create a social impact and at the same time be sustainable. (Some Microfinance Institution may not have financial objective, but a social one only.) To perform their activity, which is providing loans, several inputs are needed. Examples of the inputs of an MFI are the money invested and the time the employees are working for the MFI. Outputs of the activities of the MFI are the number of loans provided or the total amount of loans disbursed. These aspects are a direct result from an organization’s operations that can be measured. The next step is to identify the changes in attitudes, behaviors, knowledge, skills, status, or level of functioning of the people where the activity or investment was focused on. Possible outcomes of providing loans could include increased income or living standard of the borrowers of the MFI. For determining the impact of the MFI, one should decide which part of the change in income or increased living standard of the borrowers is a result of the loans provided by the MFI. Possible impact could be the increased living standard of the borrowers, which is a direct result of the loans provided. To get insight in this, the effects of other factors, such as teaching programs provided by another organization, should be excluded.
1.2 Developments in social performance measurement
For decades, the common belief has been that financial and social goals are in opposition (Emerson, 2003.) In the last few years, however, there has been a change towards more social awareness within corporations. Several authors that support social accounting theories state that all investments operate simultaneously in economic, social, and environmental realms. (Emerson, 2003; Olsen, 2003 and Lingane and Olsen, 2004). Instead of a “trade off” between the three, a “blended value” model, or “double bottom line” is preferred in which organizations achieve both economic success and maximize social benefits, as articulated by Emerson.
tangible accountability for the social impact created for each invested or granted dollar in the philanthropic sector (Clarkson, 1995; Emerson, 2003; Callens and Tyteca, 1999; Clark et al, 2004.) As a result of these developments, there is a need for performance methods capable of measuring and maximizing this new combination of financial and social value. (Kennerley and Neely 2002.) As described by Emerson and Wachowicz (2000) “What gets measured, gets managed and therefore, management attention should not exclusively be focused on financial results”.
Several authors studied the possible relationship between social and financial performance. (Margolis and Walsh, 2003; Otlitsky et al, 2003) Contrary to what was believed earlier, investors start to acknowledge the possibility to invest in a socially responsible manner without giving up the opportunity to achieve higher rates of return.
Lysy (1999) is one of the first who tried to include social aspects in performance evaluation with his Economic Rate of Return method (ERR). The ERR method aims to include all the project’s costs and benefits. Lysy calculates the ERR by adding the benefits and cost of employees, customers, suppliers, competitors, producers of complementary products, neighbours and the rest of society to the financial return of a project. In his articles Lysy stated the importance of measuring impact. Later several other authors repeated this, such as Wachowicz, (2000), Nayar (2006), and Moneva, Lirio and Munoz-Torres (2007). In the article of 1999, Lysy explains and demonstrates his ERR model with the use of a numerical example. This article is considered to be one of the starting points of the social performance measuring.
After Lysy, several other authors discussed the topic of including social aspects in evaluation. Jed Emerson introduced the expressions of “Double bottom line”, “Triple bottom line” and “blended value”, dealing with the valuation of financial, social and environmental aspects. (Emerson, 2003; Smith, 1996; Emerson, 2003; Callens and Tyteca, 1999; Mangolis and Walsh, 2003; Clark et al, 2004; Lingane and Olssen, 2004.)
In spite of the development in the field, relatively few articles discussed the methods available to value the results of social and environmental efforts of organizations. An article of Margolis and Walsh (2003) does give an extensive overview of articles dealing with corporate social performance. However, most of the articles mentioned by Margolis and Walsh focus on the relationship between social and financial performance. Moreover, social performance in the articles discussed, was indicated as the degree of disclosure of social performance aspects and the company‘s rating on reputation, rather than determining the outcome or impact of organizations.
The growing interest of measuring social performance in both the commercial and non-profit sector is a topic that has been discussed by several authors. Copestake (2006) for example, discusses the possibilities for financial and social performance assessment in the context of the increasing integration of specialized microfinance and commercial banking sectors. The article of Clay (2005) describes the case study executed by Oxfam Novib and Unilever to determine to what extend the Unilever Indonesia’s business is having an impact on the poverty in the country. Here, a non-profit and commercial organization worked together to determine the effects. A similar cooperation can be found between NCDO and Heineken in 2006, who developed a method to measure economic impact of the activities of Heineken.
both do an attempt to give an outline for indicators for measuring social impact (in order to base their recommendations for a new impact measurement method). A study done by Clark et al (2005) compares nine social impact assessment methods on several fixed criteria. The evaluation within the catalogues consists of a description of the methods and its usability; the aspects of the value chain considered in the methods, applicability to the different lifecycle changes, assessment purposes and the time and costs of performing the analysis. Their information is mainly based on interviews with funders. The authors mention that other users could interpret methods differently. The catalog discusses both methods focusing on process and methods focusing on impact. The authors selected four methods that suppose to measure impact.
As is described above, several researches have been performed in the field of measuring impact or social performance. Some tried to compare methods that are being used; others focus on the aspects or indicators used in the different methods. But, as Figge and Hahn (2004) point out, evaluating the strength and weaknesses of different methods has taken place is some case but so far, comparing methods did not happen very often.
Both commercial and non-profit organizations seem to have an interest in measuring social performance. However, as mentioned by Clark et al (2005), it could expected that different groups consider different aspects to be important within methods to measure social performance. This was also mentioned in a paper from the Global Reporting Initiative to introduce G3 (2006), that pointed out that different stakeholder groups evaluated the previous GRI method differently.
1.3 Research objective
Papers such as the paper of Carlson, Sinha and Nayar show that the development of methods to measure social performance is still in progress. This is not surprising, considering the fact that the developments in this field have only been started a relatively short time ago. Financial performance methods have experienced a similar process of development.
In this relatively early stage, efforts are made to develop a standard method to measure impact. To determine if this is desirable and to determine which aspects of each method are important, it is important to first analyze the methods already available in the market and to understand which aspects of social performance measurement methods are considered to be important. This research will analyze and compare methods that are available in the market and will link the evaluation to the requirements of the user groups of social performance measurement methods.
The goal of this research is to identify the differences between the methods to measure impact and the degree to which the methods fit the needs of their users. Based on the issue raised above, the following research question has been developed:
How are the different methods to measure social performance evaluated and how does this relate to the requirements of the user groups?
Since the character of this research is an explorative one, no hypotheses are used. Instead, the following sub questions are formulated and will be tried to answer in this research.
• What are the different users that can be identified and how they differ from each other?
• What are the differences between the methods to measure impact?
• How are the different methods evaluated?
The aim of this research is not to give a judgment of which is the best method, but rather to indicate what differences exist between the methods and to what degree the difference aspects fit the requirements of the different users. In order to get this information, a systematic comparison and analysis will be performed. This insight is expected to add value to the discussion and development of methods to measure social performance.
The added value of this study to the existing literature is that it will give insight in the differences between methodological aspects of the methods as well as requirements of the users. So far, some papers have compared different methods to measure impact. Clark et al have done the most up to date analysis. This research goes further than the research of Clark et al in two ways:
1. This research takes the differences between different user groups into account. As mentioned earlier, it could be expected that different aspects of a method are considered to be more important by one user group then the other. By taking these aspects into account, it becomes easier to understand which methods are most suitable for a certain type of user and why.
2. To analyze the methods, the study makes use of real-life cases to base the evaluation of methods on. By using actual case studies where social performance will be measured, the similarities and differences between the different methods will become clear, since all steps and considerations that should be made will actually be made.
1.4 Outline
After some background and literature on measuring social performance or impact, the next chapter will discuss the different aspects of the methodology of this research. The type of research, data collection, expected characteristics of the users and the framework with criteria for the analysis will be discussed more in detail. Also, an introduction to the case studies is included in the methodology. After outlining the methodology, the analysis on the different methods will be performed. After the methodology, the third chapter will discuss the requirements that the different potential users have for a method to measure impact. To determine the differences between the methods to measure social performance, a case study will be performed in chapter four. The following chapter will evaluate the methods based on the criteria defined in the methodology. These results are expected to give an answer to the question what the differences between the methods are. These requirements will be compared to the characteristics of the different methods to determine if the differences in the methods can be related to the characteristics of the potential users. Finally, a concluding chapter will give a finale overview of the findings in this research.
1.5 Summary
2.
Methodology
Within this chapter, the methodology of this research will be outlined. The chapter will start by describing the type of research that will be performed, followed by a description of the groups of users who will use a method to measure social performance. Since this research will review the requirements the different user groups have with regard to such methods, this chapter will also give an outline on the way the requirements of the users are determined. Furthermore, the methods that will be analyzed are described. In this research two case studies will be used to perform the analysis of different social performance analysis. These cases used will be discussed, as well as the criteria on which the methods will be evaluated.
2.1 Type of research
As mentioned in the previous chapter, the goal of this research is to identify the differences between methods to measure social performance and to determine whether the methods fit the requirements of the different user groups. To get this information, a systematic analysis and comparison will be performed. This will be done by means of a case study. A case study is considered to be the most suitable methodology for this research to make the differences in the methodology and outcomes of the methods as visible as possible. By actually performing the methodology of the methods, it becomes clear which steps need to be taken in order to determine the social performance. Moreover, doing the case studies will give insight in the amount of time that is needed to collect and analyze the data needed for calculating the social performance. Finally, performing the case studies will make analyzing the methods more tangible as well.
This case study, with a descriptive nature, can be considered as an exploratory study, since it is the first case study on this topic. The aim of this research is not to study the full range of methods available or to give a complete evaluation of the method. Rather, this study aims to give a first evaluation, based on a number of criteria that will be outlined in paragraph 2.6. This information will then be compared to the requirements of the users. This will enable the reader to get an understanding of the degree to which current method fit the requirements of the users.
Robert K. Yin (2003) describes a case study as “an empirical inquiry that investigates a contemporary phenomenon within its real-life context.” Critics of the case study research argue that a study of only a small number of cases cannot offer grounds for generalizing the findings or the establishment of reliability and is in this way only useful as an exploratory method. However, as mentioned before, it is not the objective of this research to generalize the findings of the study, but rather to give a first insight.
2.2 The user groups
The first sub question stated in the introduction is the following: “What are the different users that can be identified and how do they differ from each other?”
stakeholders (see appendix 1), these are the actors that are considered to be the most important ones. The roles are based on money flows. This is because it is considered in this research that the methods are used to evaluate the effect of money flows.
Figure 2: Actors of interest
Donor: Actor that provides money to the recipient with the objective to create maximal impact. No requirement of financial return.
Investor: Actor that provides money to the recipient with the objective to get a financial return and to create impact.
Claimant: Actor that provides money to the recipient. The primary goal is not to have a maximal impact, to get a financial return or a combination of those two. Actors in this role invest money to create impact and mainly for other reasons such as pr, marketing or to connect their employees to the company. Recipient: Organization that receive money and with this money activities with a
(partial) social goal are performed.
Rating Agency: Agency that rates the performance of organizations. Until now, this evaluation is mostly focused on financial performance only.
Regulator: Actor that does use such a method itself but tries to regulate the use and requirement of measuring social performance.
Target Group Persons, households or other units who participate in the program or receive services from the recipient.
Within this research, not all roles will be taken into consideration. First of all, the final recipient will not be taken into consideration. In the example of the MFI in chapter one, the end borrowers will be classified as the final recipient. The reason not to take this role into consideration is mostly practical. Final recipients are hard to reach. Furthermore, it could be expected that end borrowers as in the MFI example do not have a good understanding of the different requirements of the methods. Finally, it is expected that the initiative for using such a method would come from the investor, donor or claimant. Investors, donors and claimants can use the results of the methods to measure social performance to base their decisions on. For recipient, the results will probably be used for justification of their receivables. The importance of the requirements within this research is based on
questionnaires filled in by the different users. Based on the reasons above, it is decided not to send questionnaires to the end final recipients.
Regulators are also not included in this research. The most important regulator would be the government. Since different parts of the Dutch government decided not to fill in the questionnaire or cooperate with this research, little can be said about their requirements and needs with regard to methods to measure social performance. Based on their decision it could also be expected that there is no strong interest and regulation from the government with regard to the activities and methodologies of social performance measurement. Therefore, it is expected that the influence of this role could be neglected.
Finally the rating agencies will not be included in the research. From the four biggest rating agencies that were contacted, only one responded. Standard and Poor’s is the only agencies that is active in the field of sustainability. Because Standard and Poor’s was the only agency that was willing to fill in the questionnaire, it is decided not to take them into account as a separate user group.
The remaining three roles are expected to have the most interest in measuring social performance. As a result, it is expected that their requirements of the methods to measure social performance are the most important to take into consideration when evaluating these methods. These are the following three:
• Investor,
• Donor, and
• Claimant.
The differences between the requirements of these users will be compared to each other and linked to the characteristics of the methods.
2.3 Determining the important criteria for the user groups
To be able to determine which criteria are considered to be important by the different user groups, the opinion of the users regarding their needs and requirements should be identified. Within this research, a questionnaire is used to determine the importance of these criteria. By means of this questionnaire, the users will be asked to rank how important they consider the different reasons for using impact methods and which of the criteria used in the analysis they consider being the most important. The aspects that are taken into consideration are based on the issues mentioned by the respondents of the interviews held before making the questionnaire (Appendix 1). The questionnaire can be found in Appendix 2.
The results of the questionnaire should give an indication of the differences between the different users, related to the requirements of a social performance measurement method. Based on this information, an answer can be given to the fourth sub question, referring to the relation between user-characteristics and the different social performance measurement methods. The reason for choosing for a questionnaire is that is enables to reach a bigger group in a relatively small time period.
the different reasons, depending on how important they regard these reasons. The same two ways of questioning are used for determining the importance of the different criteria as well. Using the two ways of questioning together makes it possible to determine which aspects are considered to be the most important, but also to understand how the aspects are ranked separately from each other.
The total number of questionnaires taken into consideration is 30, with ten questionnaires for each of the three user groups.
2.4 Choice of methods included in the research
There are several methods that claim to measure social performance and could be considered to take into account in this research. The choice of methods taken into consideration within this research is based on practical issues. To be able to take a method into consideration within this research, the following requirements should be in place:
• The methodology of the method should be completely available,
• The analysis should enable an external party (the researcher of this study) to conduct the analysis,
• The methodology should be suitable for the case studies that will be performed, and
• The methods used should give the possibility to conduct the analysis within a relatively short time period. (Not taking more than two weeks to perform)
In appendix 5 a list of all known methods are described shortly. It should be noted that the topic of measuring social performance is still very much in development. Therefore, it is possible that more methods are available at this moment. However, after extensive literature research and interviews with several stakeholders, it is expected that a relatively complete list is provided here.
Based on the selection and several practical issues described above, the following method will be analyzed:
Methods Description
Social Return On Investment This method consists of several steps used to get an understanding of the organization and give a monetized value of the social and financial impact of the organization. Within this method the different stakeholders will be identified. Based on these stakeholders, an impact map will be made where the input, output, outcome and impact of these stakeholders need to be formulated. By using monetized indicators for the impact formulated in the impact map, a total impact can be calculated. In order to only calculate the impact created by the organization, the value of “what would have happened anyway” should be detracted from the value calculated. Finally, a social return for each dollar/euro/other currency will be given.
Development Outcome Tracking System DOTS aims to address how well a project has contributed to fulfilling IFC’s mission, which is to promote private sector development in developing countries as a way to reduce poverty and improve people’s lives. The same evaluation can be used for evaluating other projects or organizations. Providing a “development outcome rating” for projects does this. DOTS intends to capture the overall impact of a project. Development outcome is rated on a 6-point scale rating from highly successful to highly unsuccessful, and the top 3 ratings are considered a “success”.
host economy. These four components are: • Financial performance
• Economic performance
• Environmental and social performance • Private sector development
New Profit Balanced Scorecard The following method used is the New Profit Balanced Scorecard. The methodology is based on two tools: the strategy map and the Balanced Scorecard. The strategy map provides a visual communication framework that enables everyone in an organization to clearly understand how their actions help drive the organization towards its mission and vision, while to Balanced Scorecard itself is a measurement and management tool for tracking an organization’s performance along the various perspectives. The methodology of the Balanced Scorecard is based on the scorecard of Kaplan and Norton, which looks to four aspects of an organization:
• Financial • Customer
• Learning and growth • Operation
New Profit added a fifth aspect to the four mentioned above, which is social impact.
For each of the five aspects, the objectives and goals and measurements should be formulated. In this pilot it is decided to add an extra step to this method. A status list is made, to give insight in the developments of the goals and measurements formulated so far.
Cerise SPI-2 The methodology of this method has three different phases. Within the first phase, the strategy and context of the organization studies is reviewed in order to get a better understanding of the organization and its focus. Within this phase, the organization is asked to rank it focus based on four aspects. Within the second phase, the method determines how the organization scores on these four aspects, by making use of a questionnaire. The result of this questionnaire is a score on the four aspects, which can be compared to the importance the organization has given to these aspects beforehand. The final phase is focuses on the financial sustainability, assuming this to be necessary to perform well. Best Available Charitable Option This method evaluated the project on their costs, social output and
bottom of the pyramid penetration, compared to the Best Alternative Charitable Option (BACO). Derived from this data, the cost per number of impact is calculated and compared. Based on this information it could be determined whether the project will be worth the investment.
Table 1: Description of the methods analysed
Appendix 3 will give a more extensive description of the methodology of the methods that will be taken into account in this study.
2.5 Cases
Furthermore, it is decided to use a case study within the microfinance sector and one case study outside the microfinance sector. The microfinance sector is a sector that is far advanced in measuring social performance. Within the sector, a working group is created to improve social performance measuring and there are a number of methods to measure social performance focusing on the microfinance sector only. Doing a case study within this sector is therefore considered to be interesting. But because the microfinance sector is a financial sector, one could expect measuring performance to be relatively easy. Therefore, it is decided to include another case study, which is not performed for the microfinance sector. By using a method a method inside and outside the microfinance sector, possible differences in the results can be identified.
Two organizations will participate in the case studies. The first case study is a case on Aqroinvest, a Microfinance Institute in Azerbaijan. Aqroinvest is a credit union based in Baku, the capital of Azerbaijan. The credit union provides micro loans to small and medium business in southern provinces of Azerbaijan. The second case study will be done for the North Star Foundation (NSF), a foundation dealing with HIV/AIDS prevention in Africa. The North Star Foundation is a public-private partnership that is establishing health care clinics at major truck stops and border crossings in Africa, India and Asia. Wellness Centres an offer practical, low-barrier and low-costs response to the transmission of HIV and other sexually transmitted infections (STIs) among the transport industry.
The aim of both case studies is to get insight in the social performance of the organizations by using different methods that measure social performance. A full description of the two organizations as well as a description of the project being analysed can be found in Appendix 4.
For the collection of data for the two analyses, we made use of resources that were available within the organization and on Internet. For the analysis of Aqroinvest, interviews were held with both within the organization as well as with the clients of the Microfinance organization. Within the second case, there was no possibility to do direct interviews with the clients. However, there was the possibility to request certain questions indirect, through the NSF team.
2.6 Framework for analysis
To be able to evaluate and compare the different methods to measure social performance, a number of criteria should be used to base the evaluation on. In the general literature on evaluation many different and sometimes overlapping aspects are considered to be important. The most well known requirement is that performance indicators should be SMART (Specific; Measurable; Acceptable; Realistic and Timely). (Drucker, 1954) Bastien and Scapin (1995) identified three measures for examining an evaluation method: thoroughness, validity, and reliability. Sears (1997) also pointed out these same measures, giving them somewhat different operational definitions. Hartson, Andre and Wiliges, (2003) and Gill and Johnson (2003) also define validity and reliability as important criteria for performance measurement methods. Other authors, such as Scott (1995), Swanborn, Artely et al (2001) and the GEMI-organization (2007) mention a set of criteria more focused on the users rather than methodological requirements. They mention criteria such as relevance, enabling tracking of results, providing feedback, objectivity, supporting costumer and business requirements, appropriate standards, number of indicators, understandable and cost-effectiveness.
thus be discussed in the analysis. Below, the aspects that will be analyzed are discussed more in detail. The reasoning why the criterion is considered to be important is outlined shortly. Furthermore, the criteria are defined.
Criteria/importance/ definition
Importance Definition
Reliability Methods to measure social performance are used to get a more reliable insight in the effects of investments. Therefore, an important aspect of the method is reliability.
Reliability is the consistency of your measurement, or the degree to which an instrument measures the same each time it is used under the same condition and subjects. (Black, 1993; Rossi and Freeman, 1989; Posovac and Carey, 1985) Mehrens and Lehman (1984) provide a somewhat different definition: “ Reliability can be defined as the degree of consistency between two measures of the same thing”. The definition of reliability used in this research is based on Joppe (2000); a measure is considered reliable if it would give us the same result over time (assuming that what we are measuring is not changing) and when the results of a study can be reproduced by another person or similar method.
Validity The method used to measure social performance should measure what is aimed to measure. Social change is hard to be defined and measured. The indicators used should measure what you aim to measure and correctly reflect changes in the program. A method should not take effects into consideration that are not the result of the organization or project studied. This is also called attribution.
Cook and Campbell (1979) define validity as the "best available approximation to the truth or falsity of a given inference, proposition or conclusion." Or, as Black (1993) puts it, the instrument should measure what it aims to measure. A distinction can be made between internal and external validity. Internal validity refers to whether the subject that is identified as the ‘cause(s)’ or ‘ stimuli’ actually produce what have been interpreted as the ‘effects’ or ‘responses’ (Gill and Johnson) or as “the degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based”1 With regards to social performance measurement this could
For this research, it is most relevant to look at internal validity, since it is not the aim of a method to generalize the outcome outside the selected research. Internal validity in this sense looks at whether a method should measure what it intends to measure. (Artley, 2001; Rossi and Freeman, 1998; Posovac and Carey, 1985)
Accuracy Next to measuring what you aim to measure, the data should be collected accurately to get the most reliable outcome. This enables us to be able to expect objectivity.
Accuracy is defined in the dictionary as the ability of a measurement to match the actual value of the quantity being measured.2 In this sense, that means measuring and using correct data. It should be clear what the method measures, how it is measured as well as how the indicators are selected. This is referred to in the scientific literature as sensitivity and specificity. Sensitivity is the ability of an indicator to identify a characteristic correctly when it is present, and specificity is the capacity to identify correctly the
1
(http://www.socialresearchmethods.net/kb/considea.php) 2
absence of a characteristic. (Conner, 1981) Since sensitivity and specificity is hard to determine, data is considered accurate when it is verifiable and auditable. The definition of verifiable used is “observable to outsiders, in the context of a model of information.” A method is auditable when there is a possibility to examine for accuracy.
Comparability One of the objectives of using a method to determine impact is being able to compare results, do a benchmark or track results. Evaluating a method on its ability to compare different result is considered to be important.
Comparability can be defined as being able to compare the outcome of a project or activity to outcomes of other projects or activities.
Time needed The time element often seems to be neglected in consideration of method selection for evaluation and monitoring purposes (Conner, 1981). However, for an indicator to be useful, the results of the methods, or the degree to which the results are used, should be bigger than the costs of performing the measurement.
Time needed is defined as the total time the researcher of this paper needed to complete the analysis of a method.
Quantitative/ qualitative/ monetization
The social performance can be expressed in different ways. A qualitative or monetized value is easier to compare. But between monetized and qualitative there is a big difference as well. Since the difference in quantitative, qualitative and monetization has influence on the comparability and time needed for performing the analysis, it is an important criterion to take into consideration.
A quantitative outcome is a description expressed without numbers. A qualitative outcome is an outcome that is expressed in numbers. Monetization goes even a step further than the quantitative outcome by expressing a financial value.
Table 2: Criteria for evaluating methods to measure social performance
2.6.1 Operationalization
This paragraph, discusses the operationalization of criteria will be given.
Reliability
Whether a method can be reproduces by another person, depends on the number of assumptions made by the person performing the measurement and the number of decisions that need be made by the person who is performing the measurement. These two aspects are important, because they are based on the judgement of the person performing the measurement. This can differ between persons and therefore affect the reliability. A decision to make is defined here as choosing for an alternative without being able to make use of guidelines. An assumption to be made is defined here as taking a decision not based on information directly available.
Validity
the actual result in terms of output vs. outcome vs. impact. Furthermore the evaluation will take into consideration whether there is a methodology available to determine the actual attribution of the organization to the total impact.
Accuracy
Two questions will be asked to determine a method’s verifiability and accuracy:
• Is there is a standard methodology in the analysis on which the outcomes are analysed and which is a required aspect of the analysis?
It can be difficult to actually determine the performance. This question deals with the way it is determined how the performance is. What is good, what is bad?
• Is there a policy in the methodology, which requires more data sources to be used for the analysis?
Using multiple sources of information enables to give a broader picture. As a result, there is bigger change of giving an accurate valuation.
Comparability
To be able to compare result of different analysis with the same method, the indicators that are used should be equal or at least similar. Also, outcomes that are quantified, or at least classified are easier to compare.
Time needed
To be useful, a method should be cost effective. The benefit of using a method should outweigh the costs. Therefore, it should not cost too much time to perform the analysis. Time needed is defined as the total time the researcher of this paper needed to complete the analysis of a method
Quantifiable/Monetized
The final criterion deals with the possibility to express the result of the method in a number or a financial value. The result is quantifiable when the result (social performance) can be expressed in a number. Also when comparing is possible, it is classified here a quantifiable method. A method has a monetized result when the social performance is expressed in a financial value.
The table below summarized the operationalization of the methods that will be used for the analysis of the methods.
Criteria Operationlization
Reliable No. of assumptions to be made by the researcher No. of decision need to be made by the researcher
Validity Result (output/outcome/impact) in line with aim (yes/no) Methodology for attribution in place (yes/no)
Accurate Fixed Methodology to measure the social performance (yes/no) More data sources required? (yes/no)
Comparable Same / equal indicators every time (yes/no) Quantified/ classified outcomes (yes/no)
Time needed Time needed to complete the analysis of a method (in hours)
Quantifiable Social performance is expressed in a number/ can be compared to other social performance results
Note that the definitions of the aspects cannot be tested completely objectively. However, they do provide guiding for evaluation and analyzing the different methods. The outcome of the criteria is either a yes/no-outcome, or a number. This makes comparing the evaluation of the different methods easier.
The questionnaires will be used to determine which criteria the different user groups consider to be important. After performing the case studies, the different methods will be evaluated on the same criteria. These findings of the analyses can be compared to the requirements of the users. As a result, insight will be given into the extent to which the different methods fit the requirements of the users.
2.7 Limitations of the research
The fact that this topic is relatively new and there is still a lot of development makes it is hard to determine if a complete list of methods available is given. Also, since there is still a lot developed, some methods could not be used because they are still finishing or improving the methodologies. The research aims to give a first indication of the types of methods available in the market. Future research may include other or extra methods, such as the methods still in development described in appendix 5.
Also, there is a limitation in the possibility to generalize the findings of this study. However, as mentioned before, the aim of this study is to give a first indication of the requirements. Further research may be necessary to include all the different user groups and more questionnaires are needed to get a generalizable result.
In order to be able to compare the methods, aspects of the criteria are formulated. The researcher of this study has formulated these aspects. This may reduce the reliability, since another researcher may use other aspects. For further research, it may be useful to look into this further.
2.8 Summary
The aim of this chapter was to explain the methodology of research that will be performed and to set the boundaries for performing the case study. The aim of this research is not to determine one “best” model. Since it is the first paper to analyze the differences between different methods based on a case study, the study has an exploratory focus. Therefore, a case study with a descriptive nature seems to be the most suitable the type of study to be used.
Furthermore, in order to perform the analyses in the next chapter, the criteria on which the case study will be analyzed are operationalized in this chapter. To determine the reasons for using a method to determine social performance and to get insight in the criteria considered important for selecting such a method, a questionnaire is used. It is assumed that different user group may have different requirements and needs. Therefore, the responses of three different groups will be compared in the following chapter. The description of the different methods will be done by means of a case study. The methodology of the methods is described in detail in Appendix 3. A framework with the criteria on which the methods will be analyzed is presented in table 2. These criteria are based on literature and interviews with the users of the methods. Every method will be evaluated on the same aspects described in table 2.
3.
Requirements and needs user groups
Different user groups have different roles and different interests in measuring social performance. Therefore it can be expected that different user group may have different requirements and needs. Within this chapter the interests and requirements of the different user groups will be compared. The comparison is done based on the respondents of the three different user groups on the questionnaire in appendix 2. This chapter will compare the responses of the three user groups with regard to the criteria considered important for a method to measure social performance and the reasons for using such methods. First, a more detailed insight will be given into the respondents of the questionnaire. After that, the output of the questionnaires will be discussed. In order to determine if there are differences in the needs for using social performance methods, the similarities and differences will be outlined. Finally, a short summary will be given.
3.1 The respondents
Three different groups of users were asked to fill in these questionnaires. By making a distinction, possible differences between the groups can be identified. The three user groups that are distinguished are:
• Investors
• Donors
• Claimants
Chapter 2 already gave a description of the different groups. Investors were described as actors that provide money to the recipient with the objective to get a financial return and to create impact. The respondents of the questionnaire within this research are mainly investors in microfinance, as for example Oikocredit. Claimants were also described as actors providing money to claimant, with a mainly social focus instead of a financial or combined focus. Respondents classified as donors within this research are both the CSR-type of departments of commercial organizations and foundation. Finally, the claimant was describes as the actor providing resources to the recipients. Examples of claimants are the two organizations where the case study is performed on. For each user groups, ten respondents are used in the analysis, which makes the total amount of respondents thirty.
3.2 The output of the questionnaires
The respondents were asked to rank how important they considered different reasons for using a method to measure social performance and how important they considered different criteria of measurement methods. First, it was requested to rank each aspect on a scale of zero to five and after that 100 point should be divided over the same aspects.
In the figures below, the number 1 to 7 on the horizontal axis, indicate the aspects that were taken into consideration within the research. Per user group, the criteria considered to be important for a method to measure social performance were discusses, as well as the reasons for using a method. These are the aspects as mentioned in chapter two. The criteria that were taken into consideration are the following (with the first one mentioned indicated as 1 and the last one mentioned as 7): The reasons for using a method are the following (with the first one mentioned indicated as 1 and the last one mentioned as 7):reliability, validity, accuracy, comparability, time needed, quantifiable and the possibility to monetize.
The numbers 1 to 5 on the vertical axis indicate the number that has been given to rate the importance of a criterion. One (1) indicates the criteria is defined as not important at all and five is indicted as very important.
Finally, the 1-25 refers to the average number of points given to one criterion by a certain user group, when 100 points were divided. The maximum is 25 because there were no average scores higher than 25.
3.2.1 Investors
The responses of the investors give the following result with regard the criteria considered to be important for a method to measure social performance. Based on ranking from 1 to 5, all criteria are considered to be important, with reliability being the most important one. This is visible in figure 3. When 100 points are divided over the criteria reliability is again the most important one, followed by comparability and the time needed to perform the analysis. This is graphically outlined in figure 4.
Fig. 3: ranking requirements SP methods Investors Fig. 4: dividing 100 points over requirements Investor
The figures below indicate that the responses of the investors grade all the reasons for using methods to measure social performance above 2.5 out of five. Dividing the 100 points gives a broader distinction in importance of the different reasons. Based on this output, investors consider determining impact and possibilities for managing the most important reasons for using a method to measure social performance. Determining payment is considered to be the least important reason.
Fig.5: ranking reasons for using SP methods Investors Fig. 6: dividing 100 points over reasons Investors
3.2.2 Donors
criteria reliability is considered most important and validity is the second most import criterion. Having a monetized value is considered least important.
Fig. 7: ranking requirements SP methods Donors Fig. 8: dividing 100 points over requirements Donors
Again, the figures below illustrate a high score for all reasons mentioned in the questionnaire, based on the ranking from 1 to 5. By dividing the 100 points, a strong preference of the donors can be identified for the fourth reason for using social performance, which is determining impact.
Fig.9: ranking reasons for using SP methods Donors Fig.10: dividing 100 points over reasons Donors
3.2.3 Claimants
The final user Group that is taken into consideration is the claimants-group. With regard to the criteria considered to be important, two criteria have the highest score and are thus considered to be the most important indicators. These two criteria are reliability and time needed to perform the analysis.
The different reasons for using a method to measure social performance show a relatively equal distribution, based on the ranking of the reasons separately. Dividing the 100 points indicate that claimants consider managing the most important reasons for using a social performance measurement method. The reason that is considered to be second most important is to determine impact.
Fig.13: ranking reasons for using SP methods Claimants Fig. 14: dividing 100 points over reasons Claimants
3.3 Similarities and differences
Based on the results outlined above, a number of similarities between the responses of three user groups can be identified. All three groups rank certain criteria higher. Reliability, for example, is considered to be the most important requirement for a social performance measurement method by all three groups of users. Time needed and validity are criteria that are also considered to be important. Whether a method gives a result in a monetized value, on the other hand, is considered to be the least important by all groups.
Table 4: Ranking importance criteria social performance method, per user group
To compare the output of the questionnaires of the three user groups, a classification is made of the importance of the criteria per user groups. A distinction is made between criteria that are considered to be important, medium and less important, based on the responses of the criteria. The aspects that receive the most points are classified a s important, the following as medium and the criteria mentioned least are rank less important. This is visible in the table below.
Although there are a lot of similarities in the responses of the different groups, some differences can be identified as well. The investor group gives a higher importance to comparability than the other two groups, while they consider accuracy to be less important than the other two groups do. Donors, on the other hand, consider comparing social performance to be less important, compared to the other two user groups. Donors, compared to the other users, rank validity as more important. Finally, it should be noted that being able to compare or quantify, let alone monetize, is considered less important by the claimants than by the other two user groups.
User group Important Medium Less Important
There was a possibility within the questionnaire to mention any missing criteria that are considered to be important. Several respondents mentioned the trade of between time and costs as missing criteria. Furthermore, it was pointed out twice that a method should be user friendly. Finally, feasibility of the method is stated as a requirement. The additional criteria are all somewhat similar to the criteria already mentioned and will thus not be considered separately.
With regard to the reason for using a social performance measurement method, a similar table can be made.
User group Important Medium Less important
Investor Managing Impact Communicating Monitor Alignment Payment Budgeting Donor Impact Monitor Managing Communicating Payment Alignment Budgeting Claimant Managing Impact Monitor Budgeting Communicating Alignment (Other) Payment Table 5: Ranking importance reasons for using social performance measurement, per user group
With regard to the reasons for using a method to measure social performance, a distinction can be made between a number of reasons that are considered to be important by all user groups and reasons that are can be classified as less important, based on the responses of all user groups. The reasons that scored highest are monitoring, determining impact and using as a management tool. The investors rank managing as the most important reason, followed by impact. The donors ranked impact the highest, followed by monitoring. The last group, the claimants consider managing the most important reason, followed by impact. Budgeting, payment and alignment are considered to be less important.
It is possible to explain the importance the different user groups are giving to the criteria for methods to measure social performance. Investors, for example, consider comparability to be more important than the other two groups do. This can be explained by the fact that investors have an objective to select the projects with the highest financial and social return. Therefore, being able to compare the results of different projects will be important.
On the other hand, it remarkable to see that the claimant give a low importance to all criteria that enable the results to be compared. Because claimants have to compete for funding, comparing its social performance can be confronting or evening threatening.
Also with regard to the reasons for using social performance measurement methods, the questionnaire gave the possibility to point out other reasons that might be important. Furthermore, it was requested to bring up reasons for not using a social performance measurement method. Again, the time and costs trade of was mentioned. Furthermore, most aspects were similar to the reasons stated in the questionnaire, mostly relating to (internal and external) communicating and managing the organization/project based on results. One other reason that is stated a number of times and could be considered to be additional to the reasons mentioned in the questionnaire, is creating awareness. This is related to communication and will therefore not be considered separately.
respondents using methods for internal use, managing and monitoring for example, might consider other criteria of social performance measurement methods to be important that users with a external focus. A clear example of a reason with external focus is communicating. It seems that the investor group considers this to be more important than the other two groups. It is also visible that investors value a method that has a comparable and quantified outcome more than the other two groups. A possible conclusion can be that that users with an external focus, as communication, prefer an outcome of the method that enables comparing. This way, the results can be compared to other organizations/projects or used to track performance over time. Users with internal focus, might then be more interested in criteria such as validity and accuracy, to be sure that if they manage based on these results, the results are reliable and valid.
The distinction in the output of the questionnaires does, however, not show a very clear distinction between an external and internal focus. All groups mention both internal and external reasons for using a method to measure social performance. This could, however, be an interesting issue for further research.
3.4 Summary
Summarizing, there are several similarities in the responses of the user three groups questioned. All the groups have a relatively similar view on which reasons are important for using a social performance measurement method. With regard to the criteria for a useful method, every user group ranks reliability high. Nevertheless, some differences can be identified as well. Investors consider comparing social performance to be an important criterion for measurement method for social performance. Donors and claimants give more value to a precise method. Claimants differ in their requirements by giving a low importance to be able to compare methods. They do not consider a quantified or monetized method as an important criterion for a method.
4.
Case Analyses
In this chapter, two case studies will be used to perform the analysis of the different methods. After performing the analyses of all the different methods, an examination can be made to evaluate and compare the methods, based on the criteria outlined in the methodology. The case studies will be used to base the findings of the evaluation on.
The first case study deals with the microfinance institution Aqroinvest. Within the first case study, five methods are used:
• Cerise SP-2 method
• Social Return on Investment Method (SROI)
• Development Outcome Tracking System (DOTS)
• New Profit Balanced Scorecard (NP BSc)
• Best Available Charitable Option (BACO)
The second case study analyses the social performance of the North Star Foundation, an organization focused on HIV/AIDS prevention and treatment. Within the second case study, the same methods will be used, with the exception of the Cerise SP-2 method, since this is a method specifically focused on microfinance. The methodology of the methods is briefly discussed in Chapter 2. The complete methodologies the methods are described in Appendix 3.
Both case studies will start with giving more background information on the organization the case study will be focused on. After that, all five methodologies will be performed subsequently. Unless needed in order to finish the analysis, no adjustments are made to the methodology of the methods. Within the second case study, the introduction of the methods and steps will be brought back to a minimum to prevent repetition. The concluding paragraph of this chapter will give a short overview of social performance output of two cases, based on the methodology of the five methods.
4.1 Case 1 Aqroinvest
4.1.1. Introduction
Aqroinvest Credit Union was established in 2001 and is a privately owned financial organization. 32 farmers from Imishli have made the initial investment in Aqroinvest and choose Ayyub Akhmadov as CEO. Aqroinvest CU provides loans in four regions of Azerbaijan:
• Imishli (main office),
• Sabirabad,
• Saatli and Varkhankend village in Saatli region; and
• Bilasuvar internally displaced people (IDP) settlement.
IDPs are refugees from other parts of Azerbaijan. Each branch has at least a Branch Manager and a Loan Officer.
Fig. 15: Funding Aqroinvest Products
Aqroinvest serve four types of products Agriculture/Livestock, Trade, Production and Manufacturing. The table below shows the characteristics of the four types of products offered.
Product 1 Product 2 Product 3 Product 4
Name of loan Livestock Trade Production Agriculture
Purpose Individual Individual Individual Individual
Maturity 12-36 18-36 7-12 12-24
Security Guarantor/Collateral Guarantor/Collateral Guarantor/Collateral Guarantor/Collateral
Repayment Monthly Monthly Monthly Monthly
Principal Payment Once in 3-6 months Monthly Once in 3 month Once in 6 month
Loan Size Min. 100 AZN 100 AZN 100 AZN 100 AZN
Loan Size Max. 30 000 AZN 30 000 AZN 30 000 AZN 30 000 AZN
Loan term-Min. 12 6 12 6
Loan term-Max. 36 24 60 24
No. of loans 474 157 1 0
Table 6: Loan products of Aqroinvest
Aqroinvest is a credit union active in Azerbaijan and has a special focus on two groups: IDPs: Internally displaced People (+refugees)
Rural Finance: People not able to get access to financial products through traditional finance
The organization formulated the following vision and mission:
Vision: To promote and lead the best practices and sustainability of our members and support our members providing viable financial services as well as playing an important role in financial sustainability and economic well being of Azerbaijani people.
Mission: To contribute to the business development of the union members and vulnerable groups of people by demand-driven lending services with desirable depth, width and length of social outreach.
The added value of Aqroinvest is that is the Credit Union is part of communities they serve. The employees of the local offices are people living in the remote areas that are served. As a result, the employees are well known and respected within the communities. Potential clients feel understood and trust the employees of Aqroinvest. In addition, the credit union claims that it is capable of adapting their products and services to the wishes of their members because the organization has a good understanding of the needs of its clients.
The data used in the analysis of the different methods is based on data of 2007.
