The role of internal audit in providing assurance on the performance of international aid and development NGOs

The role of internal audit in providing assurance

on the performance of

international aid and development NGOs

Leo den Besten, MSc

Supervisor: Jacques van Kempen, RA RO

Cover photo:

C

ONTENTS

Summary ... iii

Preface ... v

1. Introduction ... 1

1.1 Background ... 1

1.2 Research objective and questions ... 3

1.3 Methodology ... 4

1.4 Structure of the report ... 5

2. Theoretical framework ... 7

2.1 Introduction ... 7

2.2 NGO performance, Efficiency and Effectiveness ... 7

2.3 Governance and accountability in NGOs ... 12

2.4 Internal audit and Assurance in NGOs ... 15

3. Case studies ... 19

3.1 Introduction ... 19

3.2 Needs for assurance on performance ... 19

3.3 Current audit practice and frameworks ... 23

3.4 Analysis: coverage of needs for assurance by current audit practice ... 26

4. Performance Auditing ... 29

4.1 Introduction ... 29

4.2 Features of performance auditing ... 29

4.3 Performance auditing practice ... 33

5. Applying elements of performance audit in NGO audit approaches and frameworks ... 37

5.1 Position and mandate ... 37

5.2 Purpose and scope ... 37

5.3 Methods, approach and preconditions ... 39

6. Conclusions and Recommendations ... 41

6.1 Conclusions ... 41

6.2 Recommendations ... 42

Annex 1 - Abbreviations used ... 47

Annex 2 - Code of Ethics Institute of Internal Auditors... 49

Annex 3 - Assurance levels in AA1000 Assurance Standard 2008... 51

Annex 4 - Characteristics of case study organisations ... 53

Annex 5 - Questions used for semi-structured interviews ... 55

Annex 6 - Records of interviews held with board members ... 59

Annex 7 - Records of interviews held with persons responsible for Internal Audit ... 61

SUMMARY

Non-Governmental Organisations (NGOs) have participated in international aid and development co-operation for decades. In NGO governance, it is essential to aim for a positive spiral of results, reputation and resources, in managing the relations with key stakeholders like beneficiaries, donors and the general public. One would expect assurance about the efficiency and effectiveness of NGO performance to be an important aspect in NGO governance. Yet, these aspects seem to receive relatively little attention in the agenda setting for internal auditing in those organisations.

This study focused on the contribution internal audit can make in providing assurance on NGO performance and on the approaches and frameworks for internal audit that are needed for making such contribution. For addressing these questions, a comparative case study approach was used. Board members and audit staff of five Dutch NGOs were interviewed about their need for receiving additional assurance about the performance of their organisation and about current audit practice. Besides, theory and practice of performance auditing of Government services were studied, as aspects of performance auditing may be relevant for internal auditing of NGO performance. For performance auditing practice, the case of the Dutch Court of Audit was studied. The interviews revealed that most board members of the NGOs studied do have a need for additional assurance on organisational performance. Although that need seems to be related to the finding that in several organisations the executive board does not receive sufficient primary assurance about performance on the basis of information from their usual monitoring systems. The evaluation function was also seen as an organisational function that may contribute to assurance on performance. The research also showed that the current scope of internal auditing focuses a lot on compliance with internal and external (donor) procedures and regulations. The limited attention that internal auditing currently gives to efficiency and effectiveness is in line with the current tendency of most of the boards to seek assurance from internal auditing on financial control and compliance with donor regulations.

The study further revealed that performance auditing of government services does not only serve accountability but also organisational learning and operational control. Its scope covers the aspects of economy, efficiency and effectiveness as well as compliance with established legislation and regulations. It makes use of a broad spectrum of methods, including evaluation. An independent position and mandate is an essential precondition for performance auditing.

It was concluded that internal audit can contribute to providing assurance on NGO performance in two roles: - assessment of the quality of available performance information related to economy, efficiency and

effectiveness and of the underlying processes and systems;

- conducting over-all analysis across the three aspects of economy, efficiency and effectiveness, with internal audit positioned as an independent layer around other functions providing information on various aspects of performance.

Especially the second role requires a relatively broad range of methods, including evaluation techniques. The provision of assurance on performance will be served by bringing audit and evaluation closer together, preferably accommodated in the same organisational unit. Internal audit can only play these roles if monitoring systems are functioning properly under the responsibility of first and second line functions. The cost-effectiveness of adding performance auditing to the internal audit scope was a point of concern and requires further investigation.

The results of this study underline the importance of performance auditing in NGOs, as it may contribute to accountability and learning about organisational performance and thus feed the positive spiral of results, reputation and resources.

P

REFACE

The writing of this report marks the end of a two years’ exploratory journey in the world of internal auditing. Undertaking a research project was the final assignment of the Executive Internal Auditing Programme of the Amsterdam Business School, which I joined in September 2012. In several ways, my participation in the programme was an investment. Having a background in human nutrition and after having worked in a programme related position in international aid and development organisations for about 25 years, joining an internal auditing programme of a business school was a new and challenging experience. It brought me a lot of new perspectives, knowledge and skills, which I can use in my work on the edge of idealistic mission and organisational management. For the benefit of the communities in Africa, Asia and other regions, which my colleagues and I want to serve.

It also was an investment for ZOA, the relief and rehabilitation organisation for which I have worked for many years. ZOA management beliefs in the importance of internal auditing and in broadening its scope towards ‘programmatic auditing’. They realise this is a relatively new field, which requires integration of different realms of expertise. Letting me do this executive programme at the Amsterdam Business School, was an expression of that vision. I want to thank the CEO of ZOA, Johan Mooij, for that. And my fellow management team members and other colleagues for respecting my ‘study day in Amsterdam’ on Fridays.

Certainly, the programme - as well as this research - also was an investment that affected my private and social life. Spending approximately 1600 hours on studying in a period of two years, certainly demanded sacrifices of people around me. I want to thank Berna, my wife, for so many hours of patience and letting me study. And several other people in my private network who I have been neglecting in the past two years. After two years of ‘social negligence’, the time of ‘social recuperation’ has come.

I am grateful to those who were involved in this research. The board members and audit staff of ICCO, Cordaid, CARE, War Child and ZOA, who were willing to share their views and experience with me during the interviews. To Jacques van Kempen, for supervision during this research and for his valuable comments on concepts of this report. And to Hannah den Besten for the design of the cover of this report.

I hope this report will feed discussion within the ‘development NGO sector’ about the importance of auditing efficiency and effectiveness of performance and that it will contribute to the further development of internal auditing in this sector. During the interviews I noted a lot of interest for the subject, which hopefully will translate in continuation of the discussions and in exchange of experience between the five organisations studied and other interested organisations in the sector.

Leo den Besten Deventer, August 2014

1.

I

NTRODUCTION

1.1 B

ACKGROUND

In the Netherlands, Non-Governmental Organisations (NGOs) have been involved in international aid and development co-operation (IADC) since many years. In 2013, the Dutch sector organisation for international development co-operation, Partos, had a membership constituency of approximately 120 NGOs (Partos, 2014a), representing a significant part of professional NGOs in the Netherlands that are active in international aid and development co-operation. Examples of such NGOs are Cordaid, ICCO, Save the Children, CARE, War Child and ZOA. Apart from professional organisations like these, there is involvement of several thousands of ‘private initiatives’, small organisations in Dutch society that are active on voluntary basis (Van den Berg, 2012). In 2011, almost €4.3 billion was donated to all ‘charitable causes’ (charities) in the Netherlands by households, bequests, foundations, businesses and lotteries (Schuyt, 2013). Among the recipient organisations, those focusing on international aid ranked third, with €569 million (13%) of the donations received. Apart from private donations, especially the professional NGOs also receive funding from institutional sources, like the European Union, the Dutch Government and the United Nations. Altogether, the budgets made available for international aid and development co-operation by such institutional donors are much larger than what is available from private sources. Yet, those institutional budgets are under pressure because of adverse economic conditions and decreasing political support for development co-operation. The budgets made available by the Dutch Government for ‘civil society organisations’ (NGOs), being one of the channels of spending the government budget for development co-operation, will decrease from €450 million in 2014 to €220 million in 2017 (Dutch Minister for Foreign Trade and Development Co-Operation, 2013). However, this is not the only budget line in the Dutch Government budget where NGOs have access to.

Complete information about total expenditures of all Dutch NGOs that are active in international aid an development co-operation is scarce. The database on project expenditures by Dutch development NGOs, maintained by CIDIN – Radboud University, indicates that 81 Dutch NGOs spent €1.04 billion in 2012 (CIDIN, 2014). This was estimated to be approximately 80% of all expenditures made by Dutch NGOs that are active in international aid and development co-operation, which implies that in 2012 the total amount spent by Dutch NGOs was approximately €1,3 billion.

The public opinion about Development Co-Operation is somewhat ambivalent. Based on review of available research findings, Van den Berg (2012) concludes that on the one hand, a significant majority in Dutch society still believes it is good to support the poor in less developed countries. However, after some years of economic downturn, between a third and half of the population indicates that the Government budget for development co-operation can be reduced. A minority of approximately 20% openly questions the effectiveness of official development aid provided by the Government. Although there may still be more support for NGOs that are active in international aid and development co-operation, also that support cannot be taken for granted. The public demands transparency about funds spent by NGOs, is critical about their overhead costs and questions the effectiveness of aid. At the same time, public support is still significant, with 37% of all households having donated in 2011 to organisations that are active in international aid (Schuyt, 2013).

In 2012, the Netherlands Institute of Chartered Accountants (NBA, 2012) produced a public management letter for charitable organisations in the Netherlands, with a number of signals and recommendations to management and supervisors. The public wants to be properly informed about the way donations were spent and on what the effects of the organisation’s activities are. Maintaining trust, reputation and being transparent, are of critical importance for continuation of public support.

Among the six specific recommendations made, were the following:

- Focus on the effects to be achieved by the organisation, both in managing the organisation and in reporting on the planned effects. Communicate financial information together with information about proposed results, actual performance and effects, and indicate to what extent the reliability of this information has been checked.

- In order to meet the expectations of public interest, the accountant should contribute to bridging the expectation gap that exists among the public, about assurance to be provided about the actual performance of charitable organisations.

The question is whether expanding the role of the external accountant significantly, in an attempt to bridge the expectation gap, will be realistic and feasible, given the specific expertise required and the additional costs that will be involved. It seems the role of internal audit should also be considered here. Especially internal auditors who combine professional practice, as defined in the mandatory elements of the International Professional Practices Framework (IPPF) developed by the Institute of Internal Auditors (IIA, 2013), with specific knowledge of the organisation and its processes, can play an essential role in providing additional assurance on organisational performance, as a basis for accountability towards the public, donors and other stakeholders. In the International Aid and Development Co-Operation (IADC) sector in the Netherlands, internalauditing has not been broadly developed and applied. In as far as auditing takes place, it mainly involves (external) financial auditing, focused at providing additional assurance on reliable financial reporting, closely linked to external regulations for the publication of annual accounts of charitable organisations, like the Dutch RJ650 reporting norm. Some form of compliance auditing is also seen in several organisations, focusing on the extent to which donor regulations that were agreed on in donor contracts, are actually complied with. Some organisations go beyond financial and compliance auditing, and practice some form of operational auditing, aimed at providing additional assurance on the effectiveness of the management control system in the organisation, based on a risk management approach. In a number of cases, this kind of auditing is done with reference to common norms for quality management systems, like ISO 9001:2008. Partos developed a sector specific version of the ISO 9001 norm (Partos, 2014b), in which specific reference is made to the operational context of IADC NGOs. In the humanitarian aid sector, various initiatives emerged in the past two decades aiming at quality enhancement of humanitarian aid, resulting in various standards, codes and norms (Hilhorst, 2002). Among those relatively well known are:

- the Sphere humanitarian charter and minimum standards in humanitarian response (Sphere, 2014); - the Humanitarian Accountability Partnership (HAP) standards in accountability and project management

(HAP, 2014);

- the People in Aid code of good practice, focusing on support and management of staff and volunteers (People in Aid, 2014).

Recently, some of the leading humanitarian organisations took initiative for merging some of those and other norms and standards into an integral quality standard for humanitarian assistance. In future, this may be at the core of a certification scheme that may be adopted by some of the main institutional donors.

Norms and standards like these, focus on the quality of programme management processes and may be part of the reference frameworks of operational, internal auditing.

Unlike internal auditing, evaluation of projects, programmes and policies is widely practiced in the aid and development sector. Evaluation approaches and methodology have been well developed and an extensive market exists for training and consultancy in the field of evaluation. Morra Imas & Rist (2009) conclude in their extensive World Bank publication on evaluation that “Auditing and evaluation can be viewed as part of a continuum, providing related but different kinds of information about compliance, accountability, impact and results.” It seems there is a relatively unattended space on the audit-evaluation continuum between financial, compliance and operational auditing on the one hand, and evaluation practice on the other hand. There is relatively little attention for providing assurance on (reporting) actual performance in terms of effectiveness and efficiency.

Current internal auditing practice, in as far as it is practiced anyway, seems to focus more on the internal performance of the organisation. As indicated before, recent developments in public and political support for charitable organisations and the public call for more transparency and accountability, underline the importance of letting internal audit in IADC NGOs pay more specific attention to actual performance. This would also be in line with the increasing level of standards that are set by institutional donors in terms of proof of organisational performance, as a precondition for funding.

As there is relatively little experience with regard to that undeveloped area on the audit continuum, there is also no well-developed audit framework that can be used to fill that gap. This raises the question if lessons can be drawn from sectors that are somehow near to the NGO sector. In various ways, the Government Services sector may be a useful neighbouring sector to learn from. In government services and programmes, like health, education or social services programmes, the processes followed in various ways resemble the type of processes that are applied in IADC NGOs. In essence, these processes follow an input-activity-output-outcome- impact pattern, which is comparable to the social value chain that is often used in NGOs. This is the way of thinking that the Dutch government has applied since 1999 in a new approach in budgeting, summarized with the Dutch term VBTB (Tweede Kamer der Staten-Generaal, 2002): Van Beleidsbegroting tot Beleids-verantwoording (translation: From Policy Budgets to Policy Accountability). Thus, in both sectors a shift from ‘eligibility of costs’ towards ‘linking resources and activities to outputs and outcomes / impact’ can be observed. In both sectors there is a call for transparency and accountability with regard to efficiency and effectiveness.

Moreover, government services and IADC NGOs show a similar pattern of main stakeholders involved. The Government Services sector involves three main stakeholders: service providers (government service workers), beneficiaries (citizens) and those providing financial resources (tax payers). In the IADC sector a similar triangular stakeholder relation exists between service providers (NGOs), beneficiaries (the people receiving aid) and donors (private or institutional). Accountability mechanisms in such triangular stakeholder situations work differently from a normal ‘corporate, profit situation’ where there is a direct client-supplier relation.

In the Government Services sector, extensive use is made of Performance Auditing, which gives particular attention to aspects of economy, efficiency and effectiveness. Performance auditing theory and practice have been extensively documented (INTOAI, 2004c; Bouckaert & D'hoedt, 2011).

All this makes Performance Auditing in Government Services an interesting field to study and learn from, aiming at enriching current internal auditing approaches and frameworks used in IADC NGOs.

1.2 R

ESEARCH OBJECTIVE AND QUESTIONS

The objective of this research is to contribute to the development of a comprehensive approach and framework for internal auditing in NGOs that are active in International Aid and Development Co-Operation (IADC). This is done by determining what internal audit can contribute to providing assurance on efficiency and effectiveness of IADC NGO performance and by ascertaining what can be adopted from approaches and frameworks for Performance Auditing of Government Services in providing such assurance.

Considering the research objective, the main research question is:

What contribution can internal audit make in providing assurance on efficiency and effectiveness of the performance of Non-Governmental Organisations active in International Aid and Development Co-Operation and what should approaches and frameworks for internal auditing in such organisations encompass for making that contribution?

In order to be able to answer the main research question, the following sub-questions will be addressed: I. What needs for assurance on efficiency and effectiveness of their performance do Non-Governmental

Organisations active in International Aid and Development Co-Operation (IADC NGOs) have that are appropriate for being addressed by internal audit?

II. How do internal audit approaches and frameworks commonly used in IADC NGOs look like, in terms of types of objectives, scope, methodology and corresponding working programmes and audit procedures? III. To what extent are IADC NGO needs for assurance on efficiency and effectiveness of performance

adequately covered by the internal audit approaches and frameworks commonly used in those NGOs? IV. What do approaches and frameworks for performance auditing in Government services encompass? V. What elements of approaches and frameworks for performance auditing in Government services can be

adopted in approaches and frameworks for internal auditing in IADC NGOs, so that needs for assurance on efficiency and effectiveness of performance in those NGOs are optimally served?

1.3 M

ETHODOLOGY

In figure 1 the methodology followed for answering the research questions has been outlined in a research framework.

Figure 1 – Research framework (numbers refer to research questions in section 1.2)

The needs for assurance on performance in IADC NGOs (research question I) to be provided by internal audit are determined on the basis of an analysis of theory about assurance on efficiency and effectiveness of performance and on the role of internal audit in NGOs, followed by studying a number of Dutch IADC NGO cases on their needs for additional assurance on performance.

Likewise, the audit approaches and frameworks commonly applied in IADC NGOs (research question II) are determined by studying the same IADC NGO cases on the characteristics of current audit approaches and frameworks used by their internal audit functions.

Needs for assurance (I) are confronted with internal audit approaches and frameworks currently used (II), resulting in an overview of unserved needs for assurance on performance in IADC NGOs and related gaps in the commonly used audit approaches and frameworks (research question III).

Document analysis and limited expert consultation should provide insight into theory and practice of approaches and frameworks for performance auditing (research question IV). That should enable the identification of elements of performance audit approaches and frameworks that may enrich internal audit approaches and frameworks commonly used in IADC NGOs, so that needs for assurance on efficiency and effectiveness of performance in those NGOs are better served by internal audit (research question V).

A combination of comparative case study and desk research strategies is used in this research project.

1.4 S

TRUCTURE OF THE REPORT

In chapter 2 of this report, a theoretical framework for this study is provided. A number of underlying concepts are discussed, based on literature study.

The sub-questions I, II and III are addressed in chapter 3, on the basis of a comparative case study that was conducted, involving five Dutch NGOs that are active in aid and development work. The annexes 4-7 contain background information and records of interviews that were used for the analysis provided in this chapter. In chapter 4, addressing sub-question IV, an analysis of approaches and frameworks used in performance auditing of Government services is provided. This was based on theory as well as on practical experience with performance auditing by the Dutch Court of Audit. Annex 8 contains records of interviews used as an input for this analysis.

Sub-question V is addressed in chapter 5, based on a synthesis between the results of chapters 3 and 4. It outlines the aspects of government performance auditing approaches and frameworks that may be applied in auditing performance auditing in NGO settings.

2.

T

HEORETICAL FRAMEWORK

2.1 I

NTRODUCTION

Addressing the question what contribution internal audit can make in providing assurance about efficiency and effectiveness of IADC NGO performance, requires clarity on underlying concepts. In the first place, the concepts of performance, efficiency and effectiveness require further theoretical framing, as does their relation with the notion of ‘value for money’. This should help in understanding the nature and importance of providing assurance on NGO performance and in identifying appropriate methodology for measuring and evaluating performance.

Secondly, it should be made clear who may benefit, directly or indirectly, from receiving such (additional) assurance about NGO performance, and why. This relates to the field of NGO governance and accountability. Who are the main stakeholders of NGOs and how do accountability mechanisms between and towards those stakeholders work?

Finally, the concept of internal audit should be defined and explained in more detail, before the question about internal audit’s specific contribution in providing assurance about NGO performance can be addressed. What characterizes internal audit? How does it relate to and compare with other functions in NGOs that may be regarded as contributors of assurance on NGO performance?

2.2 NGO

PERFORMANCE

,

E

FFICIENCY AND

E

FFECTIVENESS

2.2.1 Basic concepts

In order to understand what NGO performance is all about, it is helpful to consider what characterizes an NGO in terms of its ‘raison d’être’. A typical characteristic of NGOs is their purpose of public benefit, unlike profit organisations’ purposes (Liket, Rey-Garcia, & Maas, 2014). The intention to contribute to public benefit, in the form of social change or improvement, usually is at the heart of an NGO’s mission statement, which is leading for the impact the organisation intends to have or at least contribute to. The logic of working towards realization of such impact is often described in a logic chain or social value chain (see figure 2), representing a theory of change. According to Morra Imas & Rist (2009) a theory of change “… open[s] the “black box” to show how an intervention expects to convert inputs, activities, and outputs into results”. It comprises the “…building blocks needed to achieve the long-term goals of a social change initiative.” The characteristics of the various components of the social value chain have been summarized in table 1.

Figure 2 – Social Value Chain (Adapted from Morra Imas & Rist, 2009)

Results Implementation

Inputs Activities Outputs

B lac k B o x Outcomes Impacts

Table 1 – Components of the Social Value Chain

Inputs Financial, human and material resources that go into a project, program, or policy (Morra Imas & Rist, 2009).

Activities Tasks undertaken in order to transform inputs into outputs (Morra Imas & Rist, 2009).

Outputs Products and services produced / delivered. Outputs are tangible and can be counted (Morra Imas & Rist, 2009).

Outcomes The likely or achieved short-term and medium-term effects of an intervention’s outputs (OECD, 2010). Behavioural changes, both intended and unintended, positive and negative (Morra Imas & Rist, 2009).

Impacts Long-term changes that result from an accumulation of outcomes (Morra Imas & Rist, 2009). Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended (OECD, 2010).

Thus, performance can be seen as the effectiveness of an organisation in this process of converting inputs, activities, and outputs into results. Although performance and effectiveness are often used as equivalent terms, Liket et al. (2014) underline the importance of differentiating between these concepts. They indicate that “Organisational effectiveness is most often associated with the achievement of organisational goals, and in particular, goals that relate to the organisation’s public benefit mission”. They limit effectiveness to the last stage of the social value chain, which is the organisation’s impact. Others, like Antinoja, Eskiocak, Kjennerud, Rozenkopf, & Schatz (2011), Morra Imas & Rist (2009) refer to the entire ‘results’ part of social value chain (figure 2) and include both outcome and impact when they talk about effectiveness. According to Liket et al. (2014), performance is a broader concept and can relate to any part of the social value chain. It indicates “… how well an organisation functions in some specific dimension …”. They add by saying that “It is always necessary to specify the dimension relative to which an organisation, program, or project performs; an organisation cannot simply be ‘‘high performing’’ as such.”

Being specific dimensions of performance measurement, efficiency and effectiveness are frequently considered under the wider concept of ‘value for money’. Often, this is explained as the combination of three E’s: Economy, Efficiency and Effectiveness (Antinoja et al., 2011; Bouckaert & D’hoedt, 2011; Bond, 2012). Economy relates to procurement processes and cost-effective management practice, to obtaining inputs and realising administrative procedures at the best possible price. A typical measure of economy is unit price. Efficiency relates to the delivery of project outputs in relation to the amounts of inputs used. It is about realising maximum output for a certain amount of inputs, or minimizing the amount of inputs for a certain output. As indicated above, effectiveness is about the level of realization of intended outcomes and impacts. Thus, over-all, value for money is about using resources optimally to achieve or at least contribute to intended outcomes and impacts.

Antinoja et al. (2011) tress that in explaining value for money along the line of the three E’s, it resembles both quantitative and qualitative aspects. “ … only the Economic Case is entirely about the financial aspect of delivery, while the other cases are broader and open for qualitative interpretation and judgment.”

Bond (2012) underlines the importance of applying value for money in terms of evaluating alternatives. By considering and comparing the costs and benefits of different programme options and by choosing the option that shows the best combination of economy, efficiency and effectiveness. They also stress the importance of using robust data in building value for money cases. “The quality of the data used is fundamental; the credibility of the claims rests on this. The more explicit, transparent and specific an NGO is in providing evidence that others can review, the stronger the case will be.” This triggers questions about how to measure NGO performance and what challenges are to be addressed in doing that.

2.2.2 Performance Measurement – challenges and risks

In the past two decades, there has been considerable attention for performance measurement and evaluation in public management and the non-profit sector (Behn, 2003; Liket et al., 2014). Although the concepts of performance measurement and value for money may sound relatively straight forward, they come with various challenges, especially further down the social value chain. Ebrahim & Rangan (2010) see this as one of the tensions confronting non-profit managers: “… non-profits that focus on measuring results at the front end of the logic chain (inputs and outputs) risk being seen as failing to be accountable, failing to convince funders and citizens that they are making a difference. Those that do try to demonstrate broader societal outcomes and impacts risk overreaching by taking credit for social changes beyond their actual control.”

As NGO performance, in terms of value for money, ultimately is about using resources optimally to achieve or at least contribute to outcomes and impacts, “… measuring value requires measuring changes on these more complex dimensions. However, there is a strong focus on the quantifiable outputs rather than outcomes, in many current measurement systems” (Antinoja et al., 2011). This gap between what ought to be measured and what actually is measured and reported on in practice, has much to do with underlying analytical problems. First there is the problem of causality, which is about the causal relation between the activities undertaken and outputs realized on the one hand and the effects this may contribute to in terms of outcomes and impact on the other. Ebrahim & Rangan (2010) refer to this causal logic in terms of the theory of change. Although an organisation may contribute to change by delivering outputs, it often is a challenge to determine the extent to which that change can be attributed to the organisation’s interventions. They refer to a second problem as the control over interventions and results in terms of their operational strategy. “How do managers exercise control over their interventions and results? In particular, management control is reflected in an organisation’s boundaries and its operational strategy.”

Referring to these underlying analytical problems, Ebrahim & Rangan (2010) conclude that the way performance can be measured is related to positioning the programme on these both dimensions:

- the theory of change of an organisation’s activity or programme, which can be focused or complex; - the operational strategy that an organisation applies, which can also be focused or complex.

By differentiating between a focused and a complex theory of change and organisational strategy, they construct a contingency framework for measuring results, comprising four quadrants (figure 3).

According to this framework, the performance of several categories of interventions cannot be realistically measured at impact level and some even not at outcome level. This is an important conclusion and nuance, given the methodological challenges that are often experienced based on the assumption that measuring at outcome and impact level should be aimed for across all types of programmes and interventions. With reference to a project’s or programme’s position in this framework, NGOs can explain to stakeholders what type of results they aim for, and thus what proof of performance they should realistically be held accountable for. At the same time, there may still be a dilemma as different stakeholders may have different demands for accountability on performance.

In their study on approaches to measuring value for money, Antinoja et al. (2011) highlight the risk of what they call the “…tyranny of indicators…”. Given their aim for clear communication to taxpayers and funders, donors prefer to receive concrete evidence of the NGO’s performance on projects that were funded by them. As a result, “… there is a persisting bias towards quantitative methods and ‘hard’ data”. In order to be able to address this need for the concrete and quantifiable, emphasis may be on interventions that are less complex and on providing aid to beneficiaries that are relatively easy to reach. Groups of beneficiaries whose context and type of needs are more complex, may be excluded. Moreover, this bias may also limit innovation and programmatic risk taking, as donors tend to be less interested in funding projects and programmes that are less predictable in terms of their outcomes. This effect may be stronger with reduced budgets being available and more competition for resources.

2.2.3 Performance Measurement – determinants of approaches

The risks and challenges referred to above correspond with the need that Liket et al. (2014) see for a “… participatory negotiation process between non-profits and their stakeholders on the purpose and design of evaluations in achieving evaluation utility”. They found that in practice, performance evaluations – here referred to in a broader meaning of structured measurement of performance – often do not provide stakeholders with the information they need. Performance measurement, i.e. collecting reliable information about performance, is not an end in itself. The question is what public managers and other stakeholders do with this information about performance. Behn (2003) states that “The leaders of public agencies can use performance measures to achieve a number of very different purposes, and they need to carefully and explicitly choose their purposes. Only then can they identify or create specific measures that are appropriate for each individual purpose.” He identified eight purposes that public managers may have for measuring performance. In table 2, these are summarized, together with appropriate performance measures for each purpose and the management questions that these measures can help answering.

Behn (2003) admits that these eight purposes partly overlap and underlines that the purpose of improvement is served by all of the other seven. Liket et al. (2014) and Behn (2003) indicate that these purposes of performance measurement may refer to various steps in the social value chain. The control purpose focuses at the output level. On the other end of the social value chain, the promote and evaluate purposes focus on the results part of the chain, at the outcome and impact level. The remaining five purposes – learn, improve,

Table 2 – Purposes that public managers have for measuring performance (adapted from Behn, 2003)

Purpose Manager’s questions to be answered Characteristics of performance measures

Evaluate How well is my public agency performing, compared

to mission, strategy and objectives?

Outcomes, combined with the effects of exogenous factors, compared with a performance standard.

Control How can I ensure that my subordinates are doing the

right thing?

Inputs that can be regulated. Compliance with rules and requirements.

Budget On what program, people, or projects should my

agency spend the public’s money?

Efficiency measures, specifically outcomes or outputs divided by inputs.

Motivate How can I motivate staff, managers, collaborators,

citizens and other stakeholders to do things necessary to improve performance?

Almost-real-time outputs compared with production targets.

Promote How can I convince political superiors, legislators,

journalists, citizens, other stakeholders that the agency is doing a good job?

Easily understood aspects of performance about which stakeholders really care.

Celebrate What accomplishments are worthy of the important

organisational ritual of celebrating success?

Periodic and significant performance targets that, when achieved, provide a real sense of personal and collective accomplishment.

Learn Why is what working or not working? Disaggregated data that can reveal deviances from

the expected.

Improve What exactly should who do differently to improve

performance?

Inside-the-black-box relationships that connect changes in operations to changes in outcomes. Knowing that the purpose of performance measurement may differ and that it can relate to different stages in the social value chain, evaluations need to be designed with a specific focus on that purpose and position in the value chain. Different types of questions will be required for such specific evaluation design. Morra Imas & Rist (2009) mention three categories of evaluation questions:

- “Descriptive questions seek to determine what is. They may describe aspects of a process, a condition, a set of views, or a set of organisational relationships or network.”

- “Normative questions compare what is with what should be. They compare the current situation with a specific target, goal, or benchmark.

- “Cause-and-effect questions determine what difference the intervention makes … they attempt to measure what has changed because of the intervention.”

Liket et al. (2014) mention two types:

- A “… reporting question relates to the level of production … at the different stages of the social value chain”

- “A learning question relates to the comparative cost-effectiveness … at the various stages of the social value chain.” For letting evaluations answer learning questions, a benchmark is required to compare with. For the purpose of this research, two categories of questions for appraising performance are distinguished, in line with the types of questions used by Liket et al. (2014), though incorporating the categories of Morra Imas & Rist (2009).

- A reporting question describes an aspect of performance or compares that performance aspect with a specific target, goal or benchmark. It has a factual, determinative orientation. The question may refer to any stage of the social value chain.

- A learning question aims at understanding how the intervention actually works out, by analysing the effects of the intervention in comparison with a baseline or benchmark or with the effects of an alternative intervention. It may deal with cost-effectiveness, though it can have to do with any other aspect of the intervention. It has an explorative, analytical orientation. The question may refer to any stage of the social value chain.

The difference between the two types of questions is illustrated by Behn’s (2003) statement about the difference between the performance measurement’s purposes ‘evaluation’ and ‘learning’: “The objective of evaluation is to determine what is working and what isn’t. The objective of learning is to determine why.”

2.2.4 Conclusion

With regard to theory on NGO performance, efficiency and effectiveness, the following is concluded:

- NGO performance should be related to and determined within the social value chain. This chain reflects an NGO’s theory of change and core process of converting inputs and activities into outputs that should result in outcomes and contribute to impacts.

- Efficiency and effectiveness, together with economy, are key aspects of ‘value for money’ that each relate to a specific part of the social value chain. Performance can be measured for each of these aspects and involves quantitative as well as qualitative aspects. Over-all, value for money is about the optimal use of resources for the realization of outcomes and impacts.

- The interventions of NGOs differ in terms of complexity of both theory of change and operational strategy. These combined differences have consequences for the level within the social value chain at which performance can be realistically measured.

- “Any evaluation effort should start with a definition of its purpose, and, we argue, this is the basis for identifying the evaluation question, determining the level at which the effects should be measured and the selection of the evaluation design.” (Liket et al., 2014)

- The challenges experienced in measurement of performance aiming at satisfying external accountability needs, should not result in ‘tyranny of indicators’ and moving away from the more complex and risk full, innovative interventions. As Einstein said: “…not everything that counts can be counted.”

2.3 G

OVERNANCE AND ACCOUNTABILITY IN

NGO

S

2.3.1 NGO Governance

NGOs are increasingly challenged about their effectiveness and public value (Behn, 2003; Burger & Owens, 2010; Ebrahim, 2003). Fowler & Miller (2012) and Fowler (2013) recognize a need for improved transparency and accountability, but he also observes several (international) NGOs already working on the improvement of their governance, accountability, transparency and systems for showing evidence of results. However, in doing so there is often insufficient attention for the interrelation between these aspects. Such improved accountability and transparency requires embedding in appropriate governance mechanisms.

In thinking about governance and strengthening NGO effectiveness and public value , it is helpful to start with positioning the work of NGOs in its context and determine what is essential for the interrelations between an NGO and its environment. Fowler & Miller (2012) underline the point that the anticipated effects of development programmes of NGOs, summarized as ‘sustainable development’, are the results of “co-production” between several stakeholders, with the people benefitting from such programmes being a key stakeholder group. As it is about their lives, their “ownership” of the development process is essential. NGOs play an “intermediary role” in this joint stakeholder process, by obtaining funding and other resources from stakeholders other than those benefitting from the programme. Thus, “… the effectiveness of (I)NGOs is highly dependent on multiple ‘to and fro’ relationships with the environment”. Fowler (2013) concludes that in managing its connection with the complex operating environment, NGOs should aim for a “positive spiral” of the three key relationships of NGOs with that environment: Results, Reputation, Resources. Being able to show good results will boost reputation, which is an important success factor for acquiring resources.

In order to manage the internal organisation towards realization and maintenance of such a positive spiral, sufficient integration of governance, accountability, transparency and evidence of results is required (Fowler& Miller, 2012). Governance responsibility is exercised through chief executive(s) and senior management, for which performance measures and authorizations are agreed.

Performance management, including monitoring and evaluation as well as planning and budgeting processes, is “…principally dedicated to producing results while simultaneously gaining information on which to steer the organisation on the one hand and learning to improve on the other” (Fowler & Miller, 2012).

In serving each of these purposes of performance management, several demands for accountability and transparency of internal and external stakeholders should be met. NGOs may decide to commit themselves to external performance standards, norms and disclosure requirements and let these be leading for their accountability ambitions. That will help them in feeding their reputation.

2.3.2 Accountability

What is accountability and how does it work? In literature it is described from two perspective as “…the duty to give an account …” (Candler & Dumont, 2010) and “…the right to require an account…” (Cavil & Sohail, 2007). Ebrahim (2003) indicates that accountability has an external and an internal dimension: “It may be defined not only as a means through which individuals and organisations are held responsible for their actions … but also as a means by which organisations and individuals take internal responsibility for shaping their organisational mission and values, for opening themselves to public or external scrutiny, and for assessing performance in relation to goals.”

Cavil & Sohail (2007) see the principal-agent theory being often used to analyse the concept of accountability. This theory is about a principal, like a donor, government or recipient group, delegating a particular responsibility or activity to an agent, like an NGO. The agent may keep information behind that is needed by the principal to monitor the agent’s performance. Agreement on transparency is required to bridge this natural information gap. Thus, for letting accountability work in practice, transparency is an important condition. However, several authors (Burger & Owens, 2010; Fowler, 2012) conclude that the suggested level of transparency is not corresponding with the level of accountability that is required for building trust and reputation. “Despite rising pressure for more openness and transparency, it appears that most NGOs remain reluctant to share information, and many refuse to recognize the need for accountability. This is concerning as transparency is generally regarded as a key requirement for successful feedback systems and good governance” (Burger & Owens, 2010). Especially accountability towards people benefitting from NGO programmes (‘downward accountability’) and towards partner organisations (‘horizontal accountability’) requires improvement. In addition, Burger & Owens (2010) and Fowler & Miller (2012) observe a problem with the reliability of self-reported data and information. They underline the need for verification of reported information and third-party monitoring.

Why does accountability attract attention and should NGOs give specific attention to accountability? Several reasons are mentioned in literature. From a positive side, Cavil & Sohail (2007) conclude that “Accountability enables an organisation to learn from its mistakes and successes, as well as to reduce information asymmetries between INGO and client communities.” From a more critical angle, Behn (2003), Burger & Owens (2010) and Cavil & Sohail (2007) observe a loss of legitimacy and cases of INGO misbehaviour that feed concerns about quality of development practice. Another reason for increased attention for accountability is the result of professionalization of development practice in combination with growth of the sector and of NGOs, and increasing donor demands and standards (Cavil & Sohail, 2007; Ebrahim, 2003). Finally, due to increased attention of NGOs for capacity building and lobby and advocacy work, partly shifting away from service delivery, the visibility and influence of NGOs has increased. Consequently, the demand for accountability has increased too (Cavil & Sohail, 2007).

As for the purpose of accountability, Cavil & Sohail (2007) indicate that it “…can be used as a goal in itself, as a political instrument, or as a device to improve project effectiveness, efficiency, and economy.” Thus, accountability can be part of the development process, but it can also have the development process itself as its object of observation. This is also reflected in two trends in literature on INGO accountability.

On the one hand, there is practical accountability which is “… concerned with evidence of short-term outputs, meeting quality standards, and accounting for expended resources.” This is called functional accountability by Ebrahim (2003). Strategic accountability focuses on the performance of NGOs in relation to their mission. (Cavil & Sohail, 2007)

To whom should an NGO be accountable and for what? Based on extensive literature review, Candler & Dumont (2010) developed a non-profit accountability framework. They contrast a number of stakeholders of non-profit organisations (“accountability to whom”) with a number of resources and aspects that a non-profit is accountable for (“accountability for what”). Their stakeholder categories (“whom”) include the following: - Members are acknowledged as important stakeholders in member-based organisations. They can

contribute resources, by paying membership fees, or their membership can be critical for showing representation, as an important criterion for organisational legitimacy.

- Clients or recipients are those who directly receive the services of the non-profit organisation and are seen

as important stakeholders.

- Constituents are distinguished from clients and members. They are important for organisations that

“…base their work on representing people external to the organisation.” Because they are external to the organisation, it is relatively difficult to interact with them.

- Donors, in terms of accountability, are a clearly positioned stakeholder group. “…the ultimate control over

non-profits rests with donors. They can, by giving or withholding money or other resources, exercise the blunt instrument of financial control.”

- Government agencies may hold a stakeholder position in various roles: as a contracting authority, through

tax regulations relevant for donations, as a legislature and regulator.

- The General public or “… public trust…” demands accountability from non-profits to feed the assumption

that “… they are serving the public good…”.

- Media as “… watchdog of non-profit impropriety…” as well as contributors to legitimacy.

- Staff is a key stakeholder group in any organisation. Candler & Dumont state that “For non-profits,

accountability to staff becomes more important because of the implicit donation reflected in the lower wages NGO staff are often willing to accept.”

- Partners and allies may be important stakeholders as they may complement capacity for delivering results.

In outlining the resources and aspects that a non-profit is accountable for (“accountability for what”), Candler & Dumont (2010) differentiate between “procedural accountability” and “consequential accountability”. The latter category encompasses inputs and outputs.

Accountability for consequential inputs includes:

- Financial resources being a key resource that an NGO is hold accountable for. In most countries, a legal

requirement exists for financial accountability.

- Volunteer resources, especially voluntary labour, whose contributions are rarely accounted for in a similar

way as is done for financial resources (which also cover paid labour). Yet, in several non-profit organisations, voluntary labour is an important resource.

- Reputational capital which is defined as “… the potential of using one’s ‘brand name’ or ‘logo’ to acquire

resources and/or political influence.”

Accountability for consequential outputs includes:

- Goods and services or the “… key outputs…”, which corresponds with the ‘output level’ in the social value

chain as was described in chapter 2.2.1.

- Social capital represents “… the impact an organisation has on the community in which it operates…”. It

corresponds with the ‘outcome and impact level’ in the social value chain as described in chapter 2.2.1. - Policy impact reflects the policy impact of advocacy non-profits.

Thus, consequential accountability roughly parallels with the three key relationships of NGOs with their environment as identified by Fowler & Miller (2012): Results, Reputation, Resources.

Finally, Candler & Dumont (2010) identify key types of “procedural accountability”, including: - Law referring to legal accountability

- Mission as the “extension and interpretation of [organisation] mandate” and the reason for its existence.

- Ethics which may include faith, professionalism, codes and standards, and social values. These can be

located at broader social, organisational, peer or personal level.

In categorizing types of accountability, according to the ‘direction’ of accountability being provided or claimed, Cavil & Sohail (2007) as well as Ebrahim (2003) refer to a distinction often being made between upward, downward, horizontal and inward accountability. Upward accountability is directed at donors, funders, boards of trustees, and governments. In those cases, accountability often is about ensuring ‘value for money’ and about realization of development targets. Downward accountability is directed at partners, beneficiaries of services provided or at those on whose behalf NGOs speak. Because these stakeholders often have a weak accountability position, as they don’t have a relation with donors, this category of accountability is of specific importance. Horizontal accountability towards peers and colleagues often focuses on shared values, standards and reputation in the sector. Inward accountability refers to non-profit staff working in accordance with organisational mission, values and norms.

2.4 I

NTERNAL AUDIT AND

A

SSURANCE IN

NGO

S

2.4.1 Internal Audit in NGOs

As a starting point for defining internal audit, the International Professional Practices Framework (IPPF) developed by the Institute of Internal Auditors is used (IIA, 2013). It comprises “mandatory guidance” and “strongly recommended guidance”. The three mandatory elements of the IPPF are: Definition of Internal Auditing, Code of Ethics and the International Standards for the Professional Practice of Internal Auditing (“Standards”). If they wish to operate in accordance with generally accepted professional standards, internal auditors should operate in accordance with the IPPF, and particularly the mandatory guidance parts.

The definition of internal auditing outlines the fundamental purpose, nature, and scope of internal auditing: “Internal auditing is an independent, objective assurance and consulting activity designed to add value and improve an organisation's operations. It helps an organisation accomplish its objectives by bringing a systematic, disciplined approach to evaluate and improve the effectiveness of risk management, control, and governance processes” (IIA, 2013). In the main IIA textbook on internal auditing (Reding et al., 2009), “independence & objectivity, proficiency and due professional care” are identified as the “three pillars of effective internal audit services”. In dealing with the question what internal audit characteristics are essential when external auditors want to make use of the work of internal auditors, the ISA 610 standard of the International Auditing and Assurance Standards Board identified similar aspects. They focus on objectivity, the level of competence and systematic and disciplined approach of the internal auditor (IAASB, 2013).

In the OECD glossary of key terms in evaluation and results based management, the same definition as provided by the IIA is used (OECD, 2010). They add the following note: “… a distinction is made between regularity (financial) auditing, which focuses on compliance with applicable statutes and regulations; and performance auditing, which is concerned with relevance, economy, efficiency and effectiveness. Internal auditing provides an assessment of internal controls undertaken by a unit reporting to management while external auditing is conducted by an independent organisation.”

The Code of Ethics (see annex 2) provides principles and basic rules of conduct that should guide the behaviour of individuals and organisations in the process of internal auditing.

The “standards” set criteria for a number of aspects of professional internal auditing, like independence and objectivity, professional care, quality assurance and the audit process itself.

What should internal auditing in NGOs focus on? According to the IIA definition, the focus of internal audit is on the evaluation and improvement of the effectiveness of risk management, control, and governance processes. As these are generic concepts that are not confined to specific sectors, nor to profit organisations, there is no reason to deviate from this definition and focus for internal auditing in NGOs. However, given the specific nature of the mission and social value chain in NGOs (see section 2.2) and of the governance processes and stakeholder relations (see section 2.3), these concepts require interpretation towards an NGO setting.

How does internal audit relate to and compare with other functions in NGOs that may be regarded as contributors of assurance on NGO performance? Evaluation seems to be the most common function in NGOs that may be considered as another contributor. Morra Imas & Rist (2009) indicate that, although audit and evaluation can be seen as part of a continuum, important differences exist between the two. "Auditors tend to seek answers to normative questions (what is versus what should be), while evaluators tend to seek answers to descriptive and cause-and-effect questions.” They see evaluation as being more concerned with explaining why certain strategies do or do not result in anticipated results and effects, as a basis for learning. To some extent, current evaluation practice does also address questions of accountability, by reviewing the extent of realization of project or programme objectives. However, the different roles evaluation can play in accountability and learning, have not been standardized and the question is whether ‘strategic learning’ and accountability of actual performance should be made one responsibility.

Mayne (2006) also recognizes the different roles evaluations may play. It may “…help management improve their policies, programs, and services.” In that case, management is the client of an evaluation. He also sees evaluation playing a role in the accountability process. In that case, funders of a programme may be the main clients. “Thus, unlike audit, where the role and client are reasonably clear to all, the role evaluation plays depends on who the client is.”

Moreover, performance audit institutions, which also make use of evaluation tools and methodology, have a legislated mandate, which gives them real independence from the objects of their audit process. Unlike consultants and institutions that conduct evaluations, who do not have a legislated mandate, and who are usually paid for by the entity that is responsible for the evaluated object. Therefore, Pollit & Summa (as cited in Mayne, 2006, p. 27) “… argue that the key differences are not the tools and methods used but rather the institutional settings of each.”

2.4.2 Assurance on NGO performance

Given the definition of internal auditing as “… an independent, objective assurance and consulting activity…” and the main research question about the role internal audit can have in providing assurance on NGO performance, the concept of providing assurance on NGO performance requires exploration. However, as performance auditing in NGOs is uncommon, no specific literature was found on assurance provision about NGO performance. Therefore, assurance provision based on performance auditing in government services was initially looked at, although even there information was scarce. As will be discussed in chapter 4, government performance auditing covers a much wider scope and is a more open and creative process than financial auditing. Consequently, “the [performance] auditor is not normally expected to provide an overall opinion on the achievement of economy, efficiency and effectiveness [aspects of performance] by an audited entity in the same way as the opinion on financial statements” (INTOSAI, 2004b, p. 7). Thus, assurance levels provided in performance auditing are not as standardised as in financial auditing.

In its guidelines on performance auditing, INTOSAI (2004c) further indicates that performance auditors may not always find complete and accurate data. But “… efforts to be exact may easily be in conflict with ambitions to make an intelligent analysis”. So limitations in finding suitable data should not prevent performance auditors to report on their assessment. As long as the user of the audit report is informed about the quality of information used, and as long as no other conclusions are drawn than those permitted by the quality of the data. Obviously, this has implications for the level of assurance that can be provided, which should be increasingly limited in nature, with decreasing levels of completeness and accuracy of underlying data and information. In order to find further guidance, an alternative sector claiming to make social benefit contributions was looked at: that of companies reporting about their social and environmental performance. O’Dwyer & Owen (2005) analysed assurance statements in a number of environmental, social and sustainability reports of companies. They noted different approaches to providing assurance among accountants and consultants that were responsible for those statements, as well as differences in the level of assurance provided. Consultants followed a more evaluative approach and tended to provide higher levels of assurance. One of the sets of guidelines on assurance provision used by O’Dwyer & Owen (2005), was the AA1000 Assurance Standard developed by AccountAbility (2008). These standards make use of different levels of assurance that may be provided to users of sustainability reports produced by companies. Annex 3 contains an overview of the different characteristics of the assurance levels used and the corresponding differences in the assurance procedures required1. A similar approach, defining different levels of assurance with corresponding characteristics of evidence, may be followed in providing assurance on NGO performance statements. That may help in creating at least some level of standardization in the provision of assurance on NGO performance. Obviously, more research is required on this.

1

The “principles” referred to in the overview in annex 3, are the accountability principles from the AccountAbility Accountability Principles Standard 2008, which are an important reference in the assurance standard procedures in the AccountAbility