• No results found

Non-monetary methods

In document UvA-DARE (Digital Academic Repository) (pagina 41-44)

Impact Assessment

16

Description

The goal of an Impact Assessment (IA) is not to rank the alternatives but to give a clear overview of the effects of the alternatives. In an Impact Assessment all effects are treated separately, positive effects as well as negative effects. Each effect is assessed in quantitative or qualitative terms. As in SCBA, the effects consist of direct, indirect, external, strategic, societal and environmental effects. An Impact Assessment can be integral or can explicitly be conducted for

15 The description of the SROI methodology is mainly based on Boyle and Murphy (2005), Steed and Nicholles (2011).

16 Many definitions of Impact Assessment (IA) are used in the literature. The term is used as a synonym for evaluation methodologies in general and to describe a subset of evaluation methodologies. To avoid confusion, in this study the term Impact Assessment is used to describe a subset of evaluation methodologies.

METHODOLOGIES 21

the most important or the largest effects, also called an assessment of the Key Performance Indicators. An important subtype of Impact Assessment is Economic Impact Assessment (EIA) in which only the economic effects are included. Another type often applied is Economic Effect Analysis (EEA), which only includes the economic effects considered to be of importance. An example of EEA in the space sector is the B.E.T.A. (Bureau d’Économie Théorique et Appliquée) methodology. This methodology can be used to evaluate spin-off effects. The methodology is based on extensive interviewing of key persons in each firm that received a contract. The methodology does not allow the estimation of the long term effects of programmes on the economy as a whole.

The Impact Assessments that have been found in the literature differ in scope, for instance in terms of effects considered, industries taken into account or geographical range. The main data sources for these studies are surveys, workshops and interviews to estimate turnover, employment and/or costs and profits. None of the studies was an Impact Assessment in the sense that it contained all effects. Many studies however were found in which an assessment was made of a part of the economic effects of a space activity (EEA) (see Space Foundation, 2011;

RPA, 2007; ASD-Eurospace, 2010; Bullock et al., 2002; Patureau et al., 2002; British National Space Centre, 2008, 2009; Davies, 2009; SIA, 2011; VEGA and Booz Allen Hamilton, 2004;

ESA, 2005; Technofi, 2007 and Ecorys, 2009). In only a few cases all economic effects (direct, indirect and induced) were given (EIA) (see Oxford Economics, 2009; Department for Business Innovation and Skills, 2010 and California Space Authority, 2010). In some studies economic effects were extended with external, societal or strategic effects (see OECD, 2011; UK Space Agency, 2010; PWC, 2006; Danish Agency for Science, 2008; NDP Consulting, 2011; NASA, 2007, 2008, 2010b; Schnee, 2009; Sadeh, 2006; Hertzfeld, 1998, 2002a; Technopolis, 2010; Fisher, 2009; Centre for Strategy & Evaluation Services, 2011 and Hallonsten, Brenner and Holmberg, 2004). Several studies applied the B.E.T.A. methodology (see Brendle, Cohendet and Larue, 1986; Cohendet, 1989; Amesse et al., 2002; Bach et al., 1995 and Bach, Cohendet and Schenk, 2002).

Advantages & drawbacks

Advantages of the IA are that it can incorporate different kinds of effects and that the information is processed in an explicit way. IA is also capable of dealing with different numbers of policies, criteria and actors. Another advantage of IA is that it is a relatively simplistic analysis, so the amount of data and calculations necessary are limited.

A drawback of IA is that it does not provide a ranking of policies or an attractiveness conclusion.

Different policies are not fully comparable after the analysis has been conducted. In this way every decision maker can draw his/her own conclusions and the analysis does not provide full help in the decision making process. The additional downside of the subtypes is that they do not provide a complete overview of all the effects.

Multi Criteria Analysis

Description

In Multi Criteria Analysis (MCA) policy alternatives (e.g. various space programmes) are first scored on different criteria. Second, the different criteria are weighed. The main goal of MCA is to structure the effects of the alternatives to aid the decision maker. It provides a systematic way

to measure and weigh effects for the relevant actors, where effects are not necessarily monetized (as opposed to Social Cost Benefit Analysis). It also provides a tool to aggregate the different effects.

MCA not only gives an assessment of the merits of an investments. It also provides a communication and interation tool for the different actors that are involved. MCA can help to explicitly take account of conflicts between actors regarding the impacts of a plan. These impacts may be estimated in a more or less objective, neutral way. This may reduce differences between stakeholders to discussions about the weights and the policies to be chosen. Normally, stakeholders will look at the “magic number”: the effect for their own country, region, company etcetera.

An example is a case when several actors agree that an alternative will have an estimated effect but disagree on the value of this effect. MCA might then be helpful in taking into account these different views by using different weighting schemes, proposed or inspired by the stakeholders, in turn. This shows how weighting affects the outcome of the analysis. Also, a part of the results may not change if the weighting is altered. These ‘robust’ results are an important starting point for reaching consensus among stakeholders.

When a Multi Criteria Analysis is conducted, several goals might be pursued:

Structure the policies and scores on criteria;

Identify a single most-preferred policy;

Rank all policies;

Indentify acceptable policies or policies that need to be considered in a next phase;

Combine positive features of policies;

Involve stakeholders in the analysis.

These goals are relevant in ex ante analysis, but also ex post it is useful to structure effects, compare policies and to involve stakeholders.

The descriptive approach of MCA is comparable to the Impact Assessment. However in most cases MCA will be more sophisticated and the analysis is followed up by a synthesis, consisting of assigning weights across the criteria. The MCA methods can be divided in non-compensatory methods, partially compensatory methods and fully compensatory methods. The main difference between the different types of MCA is how the synthesis is conducted. Non-compensatory methods do not permit trade-offs between criteria. Fully compensatory methods do allow for trade-offs between criteria; the scores on the different criteria are combined in one single value.

Partially compensatory methods do not combine all criteria to one single value but make a selection depending on the importance of a criterion: a strong performance on one criterion can compensate a weak performance on another criterion.

MCA has been applied to evaluate policy options and investments in space programmes, to prioritize space programmes and to benchmark space activity in different countries (see RPA, 2007; Smith, Dolgin and Weisbin, 2003; Tavana, 2006; Futron, 2010 and European Commission, 2009a, 2010a). The data with respect to the criteria and weights used mainly came from consulting experts.

METHODOLOGIES 23

Advantages & drawbacks

Advantages of the MCA approach are that it can incorporate a very diverse range of information and that the information is processed in a very explicit way. Also, MCA offers flexibility in the number of policies, the criteria, the weighting and the involvement of stakeholders. The flexibility of the MCA also forms a risk; MCA lacks methodological rigor. The weighing of the different criteria is difficult however, and is open for subjectivity or even manipulation.

In document UvA-DARE (Digital Academic Repository) (pagina 41-44)