The described monetary and non-monetary evaluation methodologies all have their own advantages and disadvantages (see table 3.2). The aptness of the various methodologies is partly expressed in terms of characteristics of the general methodology: completeness, feasibility and objectivity. The aptness in terms of usability for policy makers is linked to the clarity of calculations, the ambiguity (or not) of results and the acceptability. Each of these criteria has been specified further in Appendix B.
Completeness
The first characteristics on which the methodologies can be compared is the capability to take many different type of effects and actors into account; the completeness of the methodologies.
This completeness is indicated by the shaded area in Figure 3.1. The intensity of the shaded area (black versus hatched) represents to which extent the effects on the actors are taken into account.
When effects are only listed but not made fully comparable to other effects, which is the case in for example Multi Criteria Analysis, the area is hatched. When effects are made fully comparable, by translating effects to monetary values, as is the case with all quantifiable effects in Cost Benefit Analysis, the area is filled in. In practice this means that the non-monetary methodologies, IA and MCA are represented by shaded areas and monetary methodologies by a combination of shaded and filled in areas.
Box 3.2 Interpretation of completeness
The proposed (set of methodologies) presented in this report should cover all possible effects from space activities. One approach aimed at completeness might be to identify an optimal methodology to measure each effect separately. However, this would create a patchwork of effects measured by different methods (in other words, an impact analysis). This would be complete in the sense of covering all effects, but incomplete in the sense that the effects are not made comparable. Therefore, in this report, an approach aimed at different methods for different methodologies is considered as only partially complete.
Table 3.2 Advantages & drawbacks of methodologies in terms of criteria
Methodology features Usability in decision process
Completeness Feasibility Objectivity Claraty of calculations
METHODOLOGIES 25
Table 3.2 Advantages & drawbacks of methodologies in terms of criteria (continued)
Methodology features Usability in decision process
Completeness Feasibility Objectivity Claraty of calculations
The larger the shaded area in Figure 3.4 the more extensive the analysis is, and consequently the data requirements will be higher. Monetary methodologies will require more background data while non-monetary methodologies require less data.
As noted in the discussion of the methodologies above, the use of IO tables to evaluate public investments in space might be difficult since the space sector does not have a separate entry in the available IO tables. SCBA and SROI will require the most extensive analysis. Some effects may be relatively easy to monetise, but to monetise other effects, intensive desk research or (expert) survey may be needed. The more often these types of analyses are conducted, the more parameters have a generally accepted value.
Objectivity
The more standardized approaches and values are used in a methodology, the more objective it is. Financial Analysis, Computable General Equilibrium Analysis, Cost Effectiveness and Cost Benefit Analysis can all be regarded as highly objective. In these methods, causal links between investments and effects are based on parameters estimated on ‘hard’ data. Financial Analysis has a strong basis in accounting rules, CGE is based on calculated IO tables and CEA and SCBA have a strong basis in economic science. Over the years, extensive guidelines for SCBA have been developed which contain, among others, standardised parameters.
In contrast, in MCA the stakeholders influence the weights that are put on the effects, which makes the methodology subjective. Impact Assessment provides an overview of effects of policies rather than a ranking of policies. As no weights are used in the analysis, the methodology cannot be called either objective of subjective. However, the lack of weights may lead to a skewed picture in which unimportant effects are presented on equal terms with more important impacts.
Figure 3.1 Graphical representation of completeness of methodologies
Clarity of calculations
Some methodologies have a risk of a black box effect, meaning that stakeholders know the input and the output of calculations but not the process in between. Methodologies that suffer from the black box effect are Input-Output Analysis and Computable General Equilibrium. Due to the use of Input-Output tables and complicated methods, even the researchers themselves cannot fully understand the separate effects in IO Analysis and CGE. Social Cost Benefit Analysis and Social Return on Investment also face the risk of the black box effect. However this can partly be
Financial Analysis Social Cost benefit Analysis
Input-Output Analysis Social Return on Investm ent
Direct / indirect
Com putable General Equilibrium Multi Critera Analysis
Direct / indirect
METHODOLOGIES 27
avoided by a clear presentation of the calculations and results and by a thorough communication with stakeholders.
Methodologies that consist of a more transparent process are Financial Analysis and Cost Effectiveness Analysis, mainly due to their focus on a limited range of effects. The clarity of calculations of Multi Criteria Analysis is due to the use of weights instead of extensive calculations.
Clear advice
Methodologies that give a clear advice to the decision maker provide a full ranking of policies and also distinguish attractive from unattractive policies. Financial Analysis, Input-Output Analysis, Computable General Equilibrium, Social Cost Benefit Analysis and Social Return on Investment all provide these. Multi Criteria Analysis is able to provide a ranking in some cases, depending on the type of MCA, see section 3.2.2. But MCA does not provide the decision maker with an attractiveness conclusion.
Acceptability
The acceptability of a methodology is linked with some of the other characteristics. A methodology which takes only part of the types of effects into account will not be accepted to assess policies which have suspected effects outside that limited scope. This also holds for methodologies of which the calculations are a black box for stakeholders, or for subjective methodologies.
Financial Analysis limits its scope to financial effects. Due to this the outcome of this methodology is not accepted to assess large projects. IO Analysis uses assumptions about the state of the world which only hold in the short term. The outcome of Cost Benefit Analysis and Computable General Equilibrium will be more accepted among stakeholders with an economic background. The outcome of Impact Assessment and Multi Criteria Analysis will generally be more accepted by stakeholders as there is room for interpretation. Note that this also creates friction as different stakeholders might still not agree on the outcome of the analysis
PUBLIC INVESTMENTS IN SPACE 29
4 Data
“A man should look for what is, and not for what he thinks should be.”
Albert Einstein A lot of information is available on the space programmes themselves, but much less on related investments and on the impacts of investments on the economy. Data available within ESA can be used to complement macroeconomic data. A very important data limitation is the absence of an explicit space sector in economic data.
The data required by the methodologies presented in Chapter 3 are discussed in section 4.1, whereby the required data is divided into general data sources and methodology-specific data sources. The available data sources with respect to the economic impact of space activities are identified, categorized and analyzed in section 4.2. The aptness of the various data sources is determined using several criteria such as relevance, consistency, reliability, completeness and accessibility. Section 4.3 concludes by aggregating data requirements and data availability into one single table.