A critical assessment of TCB18 electricity

(1)

A critical

assessment of

TCB18 electricity

Dr Srini Parthasarathy, Partner

Alan Horncastle, Partner

Professor Emmanuel Thanassoulis,

Associate

Professor Subal Kumbhakar,

Associate

Charles Blake, Project Manager

Hannes Seidel, Analyst

(2)

1

(3)

Significant issues in TCB18

Themes and headline issues

◄Prevalence of data errors ◄Defining the input variable

◄Adjusting for differences in input prices ◄Indirect cost allocation

Data collection

and construction

◄Cost driver analysis

◄Sensitivity to the sample selected ◄Selecting candidate cost drivers ◄Aggregation of NormGrid

◄Adjusting for environmental factors

Model

development

◄Returns-to-scale assumption ◄Outlier analysis

◄Validation of DEA outputs

◄Identification of omitted cost drivers ◄Stochastic frontier analysis (SFA) ◄Dynamic efficiency (frontier shift)

Application and

validation

Sumicsid’s level of

transparency falls

short of good practice.

Sumicsid’s outputs do

not contain the

necessary information

to clearly follow its

analysis, or to validate

its analysis and

(4)

2

(5)

Transparency

Data collection

and construction

Model

development

Application and

validation

Sumicsid’s level of

transparency falls

short of good practice.

Sumicsid’s outputs do

not contain the

necessary information

to clearly follow its

analysis, or to validate

its analysis and

(6)

Transparency

Modelling codes and sourcing

• the report was not sufficiently detailed to follow each step of the analysis without the considerable effort we have undertaken

• our replication of TCB18 results was close enough to conclude on the quality of the benchmarking

• Sumicsid could have published modelling codes that would have ensured that third parties are able to follow each step in its analysis

• publishing modelling codes would allow third parties to follow each step, identify errors or

assumptions that have been made in the modelling process, and provide constructive comments • this would not entail the publishing of (confidential) data

• Sumicsid did not adequately source all of the external information it had used in the analysis

• for example, it did not present the environmental complexity weights used to adjust NormGrid in the main report or associated appendices

• one TSO was removed from the dataset without clear reasoning

(7)

Transparency

Modelling codes and sourcing: example of good practice

Simple commands with

descriptions of the code

Provide anonymised

outputs and clearly

reference where the results

are used in the report

(8)

3

(9)

Data collection and construction

◄Prevalence of data errors ◄Defining the input variable

◄Adjusting for differences in input prices ◄Indirect cost allocation

Data collection

and construction

Model

development

Application and

validation

Sumicsid’s level of

transparency falls

short of good practice.

Sumicsid’s outputs do

not contain the

necessary information

to clearly follow its

analysis, or to validate

its analysis and

(10)

Data collection and construction

Data errors (I) (section 3.1)

• significant data errors and inconsistencies are contained in the final dataset

• some TSOs flagged data inaccuracies to Sumicsid, which were not corrected for the final analysis

• empirical analysis relies on the accuracy of the data being used, and real data tends to be noisy

• Sumicsid’s application of DEA is deterministic, meaning that it is particularly sensitive to data errors

• data errors will therefore have a significant impact on TSOs’ estimated efficiency scores. For

example:

• one TSO’s TOTEX is overstated by 32%

• in another case, the value for weighted lines is overstated by 27%

• data errors will also change the shape of the efficient frontier, affecting other TSOs

(11)

Data collection and construction

Data errors (II) (section 3.1)

• Monte Carlo simulation considered in international and national exercises indicates that:

• most TSOs’ efficiency scores are highly sensitive to the addition of small errors, for example the second TSO in the chart may be between 55% and 100% efficient

• we cannot be certain (at the 90% significance level) that four additional TSOs are not operating efficiently based on data errors alone

• the errors simulated are less severe than actual errors in the TCB18 dataset

• a regulator needs to be confident that the proposed savings are feasible, so uncertainty inherent in the data and modelling needs to be taken into account

(12)

Data collection and construction

Defining the input variable (section 3.2)

• treatment of the cost categories (OPEX and CAPEX) is unnecessarily restrictive

• Sumicsid assumes that OPEX and CAPEX are equivalent and controllable

• OPEX and CAPEX are unlikely to be equivalent as they are subject to different normalisations • the significant variance in CAPEX shares across TSOs may be indicative of fundamentally

different operating models or reporting differences

• several alternatives that Sumicsid has overlooked exist and are used in regulation

• developing separate OPEX and CAPEX (or even more disaggregate) models, recognising the trade-offs between them

• constructing TOTEX as a weighted sum of OPEX/CAPEX • modelling OPEX and CAPEX as separate inputs

• all of these have advantages

• two input models can account for heterogeneity in expenditure mix

• separate OPEX/CAPEX models can account for more specific drivers capturing relevant heterogeneity (e.g. position in investment cycle can be accounted for)

(13)

Data collection and construction

Accounting for differences in input prices (section 3.3)

• differences in input prices across TSOs are insufficiently accounted for

• Sumicsid only adjusts 5.9% of the cost base for differences in input prices

• not normalising the costs risks conflating uncontrollable price-level differences with inefficiency

• in international benchmarking exercises (e.g. in benchmarking of rail infrastructure managers), all expenditure may require appropriate normalisation for price differences

• one option could be to adjust all OPEX and CAPEX. Results are sensitive to the method of

indexation (e.g. choice of PLI, proportion of expenditure adjusted) and require robust justification

(14)

Data collection and construction

Indirect cost allocation (section 3.4)

• the allocation rule used in TCB18 is not justified or sense-checked

• in some cases, the allocation of indirect expenditure is driven by large cost items that are unrelated to where indirect costs are incurred

• most TSOs’ estimated efficiency scores are insensitive to the allocation of indirect expenditure

• the allocation of expenditure to benchmarked activities is a conceptually important issue and may have a more material impact on a different sample

(15)

4

(16)

Model Development

Data collection

and construction

◄Cost driver analysis

◄Sensitivity to the sample selected ◄Selecting candidate cost drivers ◄Aggregation of NormGrid

◄Adjusting for environmental factors

Model

development

Application and

validation

Sumicsid’s level of

transparency falls

short of good practice.

Sumicsid’s outputs do

not contain the

necessary information

to clearly follow its

analysis, or to validate

its analysis and

(17)

Model development

Cost driver analysis (section 4.1)

• the cost driver analysis is not transparent. It is not explained:

• which cost drivers were considered in the project—how were they selected?

• how the final set was derived—how were some drivers discarded?

• how this final model was selected—what were the alternatives?

• cost driver analysis is based on unsubstantiated assumptions—for example:

• inference from OLS, robust OLS results is not valid in the presence of inefficiency

• functional form (linear) is not substantiated with evidence—alternative functional forms could result in alternative models

• the intercept is manually set to zero, with no justification

• the statistical assessment of the final model is limited and not informative

(18)

Model development

Sensitivity to the sample selected (I) (section 4.2)

• the estimated relationship between costs and cost drivers is highly sensitive to the sample on which

the model is estimated

• Sumicsid removed one TSO from the reference set, but does not explain the basis for this— e.g. what tests were performed?

• the estimated coefficients in the cost model are highly sensitive to the inclusion/exclusion of some TSOs

• NormGrid and weighted lines may have positive or negative coefficients depending on the exact sample

• on average, an increase in the cost driver is not associated with an increase in costs

• the relationship between the costs and cost drivers in Sumicsid’s model is highly dependent on the

exact sample of TSOs used and thus unreliable

Minimum Median Maximum

Intercept -13713113 -9717442 2641704

Adj. NormGrid -0.14 -0.09 0.54

Transformer power 2629.86 4103.23 4535.93

Weighted lines -3498.12 64817.61 71005.72

Observations 761 ₇₆ ₇₆

(19)

Model development

Sensitivity to the sample selected (II) (section 4.2)

• the year of assessment (2017) is arbitrary and the estimated efficiencies are highly sensitive to it

• one TSO that is a peer in Sumicsid’s analysis is estimated to be 82% efficient prior to 2016

• large changes in efficiency should not occur if the model is robust. Such changes may be indicative of model development issues, for example:

• relevant cost drivers may be missing, e.g. declining asset health may lead to higher expenditure • the input price inflation might have been understated, thus large investments may seem inefficient

• it is especially important to justify the large swings and how these coincided with a change in the

efficiency of the TSOs

(20)

Model development

Selecting candidate cost drivers (section 4.3)

• restriction of outputs to asset-based measures is unusual and unsubstantiated

• asset-based measures create issues when the asset structure differs between TSOs

• lack of comparison to models involving pure outputs means the cost drivers remain unvalidated • using alternative (asset-based) outputs to capture the same characteristics results in significantly

different results (e.g. transformer power and circuit end power are both proxies for capacity)

• it is considered best practice to arrive at appropriate cost drivers through iterative consultations with companies and stakeholders

(21)

Model development

Aggregation of NormGrid (section 4.4)

• construction of NormGrid is not robustly justified

• Sumicsid has not presented empirical evidence as to how the OPEX and CAPEX weights are derived

• the distribution of assets across TSOs is heterogenous. Thus, the weight attached to an asset class has an impact on estimated efficiencies

• changing the weights implied by regression analysis has an impact on some but does not validate the approach for others (where the impact is less material) given other limitations

• using components of NormGrid as the output variables in DEA significantly changes the distribution

of efficiencies 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Es tim at ed ef fic ienc y

(22)

Key messages

Adjusting for environmental factors (section 4.5)

• environmental adjustments to NormGrid are not supported by evidence

• we estimate a counterintuitive (negative) relationship between cost per NormGrid and environmental adjustment (i.e. TSOs operating in ‘more complex’ environments have lower unit costs)

• the biggest driver of environmental adjustment is the proportion of area covered in forest

• more operationally intuitive drivers of expenditure, such as urbanity, mountainous area and the

proportion of surface area covered in infrastructure, have a small impact on the overall

environmental adjustment

Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Figure 4.8. Source: Oxera (2020), ‘A critical assessment of TCB18 electricity’, April, Figure 4.7.

0 0.050.1 0.150.2 0.250.3 0.350.4 0.45 En viron m en tal w eig ht -1

(23)

5

(24)

Application and validation

Data collection

and construction

Model

development

◄Returns-to-scale assumption ◄Outlier analysis

◄Validation of DEA outputs

◄Identification of omitted cost drivers ◄Stochastic frontier analysis (SFA) ◄Dynamic efficiency (frontier shift)

Application and

validation

Sumicsid’s level of

transparency falls

short of good practice.

Sumicsid’s outputs do

not contain the

necessary information

to clearly follow its

analysis, or to validate

its analysis and

(25)

Application and validation

Returns-to-scale assumption (section 5.1)

• returns-to-scale assumption is not supported by statistical evidence

• Sumicsid states that it has used two tests to support its returns to scale assumption: • a Banker F-test for returns to scale on the DEA efficiencies

• a test that the sum of coefficients in a log-linear regression is different from 1

• even performing the same tests we find no conclusive evidence supporting the non-decreasing

returns to scale (NDRS) assumption made in TCB18

• the assumption has a significant impact on the shape of the efficient frontier and the efficiencies

• any decision made must be robustly justified by both conceptual and empirical evidence

Test Null

hypothesis Alternative hypothesis P-value Favoured RTS

Banker test (DEA) CRS NDRS 49% CRS

Banker test (DEA) NDRS VRS 5% VRS

Intercept (OLS in levels) CRS VRS 39% CRS

Sum of coefficients (OLS in logs) CRS VRS 9% VRS

(26)

Application and validation

Outlier analysis (section 5.2)

• outlier procedures are not justified and are scientifically inadequate

• Sumicsid relies on ARegV, but the outlier procedure specified in ARegV is neither legally binding nor sufficient in an international benchmarking context

• Sumicsid’s dominance test is not scientifically valid and is biased against the detection of

outliers. It is also not valid in small samples

• we recommend the use of a bootstrap-based dominance test instead

• the super-efficiency test adopted by Sumicsid does not test whether any of the remaining TSOs

are super-efficient in the new sample and is biased against detecting outliers in volatile samples

• we recommend a sequential application. We have also considered how appropriate thresholds for outlier detection can be defined

(27)

Application and validation

of DEA outputs (section 5.3)

• DEA outputs were not discussed in Sumicsid’s final report and remain unvalidated

• examining them reveals that most TSOs do not have NormGrid as the main driver of their efficiency • this contrasts Sumicsid’s claim that NormGrid is the strongest cost driver

• peer analysis appears to have been ignored

• there is no discussion of whether the identified peers for inefficient TSOs are appropriate

• it is possible to perform a precise one-to-one assessment of the homogeneity of the units

• alternatively, one could assess the weights (i.e. ‘scaling factors’) on peers—some ‘inefficient’ TSOs are benchmarked against TSOs that are more than 12 times smaller

(28)

Application and validation

Identification of omitted cost drivers (section 5.4)

• second-stage analysis forms the core of Sumicsid’s model validation

• second-stage analysis has no theoretical basis for model validation purposes

• Sumicsid has not considered necessary tests to justify the approach

• second-stage analysis would not support Sumicsid’s own model—neither NormGrid nor weighted

lines will come out relevant in the second-stage regressions

• this is not an argument to exclude NormGrid and weighted lines from the model. Rather, these

results indicate that second-stage analysis of this kind is not able to detect omitted cost drivers

• it cannot be argued (on the basis of this procedure) that no relevant cost drivers were omitted

Omitted cost driver Coefficient on omitted _{cost driver} P-value Relevant (omitted) _variable?

Adj. NormGrid -1.1E-10 0.48 No

Transformer power -2E-06 0.01 Yes

Weighted lines -2.6E-05 0.19 No

Environment 0.06014 0.88 No

(29)

Application and validation

Stochastic frontier analysis (section 5.5)

• Sumicsid has not used SFA or any other benchmarking method to validate its model

• SFA is not deterministic (unlike DEA) and is thus particularly relevant when we suspect a lot of noise in the data

• Sumicsid argues that the sample is too small for a full-scale application of SFA and does not present the results from SFA models

• this argument is not specific to SFA; most empirical investigations (including DEA) perform better on larger samples

• sample size alone is not a sufficient justification for ignoring SFA

• SFA indicates that noise is being identified as inefficiency by Sumicsid’s application of DEA

• only if the LR test detects statistically significant inefficiency and coefficients are intuitive could meaningful efficiency estimates be derived

• this suggests that caution is warranted against interpreting any estimated inefficiency in the

DEA as actual inefficiency rather than statistical noise

SFA estimator Does the LR test detect statistically _{significant inefficiency?} Intuitive interpretation of _{coefficients?}

Pooled SFA No Yes

Time-invariant SFA No Yes

Time-varying SFA No Yes

(30)

Application and validation

Frontier shift (section 5.6)

• dynamic efficiency results (-4.1% p.a.) cast further doubt on the validity of results

• such a large and negative frontier shift result differs from what is often applied in regulatory contexts and could be indicative of model mis-specification

• cost drivers poorly explain changes in efficient costs over time and differences in efficient costs between TSOs

• similar to the individual inefficiency estimates, frontier shift is highly volatile and indifferent from zero • the relative efficiency scores and frontier shift estimates in the TCB18 study are fundamentally

related—it cannot be that one piece of evidence is deemed more robust for regulatory applications than another, given both parameters are estimated using the same data, model and methodology

Pooled SFA _{(time-invariant)}Panel SFA _{(time-varying)}Panel SFA

NormGrid 0.551*** 0.929*** 0.891***

Transformer power 0.520*** 0.147 0.12

Weighted lines 0.0146 0.065 0.115

Time trend 0.0464 0.0284*** 0.0229

Constant 2.688* -1.593 -0.933

Implied frontier shift (% p.a. ) -4.7% -2.9% -2.3%

(31)

6 Conclusion and

(32)

Overall summary of our assessment

• we have issued a series of principles-driven recommendations

• significant additional work is needed to ensure that the outcomes from the study are

robust and fit for purpose

TCB18 suffers from fundamental weaknesses that mean that the estimated efficiency

scores cannot be used for regulatory, operational or valuation purposes

We have not presented the combined results in which all our recommendations are

jointly implemented

• we have identified numerous fundamental issues with Sumicsid’s analysis

• given the lack of relevant data, we could not develop alternative models with outputs as

cost drivers

(33)

Full assessment

(34)

Recommendations

Transparency and process

• clear conceptual and empirical justifications

for any assumptions

• publish modelling codes

(that are anonymised) and provide detailed description of

process and outputs

• establish an iterative data-collection

procedure to minimise data errors

• make effective use of panel data

and estimate static and dynamic efficiencies to assess

the validity of the models

• use alternative model specifications

and

evidence

from alternative estimators such as

SFA, economic and operational perspectives to motivate assumptions and cross-check

results

• it can be helpful to

consider debriefs involving all the parties on process

and

(35)

Recommendations

Analysis

• evaluate the impact of data errors

using various methods (e.g. Monte Carlo)

• as real data is noisy,

derive confidence intervals

around any results, including estimated

efficiency scores

• capture input price differences and other heterogeneity

robustly

• perform scientifically valid model development

• based on realistic modelling assumptions

• test the sensitivity to changes in the sample

• develop alternative model specifications

• the causes of year-to-year efficiency fluctuation

must be explored

• asset-based outputs

—if used—must be validated with pure output models

• develop an appropriate outlier-detection procedure

for international benchmarking

• analyse the outputs of DEA

to ensure that they are consistent with intuition

• the impact of omitted cost drivers

should be assessed through sensible robustness

(36)

Contact:

Dr Srini Parthasarathy

+44 (0) 20 7776 6612

srini.parthasarathy@oxera.com

Oxera Consulting LLP is a limited liability partnership registered in England no. OC392464, registered office: Park Central, 40/41 Park End Street, Oxford OX1 1JD, UK; in Belgium, no. 0651 990 151, branch office: Avenue Louise 81, 1050 Brussels, Belgium; and in Italy, REA no. RM - 1530473, branch office: Via delle Quattro Fontane 15, 00184 Rome, Italy. Oxera Consulting (France) LLP, a French branch, registered office: 60 Avenue Charles de Gaulle, CS 60016, 92573 Neuilly-sur-Seine, France and registered in Nanterre, RCS no. 844 900 407 00025. Oxera Consulting (Netherlands) LLP, a Dutch branch, registered office: Strawinskylaan 3051, 1077 ZX Amsterdam, The Netherlands and registered in Amsterdam, KvK no. 72446218. Oxera Consulting GmbH is registered in Germany, no. HRB 148781 B (Local Court of Charlottenburg), registered office: Rahel-Hirsch-Straße 10, Berlin 10557, Germany.

Although every effort has been made to ensure the accuracy of the material and the integrity of the analysis presented herein, Oxera accepts no liability for any actions taken on the basis of its contents.

No Oxera entity is either authorised or regulated by the Financial Conduct Authority or the Prudential Regulation Authority within the UK or any other financial authority applicable in other countries. Anyone considering a specific investment should consult their own broker or other investment adviser. Oxera accepts no liability for any specific investment decision, which must be at the investor’s own risk.

www.oxera.com