EXTENSIONS TO EARNED VALUE
MANAGEMENT
PRACTICAL APPLICATION AND EMPIRICAL VALIDATION
Word count: 16.526Lien Fasseur
Student number : 01505743
Supervisor: Prof. Dr. Mario Vanhoucke
Commisioner 1: Tom Servranckx
Commisioner 2: Prof. Hipólito Sousa
Master’s Dissertation submitted to obtain the degree of:
1) Master in Business Engineering: Operations Management (University of Ghent)
2) Master in Service Engineering and Management (MESG) (Universidade do Porto)
PERMISSION
I declare that the content of this Master’s Dissertation may be consulted and/or reproduced,
provided that the source is referenced.
After five years of hard work, a milestone in my academic career is reached: the completion of my Master’s dissertation. This was a challenging task considering the COVID-19 pandemic. Especially the data collection became more complicated since company visits were no longer possible, and companies had other priorities than helping me in collecting data. I am very grateful for the people who have encouraged, supported, and helped me along the way.
First of all, I would like to thank my supervisor Professor Mario Vanhoucke for the opportunity to perform research in project management. During his courses, he fueled my interest in this foremost interesting and challenging topic.
Secondly, I also want to thank Tom Servranckx, first commissioner of this dissertation. He gave me a lot of valuable insights and was always there when I needed some help.
Next, a word of thanks goes to my second commissioner, Professor Hipólito Sousa, for helping me to collect Portuguese project data. Furthermore, I would like to thank everybody who helped in providing the empirical data.
Finally, I want to thank my family and friends. My parents, brother, and sister, for providing the love and support I needed, and my friends, for making the last five years unforgettable. Last but not least, a special word of thanks goes to Margot, Emma and Miguel, who took the time to re-read this dissertation.
List of Abbreviations VII
List of Figures IX
List of Tables XII
1 Introduction 1
2 Literature review: EVM and its extensions 5
2.1 The EVM/ES methodology . . . 5
2.1.1 Forecasting . . . 12
2.2 Schedule Adherence . . . 18
2.3 Reference Class Forecasting . . . 21
2.4 Bayesian Approach . . . 25 3 Problem definition 31 3.1 Objectives . . . 32 3.2 Hypotheses . . . 32 3.2.1 Initial hypotheses . . . 32 3.2.2 Project characteristics . . . 33 3.2.3 Additional hypotheses . . . 34
3.2.4 Relation with the research questions . . . 34
4 Methodology 35 4.1 Setting up hypotheses . . . 35
4.2 Empirical data collection . . . 36
4.3 Data analysis . . . 37
4.3.1 EVM/ES . . . 38
4.3.2 Construction of the reference classes . . . 39
4.3.3 RCF . . . 42
4.3.4 The Bayesian approach . . . 42
5 Empirical data 45 5.1 Collected projects . . . 45
5.1.1 Service Flats . . . 46
5.1.2 Residential Housing . . . 46
5.1.3 Office Building Luxembourg . . . 47
5.1.4 Residencia de Estudantes . . . 47
5.1.5 Via Expresso . . . 48
5.1.6 Two Warehouses . . . 48
5.1.7 Additional Warehouse . . . 48
6 Results 53
6.1 Overview of the forecasting accuracy per project . . . 53
6.1.1 Overview of the mean forecasting accuracy . . . 56
6.1.2 Discussion . . . 57
6.2 Overview of the forecasting accuracy per reference class . . . 58
6.2.1 Discussion . . . 59
6.3 Comparison of the theoretical and practical approach . . . 60
6.4 Comparison per stage of project execution . . . 62
6.4.1 Impact of other project characteristics . . . 64
6.5 Comparison of time and cost forecasting . . . 66
6.6 Comparison of the two countries . . . 67
7 Conclusion 69 7.1 Overall conclusions . . . 69
7.2 Shortcomings and recommendations . . . 71
Bibliography XVI
Appendices 0
A Overview of the empirical data XVII
A.1 Empirical data from the database . . . .XVII
B Mean MAPE XIX
B.1 Mean MAPE per specific construction sector . . . XIX B.1.1 Residential building . . . XIX B.1.2 Commercial building . . . XX B.1.3 Civil construction . . . XX B.1.4 Industrial building . . . XXI B.2 Mean MAPE per PD class . . . .XXII B.2.1 Low planned duration . . . .XXII B.2.2 Medium planned duration . . . .XXII B.2.3 High planned duration . . . .XXIII B.3 Mean MAPE per BAC class . . . .XXIV B.3.1 Low BAC . . . .XXIV B.3.2 High BAC . . . .XXIV
C %differences between MAPE’s XXV
C.1 Comparison between the technical and practical approach . . . .XXV C.1.1 Comparison per reference class . . . .XXV
A
AC Actual Cost
AD Actual Duration
AT Actual Time
B
BAC Budget At Completion
C
CPI Cost Performance Index
CV Cost Variance
E
EAC Estimated cost At Completion
EAC(t) Estimated time At Completion
ED Earned Duration
ES Earned Schedule
ES(e) Effective Earned Schedule
EVM Earned Value Management
F
FE Forecasting Error
M
MAPE Mean Absolute Percentage Error
N
ND(c) Normalized Deviation for cost
ND(t) Normalized Deviation for time
P
PDLC Parkinson Distribution with a Lognormal Core
PC Percentage Completion
PCWR Planned Cost of Work Remaining
PD Planned Duration
PDWR Planned Duration of Work Remaining
PF Performance Factor
PV Planned Value
P Vrate Planned Value Rate
R
RAC Real At Completion
RC Reference Class
RD Real Duration
RP Risk Profile
S
SAC Schedule At Completion
SCI Schedule Cost Index
SCI(t) Schedule Cost Index (with ES)
SPI Schedule Performance Index
SPI(t) Schedule Performance Index (with ES)
SPI(t)(e) Effective Schedule Performance Index (with ES)
SRA Schedule Risk Analysis
SV Schedule Variance
SV(t) Schedule Variance (with ES)
T
TP Tracking Period
TV Time Variance
W
1.1 The three components of dynamic scheduling (Vanhoucke, 2016a). . . 2
2.1 Earned Value Management: key parameters, performance measures and forecast-ing indicators (Vanhoucke, 2009). . . 6
2.2 The EVM key parameters for the different possible project outcomes (Vanhoucke, 2009). . . 8
2.3 The calculation of the ES for a project behind schedule (Vanhoucke, 2009). . . . 11
2.4 The SPI and SV performance measures versus the SPI(t) and SV(t) performance measures (Vanhoucke, 2009). . . 11
2.5 The EAC and EAC(t) forecasts for a fictitious project (Vanhoucke, 2009). . . 15
4.1 Research methodology. . . 35
4.2 Data analysis methodology. . . 37
4.3 Level of objectiveness for the different methods. . . 38
4.4 Construction of the different reference classes. . . 40
2.1 Overview of the different scenarios for EVM cost forecasting (Vanhoucke, 2012). . 12
2.2 Progress and distance classes (Caron et al., 2016). . . 28
2.3 Standard deviation for the likelihood function (Caron et al., 2016). . . 29
5.1 Overview of the main project characteristics of the collected empirical data. . . . 46
6.1 Time and cost forecasting accuracy for project 1. . . 54
6.2 Time and cost forecasting accuracy for project 2. . . 54
6.3 Time and cost forecasting accuracy for project 3. . . 54
6.4 Time and cost forecasting accuracy for project 4. . . 55
6.5 Time and cost forecasting accuracy for project 5. . . 55
6.6 Time and cost forecasting accuracy for project 6. . . 55
6.7 Time and cost forecasting accuracy for project 7. . . 56
6.8 Overview of the mean MAPE. . . 56
6.9 Overview of the best and overall best forecasting methods for the seven projects. 57 6.10 Overview of the mean MAPE for reference class 1. . . 58
6.11 Overview of the mean MAPE for reference class 2. . . 58
6.12 Overview of the mean MAPE for reference class 3. . . 58
6.13 Overview of the mean MAPE for reference class 4. . . 59
6.14 Comparison between the technical and practical approach to form reference classes. 61 6.15 Overview of the mean MAPE for time forecasting per project stage. . . 62
6.16 Overview of the mean MAPE for cost forecasting per project stage. . . 63
6.17 Comparison of time and cost forecasting accuracy. . . 66
6.18 MAPE for time forecasting per country. . . 67
B.1 Overview of the mean MAPE for the residential building sector. . . XIX B.2 Overview of the mean MAPE for the commercial building sector. . . XX B.3 Overview of the mean MAPE for the civil construction industry. . . XX B.4 Overview of the mean MAPE for the industrial building sector. . . XXI B.5 Overview of the mean MAPE for the projects with a short PD. . . .XXII B.6 Overview of the mean MAPE for the projects with a medium planned duration. .XXII B.7 Overview of the mean MAPE for the projects with a high planned duration. . .XXIII B.8 Overview of the mean MAPE for the projects with a low BAC. . . .XXIV B.9 Overview of the mean MAPE for the projects with a high BAC. . . .XXIV
C.1 Comparison between the technical and practical approach for RC1. . . .XXV C.2 Comparison between the technical and practical approach for RC2. . . .XXVI C.3 Comparison between the technical and practical approach for RC3. . . .XXVI C.4 Comparison between the technical and practical approach for RC4. . . .XXVI
Introduction
Project management is the discipline of planning, organizing, and managing resources in order to obtain the successful completion of specific project objectives. The main goal of project man-agement is to finish a project on time and within budget while fulfilling customer requirements. “Dynamic Scheduling" (Vanhoucke, 2012) is used to refer to the integration of the three project management dimensions: baseline scheduling, schedule risk analysis, and project monitoring and control. The three components of dynamic scheduling can be summarized as follows (Figure 1.1):
• In the scheduling phase, a timetable is constructed with a start and end date for each
activity, taking into account the activity relations, resource constraints, and other project characteristics, and aiming to reach a certain scheduling objective. This baseline schedule acts as a point of reference for the other two dimensions.
• During the risk analysis phase, Monte Carlo simulations are performed to generate activity
duration and cost deviations from their baseline values in order to assess the impact of these variations on the project’s time and cost objectives. This technique is also known as Schedule Risk Analysis (SRA).
• Project control is used to monitor the performance of a project during its progress and
measure whether it is performing according to the baseline schedule. In case of problems, corrective actions can be taken to get the project back on track.
When a project has started, it should be the main concern of every project manager to mon-itor and control the progress and performance of the project. Therefore, dynamic scheduling was renamed “Integrated Project Management and Control" (Vanhoucke, 2014) to stress the
Figure 1.1: The three components of dynamic scheduling (Vanhoucke, 2016a).
Project management is primarily concerned with decisions affecting the future. Consequently, the ability to accurately forecast a project’s final duration and cost is essential to successfully manage projects, especially since a major source of risk in project management is inaccurate forecasts of project costs, demand, and other impacts (Flyvbjerg, 2006). A widely accepted technique for making these forecasts is Earned Value Management (EVM). It provides a means to forecast the project’s future performance based upon its actual performance by utilizing the fundamental principle that patterns and trends in the past can be good predictors for the future. By applying EVM, the project manager can monitor the performance of the project’s execution and receive warning signals to take corrective actions in order to get the project back on track.
However, researchers and practitioners have been experiencing difficulties with the underlying assumptions and practicability of the technique. These difficulties have resulted in the applica-tion of new methods and adapted approaches from other research fields to extend the existing approach (Willems & Vanhoucke, 2015). Many extensions to EVM have been proposed during the past decades. The most well-known extension is the Earned Schedule methodology proposed by Lipke (2003). The ES method has been so well adopted that nowadays, almost everyone uses it. Throughout this dissertation, both EVM and ES will be used and referred to as the EVM/ES methodology. Other famous extensions are Exponential Smoothing (Batselier & Vanhoucke, 2017), Reference Class Forecasting (Kahneman & Tversky, 1979), the Bayesian approach (Caron et al., 2013), Schedule Adherence (Lipke, 2004), and many more.
Empirical validation
The primary reason why empirical data must be used in research is to validate academic results for practical use, and to show the relevance in a real-life setting that often differs from the well-designed artificial data (Vanhoucke, 2016b). Artificial and empirical data should not be seen as alternatives but are rather complementary. By combining simulated and real-life data, improved knowledge and deeper insights into the structure and characteristics of projects are gained. While artificial data can be used to test novel ideas under a strict design and controlled academic environment, the empirical data can serve as the necessary validation step to translate the academic research results into practical guidelines (Vanhoucke et al., 2016).
Many methodologies in literature lack empirical validation. Hence, this research will apply the Bayesian approach in project management and the Reference Class Forecasting technique on collected real-life project data in order to empirically validate these techniques and demonstrate their potential. The collected projects originate from Belgium and Portugal. Furthermore, it can be interesting to examine whether this country aspect influences the forecasting accuracy of the different forecasting methods. The major research question to ask is whether the extensions of the EVM/ES method discussed in this research, i.e. the Bayesian approach and the RCF technique, can outperform the traditional EVM/ES method. In addition to this major research question, three other research questions are answered throughout this dissertation:
1. Suppose the extensions have proven to be better performing than the EVM/ES method. Are there any specific situations and project characteristics (sector, size, stage of project tracking) for which this is not the case?
2. Which forecasting method works best for the Belgian projects? And which one obtains the best results for the Portuguese projects? Or in other words: does the country from which the project is collected influence the results?
3. Do the researched methods obtain better results for either time or cost forecasting?
The next chapter gives a concise overview of the relevant literature about the EVM/ES method-ology and some of its extensions. Chapter 3 explains the objectives of this dissertation, and the hypotheses necessary to reach these objectives are constructed and discussed. In Chapter 4, the general research methodology is described, going from formulating the hypotheses and collecting
Chapter 5 briefly describes the empirical data collected for this research and the assumptions that were made to enable further analysis of the real-life projects. A comprehensive overview of the results and its discussion are given in Chapter 6. Moreover, the hypotheses are tested and conclusions can be made. Finally, Chapter 7 illustrates the overall conclusions and shortcomings of this research.
Literature review: EVM and its
extensions
Project control is the process to observe the progress of a project during its execution in order to identify potential problems and/or opportunities in a timely manner (Vanhoucke, 2012). It aims at measuring and evaluating the actual progress of projects in order to complete the projects on time and within budget. When necessary, corrective actions can be taken to get the project back on track. Over time, many different project control techniques have been proposed and developed. The most used and well-known technique is Earned Value Management. In this chapter, an overview of the EVM methodology is given and some of its extensions are represented.
2.1
The EVM/ES methodology
The EVM technique was developed by the United States Department of defense in the 1960s as an integral part of the cost/schedule control system criteria (Batselier & Vanhoucke, 2015b). It is a powerful and simple tool that integrates the three critical elements of project management: time, cost and scope. It also implicitly incorporates the quality aspect by taking into account project progress (Willems & Vanhoucke, 2015). The technique allows the calculation of variances and performance indices for both cost and time, and forecasts of cost and schedule at completion. Consequently, it provides early indications of the expected project results based on the project’s performance and gives early warnings when corrective actions are needed to get the project back on track.
In the following paragraphs, a short overview of the EVM/ES methodology is given based on Anbari (2003) and Vanhoucke (2012) (Figure 2.1).
Figure 2.1: Earned Value Management: key parameters, performance measures and forecasting indicators (Vanhoucke, 2009).
The three key parameters to measure the project performance are:
• Planned Value (PV): the time-phased budget baseline. This is the amount of money that
should have been earned at a certain point in time according to the baseline schedule.
• Actual Cost (AC): the cumulative actual cost spent to a given point in time to execute the
project.
• Earned Value (EV): represents how much value has been earned, given the work that has
been done at a given point in time. It is calculated as follows:
BAC ∗ P C (2.1)
In this equation, BAC stands for Budget At Completion. This is the total expected budget for the project according to the baseline schedule. PC stands for Percentage Completion.
Cost performance can be determined by comparing the EV to the AC of the project. The performance measures concerning the budget of the project are the Cost Variance (CV) and the Cost Performance Index (CPI). These measures are calculated as follows:
CV = EV − AC (2.2)
CP I = EV
AC (2.3)
A CPI larger than 1 indicates that the project is under budget. It means that the monetary value that has been earned is larger than the actual amount of money spent until a certain moment in time. This corresponds with a positive CV. When the CV is negative (CP I < 1), there is a budget overrun, whereas CV equal to 0 (CP I = 1) means that the project is on budget.
The schedule performance can be obtained by comparing the EV to the PV of the project. The schedule performance measures are the Schedule Variance (SV) and the Schedule Performance Index (SPI), which are calculated as:
SV = EV − P V (2.4)
SP I =EV
P V (2.5)
When the earned value is larger than the planned value according to the baseline schedule, the SV is positive and the SPI is larger than 1. This means that the project is ahead of schedule. A negative SV (SP I < 1) indicates that the project is behind schedule and a SV equal to 0 (SP I = 1) shows that the project is on schedule.
Figure 2.2 gives an overview of the four different possible time/cost scenarios. Corrective actions are needed whenever the project is behind schedule or when there is a budget overrun. This corresponds with negative variances and performance indices smaller than 1.
Figure 2.2: The EVM key parameters for the different possible project outcomes (Vanhoucke, 2009).
Shortcomings
The EVM methodology has some shortcomings concerning the schedule performance measures. First of all, the schedule performance measures are expressed in monetary units instead of time units. To overcome this, three methods have been proposed (Vanhoucke, 2009): the planned value method (Anbari, 2003), the earned duration method (Jacob, 2003), and the earned sched-ule method (Lipke, 2003). The first two techniques translate the SV and SPI indicators from monetary units to time units, while the earned schedule technique calculates two alternative schedule performance measures (SV(t) and SPI(t)) that are directly expressed in units of time. Each of these extensions is discussed below.
The second shortcoming of EVM is the fact that the SPI always equals 1 at the end of the project, whereas the SV always converges to 0. The measures thus indicate that the project was finished on time, even when it was behind schedule. This can be explained by the fact that at the time of completion of the project, EV is always equal to PV. Since SPI provides unreliable results in the final third of the project, the traditional EVM metrics will fail in predicting the final duration of a project. This issue can be solved by using the alternative schedule performance measures of the earned schedule methodology.
Planned Value
The planned value method was introduced by Anbari (2003) to translate the SV into time units. The average PV per time period is called the Planned Value Rate (P Vrate) and is calculated as:
P Vrate=
BAC
P D (2.6)
In the equation above, PD stands for Planned Duration. This is the expected total duration resulting from the baseline schedule. By dividing the SV by the P Vrate, the SV can be translated into time units. The result, i.e. the SV in time units, is called the Time Variance (TV):
T V = SV P Vrate
(2.7)
If TV equals 0, the project is on schedule. A positive value indicates that the project is ahead of schedule, whereas a negative value means that the project is behind schedule.
Earned Duration
Jacob (2003) introduced the new term Earned Duration (ED) for forecasting the final duration of a project by using the SPI, which can be calculated as follows:
ED = AD ∗ SP I (2.8)
In the Eq.(2.8), AD stands for the Actual Duration, or Actual Time (AT), of the project at the current time. The earned duration translates the current actual project duration into an earned duration while taking the current schedule performance into account. Consequently, when the earned duration is lower than the current actual duration, the project has a delay (SP I < 1). Projects with good performance, i.e. SP I > 1, have earned more time than actually needed (EV > AD).
Earned Schedule
In 2003, Lipke proposed the ES methodology. The schedule performance measures formed by the ES technique behave appropriately during the entire duration of the project (Lipke, 2003) and thus overcome the quirky behavior of EVM near the end of the project. The ES method uses time instead of costs for measuring schedule performance by converting the earned value at a certain point in time to its equivalent duration needed to achieve the planned value. The Earned Schedule can be calculated as follows:
ES(t) = t + EV − P Vt P Vt+1− P Vt
(2.9)
For this equation, you have to find the moment in time t such that EV ≥ P Vtand EV ≤ P Vt+1. Figure 2.3 gives a visual representation of the calculation of the ES for a fictitious project. The schedule performance measures can be calculated as:
SV (t) = ES(t) − AD (2.10)
SP I(t) = ES(t)
Figure 2.3: The calculation of the ES for a project behind schedule (Vanhoucke, 2009).
The research of Batselier and Vanhoucke (2015b) has shown that the SPI(t) metric outperforms the SPI metric. Since ES(t) does not necessarily equal the AD, the SPI(t) and SV(t) metrics do not show any quirky behavior near the end of the project, as can be seen in Figure 2.4.
Figure 2.4: The SPI and SV performance measures versus the SPI(t) and SV(t) performance measures (Vanhoucke, 2009).
2.1.1 Forecasting
EVM metrics are designed to follow up on the performance of a project and to act as a warning signal to take corrective actions. Furthermore, they are also designed to forecast the cost and time performance measures based on the actual performance of the project and the assumptions about the near future (Vanhoucke, 2012). Eight cost forecasting methods and nine time forecasting methods can be formed based on the previously discussed performance measures.
Cost forecasting
The Estimated cost At Completion (EAC) is used for predicting the final cost of a project, which is essential to the success or failure of a project since it allows us to take corrective actions when the predicted EAC exceeds a certain threshold. This forecast is based on the actual costs already spent and a prediction of the future costs for the remaining work to be done, and is calculated as follows (Vanhoucke, 2014):
EAC = AC + P CW R (2.12)
PCWR stands for Planned Cost of Work Remaining and can be calculated by using the following formula:
P CW R = BAC − EV
P F (2.13)
The Performance Factor (PF) refers to the assumption made about the expected future project performance. The different scenarios and values for the PF are given in table 2.1.
Scenario SPI SPI(t) 1 According to plan (baseline schedule) PF = 1
2 According to current cost performance PF = CPI
3 According to current time performance PF = SPI PF = SPI(t) 4 According to current time/cost performance PF = SCI PF = SCI(t)
5 According to weighted time/cost performance PF = 0,8*CPI + 0,2*SPI PF = 0,8*CPI + 0,2*SPI(t)
Table 2.1: Overview of the different scenarios for EVM cost forecasting (Vanhoucke, 2012).
Table 2.1 makes use of an additional performance index that combines the schedule and cost performance index. It is known as the Schedule Cost Index (SCI) and is defined as:
SCI = SP I ∗ CP I (2.14)
Time forecasting
To predict a project’s final duration, the Estimated time At Completion (EAC(t)) is used. The ability to forecast the final duration of the project is crucial to the success of a project since it allows us to take corrective actions when the predicted EAC(t) exceeds a certain threshold. This forecast is based on the actual time that is already spent on the work done and a prediction of the duration for the remaining work to be done and is calculated by using the following formula:
EAC(t) = AD + P DW R (2.16)
PDWR stands for Planned Duration of Work Remaining. The way the PDWR is calculated, depends on the assumption made about the expected future project performance, i.e. the per-formance factor PF (Vanhoucke, 2014). Three different project situations can be distinguished:
• P F = 1: Future performance is expected to follow the baseline schedule;
• P F = SPI or SPI(t): Future performance is expected to follow the current time
perfor-mance;
• P F = SCI or SCI(t): Future performance is expected to follow the current time and cost
performance.
There are three different EVM time forecasting techniques, resulting in nine different EAC(t) formulas. Each of these time forecasting techniques will be discussed below.
The Planned Value Method
The planned value method (Anbari, 2003) does not directly give an estimate for the PDWR, but it relies on the planned value rate (P Vrate). According to the three different project situations,
the following forecasting formulas have been derived (Vanhoucke, 2012):
• Duration of remaining work as planned:
EAC(t) = P D − T V (2.17)
• Duration of remaining work following the current SPI trend:
EAC(t) = P D
SP I (2.18)
• Duration of remaining work following the current SCI trend:
EAC(t) = P D
SCI (2.19)
The Earned Duration Method
Under the earned duration method (Jacob, 2003), the PDWR can be calculated as follows:
P DW R = P D − ED
P F (2.20)
The three forecasting methods to predict the final project duration are (Vanhoucke, 2012):
• P F = 1: EAC(t) = AD + (P D − ED) = P D + AD ∗ (1 − SP I) (2.21) • P F = SP I: EAC(t) = AD + P D − ED SP I = P D SP I (2.22) • P F = SCI: EAC(t) = AD + P D − ED SCI = P D SCI + AD ∗ 1 − 1 CP I (2.23)
The Earned Schedule Method
The PDWR formula is calculated as:
P DW R = P D − ES(t)
P F (2.24)
Resulting in the following three earned schedule time forecasting methods (Vanhoucke, 2012):
• P F = 1: EAC(t) = AD + (P D − ES(t)) (2.25) • P F = SP I: EAC(t) = AD + P D − ES(t) SP I(t) (2.26) • P F = SCI: EAC(t) = AD + P D − ES(t) SCI(t) (2.27)
The EVM/ES forecasting accuracy was tested on empirical data by Batselier and Vanhoucke (2015b). They concluded that the EVM/ES approach can provide time forecasts that are just as accurate as the cost forecasts and thus, EVM/ES can be considered an appropriate technique for forecasting project duration.
Figure 2.5 displays the EAC and EAC(t) for a fictitious project.
Stochastic forecasting methods
In order to manage the uncertainty of projects, accurate estimates for durations, costs, resources, etc. are crucial for making decisions. Without these estimates, managers have to rely on their intuition and experience, which are often biased and hard to quantify. To estimate the duration of a project, you also need estimates of the activity durations. However, in reality, these activity durations are not deterministic, but stochastic and can thus take a range of values. Project activi-ties are said to belong to the same uncertainty class when they have comparable values for certain characteristics and every uncertainty class can be characterized by its own specific distribution profile (Vanhoucke & Batselier, 2019a). In order to be realistic, the distribution profiles should be formed based on real-life data and not based on estimates made by the project manager. The distribution profiles that were defined for the activities from past projects should be assigned to similar activities from the new projects. Consequently, the identification of similarities between activities is very important and is related to the research on Reference Class Forecasting (RFC), which is discussed in subsection 2.3. Vanhoucke and Batselier (2019a) recently developed a new stochastic forecasting method that consists of a new way of quantifying probability distributions for activity duration by making use of empirical project data. The method accurately estimates project uncertainty to improve project forecasting and decision-making.
Vanhoucke and Batselier (2019a) considered the Parkinson distribution with a log-normal core (PDLC), proposed by Trietsch et al. (2012), as a global distribution that appears appropriate in a real-life project context. The PDLC is a lognormal distribution extended by the Parkinson effect and the rounding effect. These effects both cause activities to be falsely reported as being on time:
1. The Parkinson effect implies that employees are not very willing to report an activity as completed early since there is usually no incentive to do so. On the other hand, they could benefit from labeling an activity as on-time, when it is actually early, since this would provide them a safety buffer for similar future activities (when these are estimated based on comparable historical activities).
2. The rounding effect results from the coarseness of the time scale that is used for reporting the activity performances. For example, when most activities have a planned duration of at least several weeks, it is likely that the chosen base time unit would be one week (i.e. 5 working days). However, if there are a few activities in the project that actually take 3–4
days instead of the planned week, they would be reported as being on time, and not as early, because 3–4 days would be rounded to one week.
The new stochastic forecasting method is an extension of the calibration method developed by Colin and Vanhoucke (2015). The extended calibration method makes use of real project data to derive realistic statistical distributions for the activity durations and introduces managerial partitioning. A calibration method is a method to filter data of empirical projects (inputs) by removing parts (calibration) that cannot be used further in the analysis and to identify the distribution parameters for activity durations that are the most appropriate in a real-life context (Vanhoucke & Batselier, 2019b). The method aims to classify the project activities in clusters that have identical values for the parameters (average and variance) of a predefined probability distribution (outputs). The input data is a set of empirical projects that are already completed and for which the outcome is known. Moreover, the empirical project data should consist of a set of planned activity durations (i.e. estimates made during the scheduling phase) and a set of known real activity durations that are collected after the project is finished.
Managerial partitioning consists of dividing the activities into partitions according to managerial criteria like Planned Duration (PD), Work Packages (WP), and Risk Profile (RP), which all in-dicate which activities are related to each other by nature (Vanhoucke & Batselier, 2019b). The extended calibration method was empirically validated by Vanhoucke and Batselier (2019a) on the projects from the database constructed by Batselier and Vanhoucke (2015a). It proved ex-tremely favorable and confirmed that partitioning is a promising direction for proving the realism of the lognormal distribution for activity duration. However, managerial partitioning is based on criteria defined by project managers, who are susceptible to bias in human judgement. To over-come this problem, Vanhoucke and Batselier (2019b) introduced a new approach: the statistical partitioning heuristic. It is a statistical procedure in contrast to the managerial procedure that requires human input. The statistical partitioning heuristic was also empirically validated on the projects from the database constructed by Batselier and Vanhoucke (2015a). The results were very good, and almost perfectly matched those obtained by performing managerial partitioning. This means that equally adequate partitions can be obtained through the statistical procedure without requiring managerial criteria and thus without being susceptible to human bias.
2.2
Schedule Adherence
The most fundamental criticism on the EVM/ES methodology is probably its assumption that the activities and precedence relations are known in advance and that estimates (activity du-rations, resource requirements, unexpected events, etc.) can be given within a certain range. However, Loch et al. (2006) mentioned that projects often do not fulfill these assumptions and are commonly plagued by unforeseeable events and/or unknown interactions among various ac-tions and project parts. Consequently, due to the cycles of rework, the accuracy of the EVM metrics can be biased, which can lead to incorrect management decisions (Cooper, 2003). The p-factor, or schedule adherence, proposed by Lipke (2004) makes a small yet possibly important step towards the right direction (Vanhoucke, 2013).
In 2004, Lipke proposed an extension to the EVM/ES methodology which takes into account schedule adherence and rework. The schedule adherence aims at connecting the logic of the baseline schedule with the dynamics of project progress and is formalized in the p-factor approach (Vanhoucke, 2014). It provides an early warning for later cost and schedule problems (Lipke, 2004). The p-factor indicates the degree of schedule adherence and measures the portion of EV accrued in congruence with the baseline schedule, i.e. the tasks which ought to be either completed or in progress (Vanhoucke, 2014). The p-factor is calculated as follows:
p = P
i∈Nmin(P Vi,ES, EVi,AT)
P
i∈NP Vi,ES
(2.28)
where:
• N = set of activities in the project
• P Vi,ES= planned value of activity i at time instance ES
• EVi,AT = earned value of activity i at the actual time (AT)
Thus, the p-factor is the ratio of the earned value corresponding to the baseline schedule divided by the total planned value at time instance ES. Since the nominator takes the minimum of the planned value at time unit ES and the earned value accrued at the actual time, the p-factor lies between zero and one (Vanhoucke, 2009). A schedule adherence equal to one (p = 1) indicates perfect schedule adherence, whereas a p-value smaller than one (p < 1) indicates lack of perfect schedule adherence.
The rationale behind this new measure is that work that is not performed according to the base-line schedule often indicates activity impediments (i.e. activities that are performed relatively less efficiently compared to the project progress) or likely results in rework. Whenever impedi-ments occur, resources are shifted from these constrained activities to other activities where they could gain earned value. Consequently, the project execution deviates from the original baseline schedule which might involve a certain degree of risk, since the latter activities are performed without the necessary inputs, and might result in a certain portion of rework (Vanhoucke, 2009).
The approach is used to adapt the current EV to an effective Earned Value (EV(e)) taking the risk of rework into account. The formula is as follows (Batselier & Vanhoucke, 2015c):
EV (e) = EV − R (2.29)
In this equation, R stands for Rework and is calculated as follows:
R = (1 − P Cn∗ e−m(1−P C)) ∗ (1 − p) ∗ EV (2.30)
where:
• P C = Percentage Completion
• p = schedule adherence (p-factor)
• n = 1 and m = 0.5
These values for n and m yield a nearly linear decrease of the rework fraction as PC increases (Lipke, 2011). The adapted EV(e) can be used to calculate ES(e) and SPI(t)(e), which can then be used in the EVM/ES time forecasting formulas:
ES(e) = t +EV (e) − P Vt P Vt+1− P Vt
(2.31)
SP I(t)(e) = ES(e)
However, the use of the p-factor to improve the forecast accuracy of time forecasting methods is limited. Nonetheless, despite its failure to improve the accuracy of time forecasts, some results have shown that the p-factor is a good indicator for predicting how accurate forecasts will be (Vanhoucke, 2013).
The schedule adherence approach has been empirically validated by Batselier and Vanhoucke (2015c) on real-life projects from the database constructed by Batselier and Vanhoucke (2015a). They concluded that this approach shows potential to improve forecasts when P F = 1, but has an adverse effect on the forecasting accuracy of performance-based methods (P F = SP I(t)).
2.3
Reference Class Forecasting
One of the impediments during project progress is the fact that the traditional EVM performance indicators, CPI, SPI and SPI(t), cannot be used to produce reliable forecasts in the early stages of the project’s execution. This is because EVM assumes that past data on the current project is the best available information. As a result, EVM solely relies on historical and objective information, even though this kind of information is hardly available in the early phases of the project (Gardoni et al., 2007). To alleviate this problem, the outside view, or Reference Class Forecasting (RFC), could be incorporated (Willems & Vanhoucke, 2015).
RCF was developed by Kahneman and Tversky (1979) and was later fine-tuned by Flyvbjerg et al. (2005). It was originally developed for making forecasts in the scheduling phase, when project managers have to estimate the duration, i.e. Planned Duration, and cost, i.e. Budget At Com-pletion, of the project before it is started. Traditionally, these estimates are made by forecasting the future course of specific events that could influence the course of the project (Batselier & Vanhoucke, 2016). This internal approach is called the ‘inside view’ and relies on the human judgement of the forecasters. It is likely to produce underestimation since it only incorporates singular information, i.e. evidence about the project under consideration, and neglects distribu-tional data, i.e. knowledge about the distribution of outcomes in similar situations (Kahneman & Tversky, 1979). Moreover, people will underestimate the costs, completion times and risks of planned actions, while overestimating the benefits of the same activities. This ‘delusional opti-mism’ is caused by optimism bias and strategic misinterpretation (Lovallo & Kahneman, 2003). Optimism bias is self-deception and can be explained by psychological explanations, that is peo-ple tend to judge future events in a more positive light than is warranted by actual experience. Strategic misinterpretation, on the other hand, is intentional deception and can be explained by political and organizational pressures. Here, project managers deliberately and strategically overestimate the benefits and underestimate the costs to obtain approval and funding (Flyvb-jerg, 2006). Consequently, managers pursue projects that are unlikely to be delivered on time or within budget. Thus, underestimating costs in the scheduling phase will result in cost overruns.
Flyvbjerg et al. (2002) did the first statistically significant study of cost overrun in transportation infrastructure projects. They observed that costs are underestimated in 9 out of 10 projects and actual costs are on average 28% higher than estimated costs. This is also the case for other types
(2012) observed that in 55% of the Dutch projects under consideration, actual costs exceed the estimated costs (resulting in cost overruns), whereas, in 44% of those projects, actual costs are lower compared to estimated costs (resulting in cost underruns). Furthermore, they concluded that although cost overruns are as common as cost underruns for projects in the Netherlands, the average cost overrun is larger than the average cost underrun. Lastly, they found that the main problem with cost overruns takes place even before construction has started. The average cost overrun in the pre-construction phase, i.e. period between the decision to build and the start of the project, is significantly higher than the average overrun during the construction phase, i.e. the period between the start of construction and the completion. There are three types of explanations for cost overruns (Flyvbjerg, 2007b):
1. Technical explanations: imperfect forecasting techniques, inadequate data, inherent prob-lems in predicting the future, lack of experience on the part of forecasters, etc. Technical explanations mainly relate to difficulties in predicting the future and are considered as honest mistakes. Technical errors may be reduced or eliminated by developing better fore-casting models, better data, and more experienced forecasters.
2. Psychological explanations are based on the concepts of planning fallacy and optimism bias. In the grip of the planning fallacy, project managers tend to make decisions based on delusional optimism rather than rationally weighting gains, losses, and probabilities. Overoptimism can be traced to people’s cognitive biases and their cautious attitudes to-wards risks when making decisions (Cantarelli et al., 2010). These biases are ubiquitous, but their effects can be reduced by simple reality checks, thus reducing the chance that organizations will blindly pursue projects that are unlikely to finish on time and/or within budget.
3. Political-economic explanations: project managers deliberately overestimate benefits and underestimate costs in order to make their projects look more attractive and thereby in-crease the chance of being selected and getting the project started. This results in ‘survival of the unfittest’, in which often it is not the best projects that are being approved, but the most misrepresented ones. Political-economic explanations are generally seen as the main explanation for cost overruns.
Although better forecasting methods have become available, data showed that cost estimates have not improved over time. Therefore, technical explanations can be ruled out as the main reason for cost overruns in projects since better methods have become available (Cantarelli et al., 2012). Furthermore, two fundamentally different situations can be distinguished: (1) project managers consider it important to get forecasts of costs, benefits, and risks right, and (2) project managers do not consider it important to get forecasts right, because optimistic forecasts are seen as a necessary means to getting projects started (Flyvbjerg, 2007a). If project managers genuinely consider it important to get forecasts right, they can use the Reference Class Forecasting method. By taking an ‘outside view’, much more accurate forecasts can be made since it provides a reality check on the more intuitive inside view.
Kahneman and Tversky (1979) found in their research that awareness of this optimism does not by itself produce a more accurate prediction and thus an external approach was needed to overcome it. They proposed a corrective five-step procedure for prediction, called reference class forecasting. It is a method for systematically taking an ’outside view’ on planned activities and is based on actual outcomes of past projects similar to the project being forecasted. Since it relies on historical data and actual performance, it bypasses human judgement and thus bias. Flyvbjerg et al. (2005) reduced the five-step procedure to the following three-step procedure:
1. Identifying a relevant reference class of past similar projects. The class must be broad enough to be statistically meaningful but narrow enough to be comparable to the project at hand.
2. Establishing a probability distribution for the reference class. For this step, empirical data for a sufficient number of projects in the reference class is necessary to make statistically meaningful conclusions.
3. Comparing the specific project with the reference class distribution in order to forecast the most likely outcome for the considered project.
Reference class forecasting is thus taking an ‘outside view’ on the project at hand. In contrast to the inside view, the outside view does not try to predict specific events that will affect the future course of the project, but instead it places the project in a statistical distribution of outcomes from the projects in the reference class (Lovallo & Kahneman, 2003). Research has shown that
than when using conventional forecasting methods (Kahneman and Tversky, 1979; Lovallo and Kahneman, 2003).
The advantage of reference class forecasting is the largest for nonroutine projects. These are initiatives that project managers and companies have never attempted before. For such new efforts, the optimism biases and misinterpretation are likely to be the largest. Nevertheless, composing a good reference class of comparable past projects becomes more difficult (Lovallo and Kahneman, 2003; Flyvbjerg et al., 2005).
Flyvbjerg (2006, 2007a) presented the first practical application of reference class forecasting in project management, namely the adoption of the method by the U.K. Department for Transport as part of project appraisal for large transportation projects. Flyvbjerg used uplifts to ensure that the risk of cost overrun is below certain predefined levels. The process is fully described in a guidance document: Flyvbjerg et al. (2004). Since then, the reference class forecasting technique has been used on several different real-life projects, such as Icelandic transportation projects (Fridgeirsson, 2016), the Bujagali dam in Uganda (Awojobi & Jenkins, 2016), building projects in Turkey (Bayram & Al-Jibouri, 2016), roadwork projects in Hong Kong (Flyvbjerg et al., 2016) and bridge projects in China (Liu et al., 2018). The first quantitative evaluation of the RCF technique was performed by Batselier and Vanhoucke (2016). They compared the performance of RCF with those of the most common forecasting methods in terms of accuracy, timeliness, and stability for both project cost and project duration. However, the application of RCF in project management is still limited and has so far been mainly focused on cost forecasting.
2.4
Bayesian Approach
Another extension of the EVM methodology originates from statistics and is based on Bayes’ theorem. In the Bayesian approach from the project management field, conditional probabilities are applied to combine both the inside and the outside view for forecasting the cost and time at completion (Willems & Vanhoucke, 2015). As already mentioned in the previous section, the inside view only incorporates information on the project itself, while the outside view utilizes information based on expert knowledge and historical data of similar projects. The Bayesian approach is thus a probabilistic forecasting technique, in which forecasts take the form of distri-bution profiles with corresponding confidence intervals, whereas the EVM methodology produces deterministic project forecasts, i.e. point estimates.
Several Bayesian models have been proposed in literature to formulate more reliable forecasts (Gardoni et al., 2007; Kim and Reinschmidt, 2009, 2011; Caron et al., 2013, 2016). The Bayesian adaptive model introduced by Gardoni et al. (2007) and the model by Kim and Reinschmidt (2009) aim at forecasting the actual duration of a project by deriving the S-curve that describes the progress of the project. These probabilistic frameworks forecast project progress and final time-to-completion, accounting for both objective information, such as empirical data and math-ematical models, and subjective information, such as expert knowledge, experience, judgment, opinions, beliefs, intuition, or a combination of these. The subjective information is converted into a prior distribution, which is then updated using the objective information that enters into the likelihood function (Gardoni et al., 2007). So, the performance data of a project during execution that is obtained during a certain Tracking Period (TP) is used to revise the prior estimates through Bayesian inference (Kim & Reinschmidt, 2009). Both models provide a higher accuracy than the traditional EVM approach. However, they both focus uniquely on time per-formance and require quite complex computations when compared to the simple EVM formulas. The Bayesian approach developed by Caron et al. (2016) is based on the integration of data records and qualitative knowledge provided by experts. It aims at improving the accuracy of the EVM forecasting metrics for both schedule and cost and obtaining more reliable forecasts in the early phase of project execution.
The Bayes theorem can be stated as (Gardoni et al., 2007):
p(θ|W ) = κ ∗ L(θ|W ) ∗ p(θ) (2.33)
where:
• θ = model parameter;
• p(θ) = prior distribution: represents the current status of knowledge about the future
before obtaining the observations W = (W1, W2, . . . , WT). This is any information based
on past experience and management judgement, i.e. subjective information;
• L(θ|W ) = likelihood function obtained after the observations W: represents the objective information on the estimates contained in the set of observations W;
• κ = normalizing factor = [R L(θ|W ) ∗ p(θ) ∗ dθ]−1;
• p(θ|W ) = posterior distribution: represents the updated status of knowledge about θ.
A general framework of the Bayesian adaptive forecasting method consists of the following three steps (Kim & Reinschmidt, 2009):
1. Generating prior distributions of the model parameters;
2. Updating the model parameters based on the reported data;
3. Using the updated model for forecasting.
The Bayesian model for cost and time estimates at completion, as proposed by Caron et al. (2016), will be discussed in further detail below.
The two performance indexes that are used are the Normalized Deviation for cost, N D(c)AT, and the Normalized Deviation for time, N D(t)AT, both measured at the actual time, AT:
N D(c)AT = ACAT − EVAT EVAT (2.34) N D(t)AT = AT − ESAT ESAT (2.35)
The parameter of inference for the Bayesian model is the Normalized Deviation of cost and time evaluated at completion: N D(c) = RAC − BAC BAC (2.36) N D(t) = RD − SAC SAC (2.37)
In the equations above, RAC stands for Real At Completion and RD for Real duration. These variables represent the final cost and duration, respectively, after finishing the project. SAC stands for Schedule At Completion and is a synonym for the planned duration (PD).
In the Bayes theorem, the prior distribution is the initial knowledge that the decision-maker has available for the specific parameter of interest. In the case of project management, the project team can rely on experts’ opinions and data records from similar projects completed in the past. This master thesis will focus on the available data of past similar projects. In order to translate this concept in Bayesian terms, a distribution has to be defined to represent the reference class of similar projects. It is assumed that the set of data with n projects follows a normal distribution with the following parameters:
µs= Pn i=1N DRD,i n (2.38) σs= s Pn i=1(N DRD,i− µs)2 n − 1 (2.39)
with N DRD,i = the actual cost or time deviation from the baseline estimate (BAC or PD, respectively) for project i.
After generating the prior distributions, the likelihood function should be formed based on the reported data. The empirical data collected during a certain tracking period will contain a single observation of the performance index, i.e. N D(c)AT and N D(t)AT. These are cumulated values
and sum up the entire interval from the project start up to the actual time. It is assumed that N D(c)AT and N D(t)AT are normally distributed, which can be justified by the fact that the
observation at the actual time can differ from the unknown final value N DRDeither by a positive or negative quantity. The relationship between the final unknown performance and the one at the actual time can be expressed as follows:
In Eq.(2.40), x is the observation at the actual time, µ is the final unknown parameter, σ(t) is the standard deviation and e is the error that, in the case of a normal likelihood function, follows the standard normal distribution, i.e. e ∼ N (0, 1).
The second assumption is that the standard deviation of the likelihood function σ(t) is known. From the previous equation, it can be noticed that σ(t)−1quantifies the degree of confidence that can be attributed to the observation at the actual time. So, the more the observation at the actual time can be assumed to be close to the final actual value, the lower the value σ of the likelihood function. To calculate the value of σ, two additional variables need to be introduced: the progress and distance indicator. The progress indicator corresponds to the Percentage Completed (PC) of the project and can assume three qualitative values according to the project progress at the actual time, as displayed in Table 2.2. The distance indicator can also assume a low, medium, or high value depending on the difference between the observation at the actual time and the expected value of the prior distribution:
∆c= |N D(c)AT − Ep[N D(c)RD]| (2.41)
∆t= |N D(t)AT − Ep[N D(t)RD]| ∗ SAC (2.42)
∆c and ∆t represent the cost and time distance, respectively. Ep[N D(c)RD] is the expected
value of the prior distribution of the final cost performance and Ep[N D(t)RD] of the final time
performance. In Table 2.2, the classes for the distance indicator are defined.
Low Medium High
Project progress [0, 30%) [30%, 65%] (65%, 100%] ∆c [0, 0.05] (0.05, 0.1] > 0.1
∆t [0, 3 months) [3, 5 months) ≥ 5 months
Table 2.2: Progress and distance classes (Caron et al., 2016).
When the class of project progress and distance has been identified, a corresponding value can be assigned to the standard deviation by following the matrix in Table 2.3. It can be noted that the standard deviation increases with an increasing distance and decreases as the project progress increases. The former can be explained by the fact that the bigger the difference between the observation at the actual time and the expected value of the prior distribution, the less reliable the observation is, and thus, the larger the standard deviation will be. The latter is a consequence
of the performance at the actual time becoming closer and closer to the final value as project progress increases.
Distance Low Medium High
Progress
Low 0.35 0.45 0.55
Medium 0.25 0.35 0.45
High 0.15 0.25 0.35
Table 2.3: Standard deviation for the likelihood function (Caron et al., 2016).
After evaluating the performance index at the actual time and the standard deviation from Table 2.3, the likelihood function is completely defined. Indicating N D(c)AT or N D(t)AT with x and the parameter of inference with µ, the expression of the likelihood function is as follows:
Lx(µ) = 1 √ 2πσ ∗ exp " −1 2 x − µ σ 2# (2.43)
By combining the prior distribution and the likelihood function according to the Bayes theorem, the following posterior distribution can be obtained:
Y s (µ|x) = Q s(µ)Lx(µ) R [Qs(µ)Lx(µ)] dµ (2.44) Y s (µ|x) : N σ 2µ s+ σs2x σ2+ σ2 s , σ 2σ2 s σ2+ σ2 s (2.45)
Finally, new forecasts can be obtained based on this posterior distribution.
This method has only been applied to three Oil and Gas projects (Caron et al., 2016). Conse-quently, further (empirical) validation of this method in other industries is needed. Moreover, the accuracy of the Bayesian approach should be evaluated on a wider set of data.
Problem definition
The EVM/ES method has been used for project control during the last decades. However, it cannot produce reliable time forecasts in the early stages of the project’s execution since it relies on the available past data of the project being forecasted. To overcome this problem, the traditional EVM/ES method can be extended with some newer techniques that produce accurate time forecasts during early project progress. The Reference Class Forecasting method provides uplifts based on the actual outcomes of past similar projects. Another appropriate technique is the Bayesian approach for project management. This approach relies on distribution profiles that are updated every tracking period based on the project data gathered during that period. It also uses reference classes to obtain a prior distribution. Since these techniques rely on past data, they have more relevant information available during the project’s early stages than the EVM/ES method. However, the challenge for both techniques is to select past projects based on appropriate project properties to obtain a reference class that is similar to the project being forecasted. Both approaches can be used in combination with EVM/ES or on its own. When used as an extension to the EVM/ES method, they provide an uplift to adjust the EVM/ES forecasts. When used on their own, they provide an uplift for the initial time and cost estimates (i.e. the planned duration and budget at completion).
In this chapter, an overview of the objectives of this dissertation is given. Furthermore, the hypotheses necessary to reach these objectives are constructed and discussed in section 3.2.
3.1
Objectives
This research aims to compare the forecasting accuracy of the RCF technique and the Bayesian approach with the forecasting accuracy of the traditional EVM/ES method. Furthermore, the main research questions to be answered are the following:
1. Can the EVM/ES method’s extensions discussed in this research, i.e. the Bayesian ap-proach and the RCF technique, outperform the traditional EVM/ES method?
2. Suppose the extensions have proven to be better performing than the EVM/ES method. Are there any specific situations and project characteristics (sector, size, stage of project tracking) for which this is not the case?
3. Which forecasting method works best for the Belgian projects? And which one obtains the best results for the Portuguese projects? Or in other words: does the country from which the project is collected influence the results?
4. Do the researched methods obtain better results for either time or cost forecasting?
To provide an answer to these research questions, some hypotheses need to be constructed and tested. These hypotheses are discussed in the following section.
3.2
Hypotheses
3.2.1 Initial hypotheses
Four hypotheses could be formed based on the shortcomings of EVM/ES described in literature. As mentioned by Willems and Vanhoucke (2015), the traditional EVM performance indicators (i.e. CPI, SPI, and SPI(t)) cannot be used to produce reliable forecasts in the early stages of the project’s execution. To alleviate this problem, Reference Class Forecasting (RCF) could be incorporated. So, the first hypothesis can be formulated as follows:
• H1a: EVM-RCF produces better forecasts (i.e. with a smaller MAPE) than EVM/ES.
• H1b: EVM-RCF produces better forecasts (i.e. with a smaller MAPE) in the early stages
of a project’s execution than EVM/ES.
Hypothesis H1a tests whether EVM/ES extended with the RCF method (i.e. EVM-RCF) obtains more accurate forecasts than the traditional EVM/ES methodology. Hypothesis H1b narrows
down the first hypothesis to the early phase of a project’s execution. MAPE stands for Mean Absolute Percentage Error and is used to measure the accuracy of forecasting methods. It is discussed in more detail in section 4.3.1.
The Bayesian approach, as proposed by Caron et al. (2016), aims at improving the accuracy of the EVM forecasting metrics for both schedule and cost, and obtaining more reliable forecasts in the early phase of project execution. This statement can be translated into the following hypotheses:
• H2a: EVM-BAYES produces better forecasts (i.e. with a smaller MAPE) than EVM/ES.
• H2b: EVM/BAYES produces more reliable forecasts (i.e. with a smaller MAPE) in the
early stages of a project’s execution compared to forecasts made with EVM/ES.
Hypothesis H2a tests if extending EVM/ES with the Bayesian approach (i.e. EVM-BAYES) results in better forecasts, while hypothesis H2b narrows it down in order to validate if this method also performs better in the early project’s execution phase.
3.2.2 Project characteristics
After collecting the empirical data, it became clear that the results of this research are highly dependent on the collected projects as each project is characterized by different elements. In order to take these project characteristics into consideration, two additional hypotheses can be added:
• H3: There is a difference in the MAPE for forecasts made with the researched methods for
projects with different characteristics (sector and size).
• H4: There is a difference in the MAPE for forecasts of the Belgian projects and the MAPE
for forecasts of the Portuguese projects.
While hypothesis H3 evaluates whether the specific construction sector and size (i.e. planned duration and budget at completion) have an impact, hypothesis H4 tests if the country influences the forecasting accuracy of the different methods.
Since both schedule and cost information were obtained, one more hypothesis could be added:
• H5: There is a difference in the MAPE for time forecasts and the MAPE for cost forecasts.
This hypothesis tests whether the researched methods obtain better results for either time or cost forecasting.
3.2.3 Additional hypotheses
By doing research and testing the first hypotheses, the idea arose to use both methods not only in combination with EVM/ES but also on their own. So, three additional hypotheses can be formulated:
• H6: Forecasts made with the RCF technique have higher accuracy (i.e. a smaller MAPE)
than the forecasts made with the EVM/ES method.
• H7a: Forecasts made with the Bayesian approach have higher accuracy (i.e. smaller MAPE)
than the forecasts made with the EVM/ES method.
• H7b: Forecasts made with the Bayesian approach are more reliable (i.e. with a smaller MAPE) in the early stages of a project’s execution than the forecasts made with the EVM/ES method.
Hypothesis H6 and H7a test whether the RCF technique and the Bayesian approach provide better forecasts than the EVM/ES method. Hypothesis H7b tests if the Bayesian approach obtains a better forecasting accuracy than EVM/ES during the early project completion stage.
3.2.4 Relation with the research questions
Hypotheses H1a, H2a, H6, and H7a can be linked to this dissertation’s high-level research ques-tion since they compare the performance of the forecasting methods under consideraques-tion to the performance of the traditional EVM/ES method. Research questions 2 and 3 are represented by hypotheses H1b, H2b, H3, H4, and H7b as these examine whether the project characteristics have an impact on the accuracy of the forecasting methods. Finally, hypothesis H5 checks if there is a difference between time and cost forecasting, and hence, addresses the fourth research question.
Methodology
This chapter gives an overview of this dissertation’s research methodology. Figure 4.1 represents the general steps of this methodology. An elaborate description of each step is given in the following sections.
Figure 4.1: Research methodology.
4.1
Setting up hypotheses
The initial hypotheses were formed based on shortcomings from the EVM/ES methodology men-tioned in literature. During the collection of the empirical data, it became clear that every project is different. Although all collected projects are construction projects, they all differ in planned duration, budget at completion, specific construction sector, and even country. Therefore, some additional hypotheses were formulated.
After analyzing the first results of the EVM-RCF and EVM-BAYES techniques, the decision was made to expand the analysis with RCF and the Bayesian approach as autonomic methods. To do so, three additional hypotheses were added. The set-up of all hypotheses is discussed in Chapter 3, on page 32.
4.2
Empirical data collection
The majority of the projects in the database constructed by Batselier and Vanhoucke (2015a) are situated in the construction industry. Considering that reference classes would have to be made, and thus, a lot of similar past project data would be needed, the choice was made to focus on collecting construction projects.
In order to collect the necessary data, construction and project development companies were contacted in Belgium and Portugal. Three Belgian project developers, one Belgian contractor, three Portuguese construction firms, and a professor at the Civil Engineering faculty of the University of Porto were willing to collaborate. After a first positive response through email, a company visit or Skype call was scheduled with the project manager of every company. During these company visits and Skype meetings, the data requirements were explained, and the projects that met these requirements were selected. Unfortunately, not every company was able to provide a project. One Portuguese construction company was no longer able to help due to other priorities that emerged, while the Belgian contractor had only detailed cost information about his projects. The professor’s project, about the construction of his house, contained all schedule information and actual costs, but did not include any budgets or tracking data and was therefore left out of the analysis. However, the other five companies were able to deliver the necessary project data. In total, seven construction projects were collected. Chapter 5 gives an overview of the relevant project characteristics of the collected empirical data, and it also provides a concise description of every project and company.
For every project, the obtained schedule and cost information was put into the standardized excel-file of the OR&S research group. By doing so, some assumptions had to be made. An overview of these assumptions is given in section 5.2. Afterward, the excel-file of each project was converted into a ProTrack file with the tool PMConverter. Both the standardized excel-file and the PMCon-verter tool can be downloaded at www.projectmanagement.ugent.be/research/data/realdata.
4.3
Data analysis
An overview of the methodology of the data analysis is given in Figure 4.2. The EVM/ES method provides a new forecast for the project duration every tracking period. The EAC(t)-ES forecasts are based on the project’s performance during the past tracking periods, while the RCF technique uses data of similar past projects. As mentioned in Chapter 3 on page 31, RCF can be used in combination with the EVM/ES method or as an autonomic approach. The Bayesian approach can also be used to adjust the EAC(t)-ES forecasts, or independently. It uses a prior distribution of past similar projects and updates its forecasts every tracking period based on the project’s performance.
Figure 4.2: Data analysis methodology.
As shown in Figure 4.3, every forecasting method has a different level of objectiveness. The traditional EVM/ES method provides an inside view since it only relies on the past data of the project being forecasted (i.e. subjective data). In contrast, the RCF technique gives an outside view as it does not use data of the project being forecasted, but data of past similar projects (i.e. the reference class). The Bayesian approach can be situated between those two extremes since it uses both subjective and objective data. The forecasting accuracy of the five different methods will be compared to each other. Each technique is discussed in detail in the sections below.
Figure 4.3: Level of objectiveness for the different methods.
4.3.1 EVM/ES
An extensive overview of the EVM/ES method and its formulas is given in Chapter 2, section 2.1. When PMConverter converts a ProTrack-file into an excel-file, it automatically generates all EVM/ES metrics per tracking period: the key parameters, the performance measures, and the forecasting metrics for both schedule and cost. The time forecasting methods used throughout this research are those from the earned schedule method:
• EAC(t)-ES (PF = 1)
• EAC(t)-ES (PF = SPI(t))
• EAC(t)-ES (PF = SCI(t))
The formulas can be found in Chapter 2 on page 14. To enable a fair comparison between time and cost forecasting, only the following three cost forecasting methods will be considered in the analysis:
• EAC (PF = 1)
• EAC (PF = CPI)
• EAC (PF = SCI(t))
It should be noted that the CPI is the cost equivalent of the SPI(t) metric. The formulas for the cost forecasting methods can be found in Chapter 2 on page 12.
For the traditional EVM/ES method, only the forecasting accuracy still needed to be calculated. This is done by using the MAPE-formula, which is explained in the following paragraph.
