PERFORMANCE-BASED CONTRACTING:
Exploring the pre-conditions for the design of a performance measurement system – A multiple case study
MSc. Thesis
Ing. F. (Frank) Vlaming S2612178
MSc. Technology and Operations Management (TOM) Faculty of economics and business
In collaboration with,
January 2016,
1st Supervisor: Dr. Ir. W.H.M. (Wilfred) Alsem 2nd Supervisor: Dr. N.D. (Nicky) van Foreest
Supervisor Aiber: M.F.M. (Marcel) Kok MBA
TABLE OF CONTENTS
ABSTRACT ... 1
1. INTRODUCTION ... 2
2. THEORY ... 3
2.1 Performance based contract (output specification) ... 3
2.2 Contractors ... 3
2.3 Performance monitoring ... 4
2.4 Incentive mechanism ... 5
2.5 Conceptualisation of theory section ... 6
2.6 Research question development ... 6
2.7 Combining theory and research objective ... 8
3. METHODOLOGY ... 9
3.1 Case setting... 10
3.1.1 Runway extension Airport Eelde ... 10
3.1.2 Second Coentunnel ... 10
3.2 Case selection ... 10
3.3 Data collection ... 11
3.4 Data analysis... 12
4. RESULTS ... 13
4.1 Alignment between PBC and PMS ... 13
4.2 PMS robustness ... 15
4.3 Validity and reliability of PMS ... 17
4.4 Learning and data management ... 18
4.5 Alignment between incentive mechanism and PMS ... 20
4.6 Short summary of findings ... 22
5. DISCUSSION ... 23
5.1 Alignment between PBC and PMS ... 23
5.2 PMS robustness ... 23
5.3 Validity and Reliability ... 24
5.4 Learning and data management ... 24
5.5 Alignment incentive mechanism and PMS ... 24
5.6 General discussion ... 25
5.6.1 Practical implications ... 26
5.7 Limitations... 26
6. CONCLUSIONS ... 26
6.1 Managerial implications ... 27
6.2 Suggestions for further research ... 27
7. REFERENCE LIST ... 28
8. APPENDICES ... 30
Appendix A – Case description ... 30
Appendix B - Case study protocol... 31
Appendix C – Within case analysis ... 33
Appendix D – Cross case analysis... 35
ABSTRACT
Purpose - The aim of this research is to determine the pre-conditions for the design of a performance measurement system which is linked to a performance based contract (PBC). It tries to identify the pre- conditions for the design of a PMS. As these pre-conditions are yet unknown, elements constituting these pre-conditions had to be identified.
Method - By conducting a case study, this research aims to enlarge the theoretical body since little is known about Performance measurement systems and their interrelationship with PBCs in current literature. A multiple case study is conducted in the Dutch infrastructure sector, where PBCs are often used in public-private-partnership projects. Two cases which differ in complexity are used to generate rich full data. Data is collected by performing semi-structured interviews.
Findings - In total 18 elements were derived out of the data that together form five pre-conditions for the design of a PMS. These are respectively, Alignment between PBC and PMS; PMS robustness;
Validity and Reliability; Learning and Data management and Alignment between incentive mechanism and PMS. Due to the difference in case complexity the application of the elements and pre-conditions is determined per case.
Conclusions / practical implications – By finding pre-conditions for the design of a PMS, this research gives meaningful insights according to the interrelationship between the PBC and PMS and the way their mechanisms affect each other.
1. INTRODUCTION
Optimizing the potential of an asset in the field of asset management is the strive for maximal performance during the lifecycle of an asset with the use of sophisticated key performance indicators.
To ensure maximal performance, suppliers and customers can make use of Performance-Based Contracting (PBC) (Hypko et al. 2010). PBC can be defined as the contractual approach between supplier and customer in which the suppliers payment is (partially) based on the performance of the asset and/or suppliers performance (Selviaridis & Wynstra 2015). The basis of PBC relies on the evaluation of output measures instead of input measures.
PBC’s are increasingly used within the Dutch infrastructure sector between public-private partnerships (Verweij 2015). To increase the overall effectiveness of large infrastructure projects during the asset life cycle, private parties are made responsible for not only constructing but also for financing, operating and maintaining the assets (Hartmann & Dewulf 2009). These projects are also known as Design-Build- Finance-Maintain-Operate projects (DBFM(O)). Private parties are paid back by the public entity based on the delivered performance. Performance is often measured in terms of “availability” (Lenferink et al. 2013).
Robinson and Scott (2009) state that developing a solid performance measurement system which makes use of the right metrics and monitoring techniques is a key challenge in PBC. Moreover, Robinson and Scott (2009) describe three key components that affect the service delivery which are defined as output specification, performance monitoring system and payment mechanism. The output specification defines the content of the contract and therefore determines what services are required. Based on the output specification a performance monitoring system translates the outputs in actual measurements. A payment mechanism is directly linked to the performance monitoring system which relates rewards/penalties to the delivered performance (Robinson & Scott 2009).
In a recent study by (Selviaridis & Wynstra 2014) it is shown that the theoretical field of PBC has a relative little body of literature with respect to some PBC concepts. By reviewing over 241 papers they indicate that PBC concepts such as performance measurement, payment scheme and performance specification as a management control theory are heavily under theorized (Selviaridis & Wynstra 2014).
Hypko et al. (2010) supports this argument by claiming that extensions of the PBC concept regarding performance-based payments have not been extensively researched. From a managerial perspective it is therefore important to investigate how performance measurement and PBC are interrelated and how these concepts can be aligned.
This research focusses on the design of a performance measurement system (PMS) which is linked to a PBC. No research has been undertaken to determine the characteristics of a PMS. Since a PMS is an important component of a PBC, the purpose of this research is therefore stated as following:
This research determines the pre-conditions for the design of a PMS which is linked to a Performance- Based Contract to improve the alignment between the output specification and PMS design.
In order to fulfil the purpose of this paper the following research question is constructed:
RESEARCH QUESTION
RQ What are the pre-conditions for the design of a PMS that is interrelated with a PBC?
2. THEORY
This theory section is used to get a deeper understanding of the topic and the key concepts of performance-based contracting. Moreover it links the existing body of literature with the essence of this research by phrasing multiple sub questions in order to answer the research question.
2.1 Performance based contract (output specification)
In contrast to traditional outsourcing contracts such as fixed price or cost-plus contracts, Performance- Based Contracting is increasingly used in health care, logistics, infrastructure and the aero defence industry (Selviaridis & Norrman, 2015; Hartmann & Dewulf, 2009; Richardson & Jacopino, 2006).
Performance-Based Contracts can reduce the costs of ownership by linking a compensation for the supplier, paid by the customer based upon the output value of the asset (Kim et al. 2007). The capability to define clear performance targets which represents the output value for the customer determines the success of a PBC (Olander 2014). Additionally, PBC success depends upon developing a solid measurement system that measures whether performance targets are met (Olander 2014).
Performance based contracting does not only have positive outcomes. Behn & Kant (1999) describe the effect of “creaming” in which the supplier seeks for the cheapest and easiest ways to produce output in order to get incentives. However, the effect of creaming may be a consequence of not defining the performance objectives adequately in the PBC. In that case, the definition of “sufficient performance”
differs from both supplier as well as customer perspective.
Richardson & Jacopino (2006) identified a four step process for the development and implementation of a performance based contract. First of all, performance drivers have to be defined in the identification phase. Thereafter, performance indicators are determined using reliability and maintainability metrics.
Within this second phase, success and failure criterion are selected with the help of a causal analysis.
The third phase requires parties to specify target values for previous determined performance metrics.
A payment regime is linked to the system which contains incentives and fines (Richardson & Jacopino 2006).
2.2 Contractors
Several researchers describe the process of negotiation and competitive tendering (Hensher & Stanley 2008; Sultana et al. 2013). Using competitive tendering as a market mechanism does not necessarily
provide effectiveness of the PBC (Hensher & Stanley 2008). Suppliers collaborating in a consortium are heavily dependent on each other in terms of performance when building the asset as well as conducting maintenance on that asset. Though, trust is emphasized in multiple studies as a strengthening force of a collaboration between contractors (Lazzarotto et al. 2014; Randall et al. 2011). To ensure an effective relationship, information sharing between contractors enhances trust as well as avoiding costs and therefore has a positive effect on the ROI of the asset (Randall et al. 2014). Fernandez (2009)supports this argument by stating that frequent communication between contractors increases information symmetry which reduces the need for monitoring. Information symmetry and the need for monitoring are considered as important constructs in studying the design of the PMS. In fact, the more information symmetry the less frequent and/or intensive measurement a PMS has to perform.
Nevertheless, knowledge should be available about the asset to define measurable outputs for the PBC.
Knowledge availability affects the capability of measuring output of contractors and thus the performance of the asset. Besides, there should be a goal congruence between buyer and supplier to ensure effectiveness of outcome based contracts (Eisenhardt 1989).
2.3 Performance monitoring
In performance measurement, variables have to be defined that clearly represent performance as indicated in the contract. In the research of Hünerberg & Hüttmann (2003) ten indicators of performance are described on which the performance pricing/payment is based upon. Performance indicators described by Hünerberg & Hüttmann (2003) are for instance performance output (max/hour); quality;
or realised cost savings. Various other researchers indicate variables on which performance can be measured (Hensher & Houghton 2004; Guajardo et al. 2012; Sumo et al. 2012; Richardson & Jacopino 2006). Considering the mentioned performance indicators, availability of the asset and reliability of the PMS are expected to be relevant in this research. The purpose of having a performance monitoring system relies on these variables.
Often high investments are required for the implementation of a performance monitoring system.
Periodically and or continuously collecting data about performance involves high administrative costs (Selviaridis & Wynstra 2014). On the other hand performance monitoring is not only depending on a PBC. In regular asset management performance is monitored to ensure asset effectiveness. At the same time verification and validation of data needs to be secured. Neumann et al. (2008) state that metrics and measurement tools as part of the performance monitoring system develop as they are evaluated over time and learning grows. Several researchers reported the phenomenon of “gaming” in which suppliers purposely report higher performance levels than the real performance outcomes to get a higher bonus or lower penalty (Lu et al. 2003). Audits performed by external parties can guarantee independency and appropriateness of data. Moreover, Sumo et al. (2012) mentions innovation as an enabler of Performance-Based Contracts. Innovation can therefore be seen as another variable that can indicate the purpose of a performance monitoring system. In contrast to this argument, innovation and continuous
improvement can be restrained as a consequence of PBC in case of not defining reward/penalties to these parameters (Behn & Kant 1999).
2.4 Incentive mechanism
Several researchers investigated the role of the incentive mechanism which is linked to the Performance- based contract. Incentive mechanisms are used to ensure that suppliers adopt the project goals defined by the customer (Bower et al. 2002). Various incentive mechanisms are applicable in Performance- based Contracts such as ‘performance or technical incentives’ and ‘multiple incentive contracts’ (Bower et al. 2002). Performance or technical incentives are assigned to performance measurements such as quality or safety. In multiple incentive contracts, incentives are balanced according to customer priority (Bower et al. 2002). Abdel Aziz (2007) and Thomas Ng & Wong (2007) studied various aspects of the incentive mechanism. Both researchers incorporated the weighting or relative share a payment type should have in accordance to the project objectives. Additionally Sols et al. (2007) describes the concept of “dead zones” in which the service provider is neither given a penalty nor given a reward at an particular performance level. The “dead zone” should be determined based on the variance of performance levels extracted from historical performance data (Sols et al. 2007). In another research by Sols et al. (2008) it is claimed that a “proportionality zone” should be used in incentive mechanisms when performance is measured using multiple performance metrics. Proportionality zones indicate proper performance levels and the associated payment level when some performance metrics represent over-performance and some represent underperformance (Sols et al. 2008).
Thomas Ng & Wong (2007) demonstrated that the level of payment should be aligned at an acceptable level such that the service providers’ risk for a cash flow problem is minimized in case of poor performance. In addition to this fact Grinblatt & Titman (1989) claim that the incentives in case of underperformance should be as severe as in case of over performance. (Hypko et al. 2010; Hünerberg
& Hüttmann 2003) support this linkage to performance and payment, by arguing that more risk is involved for the service provider when a higher amount of payment is based on a particular performance objective. Considering these arguments, there seems to be a direct linkage between performance, risk and incentives from a service providers’ perspective (Selviaridis & Wynstra 2014).
Figure 1 provides an overview of these linkages.
Performance
Risk Incentive
Figure 1: Linkages between concepts: Adapted from (Selviaridis & Wynstra 2014)
2.5 Conceptualisation of theory section
Summarizing the constructs according to the theory section results in a model which is depicted in figure 2. The PBC itself represents contractual requirements that are demanded by the customer (e.g. a deadline for a working PMS). Additionally an output specification is defined in which the asset requirements regarding functionality/performance are established. For example the output specification may contain requirements such as “The tarmac has to have a resistance coefficient of X, otherwise the road is not
‘available’ for traffic”. Both contractual requirements as well as the output specification are then linked to a set of incentives. The combination of these incentives and their relative weight can be described as the incentive mechanism which is previously discussed in the theory section. The linkage between performance, incentive and supplier’s risks described by Selviaridis & Wynstra (2014) cannot be neglected when researching the PMS and the incentive mechanism. The customer has a risk component with respect to the PBC as well. However this risk component is considered as less important when investigating the PMS. As depicted in figure 2, the PBC requirements, output specification and the designed incentive mechanism serve as an input for the PMS. The PMS translates all requirements into a metrics design and measures whether the suppliers performance and/or asset performance complies with the requirements. Due to the input of the incentive mechanism the PMS calculates whether particular performance levels have consequences in a higher or a lower remuneration towards the supplier. Given the conceptualisation of figure 2, the scope and focus of this research lies on the PMS itself and its previous mentioned inputs. This is depicted in figure 2 by using an asterix symbol (*).
Output specification Incentive mechanism PBC
PMS (performance
indicators)
Remuneration
*
Figure 2: Conceptualisation of theory
2.6 Research question development
As previously discussed, the PBC and PMS are interrelated constructs and therefore their characteristics are heavily dependent upon each other. The objective of the PMS is to translate contractual requirements into an actual remuneration by combining the incentive mechanism with its measurement output on performance indicators. Since little research is conducted on the design of a PMS this study aims to find
pre-conditions which are forming the characteristics of a PMS. For instance a pre-condition for having snow is a temperature equal to/- or below 0 degrees Celsius. In order to fulfil the purpose of this paper the following research question is constructed:
RQ What are the pre-conditions for the design of a PMS that is interrelated with a PBC?
To get an answer to what pre-conditions determine the design of a PMS, firstly elements that affect these pre-conditions for the design of a PMS have to be identified. Referring back to the conceptualization of theory in figure 2, the purpose of a PMS is to translate the output specification and the incentive mechanism into an actual remuneration by performing measurements. However it is unknown what elements are required to fulfill this purpose. The combination of output specification, PBC requirements and the incentive mechanism result in unknown elements that together form the pre-conditions for the design of a PMS. To make this reasoning more clearly, when determining a pre-condition X, firstly elements (A,B,C…) have to be identified that constitute X. Connecting this reasoning with PBC literature, a pre-condition for the design of a PMS might be for example innovation. Elements forming the pre-condition innovation (X) might then be knowledge sharing(A) and connecting rewards on continuous improvement (B). Given the conceptual framework shown in figure 3, the elements which have to be identified by the following sub question can be defined as the independent variable. SQ1 is stated as follows:
SQ1 What elements constitute pre-conditions for the design of a PMS that is interrelated with a PBC?
After evidence is collected regarding what elements determine the pre-conditions of a PMS, it is important to understand how those elements affect the pre-conditions of a PMS. Therefore the underlying mechanisms of those elements have to be investigated to identify the differences between the elements and the way they could affect the PMS. Referring back to the previous example, the following sub question investigates how the elements ‘knowledge sharing’ and ‘connecting rewards on continuous improvement’ affect the pre-condition innovation. SQ2 is constructed as follows:
SQ2 How do those elements affect the pre-conditions for the design of a PMS?
Understanding the role of the PBC is at utmost importance to investigate the underlying mechanisms of a PMS. The following sub question is incorporated to investigate the characteristics of the PBC and the effect on the PMS, in order to make the design of the PMS effectively. With the use of multiple cases in this study, different PBC characteristics (low vs high cases complexity) can be identified and consequently the influence those characteristics have on the previously determined elements. As depicted in the conceptual framework in figure 3, the PBC characteristics serve as a mediating factor and therefore have influence on the elements that constitute the pre-conditions for a PMS. The pre- conditions of the design of a PMS are therefore conceptually determined as the dependent variable in this study. Given these arguments SQ3 is stated as follows:
SQ3 What is the influence of PBC characteristics on those elements?
Combining the three sub questions lead to the conceptual framework depicted in figure 3.
Element A
Element B
Pre-conditions
(X, Y, Z) PMS design
Low - high complexity
*
SQ1 & SQ2 SQ3 RQ
To be determined
elements PBC characteristics (low vs high complexity)
To be determined Pre-
conditions
Figure 3: Conceptual research framework
2.7 Combining theory and research objective
Up to now a conceptualization of theory is made in section 2.5 as well as a conceptual research framework is developed in section 2.6. Hence, a combination of both models results in a final conceptual model which is depicted in figure 4. This model clearly represents the research objective in combination with constructs from PBC literature. As shown in the conceptual model, the combination of output specification, incentive mechanism and the PBC itself serve as an input for a “pool” of elements which together form the pre-conditions for the design of a PMS. The line connecting between PMS design and PBC denotes that a PBC and a PMS are interdependent on each other as they serve as input/output for each other.
Output specification Incentive mechanism PBC
PMS design Remuneration
*
Elements (A,B,C)
Pre-conditions (X,Y,Z) SQ3
SQ1
SQ2
Figure 4: Final conceptual model
3. METHODOLOGY
To investigate the pre-conditions for the design of a PMS an exploratory multiple case study research is conducted. A multiple case study provide a stronger base of evidence in theory building and explanatory research (Yin, 2009). In contrast to other methodologies, an exploratory approach is therefore highly applicable in the form of a case study to ensure richness of data. The various steps in the research method are depicted in figure 5. Moreover, this methodology is chosen since, as discussed previously, little is known about performance measurement systems that are interrelated with a PBC. Since determining the design of a PMS is a complex exploratory construct and case study research is applicable for answering a “What” question, a case study methodology will be the appropriate research method (Voss 2009). By studying multiple cases with different characteristics regarding the PBC and the interrelated PMS, rich data can be obtained with respect to the relationship between these constructs. Moreover, multiple cases in one sector gives allowance to generalize results and find similarities between practices (Voss 2009).
It is expected that different aspects between cases determine the way a PMS can translate the PBC in an actual remuneration. Therefore this study gives the possibility to identify similarities as well as
“benchmark” applications of the PMS mechanisms between cases.
The aim of doing a multiple case study is to provide in-depth understanding in the performance measurement system which is the translating mechanism between the performance based contract and the payment mechanism. The unit of analysis will be defined as the “performance measurement system”.
By studying the underlying mechanisms of a performance measurement system using multiple case studies, a valid exploratory outcome of the characteristics will be the end product of the study.
Case selection
Case matrix
Data collection
Collect rich data by In-depth interviews
Studying documents Data analysis
Generalize data by coding
Conduct cross-case analysis
Validate and verify data
Discuss findings with expert in the field
Connection with existing literature body
Figure 5: Research method
3.1 Case setting
Cases were selected in the Dutch infrastructure sector since a PBC in combination with a PMS is applied in several infrastructure projects over the last years but relatively little research has been conducted.
Potential projects applicable for this research had already finished the realization phase and thus were exploited for a given period. This means that for each case a PMS is already implemented and executing actual measurements on performance levels. A second criterion for case selection was to identify contrary aspects between cases regarding contract complexity and the execution of the PMS. Voss (2009) states that identifying contrasting characteristics between cases at the case selection process can provide rich insights because meaningful differences are highlighted.
3.1.1 Runway extension Airport Eelde
As a less complex case regarding contract requirements and PMS, the runway extension of Airport Eelde is chosen. A performance based-contract has been applied with respect to the realization and exploitation of a runway extension for a time period of 10 years. Two contractors (Dura Vermeer & Imtech) tendered for the project and are finally responsible for the maintenance during the contract period. A consortium was established in the form of PASE (Provider Airstrip Eelde). During the 10 years the contractors have to prove with the use of a PMS that they perform according to the contract requirements stated in the PBC. Performance is mainly measured on availability in combination with other performance indicators that are applicable with respect to asset functions.
3.1.2 Second Coentunnel
A more complex PBC with a more extensive PMS is applied at the Second Coentunnel. A tender was established by the Dutch infrastructure agency “Rijkswaterstaat”, where in total seven suppliers succeeded in building the asset including its routes that are connected to existing infrastructure. A consortium was established in the form of the CoentunnelCompany, which as entity is responsible for the maintenance of the asset during an exploitation period of 24 years. During this period they have to prove their performance levels continually with the use of a fully automated PMS. Similarly, performance is measured in terms of availability for traffic, as well as other performance indicators regarding functions as tunnel safety.
3.2 Case selection
The case complexity is mainly dependent upon on the variety and amount of functions the infrastructure- asset has. Example given, the more functions an asset has, the more external connected objects that affect the overall performance of the asset. This has a direct consequence for the complexity of the PMS and its measurement mechanisms. Moreover, case complexity differs due to the amount of contract requirements the customer has on the asset functions, the exceptions on those requirements and particularly the interdependencies in those contract requirements. The amount of parties involved affects the ability to detect accountability and therefore influences the complexity of the PMS. As a final point
cases differ in complexity as a consequence of the amount of asset functions and the connected requirements. An automated PMS is utilized at the Second Coentunnel in contrast to a manual system in the case of Airport Eelde. Hence, the PMS of the Second Coentunnel is designed as an ICT platform, whereas the PMS of Airport Eelde does not constitute of a specific ICT system, but could be seen as a set of processes. Though, differences in case complexity does not mean that the cases do not show similarities. Both contracts are a form of DBFM and are largely similar in their incentive structure as they both consist of a malus system only. The differences and similarities per case are depicted below in Table 1.
Case 1: Airport Eelde Groningen Case 2: Second Coentunnel Amsterdam
Consortium Provider AirStrip Eelde (PASE) Coentunnelcompany
Case complexity Low High
Contract period
exploitation 10 years 24 years
Realization phase 2 years 6 years
Case characteristics
DBFM (light) DBFM
Two suppliers - one customer
Seven suppliers- one customer
Manual PMS Automated and manual PMS
PMS as a set of processes PMS as a ICT system
Incentive mechanism in the form of a malus system
Incentive mechanism in the form of a malus system Table 1: Case selection matrix
3.3 Data collection
Data was collected by five semi-structured interviews divided over the two cases. A semi-structured interview approach was chosen to be agile in finding new rich full insights. A case protocol was made (Appendix B) to ensure consistency in the data collection phase and securing reliability of the study (Yin 2009). The case protocol is designed based upon literature of Voss (2009) and discussions with an expert in the field of PBCs. Questions formulated as part of the case study protocol focused on contract requirements and how these affects the PMS in its measurement techniques and system reliability.
Moreover, the incentive mechanism was intensively discussed during interviews to investigate relationships among the contract and penalty structure.
In the case of Airport Eelde all contracting parties related to the PBC were interviewed to enhance richness of data and to create meaningful differences due to a variety of insights (Yin 2009). Two parties, namely Dura Vermeer and Imtech who have made a consortium under the umbrella of PASE (Provider Airstrip Eelde) are interviewed to get insights from a supplier perspective. The financial director of
Airport Eelde has been interviewed to collect data from a customer viewpoint. This contributed to the level of construct validity of the data (Voss 2009).
At the case of the Second Coentunnel, two persons have been interviewed that work for the consortium (Coentunnelcompany). One interviewee is involved in the daily management of the PMS and is a user of the PMS. Interviewing this person contributed to data collection regarding the supplier perspective of the PMS. The other interviewee has the function of contract manager on behalf of the Coentunnel Company and serves as intermediary between the contractors. Thus, this person gave valuable insights in the PMS from both the supplier as well as the customer perspective. As a consequence of his function the interviewee contributed to data regarding the PBC characteristics and the effect on the PMS. An overview of the data collection is depicted below in table 2.
Source of data Interviewee Length of
interview Second Coentunnel Amsterdam
Contract manager CoentunnelCompany 90 minutes Project leader Croon Elektrotechniek (one of the
suppliers and users of the PMS)
65 minutes
Airport Eelde Groningen
Financial director Airport Eelde 90 minutes
Project leader Imtech 90 minutes
Project leader Dura Vermeer 35 minutes
Table 2: Data collection per case
All interviews were recorded using a digital voice recorder and thereafter a literal transcript was made accordingly. The duration of the interviews varied from approximately 60 minutes to 90 minutes.
3.4 Data analysis
Data analysis has been executed by using the qualitative analysis software program Atlas.ti. This program is specifically applicable for case study research in order to reduce a large amount of data into subcategories (Miles & Huberman 1994). The subcategories were determined with the use of the theoretical background as well as the output of the interviews. Concerning those constructs in total 18 inductive codes (elements) were derived to sub categorize the data more intensively. Merging multiple determined main categories (pre-conditions). Thereafter qualitative data-analysis was conducted in the form of a within case analysis to identify unique patterns (Yin 2009). The within case analysis is depicted in Appendix C. In order to perform the within-case analysis adequately, a case description was made for both cases. The case description is used to identify case characteristics and how the PMS is used as a result of the contractual requirements.
The within-case analysis is subdivided in the subcategories. Per subcategory a top 3 of supported quotations by interviewees was selected which indicated a strong or a weak contribution to PMS effectiveness. With this intention differences in PMS configuration between cases were identified.
Additionally a cross-case analysis was conducted which can be found in Appendix D. This analysis is performed to find different patterns from the qualitative data. The connection between the theoretical framework and PMS design was established using the cross-case analysis (Voss 2009).
4. RESULTS
Data analysis resulted in a total of five pre-conditions which are discussed below. These are respectively, alignment between PBC and PMS; validity and reliability; PMS robustness; learning and data management and finally the alignment between incentive mechanism and PMS. The results chapter is therefore subdivided according to the determined pre-conditions which are discussed in consecutive order. The elements that constitute a pre-condition are separately discussed based on the most important quotations. A result table is given per pre-condition consisting of its elements as well as the quotations that resulted in determining these elements. To make the line of reasoning understandable each element described in the text and in the tables are attached to an superscript (Element X).
Note, multiple quotations resulted in determining an element. However only the most important quotations are depicted in the results tables. Each quotation originated from the case of Airport Eelde or the case of the Second Coentunnel. The case of Airport Eelde is therefore described as case 1 and the case of the Second Coentunnel as case 2. Example given, a quotation from an interviewee 2 at case 1 is therefore depicted in the result tables as quotation 2.1.
4.1 Alignment between PBC and PMS
Data analysis with respect to interviewee quotations resulted in three elements that together form the pre-condition: Alignment between PBC and PMS. These elements are respectively, measurement accuracy on contract requirement A; executing contract requirements B and handling contract complexity
C. An overview is depicted in table 3.
Case Data reduction first order quotations Elements Pre-condition PMS
1.1
"Availability is not measured implicit or explicit every day. If we get a message from the control tower that the lights don't work on one side, then a border is passed"
Measurement accuracy on
contract requirements A
Alignment between PBC and PMS 2.2
"With the use of automatic signals continually is measured on a millisecond timespan, if the system is OK or not OK. When not OK, system fails which results in a penalty"
1.1
"We have not actually led this fail result in a
availability penalty. But we discussed intensively with
the suppliers on how to prevent next time" Executing contract requirements B 1.2
"Weekly we have to measure how many lights have failed. When 100 meter fails subsequently, you fail as a supplier and directly a penalty is executed"
3.1
"We already had a few availability penalties, but we have discussions about who is responsible. This makes it complex"
Handling contract complexity C 1.2
"The complexity of the PMS, is related to its dependencies. All kind of exception rules are
applicable and those are all executed in the calculation unit"
Table 3: Pre-condition - Alignment PBC and PMS
One important aspect of this research was to investigate how the characteristics of the PBC have influence on the design of a PMS. The presence of a complex set of requirements with many interdependencies regarding performance on asset functions included in a PBC does not necessarily mean these functions are always actually measured on their performance. Example given at case 1, one PBC requirement is to keep the runway available for handling airplanes at all times. However the runway is not measured continually on the requirements regarding availability as stated in the PBC. As interviewee 1.1 declares: “Availability is not measured implicit or explicit each moment in time. For example, we get a message from the control tower that half of the runway-lighting broke. Then we decide about the financial consequences according to the contract”. It has been observed that the measurement accuracy on contract requirements A at case 1 is rather simplistic in contrast to the measurement accuracy on contract requirements A in case 2. At case 2, all contractual requirements are to a large extent automatically measured, which results in a higher measurement accuracy on the contractual requirements A. As interviewee 2.2 claims: “Especially with the automatic measurement system, continually is measured on a millisecond time span if the system is OK or Not OK. When Not OK, the system fails”.
One other element extracted from the qualitative data was the fact that PMS measurement output was not always executed according to contract requirements B in the form of a penalty. It appeared to be in case 1, that measurement output with respect to underperformance was not necessarily translated in the form of a penalty according to the contract. As a matter of fact, the particular aspects regarding
“underperformance” were discussed internally at the Airport and thereafter was decided whether it had financial consequences. In some cases penalties were not executed according to contract requirements B despite the underperformance according to the PBC requirements. The contractual relationship between contractors was considered as an important aspect and outweighed the execution of contract requirements B regarding penalties. The opposite appears to be at case 2 where measurement output from the PMS is always executed according to contract requirements B and therefore results in a penalty when the supplier or asset underperforms.
On top of that a third element was assigned to quotations from interviewees. Both cases show a difference in contract requirements. The difference mostly is related to interdependencies on requirements and exception rules. This directly affects the PMS and the way it handles contract complexity C. All exception rules and interdependencies on requirements have to be incorporated in the PMS. As interviewee 1.2 stated: "The complexity of the PMS is related to its dependencies. All kind of exception rules are applicable and those are all executed in the calculation unit".
The determined elements (measurement accuracy on contract requirements A; executing contract requirements B and handling contract complexity C) form together the pre-condition “Alignment between PBC and PMS”. The way each element is fulfilled influences the alignment between PBC and PMS.
Example given, the higher the measurement accuracy on contract requirements A, the higher the alignment between PBC and PMS. The same holds for the other elements. Therefore the alignment between PBC and PMS and its underlying elements have to be considered when designing a PMS.
4.2 PMS robustness
Four elements were determined which constitute the pre-condition: PMS robustness. These elements are respectively, Adaptability on contract phases D; Scenario management E; Resilience to contract change
F and System redundancy G. An overview is depicted in table 4.
Case Data reduction first order quotations Elements Pre-condition PMS
1.2
"We implemented a simple PMS during the realisation phase. And a fully automated PMS during exploitation"
Adaptability on contract phases D
PMS robustness 2.2 "Over the time technology evolves. Thus we adapt
the PMS and modify it during contract period"
Scenario management E 1.2
"We have to prove that traffic intensity has increased and contract requirements have to be changed. However the function remains the same"
3.1
"The airport can identify new requirements due to law changes. We as suppliers have to adapt, and
measure for instance more frequently"
Resilience to contract change F
2.2
"The system has to be fully functional, so we build in some redundancy. For instance our data is saved
at two locations"
System Redundancy G 1.2
"The camera's for instance, we check them each week to make sure they perform. According to the contract we have to prove the working each month"
Table 4: Pre-condition – PMS robustness
PBC characteristics are considered an important aspect which affects a PMS design. Contractual requirements and the change of these requirements by internal and external forces were found to affect the PMS. Contractual requirements during contract period could be forced to change due to different pre-determined contract phases. For both cases contractual requirements regarding asset and supplier performance were different during the realization phase than the exploitation phase.
Accordingly the PMS should be compatible with contractual requirements and as a consequence be adaptable for contractual phases D. As in case 2 interviewee 1.2 stated: “During the realization phase our performance was measured on availability of the first Coentunnel, but performance requirements as well as the penalties were less heavy than in the exploitation phase”. In case 1, a similar adaptability to contractual phases D was required from the PMS. In other words the design of a PMS changes as a result of different requirements when changing contractual phases.
Additionally, traffic intensity scenario’s affected required performance in both cases. As functional requirements on asset components (e.g. tarmac, runway lighting) remained the same, contractual requirements could change which could result to a different applicability of the PMS. Example given, performance requirements regarding tarmac resistance could be set at a lower level when traffic intensity reached a lower level. The PMS is therefore affected as a consequence of scenario management E during
contract period. Another form of Scenario management E was highlighted by interviewee 2.2: “The state technology may evolve over time and therefore contractual requirements regarding the PMS change.
This results in new releases of the PMS during the contract period”. As a matter of fact the PMS has to be agile to contract changes in the form of scenario management E. Likewise law changes (e.g. regarding asset safety) may force the customer to change asset requirements and directly affects the applicability of PMS mechanisms. This occurred in case 1 as depicted in table 4. In other words a PMS has to be resilient to contractual change F and agile in its mechanisms.
Additionally the use of system redundancy G by the supplier can be applied at a PMS. As in case 2 suppliers have to prove a certain performance level for a particular asset component according to the PBC, but delivered over performance with the intention to prevent penalties. As an example, multiple back-ups are made of PMS data to prevent any penalties regarding PMS functionality while this was not required in the PBC. In case of PMS failure the suppliers were therefore always capable of proving their performance levels. Thus a particular system redundancy G was built in by the supplier which influences the design of a PMS.
The determined elements (Adaptability on contract phases D; Scenario management E; Resilience to contract change F and System redundancy G) form together the pre-condition “PMS robustness”.
Elements D, E, F mostly affect the “agility” of the PMS due to internal and external forces. These elements should be considered when designing a PMS.
Together with the elements D, E, F , the element system redundancy G can make a PMS more robust if they are considered when designing a PMS.
4.3 Validity and reliability of PMS
A pre-condition: Validity and reliability was determined with the use of four elements. These elements are respectively, Verification H; Measurement on different levels I; Internal auditing J and external auditing K. An overview is shown in table 5.
Case Data reduction first order quotations Elements Pre-condition PMS
2.2
"We only have to prove is that the system complies with the output specifications. We show this with our verification of the automatic measurement signals"
Verification H
Validity and reliability 2.1 "We mainly measure on product level. The audits are
performed on process level"
Measurement on different levels I 2.2
"Civil components are measured mainly on product level. Technical installations are mainly measured on system level"
3.1 "For our processes related to the PMS we have every year an internal audit via our quality department
Internal auditing J 1.1
"Thus they have to inspect and report. However we can perform an audit, but they perform also internal audits"
2.2
"We have secure that every measurement tool was executed in the PMS. And therefore we incorporated an accountant to control if all registrations where made
properly" External auditing K
1.2 "On the other hand we have to prove that we have an audited PMS"
Table 5: Pre-condition – Validity and reliability
The PMS translates contractual requirements into measurement mechanisms that finally results in an actual payment due to the incentive structure. A verification matrix was applied by the suppliers in case 2 to verify H if measurements were executed correctly. On top of that an extra control system was used to verify if all automatic measurements took place. In other words this extra control mechanism was testing whether all measurement signals were incorporated in the PMS. The extra control system by means of a verification H tool was built in the PMS since suppliers have to prove the PMS reliability to the customer (Rijkswaterstaat) according to the PBC. Both cases provided evidence of measuring on different levels I to verify measurement data of the PMS. Example given at case 2 interviewee 2.2 stated:
“Civil components are mainly measured on product level, whereas technical installations are to a large extent measured on system level” Rijkswaterstaat describes the measurement on different levels as
“system-based contract control” as a form of quality assurance (Rijkswaterstaat 2015).
Auditing was found to be another element which came forward out of the interviews in both cases. A distinction is made between internal auditing J and external auditing K. Internal auditing is conducted by and initiated by the organization itself. External audits are conducted by an external party to control an organization on its validity. In case 1 external audits K performed by external parties take place at process level as well as system level. The elements, measurement on different levels I and performing external audits K are therefore likely to be interrelated.
Additionally one of the suppliers conducted internal audits J on their own processes with respect to their PMS and maintenance management. Audits performed at the case 2 mainly take place on system level by means of an EDP (Electronic Data Processing) audit.
A combination of the elements, Verification H; Measurement on different levels I; Internal auditing J and external auditing K form together the pre-condition Validity and reliability. The usage of elements H, I secure reliability of a PMS. Reliability is important since it secures whether measurements are performed consistently. Validity of the PMS is secured by the elements J, K. Validity is important for a PMS as it secures whether measurements are well founded and corresponds accurately to reality.
Summarizing these findings, a combination of the elements H, I, J, K can make a PMS more valid and reliable in its design.
4.4 Learning and data management
Data analysis with respect to interviewee quotations resulted in four elements which together form the pre-condition: Learning and data management of the PMS. These elements are respectively, Information exchange about performance L; Adaptability on information PMS M ; Use of maintenance history N and Data for external purposes O. An overview is depicted in table 6.
Case Data reduction first order
quotations Elements Pre-condition
PMS
1.2
"This kind of information flows about performance goes through the PMS. We measure, and put this in the system. Rijkswaterstaat examines this"
Information exchange about performance L
Learning and data management 2.1
"We have every month meetings where we discuss performance information"
1.1
"There are standards and indexes for. We determined that the requirements where to heavy.
Therefore we executed a contract change"
Adaptability on information PMS M
2.2
"On the one hand we execute our inspections to ensure we suffice the requirements. On the other hand we use this information to optimize our maintenance".
Use of maintenance history N
3.1
"When measurements are
performed, you will look on the on hand to the contract, on the other hand you will look at your maintenance history to look for trends.
2.2
"On the one hand we use data for the handling of incidents in the tunnel. Mainly for liability purposes"
Data for external purposes O
Table 6: Pre-condition – Learning and data management
The PMS is a system that stores measurement data about performance and translates these with the use of incentives into a remuneration. Exploiting a PMS over a long period of time can therefore provide information which is useful for both supplier or customer. Information exchange about performance L was found to be a direct consequence of the output data from the PMS. In either case information exchange about performance data L emerged but in a different form. In case 1, frequently a consultation about performance levels between contractors takes place between contractors, in contrast to case 2. As interviewee 2.2 stated: “We definitely try to have meetings about performance levels and on what aspects we might improve. However these meetings between suppliers and customer do not become reality up to now” Due to the fact that more suppliers are involved which are connected to the PMS, information exchange L occurs via the PMS itself. In fact, Rijkswaterstaat as a customer has direct access to the system data generated by the PMS with respect to all performance levels. The customer can therefore check performance levels at any time. By way of contrast, in case 1, suppliers are the only one who have access to the PMS and its information. Thus, in both cases information exchange about performance L does take place but in a different manner.
Additionally in both cases, information extracted from the PMS resulted in a change in the configuration of the PMS itself. As interviewee 1.1 claims: “For this particular asset function are standards and indexes for performance levels determined. However based on the information of the PMS we determined that those requirements were too high. Therefore we adjusted the measurement strictness of the PMS”. In other words the measurement strictness of the PMS was adapted as a consequence of its own information M about measurement output. Thus the examination of measurement output resulted in a change of the configuration of the PMS. Basically a learning loop was established between measurement output and the measurement strictness of the PMS. Similar facts were reported in case 2.
Suppliers intensified their measurement frequency when asset components (tarmac, lighting) were close to failure in their degeneration process. The physical state of an asset component influences therefore the measurement frequency. However it does not influence the measurement frequency of the PMS, but it influences the measurement frequency of the maintenance management system (MMS). In both cases, suppliers make use of a MMS (maintenance history N) next to their PMS. The MMS is used for internal purposes and monitors asset components in their degeneration process as they change over time. The PMS is only used to prove performance levels according to the PBC. The PMS and the MMS are applied as two independent systems but do exchange information about performance levels of asset components.
The degeneration of asset components and performance levels of those asset components are two interdependent constructs. Additionally, information from the PMS can serve as an input mechanism for the MMS to perform maintenance more effectively. On the other hand maintenance history N is used to monitor performance levels internally, but does contribute to achieve performance levels according to contractual requirements. As interviewee 3.1 declares: “When inspections are performed we will look on the one hand at contractual requirements. On the other hand we look at your maintenance history to
