The effectiveness of contract monitoring:
A multiple-case study on the influence of
the contracting context
University of Groningen
Faculty of Economics and Business
MSc. Supply Chain Management
Master Thesis
June 20, 2016
Annelies Overkempe
S2135027
a.overkempe@student.rug.nl
1
ACKNOWLEDGEMENTS
Writing my master thesis would not have been possible without the support of others. First of all, I would like to thank dr. H. Broekhuis for always providing me with constructive feedback and guiding me through the sometimes, difficult process. In addition, I would like to thank drs. ing. H.L. Faber. Although he was not able to guide the entire process, his feedback and ideas were very useful and valuable. Thereby, I would like to thank prof. dr. J. de Vries for co-assessing my master thesis. Furthermore, I would like to thank my fellow students Maarten de Groot and Robert van der Wielen for brainstorming together and providing me with feedback during the meetings.
Secondly, I would like to express my gratefulness to Samenwerkingsorganisatie De Wolden Hoogeveen for the opportunity to conduct my research and assisting in the data collection. Especially Els Obbema for her enthusiastic participation, useful suggestions and many cups of coffee.
2
ABSTRACT
Government agencies are increasingly turning to the private sector for the delivery of health care services to their citizens. In order to make the right selection and decisions with regard to the contracted health care providers, the government agency needs effective contract monitoring. Since it is difficult, or even impossible for government agencies to develop one way of contract monitoring that is effective for all health care providers, the influence of the contracting context on the way and effectiveness of contract monitoring is researched in this study. The contracting context contains three components, i.e. contract tasks, managerial emphases and institutional environment. The outcomes of this study propose that (1) the contract tasks and managerial emphases have a combined influence on the way of contract monitoring and that (2) the managerial emphases determines whether the institutional environment affects the way of contract monitoring. Moreover, (3) it is suggested that the contracting context should be expanded with two other factors, i.e. experience in contract monitoring and consequences on performances. These outcomes extend current knowledge on contract management, especially for the field of public health services that has received limited attention in current literature. In addition, it provides managerial implications for effective contract monitoring between government agencies and health care providers.
Key words: way of contract monitoring, effectiveness of contract monitoring, contracting
3
TABLE OF CONTENT
1. INTRODUCTION ... 5
2. THEORETICAL BACKGROUND ... 7
2.1 The way and effectiveness of contract monitoring ... 7
2.1.1 Effectiveness of contract monitoring ... 8
2.1.2 Collecting information ... 9
2.1.3 Using information ... 11
2.1.4 Methods for collecting and using information ... 12
2.2 Contracting context ... 12
2.2.1 Contract tasks ... 12
2.2.2 Managerial emphases ... 13
2.2.3 Institutional environment ... 14
2.2.4 Types for components of the contracting context ... 15
2.3 Influence of the contracting context ... 15
3. METHODOLOGY ... 17
3.1 Research design ... 17
3.1.1 Research setting ... 17
3.1.2 Unit of analysis and case selection criteria ... 17
3.1.3 Procedure for case selection ... 18
3.1.4 Description of case 1 ... 19
3.1.5 Description of case 2 ... 19
3.1.6 Description of case 3 ... 19
3.2 Data collection ... 20
3.2.1 Interviewees ... 21
3.2.2 Measures and data sources ... 21
3.2.3 Quality criteria ... 23
3.3 Data analysis ... 23
4. FINDINGS ... 25
4.1 Contract monitoring in case 1 ... 25
4.1.1 Contract monitoring in contract ... 25
4.1.2 Contract monitoring in practice ... 25
4.1.3 Contract monitoring as desired ... 26
4.2 Contract monitoring in case 2 and 3 ... 26
4.2.1 Contract monitoring in contract ... 26
4.2.2 Contract monitoring in practice ... 26
4.2.3 Contract monitoring as desired ... 27
4.3 Differences and similarities in contract monitoring ... 28
4.3.1 Collecting information ... 29
4.3.2 Using information ... 29
4.3.3 Effectiveness of contract monitoring ... 29
4.3.4 Overview of the way and effectiveness of contract monitoring ... 30
4.4 Contracting context ... 31
4.4.1 Contract tasks ... 31
4.4.2 Managerial emphases ... 31
4.4.3 Institutional environment ... 32
4.4.4 Overview of contracting contexts... 32
5. ANALYSES ... 33
5.1 Influence of the contract tasks and the managerial emphases ... 33
5.2 Influence of the institutional environment ... 34
4
5.3.1 Experience in contract monitoring ... 35
5.3.2 Consequences on performances ... 36
5.4 Causal model of the outcomes ... 37
6. DISCUSSION ... 38
6.1 Influence of the contract tasks and managerial emphases ... 38
6.2 Influence of the institutional environment ... 39
6.3 Influence of the other factors ... 39
7. CONCLUSION ... 40
7.1 Theoretical implications ... 40
7.2 Managerial implications ... 40
7.3 Limitations ... 40
7.4 Recommendations for further research ... 41
REFERENCES ... 42
APPENDICES ... 48
Appendix A: Interview protocol 1 ... 48
Appendix B: Interview protocol 2 ... 52
Appendix C: Respondents semi-structured interviews ... 56
5
1. INTRODUCTION
In the last decades, a trend towards private sector involvement in the provision of public services has emerged (Valero, 2015; Trebilcock & Rosenstock, 2015). Governments are increasingly turning to the private sector to build and operate public infrastructure and the associated services, including public health services (Valero, 2015; Taylor & Hawley, 2010). The market of providers who offer the delivery of health care services is enormous and therefore governments are forced to determine which health care services it will procure from which health care providers (Talluri & Sarkis, 2002). In order to compare the different health care providers with each other and to make the right selection and decisions with regard to the current contracts (Girth, 2014), governments need to gain insight in the performances of their health care providers (Salinsky & Gursky, 2006). Insight in the performances of health care providers implies effective contract monitoring (Amirkhanyan, Kim & Lambright, 2007; Salinsky & Gursky, 2006). Due to the variety in health care services, it is difficult, or even impossible, for governments to develop one tool of contract monitoring that is effective for all health care providers (Nickerson, Adams, Attaran, Hatcher-Roberts & Tugwell, 2015; Guinea, Sela, Gómez-Núñez, Mangwende, Ambali, Nqum, Jaramilo, Gallego, Patiño, Latorre, Srivanichakorn & Thepthien, 2015). Romzek and Johnston (2005) stress that the contracting context should be taken into consideration when monitoring and evaluating health care providers. The contracting context can be defined by the contract tasks, the managerial emphases, and the institutional environment (Romzek & Johnston, 2005). The contracting context differs between the different health care services that health care providers deliver (Zaidi, Mayhew, Cleland & Green, 2012). Hence, it is likely that different ways of contract monitoring are required for the different kinds of contracting contexts. Moreover, it is expected that alignment between the contracting context and the way of contract monitoring will enhance the effectiveness of contract monitoring. However, the influence of the contracting context has not been researched in current literature yet. Therefore it is not known how the components of the contracting context can be used by government agencies to align the contract monitoring activities and consequently increase the effectiveness of contract monitoring.
6
health services differs from the private sector setting, due to the role of the government, who protects the interests of the public (Valero, 2015; Fleming, 2015). One of the government’s responsibilities includes the level of public health. In order to ensure or even improve the level of public health (Carlson, Chilton, Corso & Beitsch, 2015), governments are seeking for best practices to deliver these health care services to their citizens (Spiegelman, 2016). Due to the fact that delivering health care services is not a core business of governments (Fleming, 2015), they are always looking for accountable health care providers to collaborate with or to outsource these health care services to (Ingram, Scutchfield & Costisch, 2015).Current literature highlights the need for effectiveness in managing the contracts within public health services (Torchia, Calabrò & Morner, 2015; Mouraviev & Kakabadse, 2015), which can be considered as “the government agency having the capacity to assess
contractors’ performance and having the potential to hold contractors accountable” (Romzek
& Johnston, 2005). Contract monitoring provides insight in the relationship between organizational actions and the performances (De Bruijn, 2002; Poppo & Zhou, 2014). Hence, contract monitoring can be used as a ground for undertaking actions to influence the performances of health care providers (Poppo & Zhou, 2014).
In addition, Amirkhanyan et al. (2007) stress that effective contract monitoring results from the way of contract monitoring, like collecting and using the right information in an accurate and timely manner. However, it is not taken into consideration how the choices in the way of contract monitoring are influenced by the contracting context. Although it is likely that for different contracting contexts, different ways contract monitoring will be effective, these two are not linked yet. In order to determine appropriate ways for contract monitoring, and consequently enhance the effectiveness of contract monitoring, the contracting context should be taken into consideration (Romzek & Johnston, 2005). Aiming to explore the influence of the contracting context, this paper studies the following research question: How does the contracting context affect the way and effectiveness of contract monitoring within public health services?
7
2. THEORETICAL BACKGROUND
In this section, the theoretical background will be addressed. First of all, there will be elaborated on the way and effectiveness of contract monitoring. Thereafter, the contracting context will be discussed. Finally, it will be explained how the contracting context might be of influence on the way and effectiveness of contract monitoring.
2.1 The way and effectiveness of contract monitoring
Amirkhanyan et al. (2007) state that effective contract monitoring in public services refers to “(1) whether the government agency collects information from contractors in a timely
manner that accurately captures the quality and quantity of services being delivered, and (2) whether the government agency uses the information it collects to evaluate contractor performance and to make programming and policy decisions”. This definition clearly
indicates the relationship between collecting and using information, i.e. the way of contract monitoring, and the effectiveness of contract monitoring, as shown in figure 2.1.
Figure 2.1: The way and effectiveness of contract monitoring
8
possible, and on the other hand the use of information determines which information should be collected (Nutley & Reynolds, 2013). Both variables contribute to the effectiveness of contract monitoring, because timely and accurately capturing information and using it for evaluating contractors’ performance will enhance the effectiveness of contract monitoring (Amirkhanyan et al., 2007).2.1.1 Effectiveness of contract monitoring
Effective contract monitoring occurs when the government agency has the capacity to assess
contractors’ performances and has the potential to hold the contractors accountable
(Romzek & Johnston, 2005). In addition, contract monitoring can be seen as the governance process that oversees and assesses contractors’ compliance with the contract (Bello, Katsikeas & Robson, 2010). Hence, health care providers should strive for the best ways to fulfill the activities that are defined in the contract. The more they meet the agreements of the contract, the better their performances will be reviewed by the government agency. However, determining how well the agreements are fulfilled by the health care providers requires an effective way of contract monitoring of the government agency. In this study, the effectiveness of contract monitoring will be based on the definition of Romzek and Johnston (2005), whereby (1) “the capacity to assess contractors’ performances” is considered as the government agency’s ability to collect and use the right information to review the health care providers’ performances, and (2) “the potential to hold contractors
accountable” is viewed as the government agency’s ability to improve the performances of
9
The setting of public health services contains a wide variety of health care services delivered by multiple health care providers and procured by a government agency. Collaborations between government agencies and health care providers have become popular worldwide as a way of improving the delivery of health care services (Torchia et al., 2015). Problems of contemporary health care are too complicated for government agencies to solve individually, and at the same time private organizations are seeking for opportunities to contribute to society (Reich, 2000; Reich, 2002; Torchia et al., 2015). Government agencies try to respond to these needs by establishing collaborations with health care providers. These collaborations are also known as public-private partnerships, which is an umbrella term describing collaborative relationships between public and private actors for the achievement of common goals (Singh & Prakash, 2010), like ensuring or improving the level of public health. Aiming to continuously ensure or even improve the level of public health, effective contract monitoring of health care providers by government agencies is required (Amirkhanyan et al., 2007).2.1.2 Collecting information
Data collection within a measurement system is concerned with what is being measured and how the measures are structured (Bourne, Kennerly & Franco-Santos, 2005). The collected data can be converted into information by interpreting and structuring it (Vallejo, Krepper, Nora & Fine, 2012). For example, the collected data can be structured by the stage in which they are obtained, like structure, process or outcome measures (Donabedian, 1988) or can be interpreted by the use of different perspectives as e.g. financial, customer, internal business, and learning and growth perspectives, which are included in the balanced scorecard (Kaplan & Norton, 1992). In this way, a meaning is given to the data and in doing so the data is converted into information (Vallejo et al., 2012). However, in order to gather information, data should be collected first.
10
interactions like site visits and meetings (Brockman & Cicon, 2013). Due to the reliance on human interaction, the soft data collection methods are likely to provide different outcomes when carried out by different persons. Hard and soft data collection methods are often supplementing each other (Royset & Wets, 2015). Hence, in order to gain a complete overview of health care providers’ performances, it is likely that both methods are need to be used.Appropriate data collection has been noted as a needed and critical component of contract monitoring, because data collection that lacks fidelity, reliability, and accuracy is useless and futile (Elswick, Casey, Zanskas, Black & Schnell, 2016). Moreover, a lack of relevant, accurate and timely data may negatively impact public health (Partington, Papakroni & Menzies, 2014). This implicates that data collection needs to meet several requirements. According to Gort, Broekhuis and Regts (2013) the collected data should be in compliance with four main areas: validity, reliability, comparability and controllability. Thereby, Gort et al. (2013) highlight the importance of controllability, which refers to the persistence of a direct and causal relationship between the action and outcomes (Kluger & DeNisi, 1996). Capturing data about indicators that could not be changed by action or that do not influence the outcomes is useless and would not contribute to the effectiveness of contract monitoring, because it is not in the government agency’s ability to ensure or improve the level of public health by those indicators.
11
2.1.3 Using information
Once the data is collected and converted into information, it is ready to use. The use of information refers to whether the government agency uses the information it collects to evaluate health care providers’ performances and to make programming and policy decisions (Amirkhanyan et al., 2007). It contains the review of information as part of a decision-making process, regardless of the source of data (Nutley & Reynolds, 2013). Moreover, Fosset, Goggin, Hall, Plein, Roper and Weissert (2000) argue that government agencies should go further than evaluating the performances of health care providers, they suggest that consequences for failure should be added as well.
In order to enable government agencies to obtain an insight in the performances of their health care providers, a thorough analysis and use of the collected information is recommended. For the aim of this study, a distinction between a control-based system approach and a very active and sophisticated approach will be made (Gort et al., 2013). The control-based system approach is one in which information is gathered through the use of standard company systems, analyzed, compared against company targets, communicated and acted upon (Gort et al., 2013). The active-sophisticated approach (Bourne et al., 2005; Bourne, Melnyk & Faull, 2007) is one that involves continual interaction with the information, developing additional systems for own data collection and creating one’s future performance (Gort et al., 2013). Similar to the hard-soft balance in the phase of collecting information, it is expected that also the control-based and active-sophisticated approaches supplement each other. Although the control-based approach can factually compare performances against company targets (Gort et al., 2013), it can also be associated with distrust and deceptions, which can inhibit the renewal of the contract (Poppo & Zhou, 2014). These negative consequences can be limited by the use of the active-sophisticated approach, because this approach is more focused on creating one’s future performances and stimulates the employment of self-regulating practices to coordinate mutual adaptions (Poppo & Zhou, 2014), which reflects both parties contributions to an improvement in performances.
12
2.1.4 Methods for collecting and using information
Based on the described methods in the paragraphs of collecting and using information, table 2.1 is drawn to provide an overview.
Methods
Collecting information Hard data collection Soft data collection
Using information Control-based approach Active-sophisticated approach
Table 2.1: Methods for collecting and using information
2.2 Contracting context
Romzek and Johnston (2005) discuss that the way of contract monitoring should fit to the contracting context in which both the government and the health care providers are involved. The contracting context includes the contract tasks, the managerial emphases, and the institutional environment (Romzek & Johnston, 2005). It should be noted that the contracting context differs for different health care providers (Zaidi et al., 2012) and that one type of contract monitoring is not necessarily appropriate for each and every circumstance (Romzek & Johnston, 2005; Romzek & Dubnick, 1994).
2.2.1 Contract tasks
For the aim of this study, contract tasks will be considered as the delivery of the health care services that are agreed upon in the contracts between the government agency and the health care providers. Although a government agency attempts to specify the tasks clearly in the contracts, it can be hard to classify them into more general categories, due to the many classifying possibilities (Lillrank, Groop & Malmström, 2010). However, classifying services is needed in order to develop suitable tools to manage processes, like contract monitoring, effectively (Ellram, Tate & Billington, 2004). Hence, in order to research which contract monitoring methods are effective to which contract tasks, a classification of the contract tasks is needed first. In addition, Salinsky and Gursky (2006) argue that the effectiveness of contract monitoring can be enhanced when the method for contract monitoring is integrated with the health care services that are delivered.
13
contract tasks lend to standardization and rules (Romzek & Johnston, 2005), because they are characterized by repetitiveness and familiarity (Krishnan et al., 2011). Second, thesemi-routine tasks form a middle ground by involving a mix of routinization and discretion
(Romzek & Johnston, 2005). These contract tasks are characterized by combining both general and (client) specific activities. Third, the non-routine tasks, these are defined as contract tasks with low task programmability and frequent exceptions that rise to a lack of routines and, in turn, increased task uncertainty (Abernethy & Brownell, 1997). This kind of contract tasks is most complex and requires non-routine responses based on individual discretion and expertise (Romzek & Johnston, 2005).
2.2.2 Managerial emphases
The managerial emphasis reflects the objectives of the government agency. The focus of the monitoring activities should be aligned with these objectives in order to enhance the effectiveness of contract monitoring (Romzek & Johnston, 2005). Due to the fact that the monitoring activities directly affect the effectiveness of the strategy execution of the organization (Srivastava & Sushil, 2015), it is of great importance to align the way of contract monitoring to the objectives. The objectives of the government agency can differ for the different health care providers. The contract monitoring activities are driven by performance measurement logic, and therefore should have a different design (Srivastava & Sushil, 2015). In other words, the way of contract monitoring should be aligned with the objectives and performances that are stated by the government agency.
The objectives of the government agency determine on which aspects the contract monitoring activities should be focused, because they reflect the managerial emphases. According to Romzek and Johnston (2005), the managerial emphases can be divided into four categories. First, inputs, which include financial, human and material resources needed to support the contract tasks (Menefee & Schagrin, 2003). Second, process, which is considered as the interventions or treatment modalities applied to the client to effect change (Menefee & Schagrin, 2003). Third, outputs, which are considered as the number of units of services produced by the contractors (Menefee & Schagrin, 2003). Fourth,
outcomes, which are defined as the results or accomplishments that are attributable, at
14
instruments should be focused on tracking and measuring the outcomes for achieving these strategic objectives (Srivastava & Sushil, 2015). Therefore it is likely that input and output will not be a main emphasis of the government, because they are not directly related to ensuring or improving the level of public health. For example, consider a government agency that invests more money on public health (input), or a health care provider that visits more clients a day (output), none of them directly contributes to the level of public health. Instead, the government agency is expected to focus its contract monitoring on the processes and outcomes, because those are directly related to the level of public health. A change in process means that the health care provider will provide the clients with a different treatment, which will contribute to their level of health, which consequently should show off in the outcomes. Therefore, this study will focus on the process and outcomes with regard to the managerial emphases, aiming to figure out how they influence the way and effectiveness of contract monitoring.In addition, with regard to Huber, Van Vliet, Giezenberg, Winkens, Heerkens, Dagnelie and Knottnerus (2016), the outcomes for health care services should be aimed on the positive definition of health as “the ability to adapt and self-manage, in the face of social, physical and
emotional challenges”. Thereby, they stress that the outcomes could be measured by six
dimensions, respectively bodily functions, mental functions and perception, spiritual/existential dimension, quality of life, social and societal participation and daily functioning. When measuring the clients’ progress on these dimensions, a broad view of their health status could be obtained (Huber et al., 2016), which implicates that measuring the process could be unnecessary for government agencies, because the results of the treatments will be included in one of the six dimensions of Huber et al. (2016). Therefore this research will take into account to which extent the current contract monitoring activities are focused on measuring the outcomes by the dimensions of Huber et al. (2016) in order to find out whether and how a managerial emphasis on these outcomes will influence the way and effectiveness of contract monitoring.
2.2.3 Institutional environment
15
environment affects the way of contract monitoring. Therefore the collaborative institutional environment will be taken into account in this study as well.2.2.4 Types for components of the contracting context
Based on the described types in the paragraphs of the components of the contracting context, table 2.2 is drawn to provide an overview.
Types
Contract tasks Routine Semi-Routine Non-Routine
Managerial emphases Process Process & Outcomes Outcomes
Institutional environment Collaborative
Table 2.2: Types for components of contracting context
2.3 Influence of the contracting context
Based on the three parts of the contracting context, three sub-questions could be formulated for this research. Together, the sub-questions will provide an answer on the research question. A conceptual model of the research question is drawn in figure 2.2. Q1: How do the contract tasks affect the relationship between the way and the effectiveness of contract monitoring within public health services?
Q2: How do the managerial emphases affect the relationship between the way and the effectiveness of contract monitoring within public health services?
17
3. METHODOLOGY
In this section, the methodology will be explained. First of all, there will be elaborated on the research design of this study. Thereafter, the data collection method and the measures will be explained. Finally, there will be elaborated on how the data was analyzed in order to provide answers to the research question and the sub-questions of this study.
3.1 Research design
A multiple-case study seemed appropriate for this research, because it enabled to study the influence of the contracting context in-depth and within its real-life context (Yin, 2009). For each case it could be studied deeply whether and how the components of the contracting context affected the way and effectiveness of contract monitoring, at which the underlying reasons could be extracted (Meredith, 1998). Furthermore, the setting of a multiple-case study enabled for creating cross-case diversity to compare different contracting contexts and their influences on the way and effectiveness of contract monitoring (Yin, 2009). This made it possible to check whether and how the findings were consistent between the cases and enabled to draw propositions that were supported by the multiple cases.
3.1.1 Research setting
A Dutch municipality offered to participate in this research. Since the municipalities in The Netherlands are representative agencies of the government and this municipality had over 100 contracts with health care providers, it was considered as a suitable government agency for conducting this research. Moreover, due to decentralization of the delivery of public health services in The Netherlands in 2015, the municipalities have recently become responsible for managing contracts with a wide variety of health care providers (Rijksoverheid, 2014). Therefore studying a Dutch municipality ensured the possibility of creating cross-case diversity in the components of the contracting context. Finally, these new responsibilities for the Dutch municipalities made the multiple-case design even more relevant, because the municipality was still defining its contract monitoring activities and was looking for methods to do this even better, for which an in-depth understanding of the current situation was needed.
3.1.2 Unit of analysis and case selection criteria
18
emphases and the institutional environment could be interpreted in the same way for each case. This same interpretation should have avoided that other factors than the components of the contracting context had an influence on the outcomes of this research. In contrast, the health care providers needed to differ for each case, so that different contracting contexts were ensured and the outcomes of this study could be compared. Based on differing on the three components of the contracting context, the health care providers were selected. As shown in table 2.2, there are three types of contract tasks, three types of managerial emphases and one type of institutional environment defined in this study. Therefore the selection criteria involved that all cases together contained all defined types of the components of the contracting context. Since the studied municipality contained contracts with more than 100 health care providers, it was possible to formulate these case selection criteria.3.1.3 Procedure for case selection
The contract specialist of the municipality suggested some health care providers that differed on the types of contract tasks. Thereby, they together met the requirements of including all defined types of managerial emphases and having a collaborative institutional environment. Since the contract specialist was in charge of managing the contracts with the health care providers, she was able to make a suitable selection with regard to this research. Three health care providers were selected to represent the cases, as shown in table 3.1. Thereby, the table also shows two descriptions of the health care providers.
Case 1
Health care provider A Health care provider B Case 2 Health care provider C Case 3
Case selection criteria
Contract tasks Routine Semi-routine Non-routine
Managerial emphases Outcomes Outcomes Process & Process
Institutional environment Collaborative Collaborative Collaborative
Descriptions
Contracted since [year] 2014 2015 2015
Number of clients 20 24 45
19
3.1.4 Description of case 1
The first case contained the contract between the municipality and a health care provider for the delivery of domestic help. According to the contract, the tasks include mainly cleaning and tidy activities that are performed at the clients’ home. Examples are doing the laundry, mopping the floor, cleaning the windows or throwing away the garbage. All the cleaning activities are considered as routine, because well-known techniques were repeated to execute these tasks (Perrow, 1970; Krishnan et al., 2011). The variation in demand is low and is only reflected in adding or leaving away activities depending on the mobility of the clients.
3.1.5 Description of case 2
The second case contained the contract between the municipality and a health care provider for the delivery of group counseling. According to the contract, the tasks include guiding a group in the chosen activity and activating the group members. The chosen activity can, for example, be singing or crafting with wood. Thereby, the daily program of the health care provider should offer scope for skills training of the clients as well. The selected health care provider for this research is an organization for learning about and working with wood and metal. Together with the other group members, the clients are instructed on how to process wood and metal, while they are also assisted in developing personal skills like structure their day and plan their activities. On the one hand, the tasks of this health care provider can be considered as routine, because he performs the same tasks to each client in the group, namely processing wood and metal (Krishnan et al., 2011). On the other hand, he also delivers some personal care to each client by developing his personal skills, this can be considered as more non-routine (Romzek & Johnston, 2005). Due to this combination, the tasks of this health care provider are categorized as semi-routine (Romzek & Johnston, 2005).
3.1.6 Description of case 3
20
on health care provider’s discretion and expertise. Moreover, the tasks are considered as highly complex and require continuous adaption to the situation of the clients. Hence, these contract tasks are defined as non-routine.3.2 Data collection
Data for this research was collected in four steps. (1) First the documents were studied, i.e. the contracts between the municipality and the selected health care providers, and several overviews of the purchased health care services of the municipality. The documents provided information on the contractual goals and the agreed methods for collecting and using information. (2) Second, unstructured interviews with the contract specialist and the team leader of her department of the municipality were conducted in order to discuss the case selection criteria, gain extra information about the selected health care providers and for selecting respondents within the municipality. (3) Third, observations were performed during meetings of the municipality where topics like satisfaction with methods for contract monitoring or performances of health care providers were discussed between the contract specialist, the policy officers and the head of the department. These meetings were attended without participating in the discussions. Hence, observations in the “natural setting” were possible. These meetings provided information about the methods for collection information. (4) Fourth, and the prime source for data, are the semi-structured interviews that were conducted to gain in-depth information about the way of contract monitoring in practice, and the interviewees complemented their responses with arguments and opinions, which made the data even richer. These interviews were conducted at both the municipality and the three selected health care providers. For both parties, different question lists were used (Appendix A and B). The interviews lasted for approximately one hour and were audio recorded. Directly or soon after conducting, the interviews were transcribed and sent to the interviewees for verification. Table 3.2 contains a summary of the number of data sources and the obtained information.
Step Data source Amount Obtained information
1 Documents 5 documents Monitoring in contract
2 Unstructured interviews 3 meetings Assistance in selection processes
3 Observations 3 meetings Methods for collecting information
4 Semi-structured interviews 12 interviews Monitoring in practice (in-depth)
21
3.2.1 Interviewees
Since the semi-structured interviews were the prime source of data for this research, the respondents are of great importance for gaining an in-depth understanding. For each case multiple municipality employees and the representatives of the health care providers were interviewed to ensure a complete insight of both perspectives. An overview of the interviewees can be found in Appendix C.
3.2.2 Measures and data sources
22
Table 3.3: Data sources per sub-dimension1P1-Q3 refers to interview protocol 1, question 3
Dimension Sub-dimension Documents interviews Unstructured Observations interview Semi-structured
Contract monitoring in contract Goals of monitoring Procurement document & contracts Meeting with policy officers Frequency of monitoring Procurement document & contracts Meeting with policy officers Persons involved Procurement document & contracts Meeting with policy officers Collecting information Procurement document, contracts & ranking list Meeting with policy officers Using information Procurement document, contracts & ranking list Meeting with policy officers Contract monitoring in practice Goals of
monitoring ranking list Checklist & contract specialist Meeting with
Meetings for preparing visits (3x) P1-Q31 P2-Q4 Frequency of
monitoring Planning contract specialist Meeting with
Meetings for preparing visits (3x) P1-Q4 P2-Q5 Persons
involved Planning lead Meeting with team
Meetings for preparing visits (3x) P1-Q2,4 P2-Q3,5 Collecting information Checklist, ranking list & reports Meeting with contract specialist Meetings for preparing visits (3x) P1-Q5 P2-Q6 Using information Checklist, ranking list & reports Meeting with contract specialist Meetings for preparing visits (3x) P1-Q6 P2-Q7 Contracting context
Contract tasks reports Contracts & contract specialist Meeting with preparing visits Meetings for (3x)
P1-Q7,8,9 P2-Q8,9,10 Managerial
emphasis reports Contracts & contract specialist Meeting with
Meetings for preparing visits (3x) P1-Q10,11,12,13 P2-Q11,12,13,14 Institutional
environment reports Contracts & contract specialist Meeting with
23
3.2.3 Quality criteria
According to Yin (2009), four requirements can be distinguished to assure research quality. Table 3.4 shows how the quality criteria were assured in this research.
Quality criteria How?
Construct validity Use of four different data sources and sending the interview
transcripts to the interviewees for verification.
Internal validity Use of both descriptive and interpretive codes for finding patterns in the data and conduct both within- and cross-case analyses.
External validity
Possibility for theoretical replication. The results of this research can be generalized to government agencies that manage contracts with health care providers that meet the same case selection criteria as provided in table 3.1.
Reliability Use of protocols for the semi-structured interviews and analyzing documents and field notes in the same manner by operationalized variables.
Table 3.4: Assurance of quality criteria
3.3 Data analysis
24
First order concepts Second order themes Aggregate dimension
Contract monitoring goals as formulated in the contract Goals of monitoring
Contract monitoring in contract Contract monitoring frequency as formulated in the contract Frequency of monitoring
Persons involved as formulated in the contract Persons involved
Which information is collected according to the contract? Collecting
information How information is collected according to the contract?
Which information is used according to the contract?
Using information How information is used according to the contract?
Interpretation of the goals
Goals of monitoring
Contract monitoring in practice Acting to accomplish the goals
What the goals should be
Opinion on the descriptions of the goals
Frequency of performing contract monitoring activities
Frequency of monitoring How often contract monitoring activities should be
performed
Who are involved in the contract monitoring activities
Persons involved Who should be involved in the contract monitoring activities
Which information is collected
Collecting information Which information should be collected
How is the information collected
How the information should be collected The future of collecting the information Which information is used
Using information Which information should be used
How is the information used
How the information should be used The future of using the information
Level of routine in tasks of health care provider
Contract tasks
Contracting context Adaption of tasks on needs of clients
Involvement of clients in performing the tasks Level of focus on the process
Managerial emphases Level of focus on the outcomes
What the focus should be
Opinion on definition of Huber et al. (2016) What collaboration looks like
Institutional environment How collaboration could be improved
Level of trust in collaboration
Alignment between tasks and the way of contract monitoring
Alignment with contract monitoring Alignment between managerial emphases and the way of
contract monitoring
Alignment between institutional environment and the way of contract monitoring
25
4. FINDINGS
In this section, within-case analyses will be conducted to structure the findings per case based on the aggregate dimensions of table 3.5. Thereafter, cross-case analyses will be conducted to compare the cases. Finally, there will be elaborated on the contracting context of each case.
4.1 Contract monitoring in case 1
4.1.1 Contract monitoring in contract
In the contract is agreed that “soundly help” needs to be delivered, which is described as
“help of a good level, in terms of efficient, expedient, client-oriented, and adapted to the needs of the client”. The health care providers will be ranked by “parameters”, which involve “price per hour, waiting time for new clients and client satisfaction”. The health care
providers have to deliver their information on the parameters “by e-mail to the contract
specialist”. The parameters will be used “to create a ranking list of the health care providers every three months”, whereby “the municipality will advise the first health care provider on the ranking list, when clients do not know which one to choose or are not willing to choose”.
4.1.2 Contract monitoring in practice
According to the interviewees, the goals are “delivering a clean house” and “having a
satisfied client” [contract specialist, consultant and health care provider]. However, the
collected information did not correspond with these goals. The health care providers were requested “to send an update of their price, waiting time and client satisfaction grade every 3
months to the contract specialist”, but “some health care providers did not deliver their information anymore” [contract specialist]. The information was used “to create a ranking list whereby the health care provider on the first rank will be rewarded with the allocation of new clients” [contract specialist]. Health care providers that did not send their information “received a grade of zero on that topic, which places them automatically low in the ranking”
[contract specialist]. Except of the first rank, “the other ranks have no consequences in terms
of bonuses or penalties” [contract specialist]. Moreover, it was added that the ranking list
was “perceived as unfair”, because “the prices are fixed in the contracts” and “the client
26
4.1.3 Contract monitoring as desired
Due to the fact that the clients’ were considered as “a vulnerable group”, it was desired “to
remain a social element in the delivery of the health care service”, because it was expected to
be “of the same importance as delivering a clean house” [head of department, contract specialist]. It was feared that the social element will be “wiped away” when the municipality will focus “solely on the outcomes of a clean house” [head of department, consultant]. Furthermore it was desired to keep the use of the ranking list, but “change the content” [contract specialist, consultant]. For example, the interviewees recommended that “the
municipality should collect the information, so that the client satisfaction grades are gathered with the same interval” or they would like to add “the amount of complaints per health care provider” to it, because “it can reflect the level of client satisfaction even more” [contract
specialist, consultant]. Finally, it was suggested that “the health care providers on lower
ranks should be triggered as well” by applying “some sort of consequence” [contract
specialist, consultant].
4.2 Contract monitoring in case 2 and 3
Since the contracts for case 2 and 3 contained the same agreements and the way of contract monitoring in practice did not vary much, the within-case analyses for these cases were combined.
4.2.1 Contract monitoring in contract
In the contract is agreed that “both parties will strive towards the vision as formulated in (…)
of which participation and self-reliance are the main goals”. This goal will be checked by “the contract specialist” during “periodical visits”, of which the frequency is not formulated.
During these periodical visits the municipality may ask for “clients’ support plans,
documents listed in the contract and/or conversation topics listed in the contract”. The only
use of information formulated is “in case of failure or non-compliance, the municipality will
send a written notice of default”.
4.2.2 Contract monitoring in practice
With regard to the interviewees, there was strived towards “delivering what is indicated by
the consultants”, because “the goals depend on the clients’ personal situations” [policy
officers, head of department, consultant, contract specialist, health care providers]. During the “half-yearly visits” of “the contract specialist and a colleague of the department”, the collection of data was “based on getting a feeling”, which meant that it involved “mainly a
27
documents and/or conversation topics” [policy officers, head of department, contractspecialist]. After the visits, the contract specialist wrote a report, which “can be used as
input for the next visits” or “for defining performance criteria in future contracts” [contract
specialist]. The reports were not sent to the health care providers. 4.2.3 Contract monitoring as desired
The interviewees indicated that the contract monitoring should be focused on client-level, because “every client has different personal goals” [policy officers, consultant, contract specialist, health care providers]. It was desired to “measure clients’ progress on a frequent
base” by the consultants and “have standardized policies when goals are or are not accomplished” [consultant, contract specialist]. Moreover, the interviewees pointed out that “the health care providers are the professionals in delivering health care services”, therefore
the municipality should “solely focus on the outcomes and not the processes” [consultants, contract specialist, health care providers]. Finally the health care providers added that they would be more triggered if they could obtain positive consequences, e.g. “financial
28
4.3 Differences and similarities in contract monitoring
Table 4.1 provides an overview of the findings on contract monitoring in contract, in practice and as desired. Based on these findings, a cross-case analysis was conducted to compare the cases on the variables of this research, i.e. collecting information, using information and the effectiveness of contract monitoring.
Case 1 Case 2 & 3
Contract monitoring in contract
Goals - Soundly help - Participation - Self-reliance
Frequency - Every 3 months frequency) - Periodical visits (undefined
Persons - Contract specialist - Contract specialist
Collecting
- Method: delivered by the health care providers - Price per hour
- Waiting time for new clients - Client satisfaction grade
- Method: might be asked during the visits
- Clients’ support plans - Documents
- Conversations topics
Using - Ranking list - First rank is rewarded with new clients
- Written notice of default by failure or non-compliance
Contract monitoring in practice
Goals - Delivering a clean house - Satisfied clients - Delivering what is indicated - Depend on clients’ situations
Frequency - Every 3 months - Half-yearly visits
Persons - Contract specialist of department - Contract specialist & colleague
Collecting
- Method: delivered by the health care providers - Price per hour
- Waiting time for new clients - Client satisfaction grade
- Method: based on feeling - Clients’ support plans - Some documents
- Some conversation topics
Using - Ranking list - First rank is rewarded with new clients
- Reports
- Input for next visits
- Defining future performance criteria
Monitoring as desired
Goals - Delivering a clean house - Satisfied clients - Social element
- Focus on clients’ personal goals
Frequency - Every 3 months frequently (more often than - Measure clients’ progress half-yearly)
Persons - Contract specialist - Contract specialist - Consultants
Collecting
- Method: conducted by the municipality
- Price per hour
- Waiting time for new clients - Client satisfaction grade - Amount of complaints
- Method: standardized policies - Only results of clients’
progress
Using
- Ranking list
- First rank is rewarded with new clients
- Trigger other ranks as well
- Positive consequences - Standardized policies
29
4.3.1 Collecting information
With regard to case 1, the municipality collects factual information of the health care providers every 3 months, i.e. price per hour, waiting time for new clients and client satisfaction grades. This is considered a hard data collection. In addition, the collection of soft data was avoided, as “an impression of health care providers could not be taken into
consideration, because a conflict of interests has to be avoided” [consultant].
With regard to case 2 and 3, the municipality collects the information only during the half-yearly visits at the health care providers. “Based on the impression of the health care
provider” [contract specialist], they might ask for “some documents and/or conversation topics that are listed in the contract” [policy officers]. The combination of documents
and/or conversation topics implies that the municipality collects both hard and soft data at the health care providers of case 2 and 3.
4.3.2 Using information
With regard to case 1, the method for using information can be classified as a control-based approach, because the information is compared against company targets as “we link scores
to the collected information and create a ranking list of it” [contract specialist]. Moreover,
there is acted upon the ranking list, since “clients who do not know which health care
provider they want, are recommended to take the first one of the ranking list” [consultant].
In contrast, the method for case 2 and 3 can be classified as the active-sophisticated approach, because “we use the reports as a stocktaking of what is going on and for getting a
general impression of the health care providers” [contract specialist]. Furthermore, “the reports are not meant as a review, but we get an impression of which health care providers we are eventually interested in to intensify corporation with in the future” [policy officers].
Thereby, it is used for developing additional systems for data collection and creating one’s future performance, as underlined by “it serves as an input for a new type of contracts” [policy officers].
4.3.3 Effectiveness of contract monitoring
In order to define the effectiveness of contract monitoring, the cases are compared against the definition of Romzek and Johnston (2005), which includes (1) the capacity to assess
contractors’ performances and (2) the potential to hold the contractors accountable. Since
30
be considered high when both parts are present, medium when one part is present and low when none of the parts are present.With regard to case 1, the municipality did not have (1) the capacity to assess contractors’
performances. Since the municipality collects data about the price per hour and the waiting
time for new clients, the outcomes of the delivered health care service can hardly be measured. Although the municipality also collects the client satisfaction grade, it is not measured whether the goal of delivering a clean house is accomplished. Furthermore, the municipality does have (2) the potential to hold contractors accountable in this case. Due to the ranking list and reward system of allocating new clients to the first rank, the health care providers are held accountable for their performances. Because of not meeting the first part and accomplishing the second part of the definition, the level of effectiveness of contract monitoring of case 1 can be considered as moderate.
With regard to case 2 and 3, the municipality did not have (1) the capacity to assess
contractors’ performances. Since the contract monitoring is mainly based on feelings and
the municipality only writes a report about the visits, there are no clear performance criteria on which the health care providers can be reviewed. Furthermore, the municipality also lacks in having (2) the potential to hold contractors accountable. Due to the fact that the municipality did neither formulate positive nor negative consequences for the performances of health care providers, they are not able to hold them accountable for any performances. Based on the absence of both parts of the definition, the level of effectiveness of both case 2 and 3 can be considered as low.
4.3.4 Overview of the way and effectiveness of contract monitoring Table 4.2 summarizes the findings on the variables of this research.
Case 1 Case 2 and 3
Collecting information Hard data collection Hard & soft data collection
Using information Control-based approach Active-sophisticated approach
Effectiveness of contract
monitoring Moderate Low
31
4.4 Contracting context
The responses of the interviewees indicated that the selection criteria for the cases did not fully correspond with the contracting contexts in practice. Therefore, this paragraph will elaborate on the components of the contracting context whereby it will be explained how the selection criteria differed from the contracting context in practice.
4.4.1 Contract tasks
With regard to case 1, the contract tasks can be considered as routine, because of the standardization as “it is only adapted to the clients in terms of time and type of cleaning
services that they receive, but the content of the executed task is the same at every client”
[head of department]. Moreover, the consultant describes the types of cleaning services that are indicated to clients who need as much domestic help as possible “for example
vacuum cleaning, window cleaning and cleaning bathroom and toilet”.
With regard to case 2, the contract tasks were expected to be semi-routine, because of the combination of group activities and individual guidance. However, in practice it appeared to be non-routine, due to the lack of standardization as “one method does not work for all
clients, we always deliver counseling based on the individual” [health care provider].
Moreover, “the health care provider looks for each client which method fits best” [consultant].
With regard to case 3, the contract tasks were considered as non-routine, because they were fully aligned to the needs of the clients, like “individual counseling is focused on the
personal needs of the client” [contract specialist] and “we collaborate with the clients in achieving their personal goals” [health care provider].
4.4.2 Managerial emphases
With regard to case 1, the municipality would like to focus its managerial emphasis on the outcomes, because “I do not think that the process is that important, I rather see a clean
house than a process that is well completed” [contract specialist]. Moreover, “when the client is satisfied, it does not matter anymore how that outcome was created” [consultant].
32
For both case 2 and 3, it turned out that the managerial emphasis was undefined, but tends towards process. The policy officers state “some other municipalities focus solely onoutcomes, but we decided to not take that step yet, we first would like to know what is delivered within one hour of health care service”. In addition, the head of the department
and the consultant argue that “we have not defined the outcomes, therefore we can hardly
measure them” and “we cannot focus on outcomes, because the goals are totally different for each client, it’s not comparable”.
4.4.3 Institutional environment
Since the municipality was willing to contract all health care providers that met the given requirements and did not make any selection in contracting partners, the institutional environment can be considered as collaborative for all cases. Furthermore, multiple interviewees remarked that mutual trust is of great importance. This becomes clear from
“we have chosen to rely on trust in our policy” [policy officers] or “we have a contract with the health care providers, because we trust that they can deliver the needed health care services to our clients” [contract specialist].
4.4.4 Overview of contracting contexts
Table 4.3 summarizes the findings on the components of the contracting contexts. The components of case 2 and 3 are combined, because they turned out to be equal.
Case 1 Case 2 and 3
Contract tasks Routine Non-routine
Managerial emphases Input & Outcomes Undefined (Process)
Institutional environment Collaborative Collaborative
33
5. ANALYSES
Aiming to investigate how the components of the contracting context influence the way and effectiveness of contract monitoring, this section will analyze the findings of the previous section. Since it turned out that the contract tasks and the managerial emphases were related, these sub-questions are combined answered in this section. Furthermore, the findings indicated that more factors than the three components of the contracting context influenced the way and effectiveness of contract monitoring, therefore these other factors were taken into account in this section as well. In the last paragraph, a causal model will be drawn.
5.1 Influence of the contract tasks and the managerial emphases
With regard to case 1, the contract tasks were classified as routine and the managerial emphases were on both input and outcomes. The municipality knew the content of the contract tasks well and used it when communicating the managerial emphases to the health care providers, as explained by the consultant “we know exactly how long it takes to
clean for example a bath room, because we base our indications for the clients on this information”. This implicates that the routine contract tasks are connected to a managerial
emphasis on outcomes, because the municipality could easily define its managerial emphasis based on the contract tasks.
Moreover, the connection between the contract tasks and the managerial emphases is also implied by the findings of case 2 and 3, in which the contract tasks were classified as non-routine and the managerial emphases were mainly undefined, but tended towards process. As remarked by the contract specialist and the policy officers “for both group- and
individual counseling, we have not defined what makes a health care provider a good one”,
which is due to “the goals depend on the clients’ personal situations, therefore we can only
expect that the health care providers deliver what is indicated by the consultants”. In these
cases, the contract tasks went along with a mainly undefined managerial emphasis, because the municipality had difficulties in defining the desired outcomes to the health care providers since the contract tasks were not the same for every client.
34
care services without visiting the health care providers”. With regard to case 2 and 3, hardand soft data were collected and used by an active-sophisticated approach. According to the consultant “we always want to check the clients’ support plans, because we want to know
what kind of treatment the health care providers deliver and why they make certain decisions”. Moreover, the contract specialist argues “we do not specifically know performance criteria, so we cannot tell how they perform therefore we want to know what they exactly deliver”. Based on these findings, the following propositions can be formulated;
Proposition 1a: When the contract tasks are routine, the government agency can clearly define the managerial emphasis on the outcomes, which enables a control-based approach that can rely on solely hard data.
Proposition 1b: When the contract tasks are non-routine, the government agency experiences difficulties in defining the managerial emphasis, which will be reflected in an active-sophisticated approach that considers both hard and soft data.
5.2 Influence of the institutional environment
Although the institutional environment was considered as collaborative for all cases, the findings indicate that the impact of the collaboration varied remarkably between the cases. With regard to case 1, the contract specialist clearly stated that the collaboration between the municipality and the health care providers did not play a role in the way of contract monitoring, as “when I have good contacts with a health care provider, it may never impact
the ranking list, because we have to keep it transparent”. Since the indicators of the ranking
list are clearly defined, there is no possibility for any influence of the collaboration in the way of contract monitoring for case 1.
In contrast, with regard to case 2 and 3, the collaboration seemed of great influence on the way of contract monitoring. As argued by the head of the department and the contract specialist “in case of doubts or mistrust, we always ask for prove”. However, “we do not score
or reward the health care providers, but we only want to check on some documents and/or some conversations topics”. Moreover, the role of the collaboration in these cases is
underlined by the consultant, as “sometimes I only give a short-term indication to a client,
35
told by the policy officers “when we have a good impression of the health care provider, wedo not pay attention to asking documents and/or conversation topics as prove”.
These findings implicate that the managerial emphases determines whether the collaboration is of influence on the way of contract monitoring. Whereas the managerial emphasis in case 1 did not left space for the collaboration to be of influence, the collaboration was of great influence on the data collection methods in case 2 and 3, where the managerial emphasis was mainly undefined. Therefore the following propositions can be formulated;
Proposition 2a: The managerial emphasis determines whether the institutional environment is of influence on the method for collecting data.
Proposition 2b: When the managerial emphasis is of influence on the method of collecting data, the institutional environment determines which information will be collected.
5.3 Influence of other factors
Besides the influence of the three components of the contracting context, it was found that other factors were also of influence on the way and effectiveness of contract monitoring. Therefore it seems relevant to take them into account for this research.
5.3.1 Experience in contract monitoring
The interviewees of the municipality pointed out that the contract monitoring visits for case 2 and 3 had just started this year, which made that the municipality was still looking for the right ways to manage these visits. As stated by the policy officers “we have just
started off, but we already got an impression of how we should handle this”. Moreover, the
experiences with the ranking list were also of importance on the contract monitoring for case 1, which has been running for a few years already. The contract specialist explains “we
will consider our experiences with the current ranking list, when creating a new policy or contract. The environment is changing and we should adapt to the new trends. Maybe the ranking list needs a critical review now or should be replaced by other methods”. Hence, the
