Benchmarking and Competitive Tendering

(1)

Benchmarking and Competitive Tendering

Usefulness and effectiveness in the public sector

Master thesis

H.J. Vorenkamp

University of Groningen

(2)

Benchmarking and Competitive Tendering

Usefulness and effectiveness in the public sector

Abstract

Benchmarking is applicable in the public sector in many ways. This thesis focuses on the question if benchmarking could be useful for a principal in the case of competitive tendering in the public sector. The empirical research shows that benchmarking is indeed useful for a public organisation (principal). It makes him able to induce the provider (agent) to increase its performances, which ultimately leads to more X-efficiency and allocative-efficiency. Further, benchmarking helps the public organisation to keep the provider accountable for its performances. The effectiveness of using benchmarking in practice, however, seems to depend on how the benchmarking information is used by the public organisation and how this is interpreted by the provider. This thesis shows various ways in which the public organisation is able to influence this.

Key words: benchmarking, competitive tendering, public sector

Contact information

Name : Hergen Vorenkamp Student number : 1747169

E-mail address : H.J.Vorenkamp@gmail.com Telephone number : +31 (0)6 57128577

Additional information

Country : The Netherlands University : Groningen

Faculty : Faculty of Economics and Business Master : Business Administration

Specialization : Organizational and Management Control Supervisor : S. Tillema

(3)

Preface

Dear reader,

In front of you is the result of my last work during my study time at the Rijksuniversiteit Groningen, which was at the same time the last step for me to take to finish my Master in Business Administration, specialization Organizational & Management Control.

I would like to thank all the people who enabled me to accomplish this. First of all my thanks to all interviewees that have put in their time and effort to share their experiences and knowledge with me. Secondly I would like to thank my family and friends that supported me along the (sometimes winding) road. And last, but definitely not least, I am more than grateful for the support, advise, patience and encouragement from my supervisor, Sandra Tillema. As our collaboration already started during the last year of my bachelor, I found it very special to be able to write my ‘final work’ under her supervision.

I hope you enjoy reading my thesis.

(4)

Dutch Summary

1

In deze master scriptie is ingegaan op de vraag hoe nuttig en effectief benchmarking kan zijn in de context van aanbesteden in de publieke sector. Om deze vraag te beantwoorden is gekeken naar de theorie en de praktijk.

Het gebruik van zowel benchmarking als aanbesteden in de publieke sector kan vanuit de theorie geplaatst worden in een breder perspectief van prestatiemeting en –verbetering in de publieke sector. Het streven hiernaar is sinds de jaren ’80 groter geworden en initiatieven om dit te bereiken worden vaak geschaard onder de term New Public Management (NPM). Volgens Public Choice theory (één van de onderliggende theorieën van NPM) leidt het toepassen van aanbesteden tot meer X-efficiëntie: meer produceren met minder. Aan de andere kant leidt het volgens sommigen niet tot meer allocatieve efficiëntie (het aansluiten van het aanbod bij de behoefte), omdat de invloed van de consument niet wordt vergroot. Daarnaast zorgt het uitbesteden van publieke diensten er voor dat publieke organisaties hun directe zeggenschap over deze diensten kwijtraken, terwijl ze er nog wel voor verantwoordelijk blijven. Vanuit de agentschapstheorie kan worden geredeneerd dat het daarom belangrijk is dat publieke organisaties de dienstverlener(s) verantwoordelijk houden, door prestaties in de gaten te houden.

De literatuur laat zien dat benchmarking nuttig kan zijn voor publieke organisaties, wanneer zij diensten aanbesteden. Door benchmarking toe te passen kan een publieke organisatie meer informatie krijgen over de (relatieve) prestaties van de dienstverleners en daardoor beter toezicht houden. Tevens zullen dienstverleners door eventuele prestatieverschillen worden geprikkeld om hun prestaties te verbeteren, zeker wanneer de publieke organisatie ervoor zorgt dat slechte prestaties negatieve consequenties zullen hebben. Dit kan een publieke organisatie bijvoorbeeld doen door financiering af te laten hangen van de geleverde prestaties of door de benchmarking-informatie openbaar te maken, waardoor consumenten slechte prestaties indirect zullen ‘afstraffen’. Dit laatste kan op zijn beurt de allocatieve efficiëntie verhogen, doordat de keus van de consumenten meer invloed zal hebben en de dienstverleners hun prestaties zullen verbeteren door het aanbod bij de wensen van de consument te laten aansluiten.

1

(5)

Om te testen of de redeneringen over het nut van benchmarking in de context van aanbesteden in de publieke sector ook stand houden in de praktijk en om te kijken of het gebruik ervan ook effectief is, is onderzoek gedaan naar een specifieke benchmarkingtool, genaamd het Catalogusmodel. Deze tool wordt gehanteerd door enkele gemeenten in Nederland die Huishoudelijk Hulp hebben aanbesteed en de zorg uitbesteden aan verschillende zorginstellingen. Deze gemeenten meten de prestaties van de zorgaanbieders en presenteren de resultaten in de vorm van een ranking, die gebaseerd is op ten minste drie criteria: prijs, levertijd en cliënttevredenheid. Op basis van deze ranking, waarbij de best presterende organisatie bovenaan staat, kan de gemeente de cliënt objectief adviseren bij de keuze voor een van de aanbieders. Voor het empirisch onderzoek zijn vier casussen geselecteerd, waarbij telkens werd gekeken hoe een bepaalde (groep) gemeente(n) de benchmarkingtool hanteerde en welk effect de tool had. Voor elke casus is tenminste één beleidsmedewerker van een gemeente en één medewerker bij een zorginstelling geïnterviewd.

Uit het empirisch onderzoek bleek dat het gebruik van de specifieke benchmarkingtool in principe nuttig was. Ten eerste zorgde het voor meer competitie, omdat het beter presteren in de ranking de zorgaanbieders de mogelijkheid gaf om meer cliënten aan te trekken. Ten tweede zorgde de benchmarkinginformatie er voor dat consumenten een meer geïnformeerde keus konden maken, waardoor de invloed van de consument en de allocatieve efficiëntie werd verbeterd. Tot slot konden de gemeenten de tool gebruiken om de prestaties van de zorginstellingen in de gaten te houden. De effectiviteit van de benchmarkingtool bleek echter afhankelijk van hoe de tool werd gebruikt door de gemeenten. Niet altijd werd een zorginstelling door de tool geprikkeld om haar prestaties te verbeteren. Verschillende oorzaken kwamen naar voren, die ook aandachtspunten opleveren voor publieke organisaties die benchmarking gebruiken in de context van aanbesteden in een andere setting.

(6)

Aandachtspunt hierbij is de weging van de prestatiecriteria, die het beste dusdanig kan worden gekozen, dat de dienstverleners hun positie in de ranking kunnen beïnvloeden.

Ten tweede bleek dat sommige zorginstellingen niet afhankelijk waren van hun positie in de ranking om cliënten te kunnen krijgen. Sommige konden vertrouwen op hun goede naam en/of mond-op-mondreclame. Ook bleek dat sommige gemeenten niet altijd de nummer één adviseerde aan cliënten, maar de cliënt verwees naar een andere aanbieder. De prikkel voor deze laatste aanbieder om beter te presteren werd dus minder. In dit geval is het aandachtspunt de manier waarop de benchmarkinginformatie wordt gebruikt door publieke organisaties. Advies is om dit zo te doen, dat de dienstverleners merken dat een bepaalde positie bepaalde consequenties heeft.

Ten derde liet het empirisch onderzoek zien dat een aanbieder niet wordt gestimuleerd om iets met de ranking te doen wanneer hij denkt dat een cliënt de ranking niet gebruikt. Het onderzoek toonde aan dat het hierbij van belang is hoe de ranking wordt gepresenteerd aan de cliënt. Wanneer de scores voor de verschillende prestatiecriteria niet openbaar waren, was de cliënt ook niet in staat om op basis daarvan een keuze te maken en op die manier goede prestaties van een aanbieder op de verschillende criteria te belonen. Daardoor werden de organisaties niet gestimuleerd om beter te presteren op afzonderlijke prestatiecriteria. Voor publieke organisaties is de wijze waarop de ranking wordt gepresenteerd dus ook een aandachtspunt, waarbij het presenteren van de scores per prestatiecriteria aan te raden is.

Tot slot is het mogelijk dat sommige dienstverleners denken dat de publieke organisatie de benchmarkinginformatie niet gebruikt om toezicht te houden. Uit het empirisch onderzoek bleek dat de effectiviteit van benchmarking op dit gebied afhing van de wijze waarop de informatie werd gebruikt tijdens de gesprekken met de zorgaanbieders. Daarbij bleek ook dat dit moeilijker was wanneer er veel aanbieders waren gecontracteerd (meer dan acht), want dan kost het toezichthouden meer tijd, die meestal niet beschikbaar is. Voor een publieke organisatie is dit een aandachtspunt, wanneer ze beslist over het aantal aanbieders dat wordt gecontracteerd.

(7)

Index

1. Introduction ... 9

2. Theoretical Framework ... 12

2.1 NPM, Competitive Tendering & Benchmarking ... 12

2.2 Competitive tendering in the public sector ... 14

2.2.1 Creating competition ... 15

2.2.2 Competitive tendering, competition and efficiency ... 16

2.2.3 Competitive Tendering, Agency Problem and Monitoring ... 18

2.3 Benchmarking and competitive tendering... 19

2.3.1 Benchmarking in the ex-ante contract phase ... 20

2.3.2 Benchmarking in the ex-post contract phase ... 20

2.3.3 Expectations... 22

2.3.4 Empirical findings in literature ... 22

3. Methodology ... 23

4. Analysis ... 25

4.1 Competitive tendering and the benchmarking tool ... 25

4.1.1 Social Support Act ... 25

4.1.2 Competitive tendering process ... 25

4.1.3 Post-tendering phase ... 26

4.1.4 Benchmarking model ... 27

4.2 Case 1 ... 28

4.2.1 Introduction ... 28

4.2.2 Effects of using benchmarking ... 29

4.3 Case 2 ... 32

(8)

4.4 Case 3 ... 34

4.5 Case 4 ... 38

4.5.2 Expectations... 38

4.6 Cross-case analyse... 42

4.6.1 Competition ... 42

4.6.2 Complications ... 45

4.6.3 Monitoring and Control ... 48

5. Conclusions & Discussion ... 49

(9)

9

1. Introduction

Criticism in the late 1970’s and early 1980’s about an inflexible, bureaucratic and inefficient public sector, resulted in pressure on governments to introduce reforms in order to realize improvements (OECD 1995). The first countries realizing changes in reaction to that type of criticism were the US, UK and later on Australia and New Zealand (Greuning, 2001). Main components of the reforms in these countries were, among others measures, the introduction of more competition, the use of output controls, the empowerment of managers and an emphasis on accountability and control (OECD 1995). Years later, the term New Public Management (NPM) was introduced by Hood (1991) to label this kind of public reforms. Some researchers say that NPM was and is actually nothing more than a loose term for various ideas about public sector improvements (Hood, 1991; Greuning, 2001; Christens & Laegreid, 1999). However, according to Boston et al. (1996) the combination of those ideas and how they were presented was new. Further Greuning (2001) argues that when NPM was introduced, it was the first time those ideas were actually implemented and tested in practice.

One of the main elements of NPM is the introduction of market mechanisms in order to create competition. According to public choice theory competitive pressures are needed to reduce inefficiencies in the public sector, because the lack of these pressures negatively influences the efficiency of bureaucrats (Niskanen, 1968). Contracting out and using competitive tendering for public services is one of the most fundamental ways to create competition. When a public organisation contracts out certain services, it means that the provision of those services is done by another party. This can be either a public, non-profit or a private, profit organisation (Walsh, 1995). When using competitive tendering, the contracts will be granted to the organisation with the best proposal. This procedure leads to competition between the parties willing to provide the service (Boyne, 1998).

(10)

10

contracted providers and to control their behaviour. Agency theory focuses on this relationship between the principal (purchaser) and the agent (provider) in a contractual agreement (Walsh, 1995). According to this theory, an agency problem arises as a result of asymmetric information and a conflict of interests between the agent and the principal. Because of this problem, the agent may not always behave in the interest of the principal (Boston et al., 1996) and the principal cannot always verify the actual performance of the agent.

The principal has various options available in order to let the agent behave in his interests. Originally, agency theory focused on the optimisation of the contract in order to motivate the agent. But the theory also proposes monitoring and bonding for addressing the agency problem (Boston et al., 1996). A mechanism that could be useful for the principal in both motivating and monitoring the agent is benchmarking. The last couple of years benchmarking has been an important element of the reforms in the public sector, especially in the field of performance measurement and improvement (Bowerman et al., 2001). Furthermore, benchmarking is found to be applicable to various principal-agent relationships in the public sector and it is seen as a tool for the principal to induce the agent to behave in the principal’s interest (Andersen et al., 2008). Therefore, benchmarking could possibly be useful for performance improvements and also for solving the problem of public organisations losing direct control over contracted out services.

So far the effectiveness of benchmarking in principal agent-relationships is unclear (Andersen et al., 2008). Therefore, in this thesis it is not only investigated if benchmarking is useful for a public organisation for inducing the agents to perform, for monitoring the contracted agents and thereby can contribute to solving the problem of the loss of control, but also if it is effective in practice. The main question that will be answered in this thesis is as follows:

How can benchmarking be useful for a principal in the case of competitive tendering in the public sector, and will it be effective in practice?

In order to answer this question the following sub questions will be answered:

(11)

11

- What difficulties of using competitive tendering in the public sector can be identified in the literature?

- What does the literature say about the various applications of benchmarking in the public sector in general and how could benchmarking in the case of competitive tendering be useful?

- What is known from the literature about the usefulness and effectiveness of benchmarking in the case of competitive tendering?

- Is benchmarking useful in the case of contracting home help suppliers by local governments and what complications can be found?

(12)

12

2. Theoretical Framework

In this chapter a theoretical framework will be sketched, which forms the theoretical foundation for the empirical research. The chapter begins with a description of the main elements of NPM and an explanation of how competitive tendering and benchmarking can be related to that. After that, two theories which have been important in underpinning NPM will be used to explain the arguments behind the use of competitive tendering in the public sector. These theories are public choice theory and agency theory. In the subsequent section both theories are used to discuss how benchmarking can be useful in the situation of competitive tendering. The expectations arising from this section will form the basis of the empirical research. Finally, in the last section some empirical findings from the literature about the effects of using benchmarking in the situation of competitive tendering will be discussed.

2.1 NPM, Competitive Tendering & Benchmarking

New Public Management is the widely used term to refer to the public sector reforms in the last few decades. Those reforms come in many forms and various authors listed different types of reforms which could be grouped under NPM (Hood, 1991, p3; Stewart & Walsh, 1992; Boston et al., 1996; Pollit, 2002). The main principles of NPM are listed in table 1.

Principle Meaning

1. Disaggregation of units Making units manageable, split provision and production

2. Competition Contract based relations, internal markets and competitive tendering

3. Private-sector styles of management

Use of private-sector tools, such as performance agreements and management information systems

4. Discipline in resource use Do more with less

5. Hands-on management Active and visible control, responsibility at top, more accountability

6. Performance measurement Definition of goals, accountability through performance measurement

(13)

13

The principles of NPM are built on theories and ideas which already existed before the introduction of the term. Two streams which have been important for underpinning the ideas of NPM are “new institutional economics” and “managerialism” (Hood, 1991). Especially combining these two streams has raised critique, because of their sometimes conflicting ideas (Hood, 1991; Boston et al., 1998). New institutional economics (comprising public choice theory, agency theory and transaction costs economics) is important in arguing for more competition, transparency and incentives (Hood, 1991), while managerialism concerns freedom to manage and performance measurement (Dunsire, 1995). The main conflict between the streams is that managerialism argues for more central control (let the managers manage), while new institutional economics strives for more decentralisation (Hood, 1990).

However Hood (1991) also argued that differences between countries can be seen as for which stream is more important in their reform program, with for example the UK and Australia more oriented on managerialism and New Zealand more driven by new institutional economics. Also some countries have been selective in which reforms to implement, while others have introduced almost all of them (Pollitt et al., 2007). According to Pollit et al. (2007) and Dunleavy (2005) these differences in practice reveal the adaptability of NPM to various contexts.

Throughout the years the number of reforms ranged under the term NPM has grown rapidly. Competitive tendering is one of the reforms that is explicitly ranged under the term NPM by various authors (see table 1). It is used when public organisations contract out certain public services. The idea behind using competitive tendering is that it can create competition, since ultimately, this reform is applied in public service markets in order to select the party that can deliver that service most efficiently. Various countries have introduced competitive tendering and often made it obligatory for certain public services (Boyne, 1998).

(14)

14

defined as: “the continuous process of measuring products, services and practices against the

toughest competitors or those companies recognized as industry leaders” (Anderson, 2008).

Benchmarking has its origins in the private sector, but it has also been applied increasingly in the public sector since the introduction of Best Value in the UK (Bowerman et al., 2002). Benchmarking in the public sector is mostly used with the aim to improve performances. In principle, benchmarking exercises in the private, and also public sector are based on the idea that comparing performances gives insight into best practices and gives the possibility to copy those practices in order to improve performances (Andersen et al., 2008). Next to that, it is assumed that benchmarking practices can actually function as a substitute for market pressure as well (Van Helden and Tillema, 2005). Namely, performance gaps revealed by benchmarking can lead to institutional pressure to close those gaps and so, improve performances. This institutional pressure can come from internal stakeholders or from the necessity of demonstrating accountability to external stakeholders (Bowerman et al., 2002; Tillema, 2010).

This reveals that benchmarking not only aligns with the overall goal of NPM: performance improvements in the public sector. It also fits within the ideas of NPM about performance measurement and accountability (see sixth element, table 1). Furthermore, as benchmarking can substitute market pressure, it also meets the goal of creating competition in the public sector (see second element, table 2).

2.2 Competitive tendering in the public sector

(15)

15

2.2.1 Creating competition

The idea of using competitive tendering arises from the fact that public sector organisations are in principle not exposed to the market (Boston et al., 1996, p37). Production of some goods and services is difficult to be tested against the market, as these goods and services, such as education and healthcare, are not produced by the market or at least need governmental intervention to ensure adequate production (Walsh, 1995). When only public organisations produce such a good or service, this automatically leads to a monopoly position and no market pressure.

Public choice theory, which is based on the model of bureaucracy, explains how the lack of market pressure leads to inefficiency (Niskanen, 1968). The theory assumes that individuals are primarily motivated by self-interest. (Boyne et al., 2003). According to public choice theory this leads to three sources of deficiency in the public sector. First of all, politicians will not necessarily act in the interest of the society, but rather act from their self-interest (Boyne, 1998; Boyne et al., 2003; Boston et al.1996). Secondly, the bureaucracy may not always carry out the policy as made by politicians, because of self-interest of the bureaucrats. And finally, bureaucrats will not necessarily be efficient if it is not in their interest to be so (Walsh, 1995). According to public choice theory these three sources lead to X-inefficiency, which is producing less than is possible with the given inputs and allocative inefficiency, which is not producing the best mix of services (Walsh, 1995).

(16)

16

In order to induce public officials to behave according to the public interest and so to strive for efficiency, it is suggested by public choice theory that competitive pressures are needed (Boyne et al., 2003 and Boyne, 1998). Using market mechanisms is one of the reforms, which could create that kind of pressure. Those mechanisms are then introduced to mimic the market, such that funder and provider are split, prices become more important and competitiveness is created (Le Grand, 1991 and Walsh, 1995). In the past decennia various mechanisms have been introduced in the public sector, of which the most fundamental ones are privatization, pricing and charging for public services, and using contracts (Walsh, 1995). The last one is in particular related to competitive tendering, which is discussed in the next section.

2.2.2 Competitive tendering, competition and efficiency

The use of contracts to create competition occurs in many different ways. Sometimes contracts, or quasi-contracts, are used within the organisation itself, in order to split purchasing and provision divisions. However, contracts are also increasingly used to externalise public services to other (mostly private) providers, while making use of competitive tendering or market testing (Walsh, 1995). The primary reasons for competitive tendering lies in the assumption that cost savings can take place on three points: economies of scale, other labour practices, and more competition between suppliers (Ferris and Greddy, 1986). These effects are assumed to be larger if there are more suppliers to compete for the contract (Ferris and Greddy, 1986).

(17)

17

Boyne (1998), however, also argues that competitive tendering would not necessarily lead to less expenditure in the whole public sector, because as in line with the principles of public choice theory, bureaucrats will optimise their budgets and would reallocate the savings. Furthermore Boyne (1998) states that competitive tendering will not necessarily impact allocative efficiency, since the politicians and bureaucrats remain responsible for the services and the amount of output. As Parker (1990) puts it: “Since officials will still determine what is supplied and services

will still be funded out of taxation, competitive tendering does not introduce consumer choice and allocative efficiency. In this respect it falls far short of complete privatization.”

The introduction of customer choice has been promoted by public choice theories, as it could increase allocative efficiency. This type of reasoning has its foundations in Hirschman’s (1970) theory about exit and voice. Exit means that the consumer has the possibility to quit receiving a particular service from a certain provider, which makes him able to change to another service provider. And with voice the consumer is able to influence the type of service he receives by complaining or communicating with the provider about the service. Hirschman (1970) argues that increasing the power of the consumer by increasing the ability of the consumer to use his voice and to exit certain services, will lead to more demand-driven production by the public sector. In the last couple of years this demand-driven production has been getting more attention and it is seen as another way to improve efficiency, especially allocative efficiency.

(18)

18

2.2.3 Competitive Tendering, Agency Problem and Monitoring

So far public choice theory has been used to give arguments for the use of competitive tendering, especially in the light of efficiency improvements. However, public choice theory pays no attention to the consequences for public organisations when they enter a contract-based relationship. Problems regarding this relationship can be best analysed by using agency theory.

Agency theory has its origins in the literature about risk sharing problems between individuals and within groups (Eisenhardt, 1989) and focuses on the relationship between the principal and the agent in a contractual agreement (Walsh, 1995). The so called ‘principal-agent relationship’ exists when one party, the agent, performs a job on behalf of another party, the principal (Eisenhardt, 1989; Jensen and Meckling, 1976). Two assumptions are made by the theory. First, both the agent and the principal maximise their personal utility and so can have different goals. Secondly, there exists information asymmetry between both parties, which means that some information is not accessible for both parties (Eisenhardt, 1989; Jensen and Meckling, 1976; Moe, 1984). Out of these the agency problem arises, which is the problem of the principal that he cannot verify the actual performance of the agent (Eisenhardt, 1989, Jensen and Meckling, 1976).

Two aspects of the agency problem can be distinguished: adverse selection and moral hazard. The problem of adverse selection arises when the principal has not enough information to verify the abilities of the agent in the ex-ante contracting stage and moral hazard comes forth out of the possibility of the agent to shirk without the principal knowing that in the ex-post contracting stage (Eisenhardt, 1989; Moe, 1984). Overcoming these agency problems comes with certain costs, the agency costs (Eisenhardt, 1989; Jensen and Meckling, 1976; Moe, 1984, Fama and Jensen, 1983).

(19)

19

In the case of competitive tendering in the public sector, overcoming the agency problem is regarded to be of great importance. The main reason for this is that a public organisation will stay accountable for contracted out services, although it loses full control of those services. If the public organisation does not hold full control over the services provided anymore, it cannot guarantee continuity of the service delivery, the quality and the distributional objectives. Furthermore, due to the moral hazard problem, the service provider could actually behave in a way which conflicts with the interest of the society, while the public organisation is responsible for that (Ferris & Graddy, 1986). To overcome this problem, the public organisation has to keep the provider accountable for its performance by writing an appropriate contract and by monitoring the performance actually delivered (Ferris & Graddy, 1986; Mulgan, 1997 and Walsh, 1995).

Writing contracts and monitoring are, however, sometimes found to be difficult in the public sector. First of all writing a contract is difficult, as there is no or limited information available from the market about public services. This makes it hard to determine a good price for a public service (Ferris & Graddy, 1998). Further it is also found to be difficult to define quantity and quality aspects of the output in a contract, because most public services are by nature intangible and complex (Ferris and Graddy, 1984). This also hinders monitoring the performances of an agent effectively, as for doing so measurable criteria are needed.

2.3 Benchmarking and competitive tendering

(20)

20

phases will be respectively discussed in the first two subsections. In the subsequent subsection the expectations for the empirical research will be presented. Finally, in the last subsection some empirical findings about the use of benchmarking in the situation of competitive tendering will be discussed.

2.3.1 Benchmarking in the ex-ante contract phase

In the ex-ante contract phase the main problem for a principal is the adverse selection problem. This problem is caused by the fact that the principal is unable to know the abilities of an agent in advance (Eisenhardt, 1989; Moe, 1984). In the case of competitive tendering, the principal has to choose from multiple providers, based on the information that is delivered by the agents. Without any objective information it is however difficult to know in advance if the agent is right about his abilities. Benchmarking can then be used as a tool to assess performance objectively (OECD, Helgason, 1997) and to give insights into how one party performs in comparison with another. If this benchmarking information about possible providers would be available in advance, then it could be used to select the best candidates. Furthermore this benchmarking information can reveal which prices and quality standards are acceptable in general, which is helpful in writing the contract.

2.3.2 Benchmarking in the ex-post contract phase

In the ex-ante contract phase, benchmarking can be used by the principal as a tool to both monitor the performances of the agent and to induce him to improve these. The use of benchmarking, however, does not solve the difficulty of monitoring in the public sector, as measurable indicators will be needed as well. As has been discussed in section 2.2.2, public choice theorists argue that competitive tendering leads to more X-efficiency, as it will introduce competition. However, this competition only takes place in the ex-ante contracting phase, since only then various providers compete for the contract.

(21)

21

‘negative performance gaps’ are revealed by benchmarking, organisations will be induced by institutional pressure to improve their performance (Van Helden and Tillema, 2005). According to Andersen et al. (2008), in a principal-agent relationship this behaviour can be stimulated by the principal in various ways. First of all, the principal could use the benchmarking information to define if the agent gets access to monetary or other types of funds. In this way the principal punishes poor performance by lowering resources, which stimulates the agent to keep up his performance. Another way to induce the agent to improve his performance more voluntary is by publishing the benchmarking scores in order to influence the customers of the agent. Not only do agents get induced to improve performance by institutional pressure - the fact that their customers see when their performance is poor - customers can also make more informed decisions based on the benchmarking information, which automatically turns into a punishment for poor performance. Further, this implies also that benchmarking could actually improve allocative efficiency, as it increases consumer choice.

However for these reasoning’s to hold, two conditions must be met. First there have to be multiple suppliers providing the same service. Secondly, the consumers should be able to switch between these providers. Only then, agents will experience the negative consequences of poor performances (van Helden and Tillema, 2005; Tillema, 2010). Hence, the ability of a consumer to choose and exit freely are important in increasing the effects of benchmarking on performance improvements. It is however argued by Tillema (2010) that benchmarking information should also be actually used by consumers, will it be effective in creating market pressure. She argues that when the information is difficult to interpret or is not covering the information the consumer want to know, they will use it less.

(22)

22

2008). In the case of competitive tendering this implies that by using benchmarking the principal is able to partly compensate his loss of control over the public service, as he keeps control over the agent and can keep him accountable for the performances. Furthermore by doing so, the principal is also able to demonstrate accountability himself (Bowerman and Ball, 2000).

2.3.3 Expectations

From the reasoning in the previous two sections it is expected that benchmarking in a competitive tendering setting in the public sector could be useful. By applying benchmarking, the principal is expected to be able to attain more information about the performance of the agent. And with that information the principal is expected to be able to reduce adverse selection, give incentives for performance improvements, improve allocative efficiency, reduce moral hazard and demonstrate accountability. It is questionable if these expectations hold in practice, therefore the question for the empirical research is: Does this reasoning about the usefulness of

benchmarking in the case of competitive tendering hold in practice, is it effective in practice and what kind of complications can be found and how can these be solved?

2.3.4 Empirical findings in literature

(23)

23

3. Methodology

In this chapter the research method will be described. The empirical research is focused on a particular benchmarking tool, which is applied by several Dutch municipalities to rank providers of home help services. These providers have been contracted by the municipalities, while making use of competitive tendering.

The empirical research is based on case research. By applying this research method, phenomena can be explored in depth, which is especially useful for revealing deeper understandings (Lapan et al., 2012). For this study four case studies have been selected, based on purposive sampling, which means that the case studies are chosen based on the feature of interest (Silverman, 2000). In this research the interest was in a particular benchmarking tool, which is applied by only a few Dutch municipalities. This tool is especially relevant for this study to investigate, as it is placed in a competitive tendering setting. Further, as there are multiple cases, all of them with specific characteristics available for research, variations in the use of the benchmarking tool can be investigated. Multiple case research is therefore considered as “more compelling and regarded

as being more robust” (Yin, 2003). Moreover, Yin (2003) argues that analytic conclusions that

arise from two cases will be more powerful in comparison to a single case. Investigating multiple cases gives the ability to explore the effects in various contexts and to investigate if the expectations from the theory hold in every single context (Blumberg, 2008). Finally, comparisons can be made between the different cases (Lapan et al., 2012). For this research, multiple cases gives the opportunity to test the usefulness and effectiveness of benchmarking in multiple contexts. In addition to that, complications that are present in one case, but absent in the other, can be explained by the various contexts. However, as Yin (2003) argues, “each case must

be selected carefully, so that it either can predict similar results or predicts contrasting results but for predictable reasons.”

(24)

24

effects on the mid-term. Further, the empirical research focused on the use of benchmarking in the ex-post contract phase only, since the benchmarking tool was introduced by the municipalities with the latest competitive tendering process. Therefore the information was not used for a new competitive tendering process yet.

The most important data source were semi-structured interviews. According to Blumberg (2008) semi-structured interviews are especially useful when various issues need to be identified in order to understand the situation. Subjects in the interviews were the competitive tendering process, the reasons for using and the actual use of the benchmarking tool and the ultimate effects of benchmarking. In every case at least one official from the municipality and one manager or managing director from the contracted home help providers were interviewed. In this way the usefulness and effectiveness of the benchmarking tool could be analysed from the perspective of both the principal and the agent. In total five officials and four employees from the home help providers were interviewed. Next to that for a previous research all providers in case 4 have been interviewed (see table 2). All the interviews lasted between 45 and 90 minutes. Every conversation was digitally recorded and summaries of the interviews were sent back for approval. In addition to the semi-structured interviews some interviewees have been asked a few more questions by phone or e-mail in a later stadium. Further, some documents were used to verify some data.

Code Interviewee and function Case(s) O1 Official, responsible for contact with providers 1

O2 Official, responsible for assigning clients 1

O3 Official, responsible for contact with providers 2

P1 Employee of a relatively small provider 1, 2

P2 Account Manager of a national operating provider 1-4

P3 Managerial director of a relatively small provider 3

P4 Account Manager of a national operating provider 4

- Employees, several functions, various providers 4

(25)

25

4. Analysis

In this chapter the analysis of the interviews will be presented. This chapter is structured as follows. In the first section background information is given about the Dutch Social Support Act and the mandatory competitive tendering process which the municipalities execute in selecting home help providers. Further the benchmarking tool, i.e. Catalogusmodel, is discussed in detail. In the subsequent four sections the four cases are analysed, followed by a concluding section in which a cross-case analysis will be presented.

4.1 Competitive tendering and the benchmarking tool

4.1.1 Social Support Act

With the introduction of the Social Support Act (in Dutch: Wet maatschappelijke ondersteuning) in 2007, all municipalities in the Netherlands have become responsible for providing services which promote social and life skills and enable persons with a disability to participate in community life (Klerk et al., 2010). One of the performance areas defined by the act is related to facilities that municipalities have to provide to people with impairments, whenever these people (clients) are in need of this kind of facilities. Home help service is one of these facilities.

4.1.2 Competitive tendering process

For some services, including home help, the municipalities are required to use competitive tendering if they decide to contract external parties, from which they ‘buy’ those services. This is made obligatory by the central government in order to create competition and to give fair chances to all organisations which would like to offer their services. Usually contracts are set for a period of two years, with two times a possibility to extend the contract for one year.

(26)

26

Further, the municipality actually has much more freedom in how it deals with the multiple providers. Firstly, a municipality is free to choose if it wants to allow a limited number of providers or not, and when it limits the number, it can decide how many providers it would like to contract. Secondly, the municipality may decide on what criteria, quality and/or price, they select the providers. In practice this means that potential providers first have to agree with some quality criteria, have to indicate in what time they can deliver their services and have to offer a price which is not higher than the maximum price or which is within a certain price range.

The municipality subsequently contracts the providers which agreed with these criteria. If the municipality has decided to allow only a limited number of providers, while more providers have enrolled in the competitive tendering process, the municipality has to choose between them. In one of the four municipalities included in this study (case 2) this occurred. In that case, ultimately price differences were leading in the selection process, as all providers agreed with the quality criteria. So the organisations with the lowest price were selected.

Due to the fact there are multiple providers and it is unknown how much a provider has to deliver in the forthcoming contract period; the municipality does not make agreements about the scale of the delivered services. They only make agreements about the quality aspects and the price for which the providers are willing to deliver. The result is a ‘framework contract’ which only comes into force when the provider actually delivers his services to a client.

4.1.3 Post-tendering phase

(27)

27

municipality and provider then sign a service agreement in which the provider agrees to deliver the indicated number of hours for the price agreed in the competitive tendering phase.

It often occurs that a client does not (want to) make a choice between the multiple providers. The municipality then has various options for selecting providers. The consultant may choose for the client. Also various models have been created to help the municipality to select a provider objectively. Some of these models are developed by municipalities themselves, while others are developed by consultancy firms. One of these models is the benchmarking model, which is used by all four municipalities included in this study and which is further investigated in this research.

4.1.4 Benchmarking model

The benchmarking model, called the ‘Catalogusmodel’, is developed by a consultancy organisation called ‘Significant’. The main idea behind this model is that organisations are ranked, based on their performances. This ranking can then be used by the municipality to advise clients when choosing a provider, but also to assign clients objectively to a provider in the case a client does not make a choice. The rule agreed with the providers, is that the provider at the highest rank will get the clients who have not made a choice.

(28)

28

satisfaction only. A reason for this is that price is in principle not important for most of the clients, as they have to pay the same contribution, whatever the price providers maintain.

Finally the information produced by the model is also used by some municipalities during the meetings with the contracted providers. Usually these meetings take place quarterly. During those meetings often practical issues are discussed, but the municipality also has the opportunity to discuss issues regarding quality. Then the benchmarking information is actually used by the municipality to stimulate the provider(s) to pay attention to the performance measures.

4.2 Case 1

4.2.1 Introduction

The first case is a large municipality (>100.000 inhabitants), which subjects twenty-two home help providers to the tool. To study this case two officials (O1 and O2) from this municipality and three employees from three different home help providers were interviewed. One provider is relatively small and provides its services locally (P1). At this provider a care coordinator has been interviewed, who is responsible for the contact with the clients, but also for the contact with the municipality. The other two providers operate nationally (P2, P6). At both organisations an account manager has been interviewed, who is responsible for the contact between the organisation and the municipality.

The municipality had chosen to set no limit on the number of home help suppliers to contract. Hence, every single service provider could get a contract as long as they would confirm to satisfy the quality criteria set in the contract. One of the officials (O1) explains why the municipality had contracted multiple providers. First of all, they did not want to be dependent on a few large service providers, since that could lead to capacity problems, especially when one provider gets into financial trouble. Secondly they wanted to increase the freedom of choice for the consumers.

(29)

29

organisations to actively join the market and wanted them to see that achieving good results leads to a better position in the market. The second reason was that the municipality wanted the client to be able to choose freely, based on objective information.

4.2.2 Effects of using benchmarking

Competition

Both officials believe that the benchmarking model induces competition on price and quality, because it stimulates the providers to score better than their competitors on the ranking criteria. This is because ending up higher in the ranking can lead to several advantages. The interviewees from all providers included in this case believe that being at a higher rank leads to the ability to attract more clients. Furthermore, the official (O1) thinks that an organisation might want to prevent a low position, because of possible negative consequences. The official (O1) explains: “the ranking is working like ‘naming and shaming’, so organisations might think that clients will

not choose them anymore if they are at a low position.” Finally one of the officials thinks that

because clients with no preferences will be assigned to the providers at the highest rank, this stimulates providers to gain that position.

These arguments show that the ranking makes it important for providers to do better than their competitors. But in order to alter their position, the providers should influence the ranking criteria, since their rank depends on those. Therefore effects on price and quality would be expected.

And indeed, one of the officials (O1) notices some effects on price, which he relates to the effects of the ranking. The official says that a few providers offered a price which was a couple of euros below the maximum price. Especially one provider therefore ended up at the highest position in the ranking. The official explains that these providers probably have been stimulated to offer such a low price by the possible advantages of being at the highest rank. He also thinks that especially for new providers this could be beneficial to gain some market share.

(30)

30

organisation in principle maintains the same price in every municipality. Only when the organisation sees any possibilities to grow fast by offering a lower price, they consider the option of offering a lower price. In this municipality they decided to not invest, as the opportunities to grow were considered to be too small. The employee from the small provider (P1), on the other hand, admits that they actually did not even think about the eventual consequences of offering a lower price, but also explains that they really cannot offer a lower price. Furthermore, although the small provider is at a low rank, it does not experience the negative consequences from it. This is probably caused by the fact that this provider experiences that clients are assigned to providers, based on the personal experience of the consultants, not based on the ‘first rank rule’: “My experience is that the consultant chooses. I think it is just what that consultant thinks is best

and with what provider she has the best experiences.” This is confirmed by one of the officials

(O2) who says that the consultant sometimes assigns clients to companies who they think are good, based on their own experiences.

Effects of the benchmarking tool on quality are less clear. On the one hand, one of the officials (O1) thinks that providers are induced to improve on that and says: “Yes, I think that with the

Catalogusmodel we have quite a good instrument, at least to give incentives for improving quality. They will see when they score a 5 for quality and then they will realise that something is going wrong.” Further the official says that during meetings he had noticed that the providers are

actually looking at the ranking and responding to it. On the other hand, however, the official thinks that maybe the processes within the organisations have been improved, but the quality of the service delivered to a client has not. He says: “The windows have not become any cleaner.” This reveals that although the official thinks that the providers are stimulated to keep an eye on their quality, it is not clear if the quality actually increases, because of the benchmarking model. The interviewee from the small provider (P1) says that they just deliver good quality because they want to, not because they could reach a higher rank.

(31)

31

According to this interviewee, the only way to make the model work is to give new providers the ability to gain some market share. This indicates that right now, at least this, but maybe also some other new providers are induced by the benchmarking model not to compete, just because they cannot get a higher ranking. A second element is that the clients should make use of their ability to choose in order to have the ranking to have any effect. One of the officials (O2) confirms that clients do make use of the ranking and also the interviewee from the small provider (P1) thinks they do. However, all interviewees believe that mouth-to-mouth publicity is still of great importance, hence people do not necessarily base their choice on the ranking, but on what they hear from others and on their own experiences. The official (O2) therefore thinks that organisations which provide health care in various areas and which are more well-known, have an advantage. Also the interviewee from the small provider (P1) explains that some clients just choose them, because of their convincing brand-name.

Control

This municipality experiences to be more in control in comparison to the situation before the benchmarking model was introduced. One of the officials (O1), however, also talks about an ‘illusion of control’. He thinks that the municipality can induce providers to perform better, but does not really have options to ask more from the providers when they already fullfill their obligations of the contract. Still, the municipality tries to get the most out of the providers, according to the official (O1), but that is all they can do. The other official (O2) also thinks that this is important to do, because that official thinks that clients would appreciate the municipality doing ‘something’ for them and watch over the quality.

One of the official (O1) further says that since the scores for each aspect of client satisfaction are made transparent to the municipality, he experiences to be more in control. The official explains: “You now have something to talk about and to ask questions. That is one of the reasons why I am

really happy with this system. Now I get to know the story behind the providers.” A problem

(32)

32

The official (O1) hopes that because of these discussions the providers are stimulated to look at the benchmarking scores and to improve their performance. However, the interviewee from the small provider (P1) says that although they do discuss the numbers at the quarterly meetings, they do talk more about practical issues than about the ranking. Still, the interviewee (P1) notices that she experiences that the municipality only wants quality, since price is already agreed.

4.3 Case 2

The second case is a relatively small municipality (<30000 inhabitants), which subjects seven home help providers to that tool. To study this case, one official (O3) and two employees from two different providers (P1 and P2) were interviewed. The provider is also delivering home help services in the municipality in case 1.

With the latest tendering process, this municipality increased the number of providers who could get a contract from the originally three to eight. The official points out that they on the one hand decided this, in order to increase consumer choice and to give new organisations a chance to offer their services. On the other hand, the official says, they limited the number of providers to eight, since they wanted to be able to handle the contracts. Ultimately, eleven parties were involved in the competitive tendering process. Just as in case 1, the official explains that these had to confirm they would satisfy certain quality criteria. Next, as only eight were allowed, the municipality selected the ‘best’ providers based on the two criteria: price and delivery time. However, the official says that all three non-selected parties were not selected because of their price. Currently there are seven parties that have a contract with the municipality, since two providers have merged.

(33)

33

Competition

The official says it is hard to tell if prices have become lower, because of the benchmarking model. He explains that many other variables could have influenced the lower prices. The official, however, acknowledges that the limit on the number of providers could have influenced competition on price. Especially since the selection of the providers was mainly based on price, as it counted for 65%. The official says that the municipality also made use of the opportunity to let providers compete on price and notices that ultimately a municipality prefers an inexpensive service.

This reasoning reveals that probably not because of the fact price is an important criteria in the ranking, but especially because it played a crucial role in the selection of providers, the organisations were stimulated to offer a lower price. However the following shows that the benchmarking model also stimulates the organisations to perform well after the competitive tendering phase. The official says: “The Catalogusmodel works according to a ranking at which,

theoretically, but also according to the consultants practically, new clients look at, in order to see at what rank a provider is positioned and if it delivers good quality. The higher the rank of the organisation, the more clients it gets.” The official says that in this way the municipality

stimulates the providers also to be interested in the market after the competitive tendering phase. Further the official notices that currently the quality aspects seem to have a large influence on the ranking, as the ranking changed due to differences in quality.

(34)

34

or not they will be able to attract a lot of clients. He says that his organisation adjusts its strategy to the opportunities in the market.

Control

The official admits that the municipality is actually not interested in the position of a provider. What he thinks is of importance, is that a client feels comfortable with the service delivered and can choose, based on what he thinks is important. He says: “If the provider is getting judged by

the client satisfaction, then it is their own responsibility if they do not get any clients. We would not care about that.” The official further explains that the municipality does not expect the

providers to work harder, because of the model. He says it is already agreed in the contract that providers have to perform well.

Further the official believes that with the benchmarking tool he does not feel to be more in control over the providers. The official notices that this was also not one of the goals. The official, however, believes that the information created by the system can be useful for the conversations with the providers. Nevertheless, he says that most of the information was also available when the benchmarking tool was not used; delivery times were already administrated and client satisfaction was also measured by other types of research.

Finally, the official thinks that this municipality is too small and has too few resources to actively control and induce the providers to perform better. However, the official does think that the municipality could do more with the tool, by using it better and using its strengths. Especially by increasing the use of the tool, not only by the employees who advise the clients, but also on a higher political level, in order to induce for example the providers that are currently not actively trying to compete.

4.4 Case 3

(35)

35

providers independently. A home help provider had to satisfy certain quality criteria and a maximum price was set. In total twenty-one providers were contracted in this region; not all of them active in all municipalities. Some municipalities limited the number of providers, in some other municipalities they did not.

For this case one official and two employees of two different providers were interviewed. The official (O4) is responsible for maintaining the benchmarking tool and the contact with the providers on behalf of all the ten municipalities, but officially she is working for only one of them. One of the employees interviewed is a managing director of a regional home help provider (P3) which provides its services for yet another of the ten municipalities. The other interviewed employee is an account manager of a national home help provider (P2) which provides its services in all the municipalities.

Competition

According to the official the main reason for her municipality to use the tool is to be able to objectively assign clients to the best service provider. This already suggests that in this municipality, the number one position of the ranking is of great importance. The official also says: “The number one position is the most important one, because the agreement is that if

clients don’t know which provider to choose, we refer to the number one in the ranking. So number one, and the others do not really matter.”

(36)

36

ranking model, but it is mainly based on price. So if you have set a low price, you are almost sure to have gained the highest position in the ranking.” The provider (P2) further notices that

there was one provider which had offered a price, which was at least 2 euros below the price of most other providers. In his view, this makes it impossible for other providers to ever attain the highest rank even if you get a 10 for quality, according to the large provider.

Although this situation is bad for some providers, the official admits that this situation is in principle not bad for the municipality. If the provider with the lowest price delivers to most new clients, the municipality saves a lot of money. Furthermore the official notices that the provider with the lowest price is not performing badly on quality at all. The municipality is satisfied with a performance that is good and according to what is agreed in the contract, but also sees in practice that quality is not given up for the low price.

However, the management director of the regional provider (P3) says about the municipality where they deliver home help: “we never hear the municipality talking about quality”. This suggests that this provider perceives that quality is not the main priority in this municipality. The effect of this could be that not all home help providers are induced by the benchmarking tool to perform better on quality. At least the manager of the national service provider says they do not spend a lot of time to improve their quality in this municipality, just because they cannot influence their rank by doing so.

(37)

37

considered, and the official also believes that this could affect the choice clients make, since it would help them to distinguish providers also on quality aspects. According to the official, this could induce providers to perform better on quality.

The managing director from the regional provider (P3) says that his company is not induced by the ranking to perform better on quality. The managing director points out that for them it is important that clients feel confident with them, and will become clients for other services they can deliver in the future. Further he says that they can rely on their brand awareness and the extra educated staff, which are able to support the employees who deliver the home help services. In that way they can deliver high quality and try to influence the choice of the client. In this respect they are less interested in the ranking. The only way it is important for them is in order to compare themselves with others. The interviewee said that this is important, because in order to stay in business, they should keep up with the others.

Control

The official says that the benchmarking tool is not explicitly used by the municipality to monitor or control the behaviour of the providers. Nevertheless, the official notices that the tool gives more insights into how the providers work and it gives an incentive to the municipality to keep on monitoring the providers. However, as the information which is produced through the benchmarking system is in line with the expectations and conditions of the contract, it has never been necessary to take action, according to the official.

The official further points out that how the benchmarking tool is used, differs a lot between the ten municipalities. For example, some do not publish a new ranking directly, while others only take it for information. Some municipalities clearly do not have the ranking as their main priority. This is also experienced by the manager of the national provider (P2), who notices that not all municipalities are present at every plenary meeting and some have never been present at all.

(38)

38

many service providers is also practically impossible for a municipality and will at least not lead to more focus on quality, since a municipality cannot manage that. The manager says: “If

something is not going well, then there is almost no possibility to have a conversation with the municipality. For example, a few days ago I was at a municipality in another region which has currently 20 providers contracted and they admitted that there were actually too many providers to keep in touch with. If there are plenary meetings with all providers, there is never enough time to talk about quality. However, if there are four to five providers, a municipality can spend a lot more time on discussing quality issues.” This is however not seen as a problem by the official,

who says that so far nothing has gone wrong.

4.5 Case 4

The fourth case is a relatively small municipality (<30000 inhabitants), which subjects eight home help providers to the tool. Just as in case 3, it works together with other municipalities in the region; they did the competitive tendering process together and all planned to use the benchmarking tool. At the time the interviews were conducted, this municipality did not have the tool fully working yet, so this case is mainly based on expectations. To study this case, one official (O5) from this municipality and one employee from a home help provider (P2) were interviewed. Next to that, all providers active in this municipality were interviewed one year before the other ones. At that time the benchmarking tool had already been planned to be used, but research had to be done in what way this could be done best. The purpose then was to investigate what the providers expected from the benchmarking tool.

4.5.2 Expectations

(39)

39

to be able to objectively assign the clients who do not choose themselves, so that they can carry out a fair policy where every provider gets the same chances. How the municipality expects the benchmarking to influence these goals will be discussed in the next two paragraphs.

The official thinks that by using the benchmarking tool the risk of being dependent on one large provider can be decreased, because with that tool a more proportional distribution of clients among the various service providers can be achieved. This suggests that the official expects not only competition, due to the increased number of providers, but also the benchmarking tool to influence the spread of clients among the providers. The official explains that in the old situation, without the tool, the consultant decided which provider a client was assigned to if the client did not choose himself. This had as a result that most of these clients were assigned to the largest provider, because for consultants it was the easiest way to combine multiple requests. But when the benchmarking tool will be implemented, the official expects that more spread will be created and a fairer policy, since then consultants will assign those clients to the provider at the highest rank. If this in practice affects the spread, however, depends on which provider is at the highest rank and changes in the ranking in the course of time.

This reasoning from the official in the previous paragraph reveals that two of the main reasons for this municipality to use the tool are actually related, since assigning clients objectively is expected to affect the distribution of clients too. The goal of the municipality to make the market more transparent for clients is also expected to possibly affect the distribution of the clients, since it is expected by the official that clients get a larger role, which may imply that other choices are made. At least, the official points out that the ranking with all the components will be published, so the clients can make a choice, based on what they find to be important.

The official also believes that such a larger role for the client should create an extra incentive for the providers to do their best. The official explains: “If the clients are happy, the providers will

see this in the ranking, and that could in turn lead to attracting more clients.” Further, the