The design of a Performance Measurement System in a small organization

The design of a Performance Measurement System

in a small organization

By

Derk Jan Alberts

University of Groningen Faculty of Economics and Business Master of Science Business Administration Specialization Organizational and Management Control

January 2013

Frederikstraat 13 9724 KA Groningen derkjan.alberts@gmail.com

S1718320

Abstract

This research paper is about the design of a performance measurement system (PMS) in a small organization. The literature revealed that the current knowledge on the development of a PMS within small organizations is lacking and this lack of knowledge formed the starting point for conducting this research. Developing such a PMS includes multiple phases; however, this research focuses on the design phase. The literature that is reviewed is based on contingency factors, PMS dimensions and performance measure aspects. The outcome of the literature review resulted in a theoretical framework consisting of seven aspects: information systems, organizational culture & management style, external

environment, strategy alignment & causal relationship, balance, dept & breadth and structure.

These seven aspects were investigated during a case study at AP who is situated in the financial service industry. An analysis is performed to reveal the presence of each factor and to compare the findings from the case study with the expectations of the literature. The outcome of the analysis revealed that six factors influence the PMS design. The results on the one hand support the literature but on the other hand, this research enriches the current knowledge. The results of this case study indicate that a large database is present within AP to provide PMS with data. Moreover, AP started designing their PMS from scratch and therefore incorporated their strategy in their PMS. In addition, detailed reports were established within AP. The remaining case study contribution is the method that AP uses to define each measure.

Index

1.3 Outline of the thesis... 6

2 Research design ... 7

2.1 Research Objective ... 7

2.2 Definitions ... 7

2.3 Research scope ... 8

2.4 Research question and sub-questions ... 10

3. Theory ... 11

3.1 Influence of the contingency factors of a small organization on a PMS ... 11

3.1.1 Literature review insights of the contingency factors... 13

3.2 Performance measurement systems ... 14

3.2.1 Literature review insights of the performance measurement system ... 16

3.3 The design of a performance measure for a small organizations ... 17

3.3.1 Literature review insights of the performance measures ... 19

3.4 Theoretical framework ... 20

4. Research methodology ... 22

4.1 Data sources ... 23

5. Case study ... 25

5.1 Introduction of AP ... 25

5.2 The current use of performance information ... 27

5.3 The design of a performance measurement system ... 28

6. Analysis ... 33

6.1 Contingency factors ... 33

6.2 The performance measurement system ... 35

6.4 The overview ... 39

7 Conclusion ... 41

8. Research limitations and recommendations ... 44

References ... 45

Appendix 1: Interview protocol ... 48

Appendix 2: The Balanced Scorecard ... 50

Appendix 3: The Dashboard ... 51

1. Introduction

Within the Netherlands small organizations represents more than 52% of the business types when compared to sole-proprietorship, medium or large organizations, and therefore represent a large part of the different business types (MKB service desk, 2011). Small organizations characterize themselves by offering more flexibility and the ability to adapt quickly to the competitive and changing environment when compared to large organizations (Garengo et al, 2005). But besides these advantages, most small organizations face a lack of a measurement system that measures the performance and providing (top) management with the information about their business performance (Cocca and Alberti, 2009). This thesis will therefore elaborate more on the development of a performance measurement system (PMS) within small organizations.

1.1 Problem identification

The reason why small organizations face a lack of such a measurement system is because (top) management often wear multiple heads. They are so busy running day-to-day business that they have little time to take a step back and focus on performance measurement (Paulson and Hughes, 2009). In addition, many small organizations lack formal developed processes. As a result, the performance measurement that does take place is often based on ad hoc basis (Paulson and Hughes, 2009). But in the current business environment it is more than ever important to measure the performance of the organization because these measurements results in valuable information that is needed to plan and control the activities of the organization and to react to an increasing dynamic and competitive environment (Shahin and Mahbod, 2006).

This also counts for a small organization in the Netherlands situated in the financial services and payment industry. It started almost two years ago and has grown rapidly since. As a result of this rapid growth the need for more performance information increased. In order to maintain this growth (top) management has to take the right decisions, but they are dependent on timely, consistent and reliable information (Merchant and van der Stede, 2007).

1.2 Theoretical relevance

systems already have been established for a long time and play a major role in structuring the organization and strategy (Langfield-smith, 1997). An interesting fact is the emergence of a PMS within a small organization. Coca and Alberti (2009) state that in most literature this subject is discussed regardless of the size of the business. However, even if these publications are present few of them focus on performance measurement in small organizations. Within this subject, there are numerous frameworks and processes for the design of a performance measurement, for example the well-known balanced scorecard (Kaplan and Norton, 1992) or the EFQM excellence model. Nevertheless, these frameworks are often designed for large organizations (Fernandes et al, 2006). Small organizations do have different characteristics that differentiate them from large ones. Therefore, there is a need to enrich the current knowledge on how to design a PMS for a small organization (Hudson et al, 2001; Garengo et al, 2005; Cocca and Alberti, 2009). Hudson et al (2001) also state that there is gap between theory and practice regarding the design. Their research indicates that the current theoretical approaches for the design and implementation of PMS forms a discontinuity opposed to the requirements of practitioners. Moreover, the findings of Fernandes et al, (2006) reveal that there is limited research done on the Balanced Scorecard applications in small organizations. Gumbus and Lussier (2006) also recognize this. They state that there is a lack of knowledge of designing and usage of a BSC within small organizations, despite the amount of research, conducted on this specific topic.

1.3 Outline of the thesis

2 Research design

This chapter elaborates the research design. It starts with defining the purpose of this research stated in the research objective. Moreover, this chapter discusses several definitions that clarify the aspects of the research objective. Subsequently the scope of this research is described. Then, based on the research objective and scope the main research question and sub questions will be defined. The methodology is not discussed in this chapter; it will be elaborated in chapter 4.

2.1 Research Objective

From a scientific perspective, a gap is identified in the existing literature regarding performance measurement systems and how these systems should be designed for small organizations. It became clear that the measurement systems that are in place mostly are designed and implemented for relatively large organizations. However, this lacks knowledge on how to design these systems in small organizations. Because of this gap, the findings of this thesis will contribute to the existing literature by researching the design of a PMS within a small organization. By performing this research, it can reveal interesting insights and add knowledge towards the literature. Therefore the objective of this thesis is:

The design of a performance measurement system for this small organization within the financial service sector

However, before addressing the main research question and sub questions in detail, several definitions are elaborated in order to provide clarity and to create and maintain consistency. In addition the scope of this research is explained in more detail, it defines on which specific aspect this research is conducted and which aspects are taken and are not taken into account during this research.

2.2 Definitions

Size of enterprises

In order to decide when an organization is classified to a specific type of enterprise the European Union (EU) provided a definition of the different enterprise sizes that are common in the EU. The definition of enterprises is based according to staff headcount and turnover or balance sheet total (European summaries of EU legislation, 2011).

Enterprise category Headcount Annual turnover

Large >250 > €50 million

Medium < 250 < € 50 million

Small < 50 < € 10 million

Micro < 10 < € 2 million

Table 1: Enterprise category

This table will be used as a directive for the theoretical framework and the case study to support the focus of this research.

Performance measure

According to Neely et al (1996), a performance measure is defined as a metric used to quantify the efficiency and or effectiveness of an action. This definition will be used in this research to act as a directive and to provide clarity during the theory investigation.

Performance measurement system

Elg (2007) views a PMS as a system that transforms input data into usable information for various kinds of decisions in organizations. Neely et al (1996) define on the other hand a PMS as a balanced and dynamic system that is able to support the decision-making process by gathering, elaborating and analyzing information. Moreover, this system is used to measure the achievement of the organizational objectives. This research uses the definition of Neely et al (1996) to provide, on the one hand a clear distinction between a performance measurement system and a performance measure and on the other hand, it provides clarity while investigating the theory.

2.3 Research scope

integrates the PMS. These phases consist of the design phase, implementation phase, use phase and the updating phase (figure 1)

Figure 1: The several phases for developing a performance measurement system (extended with the literature of Neely et al (2005)

Bourne, et al (2000) state that in order for an organization to establish a well working PMS it is not enough to focus solely on the aspects of the design phase but also to take into account the aspects that are relevant for the implementation, use and updating process. In order to let the PMS contribute to the long-term success of the organization. For this research, the researcher is aware of these four phases but in order to limit the scope, this thesis will focus only on the design phase.

The design phase consists of a certain requirements that have to be fulfilled in order to complete this phase. Bourne et al (2000) and Lohman et al (2004) state that these requirements are: identifying the key objectives which have to be measured and designing these measures. Neely et al (2005) takes it one-step further and takes a set of measures into account that results according to Neely et al (2005) in a performance measurement system as an entity i.e. the design of the system. Moreover Neely et al (2005) also noticed that the relationship between the PMS and the environment in which the PMS operates should be taken into account (figure 2).

Figure 2: A framework for a performance measurement system design (Neely et al, 2005)

2.4 Research question and sub-questions

To translate the objective towards a researchable question, a main research question is formulated and presented below:

How should a performance measurement system be designed for this small organization within the financial service sector?

The main question is divided into three sub-questions

1. What is the influence of the contingency factors of a small organization on the design of a PMS? 2. How should the performance measurement system be defined for a small organization?

3. How should the individual performance measure be defined for a small organization?

3. Theory

This chapter provides an extensive literature analysis based on the three sub questions described at paragraph 2.4. At the end of each theory section, a conclusion is presented that will be used for the development of the theoretical framework. This framework will define the aspects that are investigated during the analysis, but first the theory is discussed.

3.1 Influence of the contingency factors of a small organization on a PMS

Due to the specific focus on small organizations, an investigation of the factors that affect the design of a PMS is relevant for this research due to the different characteristics a small organization has compared to a large organization (Garengo and Bititci, 2007). The purpose of this question is to elaborate these factors and to investigate their relationship between these factors and a PMS. These factors are based on the popular contingency theory within the management accountancy literature. Chenhall (2003) reviewed multiple contingency factors that influence the design, adoption and effectiveness of a management control system (MCS). According to Garengo and Bititci (2007), the organization decides based on these factors which system is the most suitable one (i.e. the system that fits the best for the organization). Chenhall (2003) performed a critical review of the findings from contingency-based studies over the past twenty years. The factors that where reviewed consist of the environment, technology, size, structure, strategy and culture. However the results of Chenhall (2003) focus on multiple types of organizations (i.e. small, medium and large) and involves multiple management control system, so the results of this review forms a broad spectrum and is not specifically tailored towards a PMS for a small organization. According to Garengo and Bititci (2007) most of the contingency studies do not focus on small organizations and therefore they performed a literature review based on contingency factors that influence the PMS in a small organization. Based on their literature review they identified four variables that have a particular influence: Information systems, Strategy, Organizational culture & management style and External environment. Due to their particular influence of these factors of a small organization on a PMS, these factors are further elaborated.

is necessary to design, implement and use a PMS (Bititci et al, 1997). Within a small organization, an inadequate information system is one of the main obstacles for performance measurement (Barnes et al, 1998). They often have limited resources available for data analysis, so data is gathered, processed and analyzed in an imprecise way that increases the ambiguity of the measured objectives (Garengo et al, 2005). The total amount of data available for performance measurement within small organizations is relatively low (Barnes, 1998; Bourne and Neely, 2002). It is expected that a small IS is available but on small organizations can also still use spreadsheets for collecting their data (Garengo and Bititci, 2007). The data that is available often consist of internal data about their operational processes but usually very little external data (Barnes et al, 1998). Moreover, data quality is largely unknown or often unclear and consists of complex or obsolete data (Hudson et al, 2001).

The other factor called ‘strategy’ forms a subject that is according to Chenhall and Langfield-Smith (2007) concerned with practices that help planning the long-term direction of an organization. According to Kaplan and Norton (1996), a PMS forms the translation of the strategy of the organization. Therefore an alignment should be expected between a PMS and the strategy of an organization. However, the study of Garengo et al (2005) revealed that small organizations often face a lack of alignment between performance measurement and the business strategy, and this is proven to be one of the obstacles. The approach that small organizations apply is that a PMS arises at an informal way, often not planned and not based on a predefined model. It is often introduced to solve specific problems and then grows out spontaneously rather than as a result of planning (Barnes et al, 1998). Consequently the alignment between a PMS and strategy inside a small organization is poor. Garengo et al (2005) state that this alignment is particularly important for small organizations, because these organizations often lack a formalized strategy. By designing and implementing a PMS, it could promote the definition of a formal strategy. Small organizations often neglect to make the strategy explicit (Garengo et al, 2005). Hudson et al (2001) confirms this by stating that strategy is defined as informal within small organizations. Moreover Hudson et al (2001); Barnes et al (1998) state the performance measures are defined with no reference to the strategy of the organization moreover the translation of the organizational strategy towards strategic/ operational goals is limited.

as the practices adopted by leaders in decision making, relationships, motivation and managing sub-ordinates (Garengo and Bititci, 2007). Within small organizations, managers are often the owner of the company and control in a small organization rest primarily with one or a few people with high autonomy (Cocca and Alberti, 2009). Organizational success or failure is therefore seriously affected by the managerial competencies of the owner-manager (Cocca and Alberti, 2009). It is expected that the authoritative management style and his/her commitment will support the introduction of a PMS and the development of the performance measures by encouraging employees and by providing the necessary resources (i.e. time, money, etc) (Garengo and Bititci, 2007; Hudson et al 2001).

Besides management commitment Sousa and Aspinwall (2010) state that due to its unified culture in small organizations there is a good foundation for change, and low bureaucracy encourages the success of a new initiative (i.e. a PMS). Moreover, because of the low bureaucracy it is expected to see more face-to-face relations and simplified communication processes that encourage the flexibility, adaptability and rapidity in responding to the change of a new PMS (Cocca and Alberti, 2009).

Small organizations are characterized to operate in highly competitive, turbulent and uncertain markets (Garengo et al, 2005). Usually they do not have control or influence over the market thus instead of driving the market they need to adopt a reactive approach and adapt to market changes (Hudson Smith and Smith, 2007). The advantage of a small organization is that they often relay on a limited customer base and operate in limited markets; they are usually closer to customers and have the possibility to develop more personal relationships with them (Hudson et al, 2001). Due to his highly dynamic and turbulent environment a PMS has to be very flexible, rapidly changeable and maintainable (Cocca and Alberti, 2009). It is therefore expected that changes should quickly be reflected in the PMS and the measures monitored should be reviewed on a regularly basis in order to see whether the measures are still accurate or outdated (Cocca and Alberti, 2009).

3.1.1 Literature review insights of the contingency factors

a) Reviewing the Information systems factor, the literature expects that small organizations often face the lack of a flexible and adequate IS. This results in a limited database and analyze functionality to provide the PMS with the information needed to measure the performance. It is expected that a small information system is available with limited analyze functionality but data quality is unknown or unclear and can consist of complex or obsolete data.

b) The outcome of the Strategy factor revealed that the design of a PMS could promote the definition of a formalized strategy due to a shift from an informal and not explicitly formulated strategy (whereas the translation towards strategic/ operational goals is limited) towards a more formal explicit strategy. It is also expected that small organizations face a poor alignment between the organizational strategy and their PMS due to the lack of planning.

c) Reviewing the Organizational culture and management style factor, the literature expects that management commitment form a critical factor for success when designing a PMS. Organizational culture on the other hand should promote change (i.e. the design of a PMS). This is expressed by the way that management encourages employees, by providing the needed resources and due to the simplified communication process.

d) The literature review of the External environment factor expects that a small organization operates in a turbulent environment and the PMS should therefore be flexible and maintainable. It is expected that changes are quickly reflected in the PMS and measures are reviewed on regularly basis.

3.2 Performance measurement systems

During the recent years, several PMS models have been developed (e.g. Balanced Scorecard, EFQM, Performance prism, etc.). These models incorporated a more balanced structure compared to the traditional models that focused heavily on financial measures (Garengo et al, 2005). However, these models do not automatically comply with the needs of a small organization. According to the study of Garengo et al (2005), several dimensions influence the selection of the PMS and therefore increase the success of the design and implementation of a PMS within a small organization. The purpose of this question is therefore to elaborate these dimensions and to analyze what the influence of these dimensions is on the design of a PMS within a small organization. Garengo et al (2005) performed a literature review and compared multiple measurement systems based on these dimensions. The dimensions that are used consist of Strategy alignment, Balance, Depth & Breadth and Causal

has a particular influence on the selection of the PMS and because of this particular influence, these dimensions (highlighted in italic) are now further elaborated.

The alignment between a PMS and strategy is according to Garengo et al (2005) important in order to link the organizational strategy towards the objectives and the operational aspects. As a result a PMS should therefore be designed and implemented in accordance with the organizational strategy. This is also confirmed by Cocca and Alberti (2009) who state that it is important for small organizations to establish an alignment because (as also identified in paragraph 3.1) these organizations often lack a formalized strategy, and by implementing a PMS it could promote the definition or formalization of a strategy (Garengo et al, 2005; Anderson et al, 2001). Moreover, the approach that small organizations apply is that a PMS arises at an informal way, often not planned and not based on a predefined model. It is often introduced to solve specific problems and then grows out spontaneously, rather as a result of planning. This results in a poor alignment between the organizational strategy and the PMS (Barnes et al, 1998).

The balance between financial/non-financial measures forms also an interesting aspect for small organizations (Cocca and Alberti, 2009). Often small organizations still relay mainly on financial information; which are more historical and short term focused (Cocca and Alberti, 2005; Hudson et al, 2001). Although some operational information is present at a small organization it is important to incorporate a balanced structure when designing the performance measures in order to align decision making processes to strategic objectives (Garengo et al, 2005). In order to accomplish this, a balanced PMS could be an important support tool (Tenhunen et al, 2001). It is therefore expected that a shift will be made towards the integration of non-financial as well as financial measures (Cocca and Alberti, 2009; Garengo et al, 2005; Neely et al, 2000).

measures used in small organizations consist often of financial and operational information, which are used to support management (Cocca and Alberti, 2009). But it therefore neglect to take into account all the other activities of the organization; i.e. including all the relevant stakeholders (Cocca and Alberti, 2009). However, when designing a PMS small organizations should integrate managerial as well as operational and support measures in their PMS whereas the focus should be on the breadth aspect instead of developing an in-depth model (Garengo et al, 2005).

The last dimension elaborates the causal relationship. A PMS should not only just measure the results but also their determinants in order to help monitor past actions and the improvement process, in other words there has to be a causal relationship (Garengo et al, 2005). Kaplan and Norton (1996) underline that indentifying a causal relationship between performance indicators and objectives supports strategy review and learning. However Garengo et al (2005) indicated that the causal relationships between determinants and results can be very complex and which is difficult to analyze. But by understanding the relationships between results and determinants periodic feedback on the measures, results and incremental changes becomes possible. This would be very useful for improving the process within a small organization (Garengo et al, 2005; Cocca and Alberti, 2009). In order to establish such a causal relationship the measures should therefore not be designed in terms of an ad hoc collection, instead the measures should be designed in such a way that a causal chain can be identified whereas eventually the measures should contribute towards the strategic objectives (Fernandes et al, 2006).

3.2.1 Literature review insights of the performance measurement system

The dimensions elaborated below are according to the literature expected to be relevant and are therefore identified as an aspect of interest. The relevance of each dimension is elaborated and an explanation is provided what theoretically could be expected at a small organization who is designing a PMS. The dimensions described below will be used for the analysis and during the analysis; these factors are empirically investigated based on the case study.

f) The results of the literature review of the Balance dimension expects that a small organization should use a more balanced view whereas the performance model that is selected should support this balanced view. This is expressed by the way that during the design process there will be a shift from the current available operational and short-term financial measures towards a balanced system consisting of non-financial as well as financial measures.

g) The outcome of literature review on the depth and breadth dimension expects that a small organization should focus on breadth aspect while designing a PMS, and the PMS should support this. It is expected that the PMS will incorporate all the activities (i.e. managerial, operational and support) of the organization in their PMS.

h) The last dimension is the causal relationship dimension. The literature expects that by using the selected model it can establish this causal relationship between its measures but also between the measures and the objectives, which eventually supports the strategy. This is expressed by using the selected model in order to suppress an ad-hoc collection of measures and design measures that contribute towards the organizational strategy.

3.3 The design of a performance measure for a small organizations

According to Neely et al (1997), one of the key questions that have to be considered during the design phase is how the specific measures of performance should be designed. They state that it is a complex process and an inadequately designed performance measure can result in dysfunctional behavior. This often occurs because the method of measuring the performance (due to calculations; i.e. formula) encourages individuals to pursue inappropriate behavior. However, designing a performance measure involves more than simply specifying a robust formula. Issues such as the purpose of the measure, the frequency of measurement and the source of data have to be considered while designing (Neely et al, 1997). In order to provide guidance and structure during the design process Neely et al (1997) developed a framework (figure 3) which seeks to specify what a good performance measure constitutes. According To Neely et al, these elements should be taken into account while designing a performance measure. The framework ensures that measures are clearly defined and based on an explicitly defined formula and source of data. It consists of ten elements that are discussed below.

2. Purpose: If a measure has no purpose then why should it be incorporated in the PMS, hence the rationale underlying the measure has to be specified (Parmenter, 2009).

3. Relate to: The measure should relate to a business objective and this objective should therefore be identified.

4. Target: The level of performance that the organization wants to achieve to satisfy these objectives is dependent on how good its competitors are. Therefore, an explicit target, which specifies the level of performance that need to be achieved and a time scale for achieving it needs to be recorded. For example, a target could be achieving a 98% reliability of the IT systems.

Figure 3: The performance measure record sheet

5. Formula: This is one of the most challenging elements to specify because the formula defines how the performance is measured and affects how people behave.

6. Frequency: the frequency defines how often the performance is measured and is reported. Parmenter (2009) state that a performance measure should be measured daily, weekly or at the very last monthly. However, a performance measure may only need to be operational for a couple of weeks within a year. Likewise, it is irrelevant to be reporting customer satisfaction results, on a monthly basis when the last survey was performed six months ago.

7. Who measures: Defines the person who is in charge of collecting and reporting the data. 8. Source of data: The source of raw- data should be specified. The importance of this question

time. Moreover Garengo et al (2005) stat that it is important to clearly define how the data is gathered, therefore avoiding elements who could reduce the quality of the data gathered. 9. Who acts on the data: it defines which person should react on the data.

10. What do they do: This is according to Neely et al probably the most important element contained in the framework. Not because it provides the most important information but it defines what action will be taken if the performance proves to be either acceptable or unacceptable.

This framework can be used to help an organization to design better performance measures by ensuring that they consider all of the subtle implications of the measures being proposed (Neely et al, 1997). Moreover this framework provide structure when designing the performance measures which can contribute to the clarity and simplicity which should be kept in mind when developing measures (Garengo et al, 2005). This is also acknowledged by Cocca and Alberti (2009) who state that small organizations suffer from a lack of resources the performance measures should therefore be simple and easy collectable. Similarly also the procedures for data collection should be well defined and resource effective.

3.3.1 Literature review insights of the performance measures

This paragraph elaborated the structure of designing a performance measure, which include the performance record sheet of Neely et al (1997). This aspect is, according to literature, expected to be relevant for empirical research and an explanation is what theoretically could be expected at a small organization who is designing a PMS. The aspect described below will be used for the analysis and during the analysis these factors are empirically investigated based on the case study.

3.4 Theoretical framework

The theory elaborated in the previous paragraphs revealed that multiple aspects theoretically have an influence on the design of a PMS within a small organization. These aspects (that were discussed at the end of every sub research question) are presented in the framework below (table 2) and this framework will be used to provide structure during the analysis.

Aspect: Expectation: Influences:

Contingency factors

a Information System

A Limited database and analyze functionality is present to provide the PMS with information needed for performance measurement

The Design of a performance measurement

system

b Strategy

The strategy will be explicitly formulated due to the introduction of a PMS and a poor alignment is expected between the strategy and the PMS due to lack of planning.

c Organizational culture and

management style

Strong management commitment is present and the organizational culture promote changes (i.e. the introduction of a PMS)

d External environment

Due to the turbulent environment, the PMS should be designed in such a way that it is flexible and maintainable. Changes in the environment are quickly reflected in the PMS

PMS dimensions

e Strategy alignment

The strategy will be explicitly formulated due to the introduction of a PMS and a poor alignment is expected between the strategy and the PMS due to lack of planning.

f Balance

A shift is expected from defining mainly financial measures towards the

integration of financial and non-financial measures

g Depth & Breadth

The PMS their focus should be on scope and less on detailed information. It should integrate all the activities of the organization.

h Causal relationship

The causal relationship is difficult to identify but the use of a PMS could promote the establishment of this relationship. The selected model can promote the relationship between the measures but also between the measures and the strategy

PM aspects i Structure

Every measure should be defined by a structured process. A framework (Neely et al, 1997) is developed but no expectations can be made due to the fact that this framework is not particular designed for small organizations

4. Research methodology

Now the in-depth literature study is completed and the theoretical framework is developed, it is time to elaborate the research methodology. This chapter defines how the empirical research is performed and what type of method is used to conduct this research.

The research conducted in this thesis distinguishes itself as a qualitative research due to the focuses on how (process) and why (meaning) things happen as they do whereas the focus is to understand and interpret the occurring phenomena (Cooper and Schindler, 2008). The purpose of a qualitative research is to provide an in-depth understanding of the situation and therefore the researcher is often involved in the situation as a participant or acts as a catalyst (Cooper and Schindler, 2008; Yin, 2003). For this thesis a qualitative research is conducted because an in depth understanding is needed in order to understand and interpret how this small organization designed a PMS.

When conducting research multiple research methods are available, one of these methods is a case study. This method will be used in this thesis and is based on data from a single case study.

Yin (2003) defines a case study research method as:

‘An empirical inquiry that investigates a contemporary phenomenon within its real-life context; when the boundaries between phenomenon and context are not clearly evident; and in which multiple sources of evidence are used’.

It is an ideal methodology when a holistic in-depth investigation is needed. Therefore, instead of using large samples and following a rigid protocol to examine a limited number of variables a case study involves an in-depth examination of a single event (Yin, 2003). Based on the main research question and the focus of this thesis (one organization that operates in a real-life context) a single case study is used, to get an in-depth investigation how this small organization should design their PMS. Another important component for designing a case study is according to Yin (2003) the unit of analysis. The unit of analysis that will be investigated are the seven aspects described in the framework (paragraph 3.4, aspect a. until i.). The object of analysis is the small organization where the research is conducted.

can be expected. Yin (2003) state that when performing a single case study it lowers the external validity and construct validity due to the use of less sources compared to a multiple case study or survey. In order to overcome this problem each aspect elaborates how this research will deal with the single case study restrictions.

• Construct validity: In order to overcome a limited amount of sources for obtaining the data, this case study uses multiple sources of evidence. First of all a desk research is performed, which allows the researcher to perform a thorough investigation of the available documents. Besides the desk research five interviews are taken in order to complement the case study with different data. Moreover every person who is interviewed did review and agreed with the outcome of the interviews and the draft case study report.

• External validity: In order to maintain the generalizability, theory is integrated in the case study design. Moreover while acquiring the data and writing the case study the theoretical framework will act as a guideline to provide structure and support during the case study design.

• Reliability: When designing the case study the researcher should take into account that he or she must be able to demonstrate that reproducing this study would deliver the same results. In order to secure this, a desk research should be performed, this implies that the researcher should study the documents that are described in paragraph 4.1. After studying them in-depth, the researcher should interview the employees (see paragraph 4.1) at the organization and follow the interview protocol described at appendix 1. When performing these methods the reliability should be ensured.

4.1 Data sources

Besides the use of articles, that where investigated during the development of the theoretical framework other sources can be used to receive the information needed. Two types of sources were used to get the information needed for the case study.

Secondly, five members of the organization were interviewed. The interviews were held with the CEO, CFO and three managers (i.e. sales person, customer contact manager and an IT consultant). Every member was interviewed twice. At the first interview, questions 1 until 12 were asked and in the second interview, questions 13 until 15 were discussed. This separation was made because when comparing the last aspect of the framework i.e. structure, it is important that the measures had to be designed first. The complete overview of the interview questions are elaborated in appendix 1. This interview protocol is not applied strictly; there is room to go into more detail if necessary. The interviews were not tape-recorded however written notes were made. After each interview, the written notes were transcribed and each person approved the results. Each interview lasted around 60 minutes.

5. Case study

This chapter will present the design of a performance measurement system at organization AP who is situated in the financial service sector. It elaborates how the design process is empirically realized based on the aspects depicted in the framework presented in paragraph 3.4. This framework will therefore act as guideline for this case study. Every aspect will be described, but the comparison between the case study and the theoretical results is done in the next chapter. Before these aspects are elaborated an introduction of the organization is presented and the current situation concerning management information is discussed.

5.1 Introduction of AP

will be received by AP. The presented figure 4 below provides an overview of the process of how AP works.

Figure 4: The process of the AP method

AP has grown rapidly since she became independent and with this fast expansion, several departments are designed in order to provide structure to the organization. AP consists of a sale-, finance-, customer contact-, business information management- and a risk-department. Every department is now shortly elaborated.

Sales department: The sales department consist of seven employees who are responsible for acquiring new web shops that are interested in the after payment method of AP. Besides the acquisition, promoting the brand of AP forms also a core task of this department.

Finance department: The finance department consists of three employees who are responsible for the controlling aspect, such as the income statements, budgets, forecasts and it monitors the cash flow of the organization. Besides the controlling aspect, there is also the financial administration and the operational aspect (when the computer system cannot match the invoices with the specific customer). Customer contact: This department is responsible to handle and solve all kind of questions from customers or web shop owners (e.g. questions about: their invoice, administration costs, or products etc). The employees also call customers with regard to fraud management. This department consists of eight people

Business information management (BIM): This department consists of five IT consultants responsible for assisting the web-shop owner with the implementation of the payment method. Although AP does not implement this method themselves, they do check if the payment method works successfully. They also are responsible for the operational aspect to ensure that the process, needed to handle every transaction, runs optimal.

in the database to see whether this customer can be accepted. One person is in charge and is responsible for designing the system in such a way that the accepted customers is as high as possible taken into account that fraud percentage has to be as low as possible.

5.2 The current use of performance information

The finance department is responsible for generating the management information that elaborates how the organization is performing. Before the PMS design process started, the current available performance information was investigated. This investigation contained an overview of the reports available, how the information was measured, how often it was measured and whether a performance system was in place.

Carrying out the desk research revealed that top management got their information from two sources: on the one hand the ad-hoc generated reports (i.e. the CEO asks a IT consultant to generate a report from the IT system to get the information he needs), on the other hand the standardized reports. The standardized reports each consist of a different structure and layout and are generated separately from each other. In addition, the data needed for every report was obtained from different IT systems. The available reports provide top management with the results of the previous day or the previous week. The current reports that are available to top management are:

• A weekly report about the turnover, based on the transactions of all the BP.

• A weekly report about the recovery rate of the outstanding transactions. This rate is based on the amount of orders (numbers) but also on the payment collection (euro’s).

• A daily report about the total amount of orders placed by customers and the percentage of customers who are accepted and therefore can use the AP method.

• A weekly report about the new BP and the amount of web shops that are so called ‘live’ (web-shops who are using the AP payment method).

The report that presents how many customers are accepted and the daily report about the amount of orders indicate if the operational aspect of the organization is still functioning well. Besides this information, the ‘turnover report’ is used to check whether AP is in line with their pre-defined target. Moreover, this report is also used for developing the turnover forecast.

certain level. The last report is the weekly report about the new BP and the ‘live’ web-shops are used for indication to show towards the entire organization how AP is growing.

These reports were generated stand alone and the outcome of each report were not integrated into one structured overview

According to the CEO, these reports still have an informing purpose and the CEO revealed that his decisions are still based on intuitive and feeling instead of the current information available. The CEO states that the current information needed for top management or external stakeholders is not enough or is unreliable. AP is currently starting to expand their markets to international countries such as Germany and Belgium and need this information to convince their external partners in order to establish a partnership. In addition, AP frequently has meetings with their bank and AP wants to present their performance information in order to acquire better terms and conditions. According to the CFO, these external stakeholders can only be convinced if there is enough reliable information about the performance of AP.

5.3 The design of a performance measurement system

AP decided to design a complete new measurement system whereas the current available reports did not deliver the satisfactory result to control the organization. Although the current reports were available and could be integrated in the new PMS, Top management (consisting of the CEO and CFO) deliberately did not do this because AP wanted to develop a PMS from scratch. This enables top management to review their business and decide which performance information is essential for AP. Nevertheless, these reports could be used later on if needed. Top management decided that the design of a PMS should be done from a top down approach. The CEO wants continuously be involved in the performance measure selection process in order to decide which performance measures are essential for him in order to make his management decisions.

measures that revealed to be important or very important were selected and written down. One of these outcomes during these sessions revealed importance of the ‘revenue of the business partners’ measure. This indicator showed the turnover based on the taken-over transactions. As earlier elaborated AP get from every successful transaction a fixed and variable fee which forms a part of the revenue of AP. Top management was also very interested in the amount of orders that were placed every day, because this was an indirect signal for the revenue AP would make.

These brainstorm sessions resulted in 34 performance measures that were relevant for AP. According to top management, this amount had to be reduced to 10 or maximal 15 performance measures in order to maintain the overview but also to retain the amount of time that would be needed to provide each measure with the right data. Before making this selection, top management wanted to use a performance system were every performance measure should be integrated. Three requirements were defined by top management and were found to be essential for selecting the performance system. At first, it should be a simple system, easy to use and the information presented should be clear to everyone within the organization. Secondly, it must incorporate financial information but also operational information. The CEO stated he wanted to keep track of the performance not only from a financial perspective but also from a operational perspective. Thirdly, the strategy should be incorporated in the performance system. Top management wanted the performance measures to reflect the organizational strategy. The CEO stated that the strategy of AP elaborates on which specific aspects AP should focus on in order to grow. The information of each performance measure should represent this growth. The strategy of AP consists of four core values: growth, innovation, brand

awareness and customer satisfaction.

Based on these requirements AP selected the Balanced Scorecard (BSC) to incorporate their measures and to let this model to serve as their performance system. This model was selected because of its strategy orientation, the possibility for incorporating financial and operational measures and because of the well-known knowledge about this model that can be used for the design, and implementation process. The next step top management took was to integrate each core value of the strategy in one of the perspectives of the BSC. This means that the core value growth was placed in the financial perspective, brand awareness & customer satisfaction were placed in the customer perspective and

innovation was placed in the learning and growth perspective. In addition between each perspective a

management to present the results in a clear and simple overview. Top management therefore transformed the BSC model (appendix 2) into a visually attractive dashboard (appendix 3). This dashboard contained all the relevant performance measures and an arrow was indicating if the outcome of the performance measure was correct or action should be taken. In addition, top management could formulate their comments and could introduce action points. Top management scheduled the update and presentation of the dashboard every Monday to their employees, allowing them to see the results of last week and the action points revealing were top management would like to focus on.

Now the top management selected the performance system, the selection of the 34 performance measure could be made. During the selection process top management selected the most valuable indicators (which represent the most valuable information towards top management) and were called ‘main’ measures. The remaining measures that were relevant but not represent the most important information were called ‘sub’ measures.

The ‘main’ measures were placed in one of the four perspectives of the BSC. Each of the BSC perspective contained therefore a maximum of the three most important measures (i.e. ‘main’ measures). The result of these ‘main’ performance measures is depicted in appendix 4. Although top management now had the eleven important measures selected, the remaining measures still maintained relevant for AP. Top management decided to integrate the remaining ‘sub’ measures into department reports. Each measure was checked and directed to a report were it would represent the performance information of the corresponding department. This resulted in five department reports each consisting of 4 or 5 ‘sub’ measures.

At this stage not every measures was equipped with an explanation or a description that explains what was meant with every measure. AP used the SMART approach to describe each measure, in order to reduce the ambiguity and increase the clarity and consistency. The approach is elaborated below:

• Specific: The performance measure needs to be defined. This results in a uniform and

consistent explanation and definition of the measure, which can be understood throughout the entire organization.

the data should be selected and what the type of value the measurement should be (e.g. amount, percentage, number, etc.)

• Assignable: This element describes who is responsible for collecting and reporting the data that is needed for each performance measure.

• Realistic: This element defines what target should be set for each objective.

• Time: It defines how often the objective should be measured.

Every performance measure was subjected to this method and the outcome for the ‘main’ measures is presented in appendix 4. However, AP did not use every aspect while defining each measure. During the interview with top management, they revealed not use the “Realistic” aspect; the focus lay on choosing and correctly formulating the performance measures instead of providing each measure with a target. According to the CFO targets could later on be introduced when each measure was foreseen with data for a certain period, whereas a trend could then be discovered.

During the process of providing each measure with a description, each measure had to be provided with data (i.e. making measures ‘measurable’). Top management elaborated for each measure: which data was needed and from which IT system the data would be gathered. The IT systems of AP consisted of large databases with a large amount of information that can be used to feed each measure with data. These databases consisted of information about every transaction placed by a customer who ordered a product on a web-shop (data such as the price of the order, at which web-shop the order was placed etc). Moreover, another database provided information about all the new business partners but also presented details about the current business partners. However, this data had to be extracted from each database and top management therefore involved the IT consultant in the design process. For each measure, a query was formulated and written down in code in order to extract the data from the specific database. As a result several meetings between top management and the IT consultant took place were the specific parameters, necessary to get the correct data out of the database, were discussed. This raw data was extracted from the database, placed in an excel sheet by the IT consultant and was subsequently edited by the CFO to transform the raw data into a final value. This value was then placed into another excel sheet (dashboard, see appendix 3) were all the measures were presented.

measure with information. From the 11 performance measures, 8 could be measured by extracting the data from the different databases. For the remaining measures, “customer satisfaction”, “Innovation projects” and “continuous improvement” data did not exist. Although each measure was described by using the SMART approach, which explains which steps were necessary to get the necessary data, these steps had to be worked out.

Now the ‘main’ measures were selected, integrated in the BSC and each measure was described, the remaining step was to inform the department managers with the new designed PMS. According to top management, it was important for everyone to realize the need of such a system. The CEO explained that AP is an organization that is focused strongly on growth and increasing her brand awareness. These are according to the CEO one of the key points to focus on. He state “we are a young and dynamic

organization that focuses on growth and in order to growth in a steady and healthy way we have to continuously innovate which requires often a change in the way we work or behave”. According to the

CEO the introduction of a PMS is one of these innovations, needed to grow. During the interview with the department managers the dashboard and their measures was explained. Every manager was asked if they understood what was meant with each measure. Moreover, every manager had the opportunity to ask questions and could introduce changes if necessary. This was also done for the ‘sub’ performance measures that were defined in a department report. The interviews revealed that from the 34 performance measures 28 of them were clearly described and 6 needed improvement in their description. It also revealed that the department managers were enthusiastic about the design and future use of the PMS. In addition, each manager indicated that they were interested how AP is performing but moreover they were also interested how their own department is functioning.

6. Analysis

The previous chapters elaborated the design of a PMS from a literature perspective and from a case study perspective. The literature review resulted in seven factors that revealed, from a theoretical perspective, to have an influence on the design of a PMS. These factors will act as a guideline to analyze the case study. This chapter is therefore separated into three sections. First, the designed PMS will be analyzed based on the influence of the contingency factors. Then the selected system, that integrated the performance measures, will be analyzed. The third section will analyze how AP designed their performance measures. The purpose of this chapter is to analyze the presence of each factor, how AP coped with these factors and if present what the influence of these factors are on the design of a PMS. Subsequently an analysis is performed to compare the findings regarding the outcome of the case study with the expectations of the literature.

6.1 Contingency factors

The first factor that is analyzed is the presence of the information system within AP. Due to AP their core process the IT consultants developed (in corporation with another external party) at the start of the organization three large IT systems (consisting of more than twenty databases) in order process every transaction. Each transaction contains much detailed information (e.g. transaction ID, name, address, bank number, etc.) that had to be stored in their system. Due to the handling of the large amount of transactions the IS is capable of storing large amounts of data. The advantage of this large system is that it can be used not only to control their core process but also for performance measurement. As a result, this system is used to provide 8 of the 11 measures with data selected from the specific databases. This revealed the importance of the information system, because 72% of all the performance measures relay on the information system to provide each measure with an outcome. Only three performance measures used another source to provide the remaining measures with data.

large amounts of data, many steps had to be executed correctly and precisely in order deliver the final performance value.

So the findings of the case study reveal that a large database is available with operational information, but there is no functionality incorporated in order to analyze the data automatically. The theory on the other hand expects that a limited database and analyze functionality is present to provide the PMS with information needed for performance measurement. When analyzing the case study findings and the theoretical expectations it reveals that, the outcome of the case study differ from the theory expectations regarding the size of the database. AP has a large database available for performance measurement, despite the literature expectation of the presence of a limited database. However when looking at the analyze functionality, signs of similarities can be discovered. The case study revealed that AP analyzes their data manually and no analyze functionality is build in their IS. The literature states that small organizations often face a lack of the incorporation of an analyze tool within their IS.

Another analyzed aspect is the organizational culture and management style. During the development of the PMS, top management was responsible for designing the PMS. Top management wanted more information about the performance of their organization and were therefore personally motivated to design the PMS. Their commitment to let the design process succeed was therefore well present. This expressed itself by the meetings that took place every two weeks to brainstorm about the performance measures, but also to discuss progress of the design. In addition, employees were involved during this design process in order to provide top management with advice regarding the need of data from the IS. Top management made time available for the IT consultant to work on the data extraction, which indicated the importance of the PMS development. Another interesting insight is that during the explanation from top management towards their department managers the PMS was enthusiastically received. During a meeting with top management and the department managers were top management introduced the PMS it revealed that the new initiative was quickly accepted by the department managers. An indication for this fast acceptance came from the department managers ‘need’ for more performance information. The department managers are committed to AP and are willing to invest time and energy to let AP grow. This revealed why they were interested in AP their performance.

with the literature expectations the case study revealed that management commitment is well present within AP. This commitment is also expected by the literature. The case study analysis also revealed that the PMS was quickly accepted. However, this does not directly support the literature regarding the organizational culture. The quick acceptance does not directly indicate that within AP an organizational culture is present which embraces changes. On the other hand, the case study revealed that department managers are committed to AP and are interested in a performance overview. Which indicate a culture that encourages new ideas or developments.

The last contingency factor is integration of the external environment into the design of a PMS. When analyzing this aspect it revealed that top management did not elaborate nor made arrangements when or how they want to discuss the functioning of the PMS. Possible changes due to entering new markets or a change in business process, which could influence the measurements or the usefulness of the PMS, were not incorporated by top management in the design process. Top management did not decide when their measures or their performance system should be subjected to a reviewing process. A possible explanation could come from the fact that the PMS was recently developed and top management decided to test it first before it became final. This could indicate why no steps were taken or described to define when the PMS should be reviewed because it was still in a test phase.

So the findings revealed that no arrangements were made when and how the PMS should be subjected to a review process. The literature on the other hand expect that because of the turbulent environment the PMS should be designed in such a way that it is flexible and maintainable, and changes in the environment are quickly reflected in the PMS. When analyzing the findings of the case study with the theoretical expectations it revealed that the case study outcome differ from the literature expectations regarding the flexibility and maintainability. It became clear that top management not made arrangements when or how the PMS should be reviewed.

6.2 The performance measurement system

AP selected the BSC model to design their performance system. This sub chapter will analyze the designed PMS on the following dimensions: strategy alignment & causal relationship, Balance and Depth

& Breadth. Starting with the first dimension.

business plan nor presented at AP their website. Nevertheless, the four values were presented in presentations for new Business Partners. When looking at the PMS of AP (appendix 2) it revealed that each core value is integrated into one of the four perspectives of the BSC. During the development of the performance system similarities were identified between the perspectives of the BSC and the core values of AP (for example: Customer satisfaction, one of the core values revealed to have a high similarity with the customer perspective of the BSC). Therefore, top management aligned and integrated each core value into one of the perspectives. Each measure that is placed in one of the four perspectives therefore contributes to a specific part (i.e. core value) of AP their strategy. Another interesting insight is that after the PMS was designed, the strategy was not revised or reconsidered. The four core values remained to act as the strategy of AP. Although the CEO provided an explanation what was meant with each core value, (which was not written down anywhere) there were no intentions to extend the strategy. A main reason for this could come from the fact that it provides the CEO with the opportunity to explain his own point of view regarding the meaning of each value.

So the findings of the case study revealed that a strategy is present within AP and is integrated into the four perspectives of the BSC. Nevertheless, the strategy of AP remained unchanged despite the introduction of a PMS. The theory on the other hand expects that the strategy will be explicitly formulated due to the introduction of a PMS and a poor alignment is expected between the strategy and the PMS due to planning. When analyzing the case study findings and the theoretical expectations the result differ from the literature expectations regarding the strategy alignment. Moreover, the case study findings also show that the strategy of AP remained unchanged. This result also differs from the literature expectations.

When analyzing the cause and effect relationship dimension, appendix 2 show that an explanation between each BSC perspective (and due to the strategy integration, therefore also between each core value) is provided. When analyzing this explanation it revealed that it explains how the result of the measures from one perspective influences the outcome of the other measures (appendix 2).

BSC perspective. No explicit explanation is given how one measure influences another measure. This is not described in appendix 2. These findings support the literature regarding the difficultness to identify such a relationship.

Besides the strategy & causal relationship dimension, the incorporation of financial and non-financial measures in the PMS is also analyzed. When looking at the designed PMS, top management used both type of measures in their performance system. The decision to use both type of measures is stimulated since top management decided to use the BSC as their main performance system. When analyzing AP their core process focusing solely on financial indicators would not deliver the desired results. AP has a time delay (an average of 4 to 6 weeks) in their core process due the release of the transaction (sending an invoice to an customer) and the incoming payment of the transaction. Therefore, the outcome of the revenue measure for example delivers a result that is historically oriented. By integrating non-financial measures such as the ‘amount of orders received’ provides more accurate information on how AP is currently performing. Moreover, top management also integrated innovation measures in the learning and growth perspective indicating what they want to achieve in the future. This combination of measures reveals results that reflect the performance of the past-, today’s- but also future-performance.

Therefore, the findings of the case study reveal that the PMS incorporate financial but also non-financial measures. The literature on the other hand state that a shift is expected from defining mainly financial measures towards the integration of financial and non-financial measures. When analyzing the case study findings and the theoretical expectations it revealed that the designed PMS integrates both type of measures. These findings show signs of supportiveness towards the literature.