The judgmental effect of order and common versus
unique information in performance evaluation.
At
Master Thesis Tim Keizer University of Groningen Faculty of Economics and Business
MSc Business Administration
Specialization: Organizational and Management Control (O&MC)
Tim Keizer
Address: De leeuw van vlaanderenstraat 2e
Postal code: 1061 CS
City: Amsterdam
Mobile: +31613088103
S2043572
May 2013
Supervisor: Prof. P.M.G. van Veen-Dirks
Co-supervisor: Dr. M.P. van der Steen
2
The judgmental effect of order and common versus
unique information in performance evaluation.
At
Abstract
This paper investigates the influence of the order of delivering performance information and the measurement types used during a performance evaluation process (at KPN). It presents evidence regarding a specific issue which is barely researched in the past, due to a combination of previously mentioned effects. Three experiments are conducted to test the influence of order and measurement type, overall 102 respondents participated in five experimental groups. All experiments are carried out among financial employees at KPN in The Hague. Participants in the first two experiments received different sets of performance information containing a certain order or measurement type. Results indicate the presence of primacy (information received first outweighs information received thereafter in overall performance judgment). In addition, the type effect (difference in weight between common and unique measures) is not found however respondents,
did label common measures as important more often than unique measures. Last finding, resulting from experiment 3, implies that a simultaneous delivery of all measures gives a significantly divergent performance rating compared to the current phased delivery used at KPN. Evaluators receiving information in a phased way rate segment’s performance significantly higher (7.65%) compared to evaluators receiving information all at once.
Key words
Order effect, Common and unique measures, Performance evaluation.
Supervisor
Prof. P.M.G. van Veen-Dirks
Co-supervisor
3
Preface
With this report, my study Master of Science in Business Administration at the University of Groningen comes to an end. After graduating from the Hanzehogeschool Groningen in 2010, I started a premaster to end up in the Master Organizational & Management Control.
My internship, preceding this thesis, at KPN Telecom in The Hague has been the inspiration for this research. At the beginning of October 2012, I had a first meeting at KPN Corporate Control, and we agreed on a research regarding performance evaluation. My master thesis was conducted in the last six months and its goal was to investigate whether a simultaneous versus a phased reporting of all measures during a performance evaluation process influences overall performance rating (within KPN)? Hopefully, this research can be used as an inspiration for both KPN and potential future research to discover new findings and to become more efficient in this field.
Most importantly, I would like to thank several people for their support during the completion of my master thesis. First of all I would like to thank my first supervisor Prof. P.M.G. van Veen-Dirks for our cooperation and the useful feedback she provided me with. In addition, I would like to thank my second supervisor Dr. M.P. van der Steen as well for co-reading my thesis. Special thanks to the people at KPN Corporate Control and Reporting, the resources they made available, and, of course, I appreciate their cooperation and kindness during my internship. Furthermore, I would like to use the opportunity to thank my family and friends who supported me during my study.
By completing this thesis my student career, which I enjoyed a lot, has come to an end. Writing my master thesis in combination with my internship at KPN Telecom has been a very valuable experience. Nevertheless, I am glad it is finished and I am really satisfied with the results. Now I am looking forward to starting my career.
Hopefully you will enjoy reading my master thesis.
Yours sincerely,
Tim Keizer
4
Table of contents
Introduction ... 5
KPN ... 7
History... 7
Performance management Process ... 8
Corporate strategy ... 8
Monthly Reviews ... 8
Internal research and Top 10 KPI’s ... 9
Theoretical framework and hypothesis development ... 11
The order effect ... 11
The type effect ... 15
Common and unique measures ... 15
Phased versus simultaneous delivery ... 18
Methodology... 20
Recruitment process and participants ... 21
Experiments ... 22 Results ... 25 Data cleaning ... 25 Results experiment 1-3 ... 25 Experiment 1 ... 25 Experiment 2 ... 26 Experiment 3 ... 26 Supplementary analyses ... 27
Discussion and conclusions ... 30
Limitations and further research ... 32
References ... 34
Appendices ... 37
Appendix 1: Organizational chart KPN ... 37
Appendix 2: Experiment 1-3 ... 38
Appendix 3: SPSS output Experiment 1-3 ... 62
5
Introduction
Performance management is the process of linking the overall business objectives of an organization with departmental, team and individual objectives. It plays a key role in translating a corporate strategy into desired behaviours and results. Performance management is widely recognized as a mechanism whereby business performance can be enhanced by developing and implementing a balanced set of measures. When there are shortfalls actions can be taken if necessary. That balanced set can be established by selecting different type of measures, like financial or non-financial and common or unique. Performance evaluation has interested researchers for decades due to this crucial role.
Prior studies (Kaplan and Norton, 1992; Lingle and Schiemann, 1996; Ittner and Larcker, 1998; Merchant and Van der Stede, 2007; Cardinaels & van Veen, 2010) focused on financials and non-financials during performance evaluation and showed a tendency to focus more on financials. More recent studies (Lipe & Salterio, 2000;2002; Banker et al., 2004; Dilla & Steinbart, 2005; Gagne et al., 2006) focused on the role of measures that are used companywide (common) and those that are only applicable to a certain division (unique). Those studies showed evaluators tend to focus (highly) more on common measures during performance evaluation. Kaplan & Norton (1996) state that unique measures mostly have a non-financial basis and argue non-financials deserve a more decisive role, since they are leading, and are thus better indicators of future performance.
What could be the reason an evaluator has a certain focus? It could be caused by previous economic success. Since companies benefited enormously from the rising economy, little attention has been paid to underlying factors that could change future performance. Because evaluators are convinced certain measures are the best indicators of overall performance.
From a psychological point of view, it is probable that people are (unconsciously) influenced, and therefore subjectivity arises. Research (Tetlock, 1983; Holstein, 1985; Pennington and Hastie, 1986; Hogarth and Einhorn, 1992; Bresnick, 1993; Jonas et al., 2001; Bond et al., 2007) showed that when people’s decisions are influenced it could cause biased and colored conclusions. At the same time, we do not recognize that we are indeed influenced by those factors. One of those influences is the order effect; cues are interpreted differently if they are received in a certain order. Thus, if focus during performance evaluation is on a certain measurement type and at the same time people are (unconsciously) influenced by order, overall judgment could be influenced. Especially when a certain combination arises of aforementioned aspects. It is in a company’s best interest to solve this. Could this be done by changing the design of the performance evaluation?
6
to report non-financials earlier in the closing process, as early as possible. Currently, KPN reports their financial and non-financial measures at widely separated times. Reported financials are used for all segments and are therefore common, while their non-financials are usually specifically selected for a certain segment, which makes them unique. Before implementing the idea of a simultaneous reporting of all measures, it is crucial to study if this change will influence performance evaluation.
Therefore, this study focuses on order effect and type of measures used in performance evaluation process. An experiment at KPN is used to collect data and test whether those aspects influence overall performance ratings. The research question of this study is:
7
KPN
KPN is the leading telecommunications and IT service provider in the Netherlands, offering wire line and wireless telephony, internet and TV to consumers. KPN offers business customers complete telecommunications and IT solutions. KPN Corporate Market (previously known as Getronics) offers global IT services and is market leader in the Benelux in the area of infrastructure and network related IT solutions. In Germany and Belgium, KPN pursues a multi-brand strategy in its mobile operations and holds number third market position. KPN provides wholesale network services to third parties and operates an efficient IP-based infrastructure on global scale in international wholesale. An organizational chart of KPN can be found in appendix 1.
At December 31st, 2012, KPN served 45.1 million customers, of which 36.7 million were in wireless services, 3.9 million in wire line voice, 2.7 million in broadband Internet and 1.8 million in TV. With 17.491 FTEs in the Netherlands (26.156 FTEs for the whole group), KPN reported full-year revenues of EUR 12.7bn and an EBITDA of EUR 4.5bn.1
History
The history of KPN can be traced back to 1852 when the government constructed telegraph lines which it intended to operate itself. However, in 1998 cooperation between the most important operating companies, PTT Post and PTT Telecom, came to an end, and KPN was privatized; they demerged and the monopolistic position that KPN had acquired over the years crumbled.
For the first time in history KPN found itself in dire straits. In 2001, due to (too) large investments the company almost went bankrupt. To prevent bankruptcy, KPN’s philosophy was clear and to the end point: it needed a period of calm, and, thus, money. Credit facilities at several banks were arranged, and cash flow and margin needed to go up. Time had come for a complete reorganization processes that had been initiated earlier. In the decade that followed, measures like EBITDA, revenue and cash flow became the most important performance measures of KPN; this lasted for ten years and enabled KPN to conquer the threat of going bankrupt and grew to a solid Dutch company.
Nevertheless, in the beginning of 2011, KPN announced a first profit warning which caused a lot of commotion in telecom business. According to KPN, the main reason for this profit warning was a sudden move of customers from messaging (sms) to data usage (whatsapp and ping) caused by the emergence of smartphones. KPN’s revenues decreased enormously. For some reason, KPN did not see it coming. Shareholders questioned this statement because emergence of smartphones did not seem to be some new trend. Apparently, KPN was not able to adapt to new circumstances quickly enough.
In sum, over the last decades the telecom industry grew and developed enormously. As a consequence, the telecom industry became extremely dynamic and profitable, which in turn attracted lots of competitors. This
8
created the urgency to focus on changing circumstances. In order to conquer this dynamic environment, telecom companies like KPN need to be flexible and should be able to adapt quickly. For example, early response to the shift from fixed telephony to mobile or from messaging (sms) to data usage (whatsapp and ping). The role of performance evaluation is crucial in achieving this.
Performance management Process
Performance measurement is the process by which businesses establish criteria for determining the quality of their activities, based on organizational goals. It should involve creating a simple, but effective, system for determining whether organizations meet objectives and outcomes are in line with what was intended. They also help to communicate expectations, provide feedback and motivate employees through performance based rewards. Performance management must be part of a system, which reviews performance, decides actions and changes the way to operate. To get a clear picture of a company’s health, it is advisable to use as many different measures as possible together. In order to be sure KPN is on their strategically right track, the monthly review is an important tool within their performance management process.
Corporate strategy
A corporate strategy is composed every three to five years by the Board of Management (BoM) and the Group Executive Committee (ExCo). On 10 May 2011, KPN presented its present strategy, the successor to ‘Back to growth’. From now until 2015, KPN aims to strengthen, simplify and grow their businesses. This means that everything is oriented on the improvement of quality, service and technology so as to become the best service provider and to strengthen its market position in the Netherlands. International focus will be on further growth in revenues and profitability.2
Monthly Reviews
Every year, each segment is evaluated monthly in order to track their performance. Each segment reports both their performance measures to Corporate Control (CC) and they present them to the BoM for an analysis. The performance measures that are reported first are the same for each individual segment (common): revenues, EBITDA, CAPEX et cetera. However, the measures that are reported subsequently are unique to a segment’s business; they are selected in consultation with a particular segment in order to make sure it covers the segment’s strategy as much as possible. During these monthly reviews, progress is monitored and action is taken where needed. Flash 1 and 2 contain only common and are delivered on working day five and seven, respectively. Flash 3 forms the unique part of the monthly review and is reported on working day eight/nine. The review is the final part of the internal monthly closing process, which is set up as follows:
9 Business
day
Activities
1-4 Segments finalize their monthly figures (revenue, EBITDA, EBIT, OPEX) and submit them to CC.
5 Flash 1: Monthly report to BoM and ExCo on revenue, EBITDA, EBIT and OPEX. Results are compared to the same period last year and to the Rolling Forecast and/or Strategic Plan.
6-7 Preparation of monthly review questions. By means of these review questions, CC makes a suggestion to the BoM as to the issue on which the discussion during reviews should focus. They are aimed at supporting the BoM in controlling the segments
7 Flash 2: Monthly report to BoM and ExCo on Free cash flow, RACF, CAPEX, debt & equity and FTE. Results are compared to the same period last year and to the Rolling Forecast and/or Strategic Plan.
Submission of Key Performance Indicator (KPI) results to CC by the segments.
8-9 Flash 3: Monthly report to BoM and ExCo on Top KPIs (10-11 items). A review often takes place the same or next day, this report is short, and aimed at providing some final focal points for the BoM during the reviews. Submission of the management letters by the segments to the BoM.
10 Monthly reviews, in which the management letter of the segment concerned is discussed, in combination with the review questions and reports from CC.
Table 1: Internal monthly closing process
Common measures are reported to the BoM and ExCo by CC in the beginning of the internal monthly closing process, while the unique part is issued to the BoM only briefly before the actual reviews with the segment take place. Therefore, board members and other evaluators do not always find time to read it before the review starts.
Internal research and Top 10 KPI’s
As mentioned earlier, KPN did internal research (de Vries, 2012). Their study found strong support when financial and non-financial information (not all actually reported within KPN) are presented, KPN employees tend to add more weight to financials in judging performance. Additionally, de Vries (2012) argues this can be reduced by providing more non-financials. Appropriate response to the upswing of data usage could have been missed because of KPN’s tendency to focus more on financial information. De Vries (2012) came to this conclusion by doing an experiment in which she asked KPN employees with various functions to take the role of an evaluator. Forty-six participated evaluated eight virtual business divisions on several measures, which were not all used by KPN, leading to a total of 368 usable responses.
10
reported in flash 1 and 2. Main recommendation of de Vries (2012) is to focus more on non-financials when judging and managing performance in order to be more successful in such a rapid-moving industry and to report them earlier in the closing process. She claims this can be done by including three times more (twenty-five to seventy percent) non-financials than financials. However, this study argues that it is a rough estimate while only that particular distribution is tested and others are not. Currently, KPN uses a distribution of (thirty to seventy percent), which seems to serve the recommendation to put more weight on non-financials. Besides, KPN currently reports measures at widely separated times. Given the current distribution of performance measures and the recommendation of reporting measures in flash 3 earlier, it is crucial to study whether a fundamental change such as reporting all measures simultaneously significantly influence overall performance evaluation.
11
Theoretical framework and hypothesis development
Most people believe they are able to make objective decisions and strongly believe they are not influenced by distracting factors. However, research showed people’s decisions are influenced. Therefore, a plausible change exists we draw biased and subjective conclusions. At the same time, we do not recognize we are indeed influenced by those factors. The order in which people receive information seems, for example, to influence final judgments, called the order effect. In addition, it apparently depends what kind of informational items are received. Research showed that people are influenced by types of information. In general, common items are weighed more heavily in overall performance evaluation compared to unique items.
To study whether those two factors indeed influence (overall performance evaluation) decisions, this study focuses on order and use of common versus unique measures in a performance evaluation process. In order to find a solution to the research question, the theoretical foundations of this study reflect three bodies of literature. To theorize the first part (H1), this study draws on literature relating to the order effect and its potential influence on overall decision making. Secondly, (H2) focuses on the substantive part of performance measurement, namely, type of measure. Lastly, (H3) focus is more practically oriented, current phased delivery process at KPN is compared to a simultaneous delivery in order to test whether differences in rating occur. This study is theorized by drawing on literature and linking psychological influences on the overall judgment made in performance measurement processes.
The order effect
Imagine that a customer is looking for a new mobile phone. While walking through a shopping mall he sees a mobile phone of a brand, unknown to him. He approaches the seller to gather information about the product quality. The seller tells him the brand is known for its amazing warranty policy. After receiving some general information, the seller tells the consumer that the brand has built a moderate reputation over the last years. Now imagine, in the customer’s perception brand reputation is a more important signal for product quality than a brand’s warranty policy.
Whether or not the order in which the customer receives product information is of chief importance to the ultimate decision he or she makes, is a very interesting question. Would the customer make a different decision if the seller first told him about the brand’s reputation and then about the warranty policy? Intuitively, we would say no. Is order relevant, or does it not have any significant influence on our final judgment? Thus, the question is, how do people update beliefs over time? A critical feature of belief updating is its sequential nature. As stated by Anderson (1981):
In everyday life, information integration is a sequential process. Information is received a piece at a time and integrated into a continuously evolving impression. Each such impression, be it of a theoretical issue, another person, or a social organization, grows and changes over the course of time. At any point in time, therefore, the
12
The way people deal with information is studied for decades, especially in psychology. Asch (1946) showed that when forming an opinion about another person, the information that is received the earliest directly sets up a direction. Theoretically, this direction could influence the way subsequent information is interpreted. Asch (1946) showed that a person described as “intelligent-tall-mean” is seen as a more positive person than a person that is described the other way around (“mean-tall-intelligent”). This indicates “intelligent” is seen as positive when it is mentioned first, but less when following “mean”. Thus, order seems to be relevant. Additionally, it could be that it is naturally presumed information mentioned first is most important.
The effect discussed in the above-mentioned example is known as the order effect. Information about the same document is perceived at different levels of relevance if it appears in a different position in a order. This means that pieces of information could emerge in, at least, two forms: A and B. Some evaluators form their decision after receiving information in order A-B; other evaluators receive it vice versa (B-A). An order effect occurs when opinions after A-B differ from those after B-A (Hogarth and Einhorn, 1992).
13 Table 2: Summary of order effect predictions by the belief adjustment model
The third row needs some additional information. This row concerns long series of evidence items for which the model predicts mixed results. On the one hand, Step-by-Step processing typically leads to recency. However, as more information is processed over time, the model predicts a decrement which eventually leads to a primacy effect. Thus, whereas recency may be observed in long series, this becomes less likely as the length increases. Based on existing research, Hogarth and Einhorn (1992) state that a long series is one that contains seventeen or more cues. To summarize, Hogarth and Einhorn (1992) their task analysis of order effects leads to the following conclusions: (1) response mode makes a difference in the case of Short, Simple tasks. End-of-Sequence induces primacy, Step-by-Step induces recency; (2) primacy seems to obtain when tasks are Simple but Long (this is also independent of response mode); and (3) recency is associated with more complex tasks (independent of response mode).
A study of Bresnick (1993) further elaborated on the belief adjustment model. He showed when information was presented sequentially, the order in which it was presented significantly influenced the overall judgment. In contrast, when all information was presented at once, there was no order effect.
14
The order effect can also be explained in terms of the effect of learning. As an evaluator comes across more relevant information and each item gives a better view of the situation, information need gets saturated. Subsequent information that comes thereafter will be seen as less important and adds little value. Similarly, Bond et al. (2007) suggest individuals will form an initial, tentative disposition towards the option. Subsequent information will be systematically distorted to cohere with that preliminary disposition. Additionally, Jonas et al. (2001) also focus on the fact that people prefer supportive over conflicting information after they have made a decision. They show that this preference for supporting evidence is even stronger when information is processed sequentially; this confirmation bias is due to sequential presentation and not to sequential processing of information. Jonas et al. (2001) state that this increase in confirmation bias through sequential presentation is caused by heightened commitment of the participants for their (preferred) decision. Learning effect is also somehow handled by Anderson (1981) who claimed primacy can be explained by a process of “attention decrement”, whereby people pay less attention to successive items of evidence.
Jones et al. (1972) give a potential cause of the existence of primacy effect. In line with the belief adjustment model, they state that primacy occurs due to a social judgment process in which early success or failure is assimilated to the initial expectations. Therefore, later performance seems to be similar to information received earlier, which causes a judgment to become more subjective. This implies early success or failure has a greater influence on an individual’s ability attribution. Similarly, observer ratings of work performance have been found to be biased towards information received earliest (Sinclair, 1988).
Feldman and Allen (1975) theorized that if a primacy effect is indeed a consequence of an assimilation process, it could be reduced, or even be eliminated by separating school lessons into two discrete units. They split up a lesson into two parts with an intervening two-day period. Their line of reasoning was that since a unit that is perceived as being truly different, it should produce a new anchor point to which subsequent behaviour is compared and assimilated. Thus, an evaluator would develop a new reference point which should take away primacy. However, their results do not support reasoning that a temporal differentiation between two parts of a lesson eliminates the primacy effect. Rating of overall performance, like ability of attribution, was not affected by a two-day time period between lesson parts. Perception of how well tutees performed was primarily due to the success of the first part, with subsequent performance having a much smaller influence. This could indicate a phased delivery of performance measures does not reduce focus of an evaluator on informational items received first.
15
especially when more than seventeen performance measures are included. The main argument is attention and, thus, sensitivity to later information fades over time, which implies evaluators seem to rely more heavily on initial information in their overall judgments. Order effects reduce decision quality (Bonner, 2008), therefore it is relevant to study the influence of order effects in performance evaluation in both theory and practice. For example, looking at primacy in court by Holstein (1985) you could reason that informational items in a performance measurement process include different and sometimes contrary signals with regards to business performance. Holstein showed when there are more interpretations, reaching a unanimous judgment becomes more difficult. During performance evaluation there could be several interpretations of a segment’s performance, since there are different people involved. Reaching an overall judgment about business performance becomes therefore less likely. Besides, Pennington and Hastie (1986) showed that it is important to present your information first to influence overall judgment. When applied to the performance evaluation process, it is plausible to assume that when measures are presented first they have a greater influence on the overall judgment of evaluators.
According to prior research in multiple contexts this particular study reasons there will be a primacy effect in overall judgment during performance evaluation of segments. Therefore, the following is hypothesized:
H1: Performance evaluators will use both sets of information measures when making performance evaluation decisions, but the first set of information is weighted more heavily in the overall performance evaluation. A primacy effect will occur.
The type effect
Several reasons exist why business performance measurement is on the firm’s agenda: the changing nature of work, increasing competition, specific improvement initiatives, national and international quality awards, changing organizational roles, the changing external demands and power of information technology (Neely, 1999). During performance evaluation, measures can be used that are common for each segment or specifically unique to a particular segment. The following discussion, leading up to the hypothesis, is based on the role of common and unique measures in performance evaluation judgments.
Common and unique measures
As said, each business unit develops their own set of measures that specifically reflect their goals and strategies. Within those set of measures across the company, some measures are used companywide and others are only applicable to a certain division. Companywide measures are known as common measures and the latter as unique ones.
16
generalizable. However, this magnitude of dominance is small and therefore cannot be interpreted unambiguously.
A fundamental article written on this subject is of Lipe and Salterio (2000), they mainly based their study on Slovic and MacPhillamy (1974). They show that when multiple divisions are assessed, valuation is based solely on measures that are common for each division. Performance on measures that are unique to one division has no effect on evaluation judgments. Their findings imply that unique measures are undervalued and disregarded during the evaluation process of divisions and/or managers. Common measures drive a manager’s evaluations. This phenomenon is called “Common Measures Bias”. Second, Kaplan & Norton (1996) note that measures that are common across units often are financial en thus lagging indicators of performance. In contrast, unique measures are more often non-financial and thus leading. This indicates that findings of Lipe and Salterio (2000) imply focus of evaluators may be more on lagging (common), rather than leading indicators (unique). Their findings are important and at the same time troubling, because unique measures lie at the basis of the scorecard concept of strategic performance evaluation. Both findings, if correct, threaten the firm’s strategic success and undermine one of the greatest benefits of scorecards. Since each business unit will have and use a scorecard that uniquely captures their strategy.
According to research of Lipe and Salterio (2000), many researchers came up with extensions. Banker et al. (2004) confirm findings so far (Lipe and Salterio, 2000; Slovic and MacPhillamy, 1974) and ascertain that evaluators rely more on common measures than on unique measures. However, they state that the preference for common measures disappears when managers have more information and a better understanding of the business unit’s strategy. When managers have detailed strategy information, they will rely more on strategically linked measures, even if they are unique, than on non-linked measures that are common. It can be assumed measures used in performance evaluation are at least strategically linked.
17
Gagne et al. (2006) also extend findings of Lipe and Salterio (2000). They imply performance evaluations are sensitive to variation in unique measures, just like variations in common measures. According to them, relatively small variations in unique measures in Lipe and Salterio’s research (2000) could be the reason why unique measures are ignored. Lipe and Salterio used a range of per cent better than target of 3,26 – 16,67. Gagne et al. (2006) wanted to find out if unique measures played a more important role in performance evaluation when results are twenty percent above or under target. They state since unique measures are division-specific, it takes more effort to interpret implications of difference in measures and therefore it could be that subjects were unwilling or unable to deal with these small differences. Second, there is another critical point in Lipe and Salterio (2000) research, namely, the relationship between actual and targeted performance of divisions. In each measure, actual results exceeded target results. Gagne et al. (2006) state results that reach their desired goal automatically leads to positive appraisals. Study of Gagne et al. (2006) is not in itself contrary with previous findings, however, they conclude that the role of unique measures is greater than argued before. Lipe and Salterio (2002) show evaluators using a BSC weigh non-financial measures (within the customer perspective) less heavily than evaluators viewing the same measures in an unformatted scorecard and they predict that this will hold for all BSC perspectives. However, this is partly disputed by Cardinaels and van Veen-Dirks (2010). They study the role of financial and non-financial measures during performance evaluation. They came up with the idea whether variation in presentation of performance measures (BSC or unformatted scorecard, this is a limited set of measures combined out of a total set of measures) affect the way evaluators weigh measures in performance evaluation. Cardinaels and van Veen-Dirks (20120) ascertain many firms use scorecards that contain only measures that are common to all business units, presentation formats therefore may play a role in performance evaluation. They theorize that people are led by financial outcomes and a BSC format may lead users to place more weight on financial performance than users of an unformatted scorecard. Summarizing their results, Cardinaels and van Veen-Dirks (2010) discovered that if performance differences between actual and target results are located in the financial category, BSC users place more weight on financial measures compared to users of an unformatted scorecard. In contrast, when those differences are located in non-financial categories, the type of scorecard does not affect performance evaluation.
Just like accountability studied in the order effect, Libby et al. (2004) find that when evaluators need to justify to their superior and provide assurance about quality, an increase in the usage of unique measures occurs. In addition, disaggregation allows superiors to utilize unique as well as common measures, thus overcoming the “Common Measures Bias”. Since this reduces an overload of information to evaluators, allowing them to make a more complete determination (Roberts et al., 2004).
18
therefore more future-oriented, leading and predictable for future performance. Since “Common Measures Bias” discounts performance evaluation quality, it is crucial for a company to have a clear thought of how to fill in their performance evaluation process.
Recent research on the usage of common and unique measure reasons evaluators will put significantly more weight on common measures than on unique measures during performance evaluation. This study reasons this weight could be strengthened by the primacy effect as hypothesized in H1 due to an enhancing effect of combining those effects (primacy and measurement type). Therefore, the following is hypothesized:
H2: The primacy effect will be strengthened if the first set of information measures contains only common measures and the second set fully consists of unique measures.
Phased versus simultaneous delivery
As seen in previous chapter, KPN currently uses a performance evaluation design in which sets of performance measures are delivered separately. The overall performance evaluation process at KPN contains 3 phases; Flash 1,2 and 3. This phased design possible triggers an order effect, presumably primacy, and since Flash 1 & 2 only contain common measures and Flash 3 only unique ones, a type effect certainly cannot be ruled out. Thus, KPN currently reports measures at widely separated times which could be affected by order and type effects. Given the current distribution of performance measures added by findings and recommendation in prior (internal) research, that emphasized the importance of a balanced focus on performance information to be successful in a rapid moving (telecom) industry, it is crucial to study whether a fundamental design change results in a significant different rating.
19
Hoffman et al. (2011) showed that when positive information received first followed by negative information results in a significant higher evaluation compared to an evaluator receiving the same information the other way around. This is in line with the theory argued on primacy effects. If common measures contain information of what happened in the past and unique are more representative for the future performance of an organization, it could be reasoned that a company performing well in the past but currently find themselves in dire straits – what could reasonably be argued about KPN – evaluate their performance more positive when receiving common measures first and unique measures thereafter. Thus, both the design itself as the content that comes through the design are of vital importance in performance judgment.
20
Methodology
The aim of this research is to explore whether the order of providing measures and measurement type in performance evaluations influences overall performance evaluation. An experiment at KPN is used to collect data and test whether those aspects have a determining role. Specifically, this study investigates whether providing common and unique performance measures simultaneously significantly influences performance measurement. This chapter deals with the research design and variables used to test hypotheses discussed before. This study aims to establish a relation between order of providing performance information and the tendency of managers to place weight on different sets or types of performance measures. In addition, this study tries to discover a difference in performance evaluation following a simultaneous or phased performance evaluation design.
Apparently, the top management of KPN still believes financials, which are mostly common measures at KPN, are (nearly) most important to the company. Therefore, if those (financial) common measures are received first and satisfy the top management of KPN, they will lessen their attention to measures presented thereafter. Partly because their information needs get saturated. Some research (Tetlock, 1983) showed that if evaluators are held accountable for their decisions there is no order effect. However, this study does not take accountability into account. According to the belief adjustment model of Hogarth and Einhorn (1992), it will be hard to make a clear prediction of which order effect will occur. Added all three flashes together, it will contain more or less seventeen measures. Seventeen is the exact number at which the belief adjustment model predicts a move from recency towards primacy. Feldman and Allen (1975) show that an interval of two days between two sets of information does not reduce primacy. During the performance evaluation of KPN a time gaps exist (see table 1). In the end, this study predicts primacy occurs in the overall judgment of performance evaluators of KPN (H1).
21
This research aims to establish a causal relationship between order and measure type usage in performance evaluation and their influence on overall performance rating. At first, conducting interviews seems most obvious. However, interviews could give a subjective result since self-reports frequently do not correspond to actual use of information (Slovic and MacPhillamy, 1972; Joyce, 1976; Lingle and Schiemann, 1996; Frigo, 2002). Most suitable research design to establish a cause-and-effect relationship between an independent and dependent variable is an experiment. Therefore, this research focuses on doing an experiment. Previous research also used an experiment to test their hypothesis, and using corresponding methodology makes it easier to put findings in the same context.
Recruitment process and participants
22 Table 3: sample frequencies
Experiments
An experiment is a classical method in research in which a variable (independent) is manipulated to see what the effect will be on another variable (dependent). In this study the dependent variable is the overall performance rating. Since this study investigates to what extent variance in overall performance rating can be explained by order and type, those can be seen as independent variables.
This study sets up three experiments. Participants are divided into two groups, in which the first (A) is an experimental group and the latter (B) a control group. Participants are presented with a case in which they are asked to take the role of a performance evaluator and rate performance of a segment of KPN. The case informs participants in a general way, but it also informs them on the particular segment to be evaluated. Several performance measures are presented to participants in a certain order, depending on the experiment. Performance measures presented to participants are sourced by the flashes (described in chapter 1), which are actually used in performance evaluation of KPN’s segments. Subsequently, according to the presented measures, participants are asked to rate the segment’s performance on a scale of 0-100 (0 meaning extremely poor and that the segment is unlikely to improve their performance and 100 meaning excellent and that the segment is exceeding expectations). Like similar studies in the past, this research will also ask participants to complete two short questionnaires. This will provide information about personal characteristics (Pre Questionnaire) and task difficulty, understandability and realism (Post Questionnaire).
23
performance measures causes a significantly different performance evaluation compared to a simultaneous delivery of both common and unique measures.
First two experiments will be set up in the same way, the third one will be slightly different. As said, there will be two groups of participants (A and B). Each group will be presented a list of eighteen performance measures of an existing KPN segment (Consumer Residential). Participants are asked to take the role of a performance evaluator of KPN and to evaluate a segment on their overall performance. To test whether the predicted effect occurs (primacy effect and type effect), the experiment is manipulated. Chosen measures create an objective reflection of segment’s performance since they are part of the Top-10 KPI’s, which is composed in consultation with the relevant segment.
This experiment simulates the way KPN currently reports their measures, however the values of the measures are manipulated. In the experimental group (group A), the first set of performance information will have a better overall performance compared to the second. Participants in the control group (group B) will receive the sets the other way around. Participants first receive a set and are asked to rate the overall performance of segment Consumer Residential, nearly immediately thereafter they receive a second set and are asked to rate the overall performance again. In experiment 1, the type of measures in both sets will be mixed (common and unique) since it is only tested whether the primacy effect occurs. To test which type of effect occurs, the types of measures in both sets will differ in experiment 2. In the second experiment, the first set will contain only common measures and the second set will consist entirely of unique measures. Experiment 3 will be set up slightly different. Participants in group A will receive both sets similarly to participants of group A in experiment 2, however, group B will receive all measures in once to test whether a simultaneous delivering influences overall rating.
This study expects there will be a significant difference between performance evaluation due to the primacy effect, type effect and design itself. This influence will be reflected in the overall performance evaluation. Evaluators of group A are expected to evaluate performance (significantly) higher than evaluators in group B, while in fact both information sets give the same view of the overall performance. An illustration of the experimental set up and expectations are redisplayed below.
24
Experiment 2 differs since set one contains only common measures and set two only unique measures. Experiment 3 differs since set for group B contains all measures together. Set up of all three experiments can be found in appendix 2.
Figure 1: Expected difference in mean number of performance rating of evaluators of group A and B
It could be that overall performance rating is also influenced by other factors, for example personal characteristics. Presence of those kind of factors decrease detection of your result, theoretically, the effect could be caused by other factors than primacy or type effect. Therefore, it is crucial to control your results for those factors, called control variables. This study selects five control variables: gender, age, segment (Consumer Residential, Consumer Mobile, Business Market or other), function (financial and business controllers or other) and working experience.
The research design can be illustrated as follows:
25
Results
This study focuses on differences in performance rating between similar groups, however, performance measures are received differently. To test whether theory reasoned previously also occurs in practice (at KPN), this study uses an ANOVA Test. The one-way analysis of variance (ANOVA) is used to determine whether there are any significant differences between the means of two or more independent (unrelated) groups. A post-hoc analysis is not performed, since there are just two groups to compare. All three experiments are analysed this way. Results can be found in table 5. SPSS outputs can be found in appendix 3.
Data cleaning
In order to report the results as clear and valid as possible, data need to be cleaned from aspects that could cause a bias. As mentioned previously, Pre and Post Questionnaires needed to be filled in that contained questions about personal characteristics, task difficulty, understandability and realism. This study cleaned data by removing respondents that did not (fully) understand the experiment since this weakens the validity of their performance rating. By data cleaning, in total six respondents are not included in the analysis (one respondent in group 1A, one in group 2A/3A, two in group 2B and 3B) Other forms of data cleaning and their findings are listed in appendix 3 together with the SPSS output.
Results experiment 1-3
Table 5 standing below shows the results of all three experiments conducted via an ANOVA Test. Analysis of the results of the experiments will be discussed separately. In addition, supplementary results are analysed.
ANOVA Test Experiment 1-3
Experiment 1 1A 1B Difference F-value
Performance Rating 76.56 (18) 68.17 (18) +12,31% 5.404**
Experiment 2 2A 2B Difference F-value
Performance Rating 79.35 (20) 77.00 (19) +3.05% 0.800
Experiment 3 3A 3B Difference F-value
Performance Rating 79.35 (20) 73.71 (21) +7.65% 3.631*
p<.10; * p<.05; ** p<.01; ***
Table 5: Outcomes experiment 1-3 (ratings can be found in appendix 3 at “ratingcleanQ”) Experiment 1
Experiment 1 highlights a more psychological part of this research, it focuses on whether performance evaluation is influenced by order (primacy). Hypothesis 1 is tested, questioning if, due to primacy, an evaluator has the propensity to evaluate the first of two sets of information, given to him at different moments in time, more highly than the latter, in the overall performance.
26
performance average (5% versus 3%), and receiving a set of measures showing an overall percentage below performance average (1% versus 3%) thereafter, rate performance statistically significantly higher (76.56) compared to an evaluator receiving it the other way around (68.17). A difference of 8.39 (76.56 – 68.17) is observed, which indicates a higher performance rating of +12.31%
The observed difference in mean performance rating given by evaluators of group A and B shown in table 5, is founded at a 5% significance level. Performance evaluators of group A evaluate performance of a segment significantly higher compared to group B, caused by the primacy effect. Results of experiment 1 show evaluators are influenced by the order in which performance measures are presented, more specifically, a primacy effect exist when it comes to evaluation of segments by KPN financials.
Experiment 2
The second experiment of this study focuses on whether performance evaluation is influenced by the type of measures and if it is possibly strengthened by primacy. Hypothesis 2 tests whether, in the overall performance evaluation, an evaluator has the tendency to weigh the first set of information more heavily than the second set of information, which are again provided at different moments in time, when the first set contains only common measures and the latter only unique measures.
The second row of table 5 shows that there was no statistically reliable difference between the mean number of performance rating given by evaluators of group A and B as determined by one-way ANOVA (F=0.800, p=.377). If evaluators first received a set of measures that are common to each segment - which are easier to interpret, showing an overall percentage above performance average (4% versus 2.63%) - and then received a set of measures that are unique to that particular segment - showing an overall percentage below performance average (1.75% versus 2.63%) - they will rate the performance higher (79.35) than the evaluators receiving it in opposite fashion (77.00). As said, this difference is not significant nor valid. Thus, it can be assumed evaluators are not influenced by the type of measures presented.3
A difference of 2.35 (79.35 – 77.00) is observed, which indicates a higher performance rating of +3.05% . However, the observed difference in mean performance rating given by evaluators of group A and B shown table 5 is not significant.
Experiment 3
The third experiment focuses on whether effects as tested in the previous two experiments can be reduced by a simultaneously delivery of both common and unique measures. As can be seen in table 5, a t test succeeded to reveal a statistically reliable difference between the mean number of performance rating given by evaluators of group A and B.
Lastly, table 5 shows there was a statistically reliable difference between the mean number of performance rating given by evaluators of group A and B as determined by one-way ANOVA (F=3.631, p=.064). Results
3
27
indicate that when evaluators first receive a set of measures showing an overall percentage above performance average (4% versus 2.63%), and who then receive a set of measures showing a percentage below the performance average (1.75% versus 2.63%), that the rate performance is statistically significantly higher (79.35) compared to evaluator receiving all performance measures at once (73.71). A difference of 5.64 (79.35 – 73.71) is observed, which indicates a higher performance rating of 7.65%. The results of experiment 3 show that the evaluators rate segment’s performance is significantly different in case they receive the performance measures separately or simultaneously.
It could be reasoned that presenting all measures at once creates a better overview of overall performance since measures can be seen in the light of results achieved on different aspects. Since it cannot be said what the exact performance rating is, it seems hard to state whether or not a simultaneous delivery of all performance measures causes a more reliable and truthful evaluation rating. However, it is proven that a phased delivery causes a significant higher (+7.65%) rating, which at least indicates that performance results are interpreted differently.
Supplementary analyses
In addition, to the main goal of this experiment, other information is analysed as well. For example, participants are asked to rank their Top 3 most important measures, indicating their preference for the way performance is presented (absolute delta or performance percentage), and the potentially observable differences between males and females, business control and financial control, or differences between segments.
Participants were asked to indicate which measures were important to them during their decision to fill in a performance rating. After each set of performance measures (set 1 and 2) respondents were asked to carefully choose three measures used most in their performance rating and rank them from one to three in importance (one meaning most important). Additionally, they were asked to do the same at the end for the entire set of information (set 1 and 2 together). Since the outcomes cannot be statistically tested, due to their nominal basis, those results are analysed qualitatively.
28
However, there is no difference between results mentioned before and results of respondents of segment Consumer Residential (see figures in appendix 4). One may wonder whether the ultimate organizational goals are kept in mind during this process of evaluation.
Given the proven presence of the primacy effect at KPN, it must be wondered whether this effect also occurs in setting up the Top 3’s. To study if this exists, it is analysed whether participants in group 1A chose other measures in their overall Top 3 compared to group 1B, the same applies for group 2A and 2B. Results show this is not the case; order does not play a role when it comes to measures selected in the overall Top 3. Measures received first are not selected (significantly) more compared to measures received thereafter. Experimental setting therefore did not influence the outcomes and conclusions derived from Top 3 data presented above.
Figure 3: # mentioned common measures in Top 3’s
Figure 4: # mentioned unique measures in Top 3’s
5 10 8 0 4 0 0 6 6 4 0 1 1 1 4 5 5 3 1 2 3 15 22 16 4 6 3 4
Revenues FCF EBITDA OPEX EBIT CAPEX FTE
% on spot common measures (507)
1 2 3 Total 1 1 0 0 10 3 1 1 1 0 3 3 3 0 2 7 5 1 2 1 1 5 3 1 1 3 2 6 2 2 1 1 3 7 5 2 5 20 13 5 5 3 2 11
% on spot unique measures (507)
29
When distinguishing common and unique measures, the former are selected near as much as the latter (all respondents 355 versus 386; consumer residential seventy-four versus eighty-two) which is at least worth mentioning since there are only seven common measures versus eleven unique ones. It becomes really remarkable when you leave market share broadband out (ninety-nine and twenty-five times). Per ratio, common measures are selected almost twice as often, which at least implies that the focus is more on common measures. Besides, unique measures are hardly mentioned at spot one which shows the lack of importance to KPN financials.
30
Discussion and conclusions
This study focuses on performance measurement (within KPN). Performance measurement fulfills a crucial role within an organization, since it enables a company to track their corporate strategy and take action when necessary. Prior research in all kinds of research fields (Asch, 1946; Holstein, 1985; Sinclair, 1988; Hogarth and Einhorn, 1992; Bond et al. 2007; and much more) showed sequence can play a role in giving judgment. In addition, research showed measurement type could have an effect on judgment as well. For example, financial or non-financial measures (Kaplan and Norton, 1992; Lingle and Schiemann, 1996; Ittner and Larcker, 1998; Merchant and Van der Stede, 2007; Cardinaels & van Veen, 2010), but also more recently common or unique measures (Lipe & Salterio, 2000;2002; Banker et al., 2004; Dilla & Steinbart, 2005; Gagne et al., 2006). KPN currently uses a phased form of performance evaluation, in which both common and unique measures are delivered separately; this raises the question whether the previously mentioned effects affects an evaluator’s final judgment. It is also tested whether a more simultaneous delivery of all performance measures would end up in another judgment. Therefore, this study focused on the following research question:
Does a simultaneous versus a phased reporting of all measures during a performance evaluation process influence overall performance rating (within KPN)?
Experiment 1 shows performance evaluators at KPN are influenced by primacy, which implies the starting point of decision making has a disproportionate effects on its outcome. Information received first weighs more heavily compared to information received thereafter. According to the proven existence of the primacy effect, results of this study are in line with prior findings of for example Asch (1946), Bresnick (1993) and in line with the predictions mentioned in the model of Hogarth and Einhorn (1992). Therefore, it can be concluded, primacy exists in business areas. This means companies, especially KPN, hardly need to reconsider the way they receive information and the potential consequences of their performance evaluation design. According to KPN, there is a significant possibility that the information given in Flash 1 (and 2) are weighed more heavily, due to primacy, when it comes to strategic decisions.
31
studies, evaluators needed to compare segments which naturally creates an awareness of difference between common and unique measures. This study did not set up the experiment that way, participants of this study (therefore) showed some lack of awareness. When cleaning data for that lack of awareness results resemble the other studies (see appendix 3). Besides it could be argued KPN employees became more aware of the importance of unique measures due to signals provided (by the top? Corporate Control?) throughout the organization. However, this is not underlined by the outcomes of measures selected in the Top 3’s.
In order to be able to solve the order and type effect and conclude whether current performance evaluation at KPN results in different judgment compared to a simultaneous design, a third experiment has been conducted. It shows when measures are delivered all at the same time, evaluators significantly rate performance lower compared to a phased delivery, as KPN currently does. This is conflicting with results of Hoffman et. al (2011), who (insignificantly) argued the opposite effect. Possibly this is caused by the fact Hoffman et. al (2011) obviously differentiated between positive and negative information, while this research left space for evaluator’s own interpretation. At first sight, one may conclude KPN evaluators currently overvalues performance of segments, which could cause crucial mistakes when it comes to strategic decision making. While it seems hard to state whether this indeed causes overvaluation and judgments are therefore less reliable and less truthful. Since there is no single truth according to the performance evaluation rating. However, it is proven that a phased delivery causes a significant higher (+7.65%) rating, which at least indicates that performance results are interpreted differently. In addition, no significant difference in rating is observed between gender, age, function, segment or working experience. Thus, results can fully be attributed to the order (and type effect).
32
Limitations and further research
This study is subject to several limitations. First, not all respondents are (regularly) involved in evaluating performance of segments. Therefore, results of participants could be affected. Ideally, participants end up in a function in which they need to evaluate segments more often or have to act in a function where they at least globally have to evaluate the performance of KPN segments.
Second, in the post Questionnaire participants needed to indicate whether they understood the experiment. Based on those results the data is cleaned. However, is it plausible to assume that participants are honest when they do not understand what is expected of them? If they are not honest, it influences the validity of the outcomes. Another limitation concerns the experiment itself. Participants were asked to rate performance of a segment between 0 and 100, extreme scores 0 and 100 are defined, in between is not. This could cause a bias, since, for example, 50 does not contain the same value for each participant. It could have been solved by defining each score, or partly by decreasing the scale to for example 0 to 10.
33
34
References
- Anderson, N.H. (1965). Primacy effects in personality impression formation using a generalized order effect paradigm. Journal of Personality and Social Psychology, 2, 1-9.
- Anderson, N. H. (1981). Foundations of information integration theory. New York: Academic Press.
- Asch, S. E. (1946). Forming impressions of personality. Journal of Abnormal and Social Psychology , 41, 258-290.
- Banker, R. D., Chang, H., & Pizzini, M. J. (2004). The Balanced Scorecard: Judgmental effects of performance measures linked to strategy. The Accounting Review, 79 (1), 1–23.
- Bond, S. D., Carlson, K.A., Meloy, M.G., Russo, J.E., & Tanner, R.J. (2007). Information distortion in the evaluation of a single option. Organizational Behavior and Human Decision Processes, 102, 240-254.
- Bonner, S. E. (2008). Judgment and Decision Making in Accounting. Upper Saddle River, NJ: Prentice
Hall.
- Cardinaels, E., & van Veen-Dirks, P., M., G. (2010). Financial versus non-financial information: The impact of information organization and presentation in a Balanced Scorecard. Accounting,
Organizations and Society, 35, 565-578.
- Dilla, W. N., & Steinbart, P. J. (2005). Relative weighting of common and unique Balanced Scorecard measures by knowledgeable decision makers. Behavioral Research in Accounting, 17, 43–53.
- Feldman, S.R., & Allen, L.A. (1975). Determinants of the primacy effect in attribution of ability. Journal
of Personality and Social Psychology, 96, 121-133.
- Frigo, M.L. (2002). Nonfinancial performance measures and strategy execution. Strategic Finance, 84 (2), 6-8
- Gagne. M. L., Hollister, J., & Tully, G., J. (2006). Using the Balanced Scorecard: Both common and unique measures are informative. Journal of Applied Business Research, 22, 147-160.
- Hoffman, R.M., Kagel, J.H., & Levin, D. (2011). Simultaneous versus sequential information processing.
35
- Hogarth, R.M., & Einhorn, H.J. (1992). Order effects in belief updating: the belief adjustment-model.
Cognitive Psyochology, 24, 1-55.
- Holstein, J. A. (1985). Jurors’ interpretations and jury decision making. Law and Human Behavior, 9 (1), 83–99.
- Ittner, C., & Larcker, D. (1998). Are nonfinancial measures leading indicators of financial performance? An analysis of customer satisfaction. Journal of Accounting Research, 36, 1–46.
- Jonas, E., Schulz-Hardt, S., Frey, D., & Thelen, N. (2001). Confirmation bias in sequential information search after preliminary decisions: an expansion of dissonance theoretical research on selective exposure to information. Journal of Personality and Social Psychology, 80 (4), 557-571.
- Jones, E. E. L., Goethals, G.R., Kennington, G.E., & Severance, L. J. (1972). Primacy and assimilation in the attribution process: the stable entity proposition. Journal of Personality and Social Psychology, 40, 250-274.
- Joyce, E.J. (1976). Expert judgment in audit program planning. Journal of Accounting Research, 14, 29-60.
- Kaplan, R., & Norton, D. (1992). The balanced scorecard - measures that drive performance. Harvard
Business Review, January–February, 71–79.
- Kaplan, R., & Norton, D. (1996). Using the balanced scorecard as a strategic management system.
Harvard Business Review, January–February, 75–85.
- Kaplan, R., & Norton, D. (2001). Transforming the Balanced Scorecard from Performance Measurement to Strategic Management: Part I. Accounting Horizons, 15, 19-33.
- Libby, T., Salterio, S.T., & Webb, A. (2004). The balanced scorecard: the effects of assurance and process accountability on managerial judgment. The Accounting Review, 79 (4), 1074-1094.
- Lingle, J.H., & Schiemann, W.A. (1996). From balanced scorecard to strategic gauges: is measurement worth it? Management Review, 85 (3), 56-61
36
- Lipe, M. G., & Salterio, S., E. (2002). A note on the judgmental effect of the balanced scorecard’s information organization. Accounting, Organizations and Society, 27, 531-540
- Merchant. K., A., & van der Stede. W., A. (2007). Management Control Systems: Performance Measurement, Evaluation and Incentives.
- O’ Reilly, C.A. (1982). Variations in decision makers’ use of information sources: the impact of quality and accessibility of information. Academy of Management Journal, 25 (4), 756-771.
- Pennington, N., & Hastie, R. (1986). Evidence evaluation in complex decision making. Journal of
Personality and Social Psychology, 51, 242-258.
- Roberts, M.L., Albright, T.L., & Hibbets, A.R. (2004). Debiasing balanced scorecard evaluations.
Behavioral Research in Accounting, 16, 75-88.
- Sinclair, R. (1988). Mood, categorization breadth, and performance appraisal: the effects of order of information acquisition and affective state on halo, accuracy, information retrieval, and evaluations.
Organizational Behavior and Human Decision Processes, 42, 22-46.
- Slovic, P., & MacPhillamy, D. (1974). Dimensional commensurability and cue utilization in comparative judgment. Organizational behavior and human performance, 11, 172-194.
- de Vries, I. (2012). Use of Nonfinancial information in Performance Management. Register Controller Thesis Erasmus University Rotterdam.
- Williams, W.A. (2002). Trusting the numbers: the power of data verification. Government Finance
Review, April, 18-21
- www.kpn.com//corporate/aboutkpn/Company-profile/
- Xu, Y., & Wang, D. (2008). Order effect in relevance judgment. Journal of the American society for
37
Appendices
Appendix 1: Organizational chart KPN
4
38
Appendix 2: Experiment 1-3
Dear colleague,
This experiment focuses on the way performance evaluation is done within KPN. Please carefully read the following case. It will inform you about Consumer Residential, part of KPN Dutch Telco. Information is provided about their main business activities and strategy in 2013. First please fill in the questionnaire and read the case information. After you feel comfortable with this material, go to the segment’s scorecard to rate its
performance. Performance measures are defined on the last page. At the end please submit your results. Finally, there will be another short questionnaire. In advance I would like to thank you for participating in this experiment. Afterwards, I am more than happy to explain more about the purpose of this experiment.
Pre Questionnaire (Please fill in the correct box)
1) What is your gender?
□
Male□
Female2) What is your age?
□
20<30□
30<40□
40<50□
>503) What KPN segment do you work in?
□
Consumer Residential□
Consumer Mobile□
Business Market□
Other4) What is your function within KPN?
□
Business control□
Financial control□
Other5) What is your total years of working experience?
