l
Amsterdam Business School
RSAC and MSc thesis template
The effect of congruence on Subjective Performance Evaluation, when
Objective performance is known.
Name: Noureddine el Adoui Student number: 10294775 Thesis supervisor: P. Kroos Date: 17-07-2016
Word count: 9069,
MSc Accountancy & Control, specialization Control
2
Statement of Originality
This document is written by student Noureddine el Adoui who declares to take full responsibility for the contents of this document.
I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.
The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.
3
Abstract
In this thesis, I examine how the congruence level of an objective performance measure affects the relationship between the objective performance measure outcome and the subjective evaluation outcome. Congruence is one criterion to assess the quality of a performance measure. Other criteria are: controllability, manipulability, sensitivity, and congruence (Banker & Datar, 1989). Bol & Smith (2011) found that supervisors provide a higher subjective evaluation when the outcome on the objective measure is low, and that this is likely to be due to external factors (uncontrollability), and the addressing of fairness concerns. The goal of this thesis was to examine to what extent the higher congruence of an objective measure could result in a high spill-over effect from the objective to the subjective measure. To test this, an experimental design of 2 x 2 between-subjects was conducted. Based on existing literature, two hypotheses were developed. (H1) A supervisor’s subjective evaluation is higher when the outcome on an unrelated objective measure is low. A One-Way ANOVA analysis was conducted to test this hypothesis, and it was found the difference in the subjective evaluation between the low and high performance condition was significant. However, the result did not support H1. The second hypothesis presumed that the negative relationship between a supervisor’s subjective evaluation and the outcome on an unrelated objective measure is stronger when the objective measure has high congruence. Hypothesis (2) was also tested with a One-Way ANOVA, which indicated that there is no significant relationship. However, the results showed that there is a difference in evaluation when there is low financial performance, between high and low congruence. The subjective evaluation is (0,87) points higher in the low financial and low congruence condition than the low financial and high congruence condition.
4
Keywords: subjective performance evaluation; performance measurement; congruence level; fairness.
Acknowledgements
It took blood, sweat and tears to finish the following thesis, "The effect of congruence on Subjective Performance Evaluation, when Objective performance is known''. I am happy that this is my final assessment to graduate at the University of Amsterdam. I learned a lot over the past two years during the master course Accountancy and Control. I would like to thank my supervisor Dr. P. Kroos who helped me tremendously with my thesis and who learned me a lot about management control systems. I also want to thank Prof. Dr. V.S. Maas, who helped me to finish mine experiment. Besides the valuable criticism on the experimental case, I also pulled out personal lessons.
5
Table of Contents
1. Introduction ... 6
1.1 Background ... 6
1.2 Research question ... 7
1.3 Motivation and contribution ... 7
1.4 Structure of the thesis ... 7
2. Literature review and hypotheses ... 8
2.1 Information asymmetry... 8 2.2 Agency theory ... 8 2.3 Performance measures ... 9 2.4 Subjective evaluation ... 10 2.5 Hypothesis development... 13 3. Methodology ... 14
3.1 Design of the experiment ... 14
3.2 Independent variables ... 14
3.3 Procedure ... 15
3.4 Manipulation Checks and Information Sufficiency ... 16
3.5 Participants ... 16 4. Results ... 18 4.1 Descriptive statistics ... 18 4.2 Hypothesis tests ... 19 4.2.1 test of hypothesis 1 ... 19 4.2.2 test of hypothesis 2 ... 19 4.3 Supplementary analyses ... 22 5. Conclusion ... 24 Bibliography ... 26 Appendix ... 29 Case ... 29 Post-experimental questions ... 32 Demographic questions: ... 33
6
1. Introduction
1.1 Background
One of the most powerful ways to influence behaviour in employees is through pay-for-performance systems. This system involves employees being rewarded with extra pay for good performance. A study on the impact of pay-for-performance on motivation, showed that it is positively related to employees’ interest in their daily work activities (Eisenberger et al., 1999). However, there are several downsides associated with performance measures in that they are not always linked with the right ‘dimension’ – the area in which reward is offered to the employee might not actually benefit the company. As an example, if a supervisor gives their subordinate a sales target, the subordinate may, in response, only focus on the easiest sales, which may in fact not generate the highest profits for the company.
There are several criteria to assess the quality of a performance measure. These criteria are: controllability, manipulability, sensitivity, and congruence (Banker & Datar, 1989; Woods, 2012). ‘Sensitivity’ can be interpreted as the extent to which an employee can influence the measure. A measure has ‘manipulability’ when it indicates good results, but in reality the performance is actually worse (Baker et al, 1994). ‘Controllability’ refers to the degree that a performance measure not only reflects managerial actions, but also reflects the effects of exogenous events (e.g. an economic downturn). ‘Congruence’ can be defined as the extent to which employees’ actions increase both the measure and firm value (Feltham & Xie, 1994).
Incentive contracts often contain both objective and subjective components (Baker et al. 1994). Subjective performance evaluations can be used by companies to adjust the performance rating of employees when the objective measures are considered not to accurately represent their real performance (Woods, 2012). However, subjective evaluations can give room for the influence of preferences and biases. Another problem is that, even when the evaluation is fair, the subordinate can experience it as unfair, because they might not fully understand how the evaluation was established (Merchant & Van der Stede, 2012). A study on the effect of ‘controllability’ as a feature of objective performance measures, showed that subjective performance evaluations are influenced by the level and controllability of the objective measure of a separate performance dimension; the study found a spill-over effect from objective performance to subjective evaluation in this respect (Bol & Smith, 2011). This finding can be interpreted thus: when an employee scores high on a separate objective measure, this will likely lead to a higher subjective evaluation. To date, very few studies have
7
explored exactly how the level of congruence of an objective measure affects the spill-over effects between objective and subjective measures.
1.2 Research question
In this thesis, the author will investigate whether, and how, the level of congruence of an objective measure has an effect on subjective performance evaluation, when objective performance is already known to the supervisor. This study builds on prior research conducted by Bol & Smith (2011), who examined the spill-over effect between objective and subjective measures, by using controllability of the objective measure as a moderator. The authors focused only on one criterion of evaluation performance measures: namely, controllability. However, there are several criteria by which we can evaluate performance measures. The current study will focus on congruence. The expectation is that when a supervisor knows that a measure is congruent, their subjective evaluation will be more strongly affected by the performance on the objective measure. The researcher aims to answer the following research question: ‘How does the congruence of an objective performance
measure affect the relationship between the objective performance measure outcome and the subjective evaluation outcome?’
1.3 Motivation and contribution
Much of the prior literature on subjectivity examined the benefits of subjectivity. In general, subjectivity is considered to address the shortcomings of solely relying on objective measures. Not much literature has looked at potential biases in subjective performance evaluations. The author intend to contribute to the literature. By examining to what extent a higher congruence of an objective measure could mean a high spill-over effect from the objective measure towards the subjective measure. This study also features a societal contribution. Given the prevalence of subjectivity in organizations, it is also important to document potential side-effects of subjectivity such as fairness of an evaluation system can be improved.
1.4 Structure of the thesis
The remainder of this paper is structured as follows. In section two, the literature review and hypotheses will be discussed. The third section contains an overview of the research method. Section four provides an overview of the empirical results. And the final section presents the conclusion and research limitations.
8
2. Literature review and hypotheses
2.1 Information asymmetry
Most organizations are characterized by some degree of information asymmetry.‘Information asymmetry’ describes a situation where the agent’s has more, or better, information about a transaction than the other party. Such a situation can occur, for example, when an agent has private information about a transaction, which they do not want to give ‘for free’ to the principal (Harris & Raviv, 1979). For instance, a lower-level employee has more information about local markets and preferences than their superior in the organization. Typically, a superior will observe the action of a subordinate and reward those actions that are perceived to be value-adding. If certain actions and decisions on the part of subordinate are not readily observable, the superior may start relying on performance measures when evaluating and rewarding them.
Agency models indicate that performance measurement systems should contain any relevant, readily available, information about the agent’s actions (Banker & Datar, 1989; Feltham & Xie, 1994). Ittner and Larcker, (2002) conducted a study on the determinants of performance measure choices in compensation plans; they found that informativeness is one of the key determinants in performance measures. Information about the agent’s actions can help the principal to select performance measures in compensation plans, to align the
behaviour of the agent with the objectives of the principal. So each performance that provided incremental information about a subordinates actions and decisions should be included in a firms’ performance measurement system.
2.2 Agency theory
Agency theory refers to the relationship between the agent and the principal, such as when the agent acts for, or on behalf of, the principal. An example of this is the relationship of employee to employer. Such a relationship between two parties is described as a ‘contract’ (Jensen & Meckling, 1976). Agency theory is concerned with potential problems that can occur in this relationship. The most obvious problem is known as the ‘agency problem’ – that is, when there is a conflict between the desires and goals of the agent and those of the principal. Related with this is when the principal cannot verify the actions of the agent – in other words, how can the principal be sure that the agent behaves in line with the contract? Another problem is that of risk-sharing, which occurs when the principal and the agent have different attitudes towards risk (Eisenhardt, 1989). If, for example, the principal carries the
9
risk, the agent is free to behave inappropriately in the attaining of their targets. The goal of the agency theory is to create an optimal compensation contract to deter the agent from uncontrollable risk-taking behaviours, and to avoid, at the same time, risk-averse behaviour in the agent, and provide them with sufficient incentives to counterbalance inappropriate behaviour (Sloof & Praag, 2005). In sum, the theory aims to deter the agent from uncontrollable risk, and protect the principal from inappropriate behaviour in the agent. The theory also points out that the goal of control systems is to align the principal’s objective with that of the agent. Performance measures form a part of such control systems, that are used by organizations to achieve behavioural congruence.
2.3 Performance measures
Performance measures are metrics that are used to quantify the effectiveness and efficiency of employees’ actions (Neely et al., 1995). By ‘effectiveness’ is meant how successful the employee is in attaining their objectives; and ‘efficiency’ relates to how resources are used to fulfil those objectives.
Research has shown that individuals and groups alter their behaviour when they know that they are being studied and their performance is being measured (Kaplan et al., 2012). To align the behaviour of employees with the firm’s objectives, incentives must be tied to objective performance measures. By ‘objective measure’ is meant a measure that is not influenced by personal feelings or interpretations – objective measures do not, however, always incorporate valuable information about a subordinate’s performance (Fisher et al., 2005). The crucial characteristic of objective measures is that they are verifiable in nature. That is, the outcome on an objective measure can be verified by a third party.
Objective performance measures can be distinguished into financial- and non-financial measures. The main difference between financial- and non-financial measures is that: financial measures are focused on results such as profit, revenue, etc., and report on the impact of decisions made in current and past periods; in contrast, non-financial measures are focused on the long term, and are ‘forward-looking’ in nature. As an example of how these may be linked, however, improved customer satisfaction, which is a non-financial measure, can lead to an increase in future-financial performance. Financial measures have been criticized because of their rewards for short-term profit, and for their lack of interest in investing in ‘intellectual’ capital. Moreover, agency models indicate that financial measures are not the most effective way to motivate employees; on the contrary, rewarding employees for non-financial measures can improve performance through means that are not captured by
10
financial measures (Feltham & Xie, 1994; Hemmer, 1996). These are some reasons why many companies these days focus on non-financial performance indicators especially for those firms where long term survival crucially depends on investment in customer satisfaction, R&D, etc. (Ittner & Larcker, 1998).
There are several criteria which can be used to assess whether the quality of the objective measure that is used in the evaluation is actually informative about the agent’s actions. These criteria are: controllability, manipulability, sensitivity and congruence (Banker & Datar, 1989; Woods, 2012). ‘Sensitivity’ can be interpreted as the extent to which an employee can influence the measure. A measure has ‘manipulability’ when it indicates good results, but in reality the performance is actually worse (Baker et al, 1994). ‘Controllability’ refers to the degree that a performance measure not only reflects managerial actions, but also reflects the effects of exogenous events (e.g. an economic downturn). ‘Congruence’ can be defined as the extent to which employees’ actions increase both the measure and firm value (Feltham & Xie, 1994). For example, a subordinate might decide to postpone a necessary long term investment, so that annual profit will be higher; in such a case, the measure will increase and the firm value will decrease. Finally, it can be argued that using only objective performance measures, such as financial- and non-financial measures, as predictors of an employee’s contribution in firm value, is insufficient. To mitigate the shortcomings of objective performance measures, compensation contracts often contain subjectivity. Baker et al. (1994) explored the combined effect of objective and subjective performance measures in incentive contracts, and found that objective and subjective measures complement each other.
2.4 Subjective evaluation
By ‘subjective evaluation’ is meant an evaluation that is based on personal feeling or interpretations, instead of external facts. The ‘rightness’ of a subjective evaluation cannot be verified by a third party (Bol, 2008; Rajan & Reichelstein, 2006).
One of the important aspects of subjective performance evaluation is that it can mitigate incentive distortions brought about by imperfect objective measures. One of these distortions is the fact that objective performance is usually focused only on a few, selected job dimensions. Employees who notice this are likely to only focus on those activities that are going to be measured and compensated. Another flaw is that objective measures are often focused only on short-term effects, at the expense of long-term value (Kaplan and Norton, 1996). Compensation contracts that contain the element of subjectivity can incentivize employees to focus on activities that are not measured, and on creating long-term value.
11
Moreover, in cases of uncontrollable events, such as a financial downturn, subjectivity can be used to correct for the uncontrollable event in question. Several studies have shown that subjectivity in compensation contracts can reinforce objective performance measures (Baker et al.,1994; & Budde, 2007).
However, there are several disadvantages in using subjective evaluations. Three of the major ones are: reneging, favouritism and bias, and these are discussed in some detail below.
Reneging
‘Reneging’ means that the principal revokes on their pledge, because paying out a bonus would reduce their own wealth. As a result, the agent will likely feel less willing to create value for the principal in the future. The fact that the agent is not incentivized anymore may well prove costly for the principal (Bol, 2008). One could argue that today, companies are multi-layered, and that, therefore, the principal is not always the residual claimant
(Prendergast, 1999). But even in a multi-layered company, a manager can revoke their pledge. For example, a manager might promise their subordinate they will get promotion if they perform well. At the end of the year, the manager receives a lower budget for the following year from head office, and because of this revokes on their pledge by giving the subordinate a lower subjective evaluation than justified, even though the subordinate has indeed performed well.
Favouritism & rent seeking
Several studies have shown that managers take their own preferences into account when appraising performance (Bol, 2008; Prendergast & Toper, 1993). Bol (2008) studied the determinants and performance effects of managers’ performance evaluations. She found that managers respond according to their own incentives and preferences when performing a subjective evaluation. This effect is stronger in cases of extensive information-gathering, and when the relationship between the manager and employee is strong. Favouritism is an
example of how preference might manifest; a manager might, for instance, inflate the
performance rating of a subordinate with whom they have a good relationship, and deflate the rating if the opposite is true. One consequence of favouritism is that an employee who has worked hard and performed well is likely to feel wronged, and perhaps even leave the
organization, if they are judged negatively in this way. This can lead to lower performance in the future. In addition subordinates may start wasting valuable time with activities aimed to personally get in favour with the superior, so-called rent-seeking activities.
12 Bias
Finally, the use of subjectivity in compensation contracts can lead to bias in
performance evaluation; various studies have shown there is a problem in this area (Woods, 2012; Bond et al., 2006). Three common biases in performance evaluation are: centrality bias, leniency bias (Prendergast, 1999) and outcome bias. Centrality bias is the attempt to ‘lump’ performance ratings together, with the result of less variance in ratings than is actually secured in performance; while leniency bias is the attempt to inflate employees ratings by avoiding to give low ratings (Saal et al., 1980). Centrality bias is beneficial for
underperforming employees; in this case the evaluator gives a higher evaluation than their actual performance merits. On the contrary, high-performing employees will get a lower evaluation than their actual performance merits. As a likely result, these high performers will lose their motivation, because they get more similar compensation as the underperformers (Garland, 1973). In the case of leniency bias, only low performing employees get a higher evaluation, because the evaluator wants to avoid giving negative outcomes. Both centrality bias and leniency bias can be very costly for the principal. From the agency theory
perspective, the principal links payment to performance to evoke extra effort. As a result of leniency- and centrality bias, however, underperforming employees get rewarded without providing extra effort, which incurs financial cost to the principal. Outcome bias relates to situations where the evaluator knows that the outcome was influenced by external factors that were beyond the influence of the person being evaluated. Terence & Laura (1981) looked at the effect of knowledge of the actual outcomes of subordinates’ performance in terms of evaluations made by the supervisor. The study showed that outcome knowledge modifies the supervisor’s judgment. This was even found to be the case in situations where the measure did not actually reveal anything about the individual’s performance. In a study by Ghosh & Lusch, (2000), the effect of outcome bias on performance evaluation was investigated. The authors found evidence that subjective evaluations are negatively impacted by knowledge of unfavourable outcomes. In cases where the evaluator had control and influence over sales or profit, however, knowledge of the subordinate’s performance did not influence their
evaluation. This was also found to be the case with environmental effects, when the
subordinate had no control over sales or profit. In contrast, the authors found that when a store manager failed to meet target outcomes, then this failure often lead to a negative performance evaluation, even when the store manager had no control over the outcome. To sum up, outcomes lead to a lower subjective evaluation in cases where (1) the subordinate has control
13
over sales or profit (2) when the subordinate does not meet their target, even when they have little control over the situation.
2.5 Hypothesis development
Subjective measures are often used to address shortcomings of objective measures. One example is that when some performance dimension cannot be measured objectively, firms try to improve the whole performance measurement system by evaluating that performance dimension subjectively.
However prior research shows how the subjective evaluation of an unrelated task can be influenced by the outcome of an objective measure. Ghosh & Lusch (2000) show how the subjective evaluation is lower when the outcome on the objective measures is substandard. Bol & Smith (2011) find how supervisor provide a higher subjective evaluation when the outcome on the objective measure is low and this is likely to be due to external factors (uncontrollability), to address fairness concerns. As most literature have put most emphasis on the role of the role of subjectivity to address the impact of uncontrollable, external events to increase fairness, I formulate my first hypothesis as follows.
H1) A supervisor’s subjective evaluations is higher when the outcome on an unrelated objective measure is low.
Supervisors may try to increase the fairness of the performance evaluation by awarding them with higher subjective evaluations when outcomes on the unrelated objective measure are low, likely due to the influence of external uncontrollable events. However, these fairness concerns may also be influenced by the characteristics of the objective measures. That is, objective measures that have a high congruence are typically considered to be of high quality, and also commonly receive a higher weight in incentive contracts. This may also imply that principals assign a greater weight on a highly congruent objective measure in the overall performance evaluation compared to a weakly congruent objective measure (Feltham & Xie, 1994). This may result in supervisors more strongly correcting for uncontrollable external events by inflating the subjective evaluation when the objective measure has high congruence. So my second hypothesis is as follows.
H2) The negative relationship between a supervisor’s subjective evaluation and the outcome on an unrelated objective measure is stronger when the objective measure has high congruence.
14
3. Methodology
3.1 Design of the experiment
An experimented is conducted to test the hypothesis. The participants each received a different hypothetical case, each of these detailing a variety of conditions. They were requested to assume the role of CEO for a production company. It was their task to perform a subjective evaluation of a brand manager, who was responsible for both financial performance and leadership; the manager’s scores on objective financial measures were also made available to them. No contact with the actual managers occurred. The design of the experiment was a 2 x 2 between-subjects. Information about the managers’ scores on financial performance, and information related to congruence, was manipulated at two levels (high and low). That is, the objective measure has two levels of financial performance (high and low) and two levels of congruence (high and low). Participants were told that the financial performance could be affected by uncontrollable events (e.g. economic downturn). For each condition, a minimum of 15 participants were recruited.
The dependent variable in this study was the subjective evaluation of performance as assessed by the CEO. The information provided about this variable was the same across all conditions. Participants were told that interviews with subordinates of the brand managers, indicates that the brand manager was able to influence and motivate them. This resulted in that the subordinates were able to implement the project on time, unfortunately the brand manager was not able to solve conflicts between the subordinates, which resulted in experiences of tension for employees. After the participants had read the case information, they were asked to rate their subjective evaluation, on a scale from 1 to 10. To ensure that evaluations were filled in correctly, there were a number of post-experimental questions
3.2 Independent variables
The independent variables were: objective measures; and the level of congruence. The objective measure was the amount of profit for two cases or revenue in the other two cases generated by the brand manager in the previous year. The score on this measure varied on a scale of 1 to 10, each score being related to a particular range of financial performance. Specifically, the financial performance score of the subordinates was defined as 2 for low, and 9 for high, financial performance.
The congruence measure was manipulated by changing the information provided about financial performance. For the condition of low congruence (revenue), the participant
15
received the information that this objective measure was not actually perceived to be a good indication of the degree to which the actions of the brand manager had build firm value. For the condition of high congruence (amount of profit) the participants received the information that this objective measure was perceived to be a good indication of the degree to which actions of the brand manager had build firm value.
3.3 Procedure
Participants received an e-mail with a link to the survey. Each participant received a different condition, which was randomly assigned. At the outset, participants were reassured that this was not a test, and that there were no correct or incorrect answers. Participants were told that the company for which their brand manager X worked was managed by the CFO and the CEO, and that the CEO was ultimately responsible for the financial performance and for the leadership performance of brand manager X.
The case description made it clear that, for the purposes of the experiment, the participants must assume the role of CEO, and evaluate the leadership performance of the brand manager. Participants were also told that the overall evaluation consisted of two components: an objective score based on financial performance; and a subjective score based on leadership performance. These components were described as unrelated. The case description described the objective measure to also be impacted as uncontrollable, outside factors. Information about the objective measure was indicated on a 1-10 scale, and the brand managers’ scores manipulated to be either high (9) or low (2). The level of congruence was manipulated at the same time. Participants were told that the financial performance of brand manager X had already be evaluated in the previous year, and their evaluation was given, along with a table showing the evaluation scale, to give the participants an idea how the objective score had been established.
In order to conduct the subjective evaluation, participants were told that interviews with the subordinates of the brand managers had been conducted, and these responses were provided. After the participants had read this case information, they were asked to evaluate the leadership performance of the brand manager on a scale of 1 to 10. To monitor whether the participants had adequately understood the case description, a number of post-experimental questions were asked. Finally, the participants were presented with demographic questions.
16
3.4 Manipulation Checks and Information Sufficiency
To check whether the manipulations were successful, the participants were required to recall them. Regarding the manipulation of the objective measure, participants were asked to answer the following question: ‘What was the objective measure’s score for the manager?’ This question was answered by 126 participants, 5 of whom were excluded because they could not remember this score. An independent samples t-test was used to test the effectiveness of the manipulation. The results concluded that the manipulation check worked well (p < 0.01), the results being in accordance with Bol & Smith (2011).The following question was then asked: ‘What was the degree to which an increase in the objective measure also implied an increase in firm value?’; this was to check the effectiveness of the manipulation of the level of congruence. This question resulted in 45 of the participants recalling that the congruence level was ‘reasonable’, in the low congruence level; and 44 recalling the congruence level was ‘excellent’, in the high congruence level (p < 0.01). This question was answered by 122 participants, 33 of whom were excluded, because they could not remember the congruence level. Thus, both manipulations – of the level of financial performance and the level of congruence – were assessed as successful. Participants were also asked to recall the relationship between the objective and subjective performance evaluations. This question was answered by 129 participants, of whom 114 recalled that the relationship was independent; and 15 said it was dependent; and were excluded because they could not recall the relationship. In summary, 3 participants did not provide the score for the subjective evaluation, 9 participants did not fill in the score for the objective measure, 6 did not answer on the relationship between the 2 components, and 13 did not provide the degree of the objective measure. Thus, in total, 31 participants failed to answer one of the questions; and 34 participants were excluded because they could not recall one or more of the control questions correctly.
3.5 Participants
The survey was completed by 135 participants, of whom only 70 could recall the control questions. This thesis includes only these 70 participants, of whom the mean age was 27 years (SD = 6,39), the youngest being 16 years old and the oldest 57. The gender balance was 45 males and 25 females. The majority of the participants had Dutch nationality (n = 23), followed by 19 participants with British nationality and 13 with American nationality.1 The majority of participants were recruited via social media and a crowdsourcing platform. They
1 The education level of the population is not a trustworthy factor, as the demographic question relating to this referred only to the Dutch system, and therefore was unfamiliar to British and American participants.
17
were mostly invited personally, and by mail, though a minority of the participants were students at the University of Amsterdam. Participants who had been invited by mail but who had failed to respond, received a reminder after 5 days. A total of 32 of the participants reported that they had supervisory experience; and 54 reported having been subject to formal performance evaluation.
Participants were also asked how they would rate their confidence in their evaluation. The participants on the whole were confident in their evaluation.2 On a scale of 1-10, the mean of this measure was 6,20 (s.d. = 2,11), suggesting that, the information about the leadership performance of the brand manager was sufficient. Participants were also asked if they had corrected the evaluation for uncontrollable effects, and the majority (n =47) said they had not done this. The last question asked the participants if they believed that manager X’s objective score actually told them how well the manager had performed on leadership. The majority (n = 49) did believe that the objective score told them something about this, and the remainder (n = 21) did not.
2 The majority of the participants found the provided case information was not sufficient to make the subjective evaluation; this could be a potential drawback of this thesis.
18
4. Results
4.1 Descriptive statistics
The descriptive statistics presented in table (1) below shows that the mean of the subjective evaluation is the highest in the high financial performance condition under high congruence (6,84), followed by the low financial performance condition under low congruence (6,56). The mean of the subjective evaluation is the lowest in the low financial performance condition under high congruence (4,76), the subjective mean for the low congruence is relatively higher (5,63). These univariate results suggest that in both the case of high and low congruence, supervisors provide a higher subjective evaluation when the objective, financial performance is high. However, the difference seems bigger in the case of high congruence. An explanation for these difference can be that supervisor put more emphasis on the objective score when it has high congruence
___________________________________________________________________________
Table 1
High congruence Low congruence Row mean
High financial performance 6,84 6,56 6,7
(0,96) (1,68) (1,35)
n = 19 n = 18 n = 37
Low financial performance 4,76 5,63 5,2
(1,71) (1,63) (1,70) n = 17 n = 16 n = 33 Column mean 5,9 6,1 (1,71) (1,70) n = 36 n = 34 ___________________________________________________________________________
19
4.2 Hypothesis tests
4.2.1 test of hypothesis 1
H1 predicts that a supervisor’s subjective evaluations will be higher when the outcome on an unrelated objective measure is low. A one-way analysis of variance (ANOVA) was used to test whether the difference between high and low financial performance was significant. As shown in table (2), the test indicated a result of (F = 17,30; p < 0,01), which means that the difference between the overall mean of high and low financial performance is significant. Table (1) shows the overall mean of the subjective evaluation is lower in the low financial performance condition (5,20) than in the high performance condition (6,70), which invalidates H1. One possible explanation for this result is cognitive distortion. This can occur when supervisors incorporate information about the objective measure in their subjective
evaluation. As mentioned earlier, the majority of the participants in this study believed that the objective score told them something about the performance of the separate task, which indicates a potential for distorted information. This is in accordance with Bol & Smith (2011); these authors found that supervisors distort information about subordinates to be consistent with an objective measure of an independent task.
___________________________________________________________________________
Table 2 Results hypothesis 1
Panel A: Analysis of Variance for low financial performance vs high financial performance.
Source df Mean Square F-statistics p-value
Financial performance 1 40,35 17,3 < 0,01
___________________________________________________________________________
4.2.2 test of hypothesis 2
H2 predicts that the negative relationship between a supervisor’s subjective evaluation and the outcome on an unrelated objective measure is stronger when the objective measure has high congruence. Linear regression was used to test whether the relation is stronger under high congruence. The summary model in table (3) shows that, 23,7% 0f variance in subjective evaluation can be explained by the regression model. The interaction results are shown in coefficients panel in table (3), the variable financial performance*Congruence shows a positive value for b* of 0,368. Thus, when the congruence level becomes high, the relation between the objective measure score and the subjective evaluation score will be (0,368)
20
higher. However, this result is not significant (p < 0,12); this implies no significant interaction between the level of congruence and the level of the objective performance which invalidates H2. So, my findings do not suggest that the congruence of the objective measure influences the relationship between the performance on the objective measure and the subjective evaluation. A closer look at table (1) reveals that congruence affects the relationship differently dependent on the outcome of the objective measure. If financial performance is high, high congruence produces a higher subjective evaluation. However, if financial performance is low, low congruence produces a higher subjective evaluation. A reason for this effect could be as follows: in the high financial performance situation under high congruence, a supervisor will put more emphasis on the objective measure, because the measure is deemed informative about the actions of the subordinate. In such a case, this will lead to higher subjective evaluation than when there is high financial performance under low congruence. However, in the case of low performance and high congruence, the supervisor will also put more emphasis on the objective measure, but will now ‘punish’ the subordinate on account of the informative objective measure.
So, in conclusion, it is almost impossible to find a significant result between the groups in the high and low congruence condition. As we can see in table (1), the differences are offset against each other. High performance and high congruence (6,84) leads to a higher subjective evaluation than high performance and low congruence (6,56). Low performance and low congruence (5,63) leads to higher subjective evaluation than low performance and high congruence (4,76). A summary of table (1) is displayed in figure (1).
Figure 1 Su b je cti ve ev alu ati o n 0 1 2 3 4 5 6 7 8
Low financial performance High financial performance
Low congruence High congruence
21 __________________________________________________________________________________ Table 3 Linear regression ___________________________________________________________________________________ Model Summary Model R R Square Adjusted R Square Std. Error of the Estimate Change Statistics R Square Change F Change df1 df2 Sig. F Change 1 ,487a ,237 ,203 1,5165 ,237 6,842 3 66 ,000
a. Predictors: (Constant), Financial performance*Congruence, Financial performance, Congruence
ANOVAa
Model Sum of Squares df Mean Square F Sig.
1 Regression 47,206 3,000 15,735 6,842 ,000b
Residual 151,780 66,000 2,300
Total 198,986 69,000
a. Dependent Variable: Subjective evaluation
b. Predictors: (Constant), Financial performance*congruence, Financial performance, Congruence
Coefficientsa Model Unstandardized Coefficients Standardized Coefficients Sig. B Std. Error Beta t 1 (Constant) 5,359 ,498 10,761 ,000 Financial performance ,133 ,074 ,276 1,786 ,079 Congruence -1,188 ,694 -,352 -1,712 ,092 Financial performance*Congruence ,164 ,104 ,368 1,579 ,119
22
4.3 Supplementary analyses
To test whether there were other variables which correlate significantly with the dependent and independent variables, a Pearson correlation model was conducted. Table (4) below shows that there was a significant positive relation between the objective measure and subjective evaluation (Rs = .450, P = <0.01). There was found to be a negative relation between the congruence level and the subjective evaluation (Rs = -.076, P = <0.532), this result is not significant. Age has a positive significant relation with subjective evaluation (Rs = .252, P = <0.05) and gender has a negative significant relation with objective score (Rs = -.346, P = <0.05). The significant variables were also evaluated on multicollinearity. There is multicollinearity if the relation between the variables is higher than 0,9 ( (De Vocht, 2013, p. 193). As shown in table (4), there are no variables with a correlation higher than 0,9.
The mean age over the four conditions was 27 years; thus, it cannot be said to influence the results. The distribution of gender under high congruence was equal; and was unequal under low congruence. This could represent an alternative explanation for the findings. However, gender also has to be significantly associated with the subjective evaluation score. The correlation table shows that this is not the case. So, in conclusion, gender and age do not represent an alternative explanation for the findings in this thesis.
23
Table 4
Pearson correlation model between the variables
N = 70
Subjective evaluation
Financial
performance Congruence Age Gender
Evaluation Pearson
Correlation 1
Sig. (2-tailed)
Financial performance Pearson
Correlation ,450** 1 Sig. (2-tailed) ,000 Congruence Pearson Correlation -,076 -,002 1 Sig. (2-tailed) ,532 ,989 Age Pearson Correlation ,252* ,084 -,027 1 Sig. (2-tailed) ,037 ,494 ,828 Gender Pearson Correlation ,011 -,346** ,051 -,028 1 Sig. (2-tailed) ,926 ,003 ,674 ,819
**. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
24
5. Conclusion
The aim of this thesis was to answer the following research question: ‘How does the
congruence of an objective performance measure affect the relationship between the objective performance measure outcome and the subjective evaluation outcome?’ The results show that
when a supervisor is aware of the objective performance measure, that this will affect the subjective evaluation. Specifically, the subjective evaluation outcome will be 1.5 points higher when financial performance is rated ‘high’, than for when it is rated ‘low’. Hypothesis (1) presumed that the subjective evaluation will be higher when the outcome on an unrelated measure is low. The results indicate that ( H1) is not supported. A possible explanation is cognitive distortion. The majority of the participants in this study believed that the objective score told them something, and provided reliable information for their subjective evaluation. This indicates a potential distortion of information. Bol & Smith (2011) found that
supervisors distort information about subordinates in order to be consistent with the objective measure of an independent task. There is no support found for (H2), which presumed that the negative relationship between a supervisor's subjective evaluation and the outcome on an unrelated objective measure is stronger when the objective measure has high congruence. The results show that there is a difference for the low financial condition between the high and low congruence. The subjective evaluation is (0,87) points higher in the low financial and low congruence condition than in the low financial and high congruence condition. An explanation for these differences could be that the supervisor tries to avoid fairness concerns (Bol & Smith, 2011). That the subjective evaluation is lower for the subordinates with low performance under the condition of high congruence, can be explained by the fact the measure is very informative, and give a good indication of the actions of the subordinate. In the low financial performance situation under low congruence, the measure is a poor indicator of the performance of the subordinate. To avoid unnecessary disharmony following a low evaluation under low congruence, the supervisor will upward rate the subjective evaluation (Ghosh & Lusch, 2000).
The findings of this thesis have a number of implications for organizations who use a combination of subjective and objective performance evaluations. The results of this study confirm the findings of Bol & Smith (2012), that supervisors distort information. Employees who are aware of this fact, can discuss the subjective evaluation with their supervisor. The results also have implications for the design of compensation contracts. One of the reasons
25
that compensation contracts contain a subjective evaluation component is that objective measures capture only a few of the dimensions of the tasks of a subordinate. Subjective evaluation can therefore be used to correct for un-captured actions. However, when subjective evaluations are used to correct for undesirable actions, objective measures, conversely, are used to correct subjective evaluations. The results of this study could help supervisors to be conscious of these effects.
This thesis has been subject to several limitations. The number of respondents (n = 70) limited the analysis, and the author was subject to time constraints. Also, the manipulation of congruence was limited to two extremes – reasonable and excellent – whereas in practice the congruence of an objective measure can also be ‘intermediate’. Another limitation was the information provided about the brand manager’s leadership, which was very brief. The reason for this brevity is that participants are more likely to fill out a survey when it is short. Finally, there are various quality criteria which can affect a supervisor’s subjective evaluation,
including manipulability and sensitivity. It is therefore recommended that future research examines the effect of manipulability and sensitivity on subjective evaluation outcomes.
26
Bibliography
Arvey, R. D. (1998). Subjective Performance Measures in Optimal Incentive Contracts. Annual Review of Psychology, 49, 141-168.
Baker, G. P., Gibbons, R., & Murphy, K. J. (1994). Subjective Performance Measures In Optimal Incentive Contracts. The Quarterly Journal of Economics, 109, (4), 1125-1156.
Banker, R. D., & Datar, S. M. (1989). Sensitivity, precision, and linear aggregation of signals for performance evaluation. Journal of Accounting Reserach, 27 (1), 21-39.
Bol, J. C. (2008). Subjectivity in Compensation Contracting. Journal of Accounting Literature, 27, 1-24. Bol, J. C., & Smith, S. D. (2011). Spillover Effects in Subjective Performance Evaluation: Bias and the
Asymmetric Influence of Controllability. The Accounting Review, 86 (4), 1213-1230. Bommer, H. W., Johnson, L. J., Rich, A. G., Podsakoff, M. P., & Mackenzie, B. S. (1995). On the
interchangeability of objective and subjective measures of employee performance: a meta-analysis. Personnel Psychology, 48, 587-605.
Bond, S. D., Carlson, K. A., Meloy, M. G., Russo, J. E., & Tanner, R. J. (2007). Information distortion in the evaluation of a single option. Organizational Behavior and Human Decision Processes, 102, 240-254.
De Vocht, A. (2013). Basishandboek SPSS21. Utrecht: Bijleveld Press.
Eisenberger, R., Rhoades, L., & Cameron, J. (1999). Does pay for performance increase or decrease perceived self-determination and intrinsic motivation? Journal of Personality and Social Psychology, 5, 1026-1040.
Eisenhardt, K. M. (1989). Agency Theory: An Assessment and Review. The Academy of Management Review, 14 (1), 57-74.
Feltham, G., & Xie, J. (1994). Performance measure congruity and diversity in multi-task principal/agent relations. The Accounting Review, 69 (3), 429-453.
Fisher, J. G., Maines, L. A., Peffer, S. A., & Sprinkle, G. B. (2005). An experimental investigation of employer discretion in employee performance evaluation and compensation. The Accounting Review, 80 (2), 563-583.
Franco-Santos, M., Kennerley, M., Micheli, P., Martinez, V., Mason, S., Marr, B., Neely, A. (2007). TOWARDS A DEFINITION OFA BUSINESS PERFORMANCE. International Journal of Operations and Production Management, 27 (8), 784-801.
Garland, H. (1973). The effects of piece-rate underpayment and overpayment on job performance: A test of equity theory with a new induction procedure. Journal of Applied Social Psychology, 3, 325-334.
Ghosh, D., & Lusch, R. F. (2000). Outcome effect, controllability and peformance evaluation of managers: some field evidence from multi-outlet businesses. Accounting, Organizationans and Society, 25, 411-425.
27
Greenberg, J. (1986). Determinants of Perceived Fairness of Performance Evaluations. Journal of Applied Psychology, 71 (2), 340-342.
Harris, M., & Raviv, A. (1979). Optimal incentive cotnracts with imperfect information. Journal of Economic Theory, 20, 231-259.
Hemmer, T. (1996). On the Design and Choice of 'Modern' Management Accounting Measures. Journal of Managment Accounting Research, 8, 87-116.
Huber, V. L., Neale, M. A., & Northcraft, G. B. (1987). Judgment by Heuristics: Effects of Ratee and Rater Characteristics and performance Standards on Performance-Related Judgments. Organizational Behavior and Human Decision Processes, 40, 149-169.
Hunter, J. E., & Hunter, R. F. (1984). Validity and Utility of Alternative Predictors of Job Performance. Psychological Bulletin, 96 (1), 72-98.
Ittner, C. D., & Larcker, D. F. (2002). Determinants of Performance Measure Choices in Worker Incentive Plans. Journal of Labor Economics, 20 (2), 58-90.
Jensen, M., & Meckling, W. (1976). Theory of the firm: Managerial behavior, agency costs, and ownership structure. Journal of Financial Economics, (3), 305-360.
Kaplan, R. S., Atkinson, A. A., Young, S. M., & Matsumura, E. M. (2012). Managment Accounting information for decision making and strategy executuion. London: Pearson Education Limited.
Levy, P. a. (2004). The Social Context of Peformance Appraisal: A review and framework for the future. Journal of Management, 30 (6), 881-905.
McDaniel, M. A., & Schmidt, F. L. (1985). A meta-analysis of the validity of training and experience ratings in personnel selection. U.S. Office of personnel Management.
McDaniel, M. A., Schmidt, F. L., & Hunter, J. E. (1988). Job Experience Correlates of Job Performance. Journal of Applied Psychology, 73 (2), 327-330.
Merchant, K. A., & Van der Stede, W. A. (2012). Management Control Systems Performance Measurement, Evaluation and Incentives. Financial Times Prentice Hall.
Murphy, K. R. (1995). Understading Performance Appraisal: Social, Organizational, and GoalBased Perspectives. Sage Publications, Thousand Oaks, CA.
Neely, A., Gregory, M., & Platts, K. (1995). Performance measurement. International Journal of Operations & Production Management, 15 (4), 80-116.
Prendergast, C. (1999). The provision of incentives in firms. Journal of Economic Literature, 37 (1), 7-63.
Prendergast, C. and Topel, R.H. (1993). Discretion and bias in performance evaluation. European Economic Review, 37 (2-3), 355-365.
28
Prendergast, C., and Topel, R.H. (1996). Favouritism in organizations. Journal of Political Economy, 104 (5), 958-978.
Rajan, M. v., & Reichelstein, S. (2006). Subjective Performance Indicators and Discretionary Bonus Pools. Journal of Accounting Reserach, 44 (3), 585-618.
Russo, J. E., Margaret, G. M., & Jeffrey, W. T. (2000). Predecisional Distortion of Information by Auditors and Salespersons. Informs, 46 (1), 13-27.
Saal, F., Downy, R. G., & Lahley, M. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88 (2), 413-428.
Schmidt, F. L., Hunter, J. E., & Outerbridge, A. N. (1986). Impact of Job Experience and Ability on Job Knowledge, Work Sample Performance, and Supervisory Ratings of Job Performance. Journal of Applied Psychology, 71 (3), 432-439.
Sloof, R., & Praag, C. v. (2005). Performance measurement, expectancy and agency theory: An. Tinbergen Institute Discussion Paper, 1-26.
Terence, R. M., & Laura, S. K. (1981). Effects of Outcome Knowledge and Outcome Valence. Journal of Applied Psychology, 66 (5), 604-612.
Woods, A. (2012). Subjective Adjustments to objective performance measures: The influence of prior performance. Accounting, Organizations and Society, 37, 403-425.
29
Appendix
Case
Your role:
For purpose of this study, you are asked to assume the role of CEO of Floor BV. Your task is to evaluate the performance of a Brand manager (Brand manager X) in the year 2015.
Setting
Floor BV is a production company in the Netherlands, which produce quality floors of reputable brands. At the head of the company are the CFO and the CEO, one of their tasks is to evaluate their brand managers. The brand managers are responsible for production and sales of a specific brand.
The overall evaluation is based on:
Financial performance
Leadership
It is the task of the CFO to evaluate the brand manager’s financial performance. Your task (as the CEO) is to evaluate brand managers on their leadership performance.
Performance evaluation system
The overall evaluation of 2015 is based on two dimensions:
1. An objective performance measure on a 1 to 10 scale, based on the amount of profit/ revenues from sales. This measure is perceived to be a reasonable, but not perfect / excellent indicator of the degree in which the actions of the brand manager build firm value. It is also known that the amount of profit/ revenues from sales can be affected by the impact of external events such as the economic climate. Given that the brands are high quality, selling prices are relatively high and therefore strongly impacted by economic recessions.
2. In 2015 the brand manager supervised an implementation project to improve the efficiency of the production line. This project will be evaluated with a subjective performance measure also on a 1 to 10 scale (with 1 indicating exceptionally low performance and 10 exceptionally high performance).
30
The overall evaluation is based on both components. Note that the components are
independent, which means that performance on amount of profit/ revenues from sales is not
related to leadership performance. Both components are evaluated on a 1-10 scale, the weighted average representing the overall evaluation.
Your task:
Your task is to evaluate the leadership performance of Brand manager X in 2015.
The CFO has already evaluated the financial performance of Brand manager X in 2015. This evaluation is based on the revenue / profit realized in 2015, as it appears in the profit and loss account below.
Profit and loss account for brand managed by Brand manager X
Gross Revenue (sales to dealers) € 825.000 Discounts given to customers € 100.000 -/- Cost of goods sold € 425.000 -/- Other costs € 55.000 -/-
Amount of profit € 245.000
Based on these results, and following the evaluation scale below, the CFO has evaluated the financial performance of Brand manager X with a score of 2/9 out of 10.
1 2 3 4 5 6 7 8 9 10 < € 50.000 € 50.000 – € 75.000 € 75.000 – € 100.000 € 100.000 – € 125.000 € 125.000 – € 150.000 € 150.000 – € 175.000 € 175.000 – € 200.000 € 200.000 – € 225.000 € 225.000 – € 250.000 > € 250.000
Remember: the financial measure is perceived to be an excellent indicator of the degree in
which the actions of the brand manager build firm value.
1 2 3 4 5 6 7 8 9 10 < € 150.000 € 150.000 – € 250.000 € 250.000 – € 300.000 € 300.000 – € 350.000 € 350.000 – € 400.000 € 400.000 – € 500.000 € 500.000 – € 600.000 € 600.000 – € 700.000 € 800.000 – € 900.000 > € 900.000
31 1 2 3 4 5 6 7 8 9 10 < € 200.000 € 200.000 – € 400.000 € 400.000 – € 500.000 € 500.000 – € 600.000 € 600.000 – € 650.000 € 650.000 – € 700.000 € 700.000 – € 750.000 € 750.000 – € 800.000 € 800.000 – € 850.000 > € 850.000
Remember: the financial measure is perceived to be a reasonable indicator of the degree in
which the actions of the brand manager build firm value.
1 2 3 4 5 6 7 8 9 10 < € 750.000 € 750.000 – € 850.000 € 850.000 – € 900.000 € 900.000 – € 950.000 € 950.000 – € 1.000.000 € 1.000.000 – € 1.100.000 € 1.100.000 - 1. 200.000 € 1.200.000 – € 1.300.000 € 1.300.000 – € 1.400.000 > € 1.400.000
Brand manager leadership performance:
The subjective evaluation is focused on the leadership that the brand manager displayed when supervising the project of implementing a new more efficient routing that should lead to substantial cost savings. Interviews with the subordinates of the brand managers, indicates that the brand manager was able to influence and motivate them. This resulted in that the subordinates were able to implement the project on time, unfortunately the brand manager was not able to solve conflicts between the subordinates, which resulted in tensioned employees.
Your evaluation:
The following question is based on the information provided about brand manager X. How would you evaluate the leadership performance of the brand manager, on a scale of 1-10?:
1 2 3 4 5 6 7 8 9 10
32
Post-experimental questions
1. What was the objective measure’s score for the manager?;
A) 2 B) 9
C) I don’t remember
2. What was the relation between the objective and subjective performance evaluation?
a) Dependent b) Independent c) I don’t remember
3. What was the degree in which an increase in the objective measure also implies that firm value was increased?
a) Excellent b) Reasonable c) I don’t remember
4. Do you think that the information about the leadership performance was sufficient to carry out the subjective evaluation?;
0 yes 0 no
5. How would you rate your confidence in the evaluation?;
1 2 3 4 5 6 7 8 9 10
1= Extremely not confident 10 = extremely confident
6. Did you ‘correct’ for uncontrollable effects?;
0 yes 0 no
33
Demographic questions:
1. What is your age? ………. 2. Are you a man or a woman? ………...
3. What is the highest level of education you completed? ………. 4. What is your nationality?...
5. Have you ever had a supervisor function? ……… 6. Have you ever been subject of a formal performance evaluation in a work
