Objective performance measures’ susceptibility to manipulation versus their spill-over effect on subjective performance evaluations

(1)

Amsterdam Business School

Objective performance measures’ susceptibility to manipulation

versus their spill-over effect on subjective performance evaluations

Name: Daan Willers

Student number: 10854789 Thesis supervisor: Peter Kroos

Date: 16 June 2016

Word count: 8938

MSc Accountancy & Control, control track

(2)

2 Statement of Originality

This document is written by student Daan Willers who declares to take full responsibility for the contents of this document.

I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.

The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.

(3)

3 Abstract

Objective performance measures can have discretional spill-over effects on unrelated subjective evaluations of performance. It is therefore important to know what the factors are that determine the strength of this spill-over effect. This thesis reports the results of an experiment to investigate whether the susceptibility to manipulation of an unrelated objective performance measure affects the spill-over effect from its score on a subjective evaluation. Participants in the experiment were economic and business students from Dutch universities. My data confirms what is found by prior research about the existence of a spill-over effect, but did not support the expectation that the strength of the spill-over effect decreases when the susceptibility to manipulation increases. The expected effect only occurs in situations where the objective score is low, but not when its high, and therefore did not result in a significant support. The findings of this research contribute to our understanding of the factors which influence the subjective evaluation process by supervisors. They are also of interest to firms that want to optimize their performance evaluation system.

KEYWORDS: Performance evaluation; subjectivity; spill-over effect; cognitive distortion; fairness; discretion.

(4)

4 Contents

1 Introduction ... 5

1.1 Background ... 5

1.2 Research question ... 5

1.3 Relevance of the question... 6

1.4 Structure of the thesis ... 6

2 Literature and hypotheses development ... 7

2.1 Agency theory and performance measurement ... 7

2.2 Objective performance measurement ... 7

2.3 Subjectivity ... 8 2.4 Spill-over effect ... 9 3 Research methodology ... 11 3.1 Research design ... 11 3.2 Experimental task ... 12 3.3 Independent variables ... 12 3.4 Control variables ... 13 4 Results ... 15 4.1 Descriptives ... 15 4.2 Hypothesis 1 ... 15 4.3 Hypothesis 2 ... 17

4.4 Post Hoc Tests ... 19

4.5 Manipulation and other control checks ... 20

5 Discussion and conclusion ... 22

(5)

5

1 Introduction

1.1 Background

Performance evaluation of an employee or business division is often based on one or more performance measure(s). These measures can have different characteristics. The most common difference is the one between objective and subjective performance measures. Objective measures are based on results, subjective measures are based on the supervisor’s observation and interpretation of the employees’ behaviour (Hoffman and Nathan, 1991). A performance measurement system often consists of a set of both objective and subjective measures (Prendergast, 1999; Gibbs et al., 2004). Subjective measures can be used complementary to address shortcomings of objective measures (Baker et al., 1994). Subjectivity in the performance measurement system is useful in reducing risk to employees and improving the congruence (alignment of the employees’ incentives with the firm’s interest) (Baker et al., 1994; Bushman et al., 1996; Hayes and Schaefer, 2000).

Prior research has found a spill-over effect from objective performance measures on subjective evaluations, because of managerial discretion (Bol and Smith, 2011; Murphy and Oyer, 2003). In other words, this means that a higher score on an objective measure can have a discretional spill-over effect on the subjective performance measure, even though these measures should be complementary. Subjective and objective performance measures are used complementary when they are supposed to capture the performance of separate actions. However, earlier psychological studies argue that subjective judgements can be influenced by information known by the supervisor, other than information where the subjective measure is supposed to be based on (Nisbett et al., 1981; Bond et al., 2007). This discretion can harm the complementary role, because in that case the objective and subjective measures in fact aren’t completely separate.

1.2 Research question

Bol and Smith (2011) conducted research to deepen the understanding of the spill-over effect from an unrelated objective measure on a subjective measure. They added the controllability by the employee of an unrelated objective measure as a moderating variable into the model, and found that controllability influences the magnitude of the spill-over effect. However, the level of controllability is just one of the criteria to evaluate the quality of performance measures described by prior research. One of the other evaluation criteria used in prior literature is the susceptibility of the performance measure to manipulation (Banker and Datar, 1989; Feltham and Xie, 1994). Because prior research only focused on the level of controllability as a moderating variable, it is

(6)

6 interesting to investigate whether or not those other quality criteria of objective performance measures also have a significant influence on the magnitude of the spill-over effect. In this research I particularly focus on the susceptibility to manipulation of the objective performance measure.

In order to extend the literature contribution added by Bol and Smith (2011), I conduct a research with the following research question:

Whether and how much does the spill-over effect from an unrelated objective performance measure on a subjective performance measure vary with different levels of susceptibility to manipulation of the objective performance measure?

1.3 Relevance of the question

This study contributes to the literature in several ways. Subjective evaluation plays an important role in practice above and beyond the role of objective measures in the evaluation process. For the practical contribution of subjectivity in evaluation systems, it is however important that subjective measures incorporate incremental information not reflected in objective measures. In short, this warrants further research into the factors that influence to what extent objective performance information influence the outcomes of subjective measures.

Prior research has found that subjective evaluations by supervisors may be influenced by prior performance information (Murphy et al., 1985; Huber et al., 1987; Kravitz and Balzer, 1992). Furthermore, information from different sources could also influence the subjective evaluation (Blakely, 1993; Murphy and Cleveland, 1995; Bono and Colbert, 2005). The study of Bol and Smith (2011) adds to this literature by showing that cognitive distortion can cause a spill-over effect from unrelated objective performance measures on subjective evaluations, and that this effect is moderated by the level of controllability of the objective performance measure. This study contributes to prior research by also examining the moderating effect of another quality aspect of performance measures, namely the susceptibility to manipulation of the objective performance measure.

1.4 Structure of the thesis

The remainder of this thesis is as follows. In the second section the literature review is included, as well as the hypotheses development. Third, the section on the research methodology discusses which method is used to conduct this research. The fourth section will give an overview of the results, followed by the conclusions in section five. This section will also include the implications of this research and suggestions for future research.

(7)

7

2 Literature and hypotheses development

2.1 Agency theory and performance measurement

Many large firms nowadays are characterized by a separation of ownership and control. Advantages of this separation are that the management is specialized, and that risk can be diversified over the investors. A disadvantage of the separation, however, is the agency problem because of information asymmetry between agents (e.g. top management) and principals (e.g. investors) (Rees, 1985). Agency problems are prevalent in organizations and do not only occur between investors and top management, but also manifest themselves at lower levels in the organization between seniors and subordinates.

A specific problem associated with information asymmetry is moral hazard. Moral hazard arises when managers have private information on their actions and decisions taken that is only partially available to investors (Bol, 2008). To reduce moral hazard, firms often rely on performance measures to get some indication whether managers have taken the right actions and decisions. In addition, firms often attach monetary incentives to compensation contracts to provide managers already beforehand with the incentives to take the desired actions and decisions.

When linking compensation to performance the principal transfers risk to the agent, where the agent needs to be compensated for (Bol, 2008). Therefore, optimal compensation contracts are a trade-off between the inducement of congruent behaviour and minimizing the amount of risk the agent has to bear. To make this trade-off as efficient as possible, the measures used in the system need to capture a maximal extent of the agent’s effort. This results in performance measurement systems with a variety of measures, which all add incremental information of different segments of the agent’s performance (Holmström, 1979).

2.2 Objective performance measurement

Objective performance measures are based on facts (e.g. periodic results). These can be financial (e.g. amount of sales), but also non-financial (e.g. average customer evaluation). Objective performance measures produce periodic outcomes that should provide some information about the actions and decisions that managers have taken. The quality of performance measures depends on several criteria that indicate the informativeness of the measures. Typically, organizations put a greater weight on objective measures when they are more sensitive, less noisy, more congruent and less susceptible to manipulation. Sensitivity of a measure describes the extent in which managerial effort affects the outcome. Noisy measures are influenced by outside factors which lay outside the control of the manager. Congruency is the alignment of the measure with the firm’s objectives.

(8)

8 Furthermore, the susceptibility to manipulation of the measure is important because it says something about the reliability of the outcome value. However, not every action can be measured objectively and a system that is exclusively consisting of objective performance measures features several other potential drawbacks. Therefore, subjectivity is often used to supplement objective measures in compensation contracts.

2.3 Subjectivity

Subjectivity is based on impressions, feelings and opinions (Bol, 2008). Subjectivity can be incorporated in performance measurement systems in three ways. The first method is to include subjective performance measures in compensation contracts, secondly to allow some flexibility in the weighting of objective measures, and thirdly by allowing ex-post discretional adjustments based on other factors than the factors which are already measured objectively. Including more measures in compensation contracts is only beneficial if it increasingly induces managers to take actions that are within the firm’s interest (Banker and Datar, 1989; Feltham and Xie, 1994).

Subjectivity in performance measurement systems has multiple benefits. One is the mitigation of incentive distortion, because subjectivity has the ability to capture value-enhancing efforts. It allows the supervisor to discretionally decide ex-post what information is relevant. It can also make contracts more complete by allowing for the possibility to contract on certain effort dimensions that cannot be measured objectively. Furthermore, if the evaluation is only based on an objective measure which can also be influenced by outside factors, the agent has to bear a risk on that. With the introduction of subjectivity into the contract this risk can be reduced (Holmström, 1979; Banker and Datar, 1989). Subjectivity also prevents suboptimal incentives during the contractual period when relevant information cannot completely be foreseen (Bol, 2008). Since objective performance measures are numeric, they can be susceptible to manipulation. Subjectivity can be used to limit this vulnerability by allowing the supervisor to make discretional adjustments in case manipulation is detected. Last, because a compensation contract with subjectivity captures more of the agent’s effort, and also reduces the agent’s risk, it will higher the perceived fairness of the evaluation process.

However, prior research has also found that the use of subjectivity in performance measurement systems doesn’t always have the desired benefits in terms of complementing the objective performance measures. One of the most discussed flaws of subjectivity is reneging (Bol, 2008). Supervisors could not follow up earlier made agreements based on subjective outcomes, because the subjective evaluation cannot be verified by outsiders (Prendergast and Topel, 1993). This means that in some situations the informativeness and incentive-strengthening role of

(9)

9 subjective evaluations could be limited (Prendergast and Topel, 1996; Bol, 2008; Bol and Smith 2011).

Because the supervisor has the ability to take personal preferences into account while evaluating performance (favouritism), the agent may intend to influence the evaluation by making himself appear friendly to the supervisor (Holmström, 1982; Higgins et al., 2003). The agent may also try to outperform on the actions which are visible to the supervisor only, at the cost of other valuable but less visible actions (Milgrom, 1988; Prendergast and Topel, 1996; Bol, 2008).

In objective evaluations the agent knows which performance is ought to be important. With subjectivity included in the contract, there might be uncertainty of measurement criteria. The expectancy theory (Vroom, 1964; Heneman and Schwab, 1972) explains that agents will not be motivated to increase their efforts when they don’t understand which specific actions and behaviour will lead to increased compensation.

Subjectivity can also cause inaccurate assessments. A subjective evaluation can contain a centrality bias, where the supervisor is reluctant to sufficiently distinguish between employees within a team, for example. A possible reason for this is the fear of harming the existing team spirit. Another accuracy issue is the leniency bias. This results from supervisors who are reluctant to carry out poor evaluations to subordinates, because they feel that bad evaluations might demotivate rather than motivate the employees (Landy and Farr, 1980; Murphy and Cleveland, 1991). Supervisors can also unintentionally bias the subjective evaluation consistent with prior evaluations or the unrelated objective performance measure (Wilks, 2002; Bond et al., 2007). This phenomenon can be explained by the theory of cognitive information distortion. According to this theory cognitive distortion occurs when specific information or knowledge causes an individual (in this context the supervisor) to interpret reality inaccurately (Nisbett et al., 1981). Inaccuracy in performance assessments can be harmful for the firm, because it will reduce the effectiveness of incentives (Baker et al., 1988). Further, it may lower the perceived fairness of evaluations and may stimulate suboptimal behaviour. Finally, inaccurate performance assessments can cause inefficient personnel decisions (Jawahar and Williams, 1997). For example, wrong personnel can be promoted, while others with high potentials may not be identified (Prendergast, 1999).

2.4 Spill-over effect

Bol and Smith (2011) argue that performance measure quality indicators of the objective performance measure, such as controllability, affect the spill-over effect from an unrelated objective performance measure on a subjective evaluation. They find that when there is some degree of uncontrollability, principals are more inclined to increase the outcome of the subjective

(10)

10 evaluation to increase the fairness of the overall performance evaluation. Ghosh and Lusch (2000) document the outcome effect where stores receive a more negative subjective evaluation when their performance on the objective performance is lower. They explain this through the observation that knowledge about objective outcomes influences the evidence recalled by the evaluator when one is attempting to subjectively assess the performance. On the basis of the aforementioned, I test for the presence of the actual spill-over effect in this research setting, with the following hypothesis:

H

1

Supervisors’ subjective performance evaluations will be higher when the employee’s

performance on an unrelated objective measure is higher.

It has been found by prior research that the magnitude of the spill-over effect from unrelated objective measures on subjective evaluations is higher when the supervisor finds the objective measure from greater importance (Bol and Smith, 2011). A measure is from greater importance, when it is strongly affected by the agent’s effort. This is realised, for example, when the measure is strongly controllable by the employee, or isn’t susceptible to manipulation. In these situations, the informativeness is higher and the outcome has more value for the supervisor. Therefore, I expect that when the unrelated objective measure is highly susceptible to manipulation (known by the supervisor), the spill-over effect will be less strong because the measure’s outcome is from less importance in the opinion of the supervisor. That is, objective information that is susceptible to manipulation is less likely to influence the evidence recalled by the evaluator when subjectively assessing performance (Ghosh and Lusch, 2000).

Furthermore, Bol and Smith (2011) describe how the objective performance can influence the subjective evaluation in the case of low controllability. The underlying explanation is that evaluators try to improve the fairness when other factors influence objective performance. The question arises how the susceptibility to manipulation affects fairness considerations of the evaluator. A high susceptibility to manipulation could offset the low controllability, providing the evaluated subordinate with the means to alter reported objective performance levels. That is, fairness considerations may play less of a role if susceptibility to manipulation of the objective measure is high. So, I expect the spill-over effect to be smaller in the case of a high susceptibility to manipulation. This results in the second hypothesis:

H

2

The spill-over effect from the employee’s performance on an unrelated objective

measure towards supervisors’ subjective performance evaluations will be lower when

the objective measure is more susceptible to manipulation.

(11)

11

3 Research methodology

3.1 Research design

I conduct this research with a case-based experiment. Participants had to complete a performance evaluation task, fulfilling the role of CEO of a Dutch shoe trading company. One of the CEO’s responsibilities was to evaluate the sales centre managers on their annual performance composed of two unrelated tasks that are measured objectively and subjectively, respectively. The participants had to evaluate one of the sales centre managers on a subjective base, and therefore received information about the manager’s performance. This information included a story representing the performance on the subjective measure, but also additional facts like the outcome of an unrelated objective measure and its susceptibility to manipulation. The experiment is structured in a 2x2 design. As summarized in table 1, the level of the objective measure is manipulated (high and low), as well as the susceptibility to manipulation (high and low). The four different case descriptions can be found in the appendix.

Table 1: Conditions

High objective score Low objective score Low susceptibility to manipulation Condition 1 (C1) Condition 2 (C2) High susceptibility to manipulation Condition 3 (C3) Condition 4 (C4)

I test both hypotheses using data from 112 economics and business students at Dutch universities. They were invited by email and/or a message on social media, in which they were asked to voluntarily participate in the research by following an URL. The students did not receive any form of compensation for their participation. I only used participants who were able to recall the sales centre manager’s objective score and completed both the subjective evaluation and every control question. Thus, the final sample consists of 60 students. Of the participants in the final sample, 66,7% were male and 33,3% were female, with an average age of 25,65.

To conduct this research, I used Qualtrics software because it offers a function to randomly assign one of the different case descriptions to a participant. A one-way ANOVA with a Bonferroni

(12)

12 post-hoc test shows that the ratio male/female is not significantly different in any of the different conditions. Thus, the randomization succeeded.

3.2 Experimental task

The case introduction was the same for all participants. They were asked to fulfil the role of CEO for a Dutch shoe trading company. The company consists of multiple sales centres spread over the country from where shoes are sold and distributed to customers.

Each sales centre has its own manager, whose responsibilities are described in the case, together with the performance measures used to monitor and evaluate them. The performance measurement system in this company consists of both an objective measure based on the gross sales of the manager, and a subjective evaluation of the manager’s distribution performance. The description will clearly indicate that these two components are completely independent, because of the different nature of the responsibilities. This is to make sure that the participant recognizes that the measures are unrelated.

Each participant is asked to carry out a subjective evaluation of the distribution performance of one of the sales centre managers on a scale from 1 to 10, based on information provided about a distribution improvement project of last year. This information was the same for all participants. After carefully reading the case, participants had to give the sales centre manager a subjective evaluation based on the distribution efficiency improvement project performance, on a scale from 1 to 10.

3.3 Independent variables

In the experiment the unrelated objective measure is operationalized as a score on a 1 to 10 scale, representing the gross sales level. That is, different gross sales levels are converted to a score ranging between 1 and 10. The performance on this measure scores 2 in case the objective measure is low, and 8 is case the objective measure is high.

The moderating variable is given to participants in a short description of the manager’s decision rights. In the case context the manager is evaluated based on gross sales, which is determined by the total revenues before subtraction of given discounts and uncollectible receivables. Decision rights to give discounts could incentivize a manager to manipulate. Although giving a lot of discounts may not be beneficial for the firm, it will increase the gross sales and thus the manager’s performance evaluation. Furthermore, if a manager has decision rights to give customers more lenient credit terms on their orders, there is an incentive as well to make use of that. On the one hand, it will increase the gross sales and thus the manager’s evaluation, but it

(13)

13 won’t add or might even lose value for the firm because it will take longer period of time to collect receivables and in some cases receivables may not be collected at all. Therefore, the fact that the manager has, or has no decision rights on discounts and credit terms, is used as variable to manipulate the susceptibility of the objective performance measure to manipulation.

3.4 Control variables

Apart from the dependent and independent variables, some control questions were included in the end of the experiment to test if the operationalization of variables worked out as supposed. Participants were asked to rate their perceived likelihood of manipulation of the objective measure, to test if the susceptibility to manipulation was successfully operationalized. Also, participants had to indicate to which extent they agree with the statement that the objective and subjective performance measure are completely unrelated. To test whether participants found that they were able to perform a thought-out evaluation, I included questions about their confidence in the evaluation they gave, and whether they perceived the information provided in case descriptions to be sufficient. Two final questions covered the participant’s age and gender. A summary of all variables used in my research, including their descriptive statistics, is provided in table 2.

(14)

14 Table 2:

Overview of variables

Category and name Theoretical values Range Mean SD

Dependent variable:

Subjective evaluation Scale: 1-10 7 7,02 1,771

Independent variables:

Score on unrelated objective measure

1 = 2 (low gross sales); 2 = 8 (high gross sales)

- - -

Susceptibility to manipulation 1 = low susceptibility; 2 = high susceptibility

- - -

Control variables:

Age - 41 25,65 7,287

Gender 1 = male; 2 = female - 1,33 0,475

Evaluation confidence Scale: 1-7 5 5,08 1,169

Perceived likelihood of manipulation

Scale: 1-7 6 5,42 1,239

Perceived unrelatedness objective and subjective measure

Scale: 1-7 6 3,60 1,796

Perceived information sufficiency

(15)

15

4 Results

4.1 Descriptives

The descriptives of the research sample are presented per condition in table 3. Apart from the number of participants, all descriptives are about the subjective evaluation which the participants carried out on a scale from 1 to 10 (the dependent variable).

Table 3: Descriptives Condition N Mean Std. Deviation Std. Error 95% Confidence Interval for Mean Minimum Maximum Lower Bound Upper Bound C1 13 7,77 1,013 ,281 7,16 8,38 5 9 C2 15 5,53 2,356 ,608 4,23 6,84 2 8 C3 19 7,84 ,898 ,206 7,41 8,28 6 9 C4 13 6,77 1,481 ,411 5,87 7,66 3 9 Total 60 7,02 1,771 ,229 6,56 7,47 2 9 4.2 Hypothesis 1

The first hypothesis predicts that the subjective performance evaluations carried out by participants will be higher when the sales centre manager’s score on the objective measure is higher. As you can read from table 1, this first hypothesis is supported if the combination of condition C1 and C3 is higher compared to the combination of condition C2 and C4. If you take a look at the descriptives in table 3 it seems to be that this is indeed the case, because the means are higher (7,77; 7,84) in conditions C1 and C3 than the means in conditions C2 and C4 (5,53; 6,77). However, this result should be tested to make sure that the difference is also statistically significant.

Because H1 tests for the relationship between the objective score (categorical) and

subjective evaluation (quantitative), I conduct a one-way ANOVA. One of the main assumptions that has to be met in order to perform an ANOVA, is the homogeneity of variances across the conditions. In table 4 it becomes clear that these variances are not equal, because the Levene

(16)

16 Statistic gives a significant p-value (0,000). This can be caused by the fact that there are in fact no equal variances within the groups, but it is also true that the Levene Statistic loses power when the sample size is small (which is the case in this research). Therefore, I also perform some other tests to check whether or not I can continue with my ANOVA. In table 5, the results from the Welch and Forsythe tests are given. Both the results of the Welch and the Brown-Forsythe test indicate that we can assume that there are significant (both p=0,000) differences between the groups. Thus, the ANOVA can be carried out.

Table 4:

Test of Homogeneity of Variances

Levene Statistic df1 df2 Sig.

17,458 1 58 ,000

Table 5:

Robust Tests of Equality of Means

Statistica df1 df2 Sig. Welch 16,267 1 36,492 ,000 Brown-Forsythe 16,267 1 36,492 ,000 a. Asymptotically F distributed. Table 6: ANOVA

Sum of Squares df Mean Square F Sig.

Between Groups 43,430 1 43,430 17,795 ,000

Within Groups 141,554 58 2,441

(17)

17 Table 7:

Contrast Coefficients and Tests

Objective score 1 2

Contrast -,5 ,5

Value of

Contrast Std. Error t df Sig. (2-tailed) Does not assume equal

variances ,85 ,211 4,033 36,492 ,000

The results from table 6 suggests that there is a significant (p=0,000) difference between high objective score and the low objective score conditions, compared on subjective evaluation means. I included a planned contrast (see table 7) in the ANOVA as well, because my hypothesis is formulated one-sided. That is, the hypothesis is only supported when subjective evaluations are significantly higher in high objective score conditions, not when they are lower. The contrast in the direction of H1 is significant, which means that the first hypothesis is supported. I used the

contrast results which does not assume equal variances in the analysis, due to the significant Levene’s statistic in the test of homogeneity of variances (see table 4).

4.3 Hypothesis 2

Because the test of the second hypothesis includes two independent variables which consist of two categories, and the dependent variable (subjective evaluation) is quantitative, I test the hypotheses with a linear regression. A support for this hypothesis is realized when the spill-over effect in the low susceptibility to manipulation conditions (C1 and C2) is stronger than the spill-over effect in the high susceptibility to manipulation conditions (C3 and C4).

As appears from the results in table 8, 29,3% of variance in subjective evaluations can be explained by the regression model. This model consists of the following components:

Subjective evaluation = α + ß1 * Objective score + ß2 * Susceptibility to manipulation + ß3 * Objective score*Susceptibility to manipulation + ε

When looking at the regression of variable Objective score*Susceptibility to manipulation we see a negative value for b* of -0,164. That is, when the susceptibility to manipulation becomes high, the relationship between the objective score and the subjective evaluation will become less

(18)

18 strong. However, this result is not significant (p>0,05) and therefore I am not able to support hypothesis H2.

Table 8: Linear regression

Model Summary

Model R Square R Adjusted R Square

Std. Error of the Estimate

Change Statistics R Square

Change Change df1 F df2 Change Sig. F

1 ,541a _,293 _,255 _1,529 _,293 _7,717 ₃ ₅₆ _,000

a. Predictors: (Constant), Objective score*Susceptibility to manipulation, Susceptibility to manipulation (centred), Objective score (centred)

ANOVAa

Model Sum of Squares df Mean Square F Sig.

1 Regression 54,108 3 18,036 7,717 ,000b

Residual 130,875 56 2,337

Total 184,983 59

a. Dependent Variable: Subjective evaluation

b. Predictors: (Constant), Objective score*Susceptibility to manipulation, Susceptibility to manipulation (centred), Objective score (centred)

Coefficientsa

Model

Unstandardized

Coefficients Standardized Coefficients

t Sig.

B Std. Error Beta (b*)

1 (Constant) 7,054 ,199 35,442 ,000

Objective score (centred) 1,616 ,399 ,459 4,049 ,000

Susceptibility to

manipulation (centred) ,616 ,399 ,175 1,543 ,128

Objective

score*Susceptibility to

manipulation -1,163 ,799 -,164 -1,456 ,151

(19)

19

4.4 Post Hoc Tests

The second hypothesis did not get support, but when looking at table 3 there seems to be some (weak) evidence. According to table 3 the drop in strength of the spill-over effect is caused by the conditions with a low objective score (C2 and C4). To analyse this more in depth I ran a Post Hoc analysis to see where significant differences are when comparing all four conditions. The results are presented in table 9. The conditions in the low susceptibility to manipulation conditions (C1 and C2) contain a significant (p=0,016) spill-over effect, but nothing significant (p>0,05) is going on in the high susceptibility to manipulation conditions (C3 and C4). However, the assumption that the effect on the strength of the spill-over effect is only caused by the conditions with a low objective score is not significant. In short, there is not enough evidence to support H2. That is,

because the difference between C1 and C3 is not significant (p>0,05), but the difference between C2 and C4 is not significant as well (p>0,05).

Table 9:

Post Hoc Tests, Multiple Comparisons

Dependent Variable: Subjective evaluation Games-Howell

(I) Condition (II) Condition

Mean Difference

(I-II) Std. Error Sig.

95% Confidence Interval Lower Bound Upper Bound

C1 C2 2,236* _,670 _,016 _,36 _4,12 C3 -,073 ,348 ,997 -1,03 ,89 C4 1,000 ,498 ,216 -,39 2,39 C2 C1 -2,236* ,670 ,016 -4,12 -,36 C3 -2,309* _,642 _,011 _-4,13 _-,49 C4 -1,236 ,734 ,354 -3,26 ,79 C3 C1 ,073 ,348 ,997 -,89 1,03 C2 2,309* ,642 ,011 ,49 4,13 C4 1,073 ,459 ,127 -,23 2,37 C4 C1 -1,000 ,498 ,216 -2,39 ,39 C2 1,236 ,734 ,354 -,79 3,26 C3 -1,073 ,459 ,127 -2,37 ,23

(20)

20 The asymmetric results provided by table 9 are also visible in figure 1. The spill-over effect loses strength when increasing the susceptibility to manipulation. The asymmetry between the high and low objective score conditions is not according to my expectations. If there was already a one-sided effect in the results, I expected it to be in the conditions where the objective score is high instead of low. Both the results in table 9 and figure 1 suggest, however, that in reality it is the other way around. That is, in the conditions with high objective scores on the objective measure the subjective evaluation does not change with varying levels of susceptibility to manipulation.

Figure 1:

Mean subjective evaluation in high and low susceptibility to manipulation conditions

4.5 Manipulation and other control checks

I included some control questions in the experimental survey, which indicate whether the applied manipulations were adequate to test both the hypotheses. Participants significantly judged the susceptibility to manipulation of the objective measure to be higher in the condition where the sales centre manager has decision rights on discounts and credit terms, compared to the condition where he has no decision rights on those items (p=0,002). The mean of this control measure in C1 and C2 (the low susceptibility to manipulation conditions, was 4,89 compared to 5,88 in C3 and

(21)

21 C4 (the high susceptibility to manipulation conditions). Therefore, I can say that the variable operationalization of the susceptibility to manipulation of the objective measure was successful.

I asked respondents to rate (on a scale from 1 to 7) whether they felt confident about their subjective evaluation. The mean of this measure was 5,08, which is significantly higher than the scale midpoint of 4,0. Furthermore, I asked respondents to rate (also on a scale from 1 to 7) whether they agree with the statement that the sales performance and distribution efficiency improvement project are two completely unrelated tasks. The mean of this measure was 3,60, which is slightly lower than the scale midpoint. Last, I checked the respondent’s agreement with a statement that the information provided about the distribution efficiency improvement project was sufficient for them to carry out the subjective evaluation. The mean of this measure was 3,53, also slightly lower than the scale midpoint.

Finally, I checked the correlations between variables and found that age and gender are not significantly correlated with the manipulations and the dependent variable. It therefore seems to be that the randomization was effective and that there is no need to additionally explicitly control for age and gender in subsequent analyses.

(22)

22

5 Discussion and conclusion

I examined how an unrelated objective performance measure can influence supervisor’s subjective evaluation of performance on a separate dimension, and how this effect is influenced by the level of susceptibility to manipulation of that objective performance measure. The results confirm what is found by prior research about the existence of a spill-over effect from unrelated objective performance measures to subjective evaluations because of cognitive information distortion of supervisors.

The specific research question of this research is as follows:

Whether and how much does the spill-over effect from an unrelated objective performance measure on a subjective performance measure vary with different levels of susceptibility to manipulation of the objective performance measure?

I expected that the spill-over effect loses strength when the objective measure is susceptible to manipulation, because then supervisors will take its outcome less into account. This is explained by the theory section in paragraph 2.2, in which I mentioned that a supervisor takes the objective score more into account while evaluating the manager when this objective score has a higher informativeness. Apparently, supervisors perceive the informativeness of the measure to be lower in case it is susceptible to manipulation.

Unfortunately, the effects found within in the data are not strong enough to support this expectation. The results seem to be somewhat asymmetric when looking at the way supervisors adjust for perceived fairness, and are therefore not causing a significant support. Specifically, they discretionally adjust the evaluation of a manager with a low objective score upwards; the spill-over effect will lose strength in that case. When managers have scored high on the unrelated objective measure, little adjustments are made. The adjustment of the evaluation in case of a low objective score on a measure that is highly susceptible to manipulation, could be explained by perceived fairness of the supervisor. Because the outcome on the objective measure is low, and the measure is easy to manipulate, the manager has an incentive to do so. Nevertheless, he scored low and that gives a positive signal to the supervisor that the manager is trustworthy. This appreciation by the supervisor is then in turn valued in terms of a higher subjective evaluation.

Results of the control questions suggest that the subjective evaluation will be higher when the supervisor is confident about his evaluation. This confidence is positively related with the objective score and the perceived level of information sufficiency. Together, this confirms again

(23)

23 that supervisors value the informativeness of the objective performance measure and take its score into account while evaluating subjectively on a separate dimension.

The research setting of my case-based experiment provides a number of opportunities for future research. The results of the control checks suggest that participants felt confident about the evaluation they gave, but were not quite sure about the sufficiency of information provided in the case. Furthermore, the independence between the objective and subjective measure was not understood by all participants. This could all weaken the strength of the results and it is therefore useful for future research to search for better ways to operationalize the independent variables. Also, the sample size used in this research was quite small, and therefore future research with larger samples could result in more significant outcomes. The study of Bol and Smith (2011) and my own study both use one single quality indicator of performance measures as moderating variable of the spill-over effect. Thus, future research could also take into account the interaction of these quality indicators, while together affecting the spill-over effect.

(24)

24

References

Baker, G. P., M. C. Jensen, and K.J. Murphy. 1988. Compensation and Incentives: Practice vs. Theory. The Journal of Finance, 43(3): 593-616.

Baker, G. P., R. Gibbons, and K. J. Murphy. 1994. Subjective performance measures in optimal incentive contracts. Quarterly Journal of Economics, 109(4): 1125–1156.

Banker, R. D., and S. M. Datar. 1989. Sensitivity, precision, and linear aggregation of signals for performance evaluation. Journal of Accounting Research, 27(1): 21–39.

Blakely, G. L. 1993. The effects of performance rating discrepancies on supervisors and subordinates. Organizational Behaviour and Human Decision Processes, 54: 57–80.

Bol, J. C. 2008. Subjectivity in compensation contracting. Journal of Accounting Literature, 27: 1–32. Bol, J. C., and S. D. Smith. 2011. Spillover Effects in Subjective Performance Evaluation: Bias

and the Asymmetric Influence of Controllability. Accounting Review, 86(4): 1213-1230. Bond, S. D., K. A. Carlson, M. G. Meloy, J. E. Russo, and R. J. Tanner. 2007. Information

distortion in the evaluation of a single option. Organizational Behaviour and Human Decision Processes, 102: 240– 254.

Bono, J. E., and A. E. Colbert. 2005. Understanding responses to multi-source feedback: The role of core self-evaluations. Personnel Psychology, 58: 171–203.

Bushman, R. M., R. J. Indjejikian, and A. Smith. 1996. CEO compensation: The role of individual performance evaluation. Journal of Accounting and Economics, 21(2): 161–193.

Feltham, G. A., and J. Xie. 1994. Performance measure congruity and diversity in multi-task principal/agent relations. The Accounting Review, 69(3): 429–453.

Ghosh, D., and R. F. Lusch. 2000. Outcome effect, controllability and performance evaluation of managers: Some field evidence from multi-outlet businesses. Accounting, Organizations and Society, 25(4): 411-425.

Gibbs, M., K. A. Merchant, W. A. Van der Stede, and M. E. Vargus. 2004. Determinants and effects of subjectivity in incentives. The Accounting Review, 79(2): 409–436.

(25)

25 Hayes, R. M., and S. Schaefer. 2000. Implicit contracts and the explanatory power of top

executive compensation for future performance. RAND Journal of Economics, 31(2): 273– 293.

Heneman, H. G., and D. P. Schwab. 1972. Evaluation of research on expectancy theory prediction of employee performance. Psychological Bulletin, 78: 1-9.

Higgins, C. A., T. A. Judge, G. R. Ferris. 2003. Influence Tactics and Work Outcomes: A Meta-Analysis. Journal of Organizational Behavior, 24(1): 89-106.

Hoffman, C. C., and B. R. Nathan. 1991. A comparison of validation criteria: objective versus subjective performance measures and self- versus supervisor ratings. Personnel

Psychology, 44(3): 601-619.

Holmström, B. Moral Hazard and Observability. 1979. The Bell Journal of Economics, 10(1): 74-91. Holmström, B. Moral Hazard in Teams. 1982. The Bell Journal of Economics, 13(2): 324-340. Huber, V. L., M. A. Neale, and G. B. Northcraft. 1987. Judgment by heuristics: Effects of rate

and rater characteristics and performance standards on performance-related judgments. Organizational Behaviour and Human Decision Processes, 40: 149–169.

Jawahar, I., and C. Williams. 1997. Where All the Children Are Above Average: The Performance Appraisal Purpose Effect. Personnel Psychology, 50: 905-925.

Kravitz, D. A., and W. K. Balzer. 1992. Context effects in performance appraisal: A methodological critique and empirical study. Journal of Applied Psychology, 77: 24–31. Landy, F. J., and J. L. Farr. 1980. Performance rating. Psychological Bulletin, 87(1): 72-107.

Milgrom, P. 1988. Employment contracts, influence activities, and efficient organization design. Journal of Political Economy, 96(1): 42-60.

Murphy, K. J., W. K. Balzer, M. C. Lockhart, and E. J. Eisenman. 1985. Effects of previous performance on evaluations of present performance. Journal of Applied Psychology, 70(1): 72–84.

Murphy, K. J., and P. Oyer. 2003. Discretion in executive incentive contracts: Theory and evidence. Working paper, University of Southern California and Stanford University. Murphy, K. R., and J. N. Cleveland. 1991. Performance appraisal: An organizational perspective.

(26)

26 Murphy, K. R., and J. N. Cleveland. 1995. Understanding Performance Appraisal: Social,

Organizational, and Goal-Based Perspectives. Thousand Oaks, CA: Sage.

Nisbett, R. E., H. Zukier, and R. Lemley. 1981. The dilution effect: Nondiagnostic information weakens the implications of diagnostic information. Cognitive Psychology, 13: 248–277. Prendergast, C. 1999. The provision of incentives in ﬁrms. Journal of Economic Literature, 37(1): 7–

63.

Prendergast, C., and R. H. Topel. 1993. Discretion and bias in performance evaluation. European Economic Review, 37(2–3): 355–365.

Prendergast, C., and R. H. Topel. 1996. Favoritism in organizations. Journal of Political Economy, 104(5): 958–978.

Rees, R. 1985. The theory of principal and agent Part I. Bulletin of Economic Research, 37(1): 3-26. Vroom, V. H. 1964. Work and motivation. San Francisco, CA: Jossey-Bass.

Wilks, T. J. 2002. Predecisional distortion of evidence as a consequence of real-time audit review. The Accounting Review, 77(1): 51–71.

(27)

27

Appendix:

(28)

28 Case condition 1 – High sales – no decision rights manager on discounts and credit terms

Setting

Footwear BV is a trade company in the Netherlands which purchases large amounts of shoes internationally from big brands like Nike, Adidas, Puma, etc. Footwear in turn sells those shoes to different local shoe stores across the Netherlands. To make this business cycle more efficient, there are multiple sales centres spread over the country from where shoes are sold and distributed to customers. Each centre has its own manager, who has two main responsibilities:

1. The sales level. 2. Distribution efficiency.

Performance evaluation system

The CEO at Footwear BV annually evaluates the sales centre managers. The evaluation of 2015 is based on two dimensions:

1. An objective performance measure on a 1 to 10 scale, based on the gross sales. The gross sales (see formula below) is the revenue before subtracting discounts and uncollectable amounts from customers that bought on credit but where it is highly unlikely that they are ever going to pay. Gross sales can be influenced by decisions and actions taken by the sales centre managers (e.g., promotional activities), but it is recognized that the gross sales is also influenced by other factors which lay outside their control (e.g., economic climate).

Sales centre managers have no decision rights to: 1) award additional discounts or, 2) decide to sell customers on credit who do not qualify for credit according to firm-level criteria.

Gross sales (official selling price x number of products sold) -/- Discounts and uncollectable amounts

Sales performance

2. In 2015 an improvement project on distribution efficiency was initiated top-down in the organization. The implementation of this project in each sales centre will be evaluated with a subjective performance measure, also on a 1 to 10 scale.

The objective and subjective performance measure in the performance evaluation of a sales centre manager capture separate parts of the performance (amount of gross sales and implementation of the distributional efficiency project). The overall evaluation of sales centre managers’ performance, and therefore their bonus, is based on an average of the outcomes on both measures.

Actual performance of one of the sales centre managers

Gross sales

The sales centre manager’s gross sales level in 2015 was €320.000 which resulted in an objective measure’s score of 8 (see table below).

1 2 3 4 5 6 7 8 9 10

< €100.000 €100.000 -

€125.000 €125.000 - €150.000 €150.000 - €175.000 €175.000 - €200.000 €200.000 - €250.000 €250.000 - €300.000 €300.000 - €350.000 €350.000 - €400.000 > €400.000

Distribution efficiency improvement project

The sales centre manager successfully implemented the project improvements into the sales centre processes. He only needed 90% of the reserved time according to the project schedule. However, he also exceeded the budget with 5%. The majority of employees affected by the project commented that the sales centre manager managed the implementation very well.

(29)

29 Case condition 2 – Low sales – no decision rights manager on discounts and credit terms

Setting

The CEO at Footwear BV annually evaluates the sales centre managers. This evaluation is based on two dimensions: 1. An objective performance measure on a 1 to 10 scale, based on the gross sales. The gross sales (see formula

below) is the revenue before subtracting discounts and uncollectable amounts from customers that bought on credit but where it is highly unlikely that they are ever going to pay. Gross sales can be influenced by decisions and actions taken by the sales centre managers (e.g., promotional activities), but it is recognized that the gross sales is also influenced by other factors which lay outside their control (e.g., economic climate).

Sales centre managers have no decision rights to: 1) award additional discounts or, 2) decide to sell customers on credit who do not qualify for credit according to firm-level criteria.

Sales performance

The objective and subjective performance measure in the performance evaluation of a sales centre manager capture separate parts of the performance (amount of sales and the implementation of the distributional efficiency project). The overall evaluation of sales centre managers’ performance, and therefore their bonus, is based on an average of the outcomes on both measures.

Gross sales

The sales centre manager’s gross sales level in 2015 was €120.000 which resulted in an objective measure’s score of 2 (see table below).

1 2 3 4 5 6 7 8 9 10

< €100.000 €100.000 -

€125.000 €125.000 - €150.000 €150.000 - €175.000 €175.000 - €200.000 €200.000 - €250.000 €250.000 - €300.000 €300.000 - €350.000 €350.000 - €400.000 > €400.000

(30)

30 Case condition 3 – High sales – decision rights manager on discounts and credit terms

Setting

Sales centre managers have decision rights to: 1) award additional discounts or, 2) decide to sell customers on credit who do not qualify for credit according to firm-level criteria.

Sales performance

Gross sales

The sales centre manager’s gross sales level in 2015 was €320.000 which resulted in an objective measure’s score of 8 (see table below). The amount of discounts and uncollectible amounts has been significant over the year.

1 2 3 4 5 6 7 8 9 10

< €100.000 €100.000 -

€125.000 €125.000 - €150.000 €150.000 - €175.000 €175.000 - €200.000 €200.000 - €250.000 €250.000 - €300.000 €300.000 - €350.000 €350.000 - €400.000 > €400.000

(31)

31 Case condition 4 – Low sales – decision rights manager on discounts and credit terms

Setting

Sales centre managers have decision rights to: 1) award additional discounts or, 2) decide to sell customers on credit who do not qualify for credit according to firm-level criteria.

Sales performance

Gross sales

The sales centre manager’s gross sales level in 2015 was €120.000 which resulted in an objective measure’s score of 2 (see table below). The amount of discounts and uncollectible amounts has been significant over the year.

1 2 3 4 5 6 7 8 9 10

< €100.000 €100.000 -

€125.000 €125.000 - €150.000 €150.000 - €175.000 €175.000 - €200.000 €200.000 - €250.000 €250.000 - €300.000 €300.000 - €350.000 €350.000 - €400.000 > €400.000