The spillover effect and the impact of target difficulty

(1)

Amsterdam Business School

The spillover effect and the impact of target difficulty

Name: Elise Briedé

Student number: 10867740

Thesis supervisor: prof. dr. V.S. Maas Date: 7 June 2016

Word count: 15.235

MSc Accountancy & Control, specialization Control

(2)

Statement of Originality

This document is written by student Elise Briedé who declares to take full responsibility for the contents of this document.

I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.

The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.

(3)

Abstract

Subjective performance measures are often used, based on the shortcoming of objective performance measures. However, subjective performance measures also have shortcomings. They cannot be verified by a third party and subjective performance measures could be influenced by information that should not have an effect. Prior research found that the score on an unrelated objective performance measure spills over to the subjective performance measure. This thesis further examines this spillover effect by examining whether it matters if the subordinate scored below or above his or her target on an unrelated objective performance measure, the administrative performance. Furthermore this thesis examines whether target difficulty has an effect as well. Supervisors take into account the fairness perceptions of subordinates when they determine the targets by giving lower targets or adjusting targets during the year or afterwards. In this thesis is predicted that supervisors will adjust the subjective performance evaluation, if they are not able to adjust the targets. The findings of the case-based experiment indicate that the spillover effect indeed exist. When a subordinate scored above target on an unrelated objective performance measure, a supervisor gives a higher score on the subjective performance evaluation than when a subordinate scored below target. However, target difficulty appears to have no impact on the subjective performance evaluation. There is neither a main effect of target difficulty nor an interaction effect of target difficulty and the score on the objective performance measure. These findings contribute to the emerging research in the field of subjectivity in performance evaluation. It is important to know what factors could influence subjective performance evaluations since subjectivity is getting more common in performance evaluations.

Key words: subjective performance evaluation, performance measurement, target difficulty,

(4)

2.1 Information distortion ... 9 2.2 Halo effect ... 10 2.3 Target difficulty ... 12 2.3.1 Below target ... 13 2.3.2 Above target ... 14 2.4 Interaction effect ... 15 3 Methodology ... 17 3.1 Experimental design ... 17 3.1.1 Independent variables ... 18 3.1.2 Dependent variable ... 19 3.2 Participants ... 19 3.3 Procedure ... 22 3.4 Validity concerns ... 25 3.5 Manipulation checks ... 25 3.6 Expectations ... 26 4 Results... 28 4.1 Preliminary analyses ... 28 4.1.1 Descriptive statistics ... 28 4.1.2 Correlation ... 31 4.1.3 Information sufficiency ... 32 4.2 Hypotheses test ... 33 4.2.1 Hypothesis 1 ... 34 4.2.2 Hypothesis 2 ... 36 4.2.3 Hypothesis 3 ... 37 4.3 Supplemental analyses ... 37

4.3.1 Dependence between objective and subjective performance measure ... 37

4.3.2 Using all respondents ... 39

5 Conclusion ... 40

(5)

1 Introduction

Subjective performance measures are often used, based on the shortcoming of objective performance measures (e.g. Bol & Smith, 2011). Kunz (2015) found that a mixed evaluation, thus an evaluation based on objective and subjective performance measures, is superior to an evaluation on objective performance measures only. Subjectivity in performance evaluation is useful because it reduces the risk of the employees and it mitigates the distortion of the quantitative performance measures (Bol & Smith, 2011; Gibbs, Merchant, Stede & Vargus, 2004; Ittner, Larcker & Meyer, 2003). However, subjectivity in performance evaluation also has some downsides; among others subjective performance measures cannot be verified by a third party (Prendergast, 1999). Consequently it is possible that a subjective evaluation does not represent the real performance of an employee (Gibbs et al. 2004; Ittner et al. 2003 ;Prendergast, 1999).

The literature about the use of managers’ discretion in performance evaluation emerges (e.g. Bol & Smith, 2011; Maas, Rinsum & Towry, 2012; Rinsum & Verbeeten, 2012) and research that investigates how different factors affect the subjective performance measures are getting more valuable (O’Connor, Deng & Fei, 2015). There are different factors that could influence the subjective performance evaluation, for example favoritism and fairness (Bol, 2011; Ittner et al. 2003; Prendergast, 1999), precision and sensitivity of the measures and the type of subjectivity that is used (O’Connor et al. 2015). This thesis examines if the score on an unrelated objective performance measure and target difficulty also have an effect on the subjective performance evaluation and if so, whether target difficulty has an impact on the spillover effect.

The literature suggest that subjective performance measures can be influenced by knowledge of the manager about unrelated information. There is found that there is a bias towards the use of the first known information when performance is evaluated (Bond, Carlson, Meloy, Russo & Tanner, 2007). Since the score on an objective performance measure is often available before the subjective performance evaluation is done, the subjective performance evaluation may be biased towards this first known score. There appears to be a spillover effect which indicates that the score on an unrelated objective performance measure spills over to a subjective performance measure (Bol & Smith, 2011). The higher the score on an objective performance measure is, the higher the subjective evaluation is even if these measures are completely independent. To examine whether only knowledge about whether a subordinate scored above or below his target impacts the subjective performance measure as well, there is hypothesized that the subjective performance measure will be higher (lower) when the subordinate scored above (below) target on an unrelated objective performance measure.

(6)

This objective performance measure is often based upon a target. The difficulty of targets has an impact on the motivation of employees and superiors may therefore subjectively adjust these goals to deal with the motivation and fairness concerns of the employees, even if it is an unrelated performance measure (Bol, Keune, Matsumura & Shin, 2010). During different parts of the performance evaluation process adjustments are made to deal with difficult targets. In the literature is stated that targets are set easy upfront, or they are adjusted during the year or ex-post to deal with the fairness concerns of subordinates (Arnold & Artz, 2015; Bol et al. 2010; Kelly, Webb & Vance, 2015). When these adjustments to the targets of the objective performance evaluations are not possible, it is likely that supervisors will look for other ways to deal with the fairness concerns of their subordinates. One possibility to deal with the fairness concerns regarding difficult targets is that the subjective evaluation is adjusted, even though this evaluation is completely unrelated. Therefore the second hypothesis is that target difficulty will have an impact on the subjective performance evaluation. The last hypothesis in this thesis is the interaction between target difficulty and the score on the unrelated objective performance measure. There is expected that a the difference in subjective performance evaluation for scores below and above target is bigger for easy targets than for difficult targets.

The hypotheses are tested with a case-based experiment which was distributed across students, friends and family. 77 participants completed the experiment and answered all manipulation checks right. The case was about a supervisor who had to evaluate his subordinate on his administrative performance. It was already known that the subordinate scored above or below his target on the sales, the unrelated objective performance measure. Furthermore this objective performance measure was based upon an easy or difficult target. These two manipulations of the two variables lead to four different cases. There was also a fifth case in which no information was given about the two variables. In each case the participants were asked to give a score for the administrative performance of the subordinate based on a note that was provided to them.

The results obtained from this experiment support the first hypothesis. If a subordinate scored below target on an unrelated objective performance measure, he receives a lower score on his subjective performance measure than when he scored above target on this objective performance measure. The other hypotheses are not supported. There appears to be no main effect of target difficulty and no interaction effect of target difficulty and the score on the objective performance measure.

(7)

These findings contribute to the emerging literature about subjectivity by re-examining the spillover effect and examining the effect of target difficulty as a factor that could influence the subjective performance evaluation. This research is the first study that examines whether target difficulty of an objective performance measure has an impact on the subjective performance evaluation. The results could influence the way in which subjectivity is used in organizations, as it appears that even only knowing that a subordinate scored high or low already impacts the subjective performance evaluation.

This thesis is structured as follows. In the next section the literature is discussed and the hypotheses are formulated. The second part contains the methodology. After the methodology the results of the research are presented and the thesis ends with the conclusion.

(8)

2 Literature and hypotheses formulation

In an organization the principal cannot observe the actions of the agent continually (Prendergast, 1999). Therefore the principal tries to verify whether the agent has taken the appropriate actions (Prendergast, 1999). To guide the actions of the agent performance measures are made. This makes it possible for the principal to see whether the agent has taken the right actions (Prendergast, 1999), and employees will focus their attention on those actions on which their evaluation, and their pay, is based (Moers, 2005). Thus, to match the divergent interests between the principal and the agents incentives that align those interests are given to the agents. Not only objective performance measures can be used for this. Opposed to objective performance measures, subjectivity is an important part of the performance measurement in most organizations (Prendergast & Topel, 1993).

There are different ways of using subjectivity, namely subjective weighting of the objective performance measures, using subjective (mostly qualitative) measures of performance, and using the discretion to adjust bonuses ex-post based on other factors than those that where specified ex-ante (Ittner et al. 2003). The use of subjectivity in performance evaluation helps in decreasing the problems that could arise when the performance is determined based on objective (mostly quantitative) performance measures (e.g. Baker, Gibbons & Murphy. 1994; Gibbs et al. 2004; Ittner et al. 2003; Prendergast, 1999). Subjectivity is used to mitigate the distortion of the quantitative performance measures, for example because the quantitative measures can be manipulated (Gibbs et al. 2004; Ittner et al. 2003). Furthermore subjectivity is used to reduce the risk for the employees as quantitative performance measures can include uncontrollable factors (Bol & Smith, 2011; Gibbs et al. 2004).

Although the use of subjective performance evaluation is based on the shortcomings of objective performance measures (Bol & Smith, 2011), subjectivity also has some downsides. There are no clear performance standards for subjective performance measures, so it is just based on the discretion and judgment of the manager (Moers, 2005). This means that subjective evaluations cannot be verified by a third party (Prendergast, 1999). Therefore it is possible that the subjective evaluations do not reflect the real performance of the employees (Gibbs et al. 2004; Ittner et al. 2003; Prendergast, 1999). Another downside is that other, unrelated factors could influence the subjective evaluation, like favoritism of the supervisor for a subordinate or the fairness perception of the subordinate which the supervisor keeps in mind when determining the subjective performance evaluation (Bol, 2011; Ittner et al. 2003; Prendergast, 1999). Bol and Smith (2011) found that even the score on another, unrelated, objective performance measure

(9)

could influence the subjective performance evaluation. They call this effect of the objective performance measure on the subjective performance measure ‘the spillover effect’.

Prior research has introduced some concepts which may explain this spillover effect. Not only cognitive distortion as discussed in Bol and Smith (2011) may cause this effect, but also the halo effect and the assimilation and contrast effect. The confirmation bias exist when a supervisor looks for information in the environment that is consistent with his prior experiences with the subordinate and a supervisor that filters out information that is not conform his experience (Bol & Smith, 2011; Müller & Weinschenk, 2015). This theory cannot be used since the participants in this research get certain information on which they should base their evaluation, so they cannot look for other information.

2.1 Information distortion

Bol and Smith (2011) believe cognitive distortion is the most fitting explanation in their research for the spillover effect of the objective performance measure on the subjective performance evaluation. This distortion of information has to do with the manner in which information is interpreted and used by the supervisors (Bond et al. 2007) and may explain the unintentional bias of the subjective evaluation towards the level of performance on the objective performance measure. In their research Bol and Smith (2011) indeed found this bias since they found that the subjective performance evaluation is higher when the objective performance measure is higher.

Their findings are supported by various researches about predecisional information – information that is available and known by a person before a decision is made. Russo, Medvec and Meloy (1996) for example found that persons distort new information in such a way that it favors the leading alternative. Furthermore Wilks (2002) concludes that subordinates give a judgment that is consistent with that of their supervisor if they know the supervisor’s view on forehand. Bond et al. (2007) also conclude that information that is received later is biased in such a way that it favors the initial information. Even though these researches are not focused on performance measures, information distortion does happen when evaluating performance. Woods (2012) found that supervisors adjust the objective performance measure to make them consistent with the performance in prior period. This effect however only occurs when the performance is unexpectedly low. In accordance with Woods (2012) Murphy, Balzer, Lockhart and Eisenman (1985) conclude that previous performance of a subordinate has an effect on the performance evaluation in later periods.

(10)

However, the effect Murphy et al. (1985) found is the opposite of the effects found in the other researches. They found that persons with a low prior performance scored higher when they performed on average in a later period, than persons with a high prior performance. This effect is known as the contrast effect, as the current evaluation is the opposite of the prior evaluation (Murphy et al. 1985). Thus, there is a bias away from the prior level of performance (Smither, Reilly & Buda, 1988). Smither et al. 1988 examined the same effect, but found both a contrast effect and an assimilation effect. The latter is the effect that the current evaluations are consistent with the prior evaluation. A bias towards the prior performance or the prior known information is found in more researches, which are mentioned before. Those effects of prior performance on the current performance are driven by the fact that the prior information or prior performance of a subordinate could raise a certain expectation by the supervisor (Murphy et al. 1985). If the subordinate performs not conform this expectation, the supervisor may bias the new evaluation.

2.2 Halo effect

The halo effect is related with information distortion. This effect may also be an explanation of the spillover effect of objective performance evaluation to subjective performance measures. A lot of research is done regarding the halo effect (Borman, 1975; DeCotiis, 1977; Palmer & Fieldman, 2005; Saal, Downey & Lahey, 1980). This effect is defined as the effect that a supervisor does not make a distinction between different performance measures, even if these measures are independent of each other (Borma, 1975; DeCotiis, 1977; Palmer & Fieldman, 2005; Saal et al. 1980). This means the evaluation is based on a general impression from the supervisor about the manager (Borman, 1975; DeCotiis, 1977), which is visible in the performance evaluation as a low variance between the different performance measures (Borman, 1975). In most organizations subjective and objective performance measures are used interchangeably (Pendergast & Topel, 1993). Managers may give a similar evaluation on both performance measures, even if these performance measures measure something else (Merchant, Stringer & Theivananthampillai, 2009). This high influence of the objective performance measure on the subjective performance measure represents the halo effect (Merchant et al., 2009). Thus, a supervisor is likely to give a similar evaluation on different performance measures, even though the performance of the employee may differ among those measures (Palmer & Fieldman, 2005).

Bommer et al. (1995) on the other hand found that the objective and subjective performance measures are correlated, but suppose that these measures are not the same.

(11)

Therefore they state that the objective and subjective performance measures are not interchangeable. Consistent with the findings of Bommer et al. (1995) Merchant et al. (2009) argue that there is no halo effect with regard to performance evaluation. They only found little evidence for the objective and subjective performance measure to be correlated.

Both the halo effect and information distortion can explain the effect of the objective performance measure on the subjective performance measure. Information distortion is more unintentional than the halo effect. With information distortion the supervisor gives unintended a higher or lower subjective performance evaluation based upon the prior performance or prior known, not relevant, information. With the halo effect the supervisor gives a higher subjective evaluation as he knows that the subordinate scored higher before and he perceives that in case of a low score this is a one-time error.

Thus, information that is known before a decision is made, impacts the decision that is made (Bol & Smith, 2011; Bond et al. 2007; Russo, Medvec & Meloy, 1996; Wilks, 2002; Woods, 2012). This means that the order in which the information is given to the evaluator, impacts the decisions that are made (Bond et al. 2007). The score on the objective performance measure is often available before the subjective evaluation is done (Huber et al. 1987; Bond et al. 2007 in Bol & Smith, 2011). A consequence of this is that the first available score on the objective performance measure could influence the subjective evaluation. This effect is driven by the desire of a person for consistency between different kinds of information (Russo et al. 2008) and because an expectation about the performance of a subordinate is made. The assimilation effect is found more often, and thus it is likely that the current performance evaluation is consistent with the prior performance evaluation. Therefore I expect that the subjective performance evaluation is consistent with the first available objective performance evaluation.

Bol and Smith (2011) found that the subjective performance evaluation was impacted by the score on the objective performance measure, which represents the spillover effect. This thesis examines whether the subjective performance evaluation is also affected if the supervisor only knows that the subordinate scored above or below target. This research predicts with regard to the information discussed before that the above target or below target information on the objective performance measure spills over to the subjective evaluation. This gives the first hypothesis regarding the main effect of the score of an unrelated objective performance measure:

H1: Supervisors’ subjective performance evaluations will be higher (lower) when the

subordinate’s performance on an unrelated objective measure is above (below) target (Bol & Smith, 2011)

(12)

2.3 Target difficulty

Targets are important in almost all organizations, especially in performance evaluations (Arnold & Artz, 2015). Objective performance measures are often based on targets. Since targets are important for the performance evaluation it is important to know the effect of target difficulty on other aspects of the performance evaluation. This part of the literature helps in examining what the effect is of using targets for one performance measure on another, unrelated performance measures.

Target difficulty is the perception of a person about the achievability of his target (Cheng et al. 2007). The goal setting theory states that difficult targets lead to a better performance than easy targets (Blumenfeld & Leidy, 1969; Locke, 1968; Locke, Saari, Shaw & Latham, 1981). Too difficult targets however, demotivate employees, since employees perceive these targets as not attainable (Merchant & Manzoni, 1989; Shalley & Oldham, 1985). Motivation of employees is important because it increases the productivity and task performance (Bol & Smith, 2011; Cheng, Luckett & Mahama, 2007). Due to this importance of employee motivation on productivity and task performance, supervisors use their discretion to adjust targets to improve the motivation of employees. They can use their discretion during the target setting process, during the year and at the end of the year.

During the target setting process supervisors use their discretion to set easier targets for departments that have a challenging reference group (Bol et al. 2010). In that way they deal with fairness concerns of the employees, since unfairness leads to a decreased motivation (Bol et al. 2010). Thus to improve the motivation, supervisors set lower targets at the beginning. Furthermore supervisors can use their discretion to adjust targets during the year. Downwards target adjustments are made more often during the year than upwards adjustments and downwards adjustments are made more often when the target is difficult than when the target is easy, since downwards adjustments are seen as fairer by the employees (Arnold & Artz, 2015).

Supervisors can also use their discretion by allowing ex-post target adjustments, which are adjustments of the ex-ante specified targets (Kelly, Webb & Vance, 2015). Ex-post target adjustments positively impact the fairness perceptions of employees and the performance and supervisors take these fairness and justice perceptions of employees into account when determining the evaluation (Kelly, Webb & Vance, 2015). However, these positive effects are only found when a moderate target is set, not when a difficult target is set. This is the case since employees with a moderate target perceive that they, because of the adjustments, are able to

(13)

achieve their targets. Employees with a difficult target experience the adjustment as not beneficial as they think they are still not able to achieve their targets.

This thesis predicts that in a situation where supervisors cannot use their discretion by adjusting targets during the target setting process, during the year or afterwards, supervisors will look for other ways to increase the fairness perception and motivation of employees and consequently the performance. As downwards adjustments to targets are equivalent with upwards adjustments to performance evaluations (Arnold & Artz, 2015), it is expected that supervisors will make adjustments to performance evaluations when they are not able to adjust targets. This prediction is explained hereafter by giving two hypothesis. One for a situation where a subordinate scored below target and one for a situation where a subordinate scored above target on an unrelated objective performance measure.

2.3.1 Below target

When the objective performance measure is based upon the target that is set ex-ante, there is expected that managers will use their discretion to adjust the subjective evaluation for the level of target difficulty of this objective evaluation (Bol & Smith, 2011). Too difficult targets are perceived as unfair by employees, as the possibility of meeting the target is lower when the target is more difficult (Dosset & Greenberg, 1981; Gibbs et al. 2004). When the objective performance measure is based on this target the objective performance measure will be lower when the target is difficult than when it is easy in a situation of equal performance. To avoid the negative effects on employee motivation and productivity, the supervisor can use discretion in the subjective evaluation by giving a higher evaluation. In this way the supervisor makes the overall evaluation of the employee better which increases the motivation of the employee (Bol & Smith, 2011) and the supervisor can ‘correct’ for the low objective performance measure due to the difficult target that was set.

Furthermore subjective bonuses are used when the objective targets are of a level that these targets are not attainable (Merchant & Manzoni, 1989). The likelihood of missing the targets is higher when the targets are difficult (Gibbs et al. 2004). Thus, with a high level of target difficulty, more subjective bonuses are used. Another research of Gibbs, Merchant, Stede and Vargus (2005) states that there are more subjective bonuses if the likelihood of achieving an unrelated objective target is lower and less subjective bonuses if the achievability of the objective target is high. This means that the subjective evaluation will be higher when the objective targets are higher as the likelihood of not achieving the targets is higher.

(14)

Anderson et al. (2013) use target difficulty as a control variable in their research, because they expect target difficulty of the objective performance measure will impact the subjective performance evaluation. They conclude that supervisors use subjectivity when during the ex-post performance evaluation it becomes clear that targets have been too difficult, which would have resulted in unfair rewards. Thus Anderson et al. (2013) indeed found that a more difficult target leads to a higher subjective performance evaluation, but only for targets which cannot be revised every week. Arnold and Artz (2015) researched this flexibility of targets during the year and found that downwards target adjustments are made more often when targets are difficult than when the targets are easy. Downwards target adjustments during the year are equivalent with upwards evaluation adjustments (Arnold & Artz, 2015) and therefore this thesis predicts that in the case of a difficult target more upwards evaluation adjustments are done than in the case of an easy target.

In sum there is expected that low scores on difficult targets will be adjusted upwards by giving a higher subjective evaluation. A score below target is ‘corrected’ upwards more when it was a difficult target than when it was an easy target since managers will keep in mind that meeting the target was difficult and in this way they deal with fairness issues that may arise by the employees. This leads to the following hypothesis, which is the first part of the main effect of target difficulty on the subjective performance evaluation:

H2a: When the subordinate’s performance on an unrelated objective performance

measure is below target, the supervisor’s subjective performance evaluation will be higher when the objective performance measure is based on a stretch target than when it is based on an easy target.

2.3.2 Above target

Downwards target adjustments are made during the year more often when the targets are difficult (Arnold & Artz, 2015). Downwards target adjustments are equivalent with upward evaluation adjustments (Arnold & Artz, 2015) and therefore there is expected that more upward evaluation adjustments are done when the targets are difficult than when the targets are easy, even when the score on the objective performance measure was high already. Supervisors compare outcome and related inputs of subordinates with the outcomes and inputs of others (Bartol, Durham & Poon, 2001). Thus, when an subordinate has an easy target and scores above target his input is likely to be lower than when an subordinate has a difficult target and this subordinate scores above target. Although the supervisors get information about one subordinate only during this research there is still expected that the input-output relationship will

(15)

have an impact, because for a high score on a difficult target more effort of the employee is needed than for a high score on an easy target. Therefore this thesis predicts that when a subordinate with a difficult target scores above target his subjective performance evaluation will be higher than the evaluation of a subordinate with an easy target, leading to the following hypothesis:

H2b: When the subordinate’s performance on an unrelated objective performance

measure is above target, the supervisor’s subjective performance evaluation will be higher when the objective performance measure is based on a difficult target than when it is based on an easy target.

2.4 Interaction effect

The objective scores that are above target spillover to the subjective evaluation and thus for subordinates with both easy targets and stretch targets the subjective evaluation is higher. The objective scores below target also spillover to the subjective evaluation and therefore the subjective evaluation is lower in this situation. There is expected that the effect of target difficulty for a score below target is stronger than the effect of target difficulty for scores above target. In other words, the effect of target difficulty on the spillover effect is stronger when the objective score is below target than when it is above target. For a score above target the subordinates already scored high thus the supervisors do not have to take the fairness perception of the subordinates into account. This is conform the findings of Cropanzano and Folger (1989), who conclude that subjects who receive the lowest levels of pay are more likely to perceive that their compensation was being unfair than the subjects who receive the highest level of pays. For the scores below target the subordinates scored low and the supervisors will take the fairness perception of these subordinates into account. With regard to H2 there is expected that the upwards adjustments for difficult targets are higher, and therefore hypothesis 3 predicts that the difference in subjective evaluation between scores above and below targets is smaller for the difficult targets than for the easy targets. This leads to the last hypothesis:

H3: The difference between the subjective performance evaluation for the score below

and above target is bigger when the objective performance measure is based upon an easy target than when it is based on a difficult target.

Figure 1 provides a graphical overview of the hypotheses that have been explained in this section. Since hypothesis 1 predicts that the subjective performance evaluation is higher when a

(16)

subordinate scores above target on an unrelated objective performance measure than when this subordinate scores below target, the blue line is expected to be higher than the red line. Hypothesis 2a is visible in the graph, as the red line is higher for difficult targets than for easy targets. Thus the subjective performance evaluation is expected to be higher for subordinates with a difficult target than for subordinates with an easy target when the subordinate scores below target on an unrelated objective performance measure. The blue line is also higher for difficult targets than for easy targets, and this line shows what is predicted in hypothesis 2b. There is expected that when a subordinate scores above target on the unrelated objective performance measure the subjective performance evaluation is higher for subordinates with a difficult target than for subordinates with an easy target. Hypothesis 3 is also shown in figure 1, since the difference between the blue and red line is bigger for the easy target than for the difficult target.

(17)

3 Methodology

This section provides an overview of the method that is used to test the hypotheses. The experimental design, participants, procedures, experimental case and manipulations are discussed. To test the hypotheses an experiment is done, because in that way under controlled conditions can be tested whether the hypotheses are supported. In this way it is more certain that differences in the dependent variable are really caused by differences in the independent variables. For other aspects that could influence the dependent variable, like gender and work experience, control variables will be used. This is also explained in this section.

3.1 Experimental design

The research is a scenario-based experiment and is based upon a research of Bol and Smith (2011). Just like the experiment of Bol and Smith (2011), the participants in this experiment were asked to make a performance evaluation of an employee in a hypothetical case setting. This means a case scenario was given to the participants. They were asked to imagine they have a supervisory role in a car dealership since this is a business everybody is familiar with and furthermore it is easy to make a clear distinction between subjective and objective performance measures. The objective of the experimental task for the participants was to make a subjective performance evaluation on the administrative performance of a manager. Before this subjective performance evaluation is done, the unrelated sales score of the manager, the objective performance measure, is known.

The experiment is a 2x2+1 between-subjects design as both the score on the objective performance measure and the level of target difficulty are manipulated at two levels. The score on the objective performance measure is above target or below target and the level of target difficulty of this objective performance measure is either easy or difficult. There is also a control condition in which no additional information is given about the score on the objective performance measure and no information about the level of target difficulty. This means the score on the subjective performance evaluation in this scenario is only affected by the information that is given about the subjective performance evaluation. This control conditions presents a standard score with which the other scores can be compared to see whether there is really a difference between the above target, below target and standard situation. Table 1 shows the 5 conditions that are distributed to the participants.

The experiment is a between-subject design, because there are independent, mutually exclusive groups. The respondents can only open the experiment once. The second time they

(18)

open the survey a message will appear that they have already participated in the experiment. Therefore it is relatively certain all participants are unique and that the respondents that participate in one scenario do not participate in another scenario as well.

3.1.1 Independent variables

In the experiment two independent variables are used. Table 2 provides an overview of all variables that are used in the experiment. The first independent variable is the score on the objective performance measure that is not related with the subjective performance measure. This objective performance measure has to do with the sales performance of the manager. As the case is about a car dealership it can be seen as the amount of cars sold in a year. Prior research mentions that the subjective evaluation will be influenced by the information that is known before this evaluation is done (Bond et al. 2007; Russo, Medvec & Meloy, 1996; Wilks, 2002; Woods, 2012). This does not mean that the first known information has to be a number, as is the case in the research of Bol and Smith (2011). Also other information that is known before the subjective performance evaluation is done, in this case the information that the score on the objective performance measure is above or below target, can influence this evaluation. Thus, to deal with some issues that a score on a scale from 1-10 would give, such a score was not given. In the cases was only mentioned whether the manager scored above or below their target.

The second independent variable is the level of target difficulty. For this variable also two manipulations are used. The first manipulation is that the target was set easy at the beginning of the year and the other manipulation is that the target is a difficult target. Target difficulty is often operationalized by using a probability of achievement (Merchant & Manzoni, 1989) and thus the likelihood that a target will be met (Gibbs et al. 2005). In this experiment this operationalization of Gibbs et al. (2005) is used. For the scenarios with a difficult target the case described that the target is a difficult target, which means that the likelihood that the target will be met is 10% and for the scenarios with an easy target the case described that the target is an easy target, which means that the likelihood that the target will be met is 90%.

Table 1. Conditions experiment

Score objective performance measure

Target difficulty

Condition 1 Above target Difficult target

Condition 2 Above target Easy target

Condition 3 Below target Difficult target

Condition 4 Below target Easy target

(19)

Anderson et al. (2013) found that the difficulty of sales target had an impact on the subjective evaluation, but not for targets that could be revised weekly. They give as potential explanation for this that targets that can be revised often do not need subjective scores to deal with the problem of too difficult targets. Therefore it is important that it is clear in the case that the objective performance target is determined ex-ante and cannot be revised during the year. This is described in the case in the following way: the sales target is set at the start of the period on the 1st of January and cannot be revised during the period.

3.1.2 Dependent variable

The dependent variable is the evaluation of a subjective performance measure. The participants needed to give a subjective evaluation on an activity that is not related with the objective performance measure. For this evaluation the participants were asked to give their evaluation on a scale of 0 to 10. This scale is the same scale Bol and Smith (2011) used in their research. Furthermore in the Netherlands a lot of scores are on a scale to 10 and therefore the participants know what each score means.

The subjective evaluation is the same as in the research of Bol and Smith (2011), thus the administrative performance of the subordinate. This cannot be assessed in real, so therefore a description of the performance on this measure is given. Each participant got the same description, which means that differences in the subjective evaluation based on this description are caused by the independent variables and not by the information that is given about the administrative performance of the manager. Therefore it must be clear from the case that the objective score does not have an impact on the subjective evaluation. Otherwise the participants will take into account the score on the objective performance measure intentionally. The independence of the two performance measures is described in the case as follows: it is necessary that there are two measures, since the main tasks of the manager (sales and administration) are completely independent of each other. This means that the performance on one task is independently of the performance on the other task.

3.2 Participants

The participants of this hypothetical case study are students and family members. A concern of using college students in experiments is that the results of these researches are only generalizable to students. However, Liyanarachchi (2007) found that accounting students are useful as surrogates for practitioners in decision-making experiments. Since the participants in this experiment make decisions about performance measures, students can be used. Another reason

(20)

why students are often used in experiments is that a lot of research is done by college professors, and for them students are most easily accessible (Higbee, Lott & Graves, 1976). This is also the case for this experiment. It is easier to get a lot of responses from students than from professionals, as most friends are students.

Table 2. Description and measurement variables

Variable Description Measurement

Subjective performance evaluation Score on the subjective performance measure

Scale 0 to 10 Score on objective performance

measure

Score in case is above or below target 0 = below target 1 = above target Target difficulty Target for objective performance

measure was easy or difficult

0 = difficult 1 = easy

Gender Gender of the participants 0 = female

1 = male

Age Age of the participants Ratio

Education The highest level of education the participant finished 0 = vmbo 1 = havo 2 = vwo 3 = mbo 4 = hbo 5 = wo Work experience Number of years the participants

worked

0 = no experience 1 = less than 1 year 2 = 1 to 5 years 3 = 5 to 10 years 4 = 10 years or more Supervisory experience Number of years the participants have

experience with supervisory tasks

0 = no experience 1 = less than 1 year 2 = 1 to 5 years 3 = 5 to 10 years 4 = 10 years or more Usefulness Usefulness of information provided to

participant to make the subjective performance evaluation

Seven-point Likert-scale (strongly disagree – strongly agree)

Confidence Confidence of the participants in their subjective performance evaluation

Seven-point Likert-scale (strongly disagree – strongly agree)

(21)

The experiment is randomly assigned to the participants. In that way it automatically deals with the different characteristics of the participants, like age and experience. To be sure that the individual characteristics of the participants do not have an impact on the results I used five control variables: age, gender, work experience, supervisory experience and education. Bol and Smith (2011) used these control variables as well.

A minimum amount of fifteen participants per scenario is required to test the hypotheses. After 75 responses were obtained, 15 for each case, the answers on the manipulation questions were checked. All respondents that gave wrong answers to these questions were not count, because they either did not understand the case or did not read the case. Therefore their answers are not useful. In each case there were respondents that gave the wrong answers, so more respondents were needed. Therefore the count in Qualtrics was edited for the amount of respondents that were still needed. This means that for example condition 4 was set at ten participants and condition 5 at fifteen. When respondents then open the link, five times condition 4 is given and no condition 5 is given. When both conditions have a count of fifteen again the system randomly assigns one of the conditions to the respondents, just as before. This editing of the count was done for all conditions, so the experiment was still randomly distributed across participants since in each condition, except condition 5, there were more responses needed. The survey was launched on April 20, 2016. On May 3, 2016 each condition had 15 respondents that answered the manipulation checks right. In total 152 participants started the experiment. 120 of them finished the experiment and 77 of them answered the manipulation checks right. Table 3 panel A shows the respondents that are used in total and for each condition separately.

In table 3 panel B the characteristics of the participants that finished the experiment and answered all manipulation checks right are shown. 55.8% of them is male and 44.2% is female. Most participants are between 20 and 30 years and most participants finished hbo or wo (84.5%). Furthermore most participants have one to five years of work experience (31.2%) and no supervisory experience (46.8%). Excluding two participants that took 18 and 20 hours to complete the experiment, the participants completed the experiment on average in 6 minutes and 47 seconds. For the participants that answered the manipulation questions right it took 5 minutes and 55 seconds to complete the experiment. For the participants that answered the manipulation checks wrong it took 8 minutes and 2 seconds to complete the experiment. Although it was expected that the participants that answered the manipulation checks wrong took less time to read the case it seems to be that the participants that answered the manipulation checks right finished sooner. In the group of participants that answered the manipulation checks wrong there

(22)

are participants that took more than 10 minutes to finish the experiment. It could be the case that these participants after they have read the case took a break and that they therefore forgot the details of the case and consequently they answered the manipulation checks wrong.

3.3 Procedure

To research the hypotheses a case-based experiment was conducted. The experiment is made in Qualtrics Survey Software and was sent to the participant by mail. In this mail a link to a site was available. Furthermore the site was put on Facebook and LinkedIn. When the participants accessed the website through the link the introduction opened. In this introduction was not described what the exact purpose of the research was, but only that the research takes place in the performance measurement field. In this way the participants did not know what the aim of the research was and thus what kind of answers were expected. This prevents the demand effect, which means that participants give an answer they think is expected by the researcher. Since each participant opened the same site on which it was not visible that there were different cases, the participants did not know that variables were manipulated and which variables were manipulated. Furthermore in the introduction was not mentioned that the participants joined an experiment, but there was written that they participated in a research. This also limited the chance that the participants knew that the research was an experiment, and this prevented participants to start guessing the hypothesis which may influence their answers. After the introduction the experiment started with some standard question, about the age, gender, education and working and supervisory experience. Then the site randomly assigned one of the five cases to the participants.

Each case started with a short description in which was mentioned that the participant was a regional manager and supervised a sales manager who worked in a car dealership. Because the difference or equality in gender between the supervisor and subordinate could impact the subjective performance evaluation of the supervisor (Bol, 2011 in Woods), the case only mentioned ‘the manager’ and did not mention any gender. There was no ‘he’ or ‘she’ used in the cases. In each case was described that the score on the objective performance measure was based upon a target. In the case was stated that the target on which the objective performance measure was based was either easy of difficult to achieve. Then a table showed a table with the score of 5 managers in the region. The sales manager scored either above or below target. This was also described in the case and there was mentioned that on this information later on the objective performance measure would be determined. For this objective score standards were available

(23)

Table 3. Demographics

Panel A. Total respondents

Total

Conditions

1 2 3 4 5

Started 152a ₄₀ ₂₈ ₂₅ ₄₀ ₁₆

Did not finish 32a ₁₁ ₅ ₄ ₉ ₀

Answered manipulation questions wrong 43 14 7 6 16 -

Number of participants used 77 15 16 15 15 16

Percentage of participants 100% 19.5% 20.8% 19.5% 19.5% 20.8% Panel B. Demographics of participants

# of participants % of participants Gender Female 34 44.2 Male 43 55.8 Age 20≤25 33 42.9 25≤30 29 37.7 30≤35 4 5.2 35≤40 1 1.3 40≤45 1 1.3 45≤50 4 5.2 50≤55 1 1.3 55≤60 3 3.9 60≤ 1 1.3 Education Vmbo 2 2.6 Havo 1 1.3 Vwo 4 5.2 Mbo 5 6.5 Hbo 34 44.2 Wo 31 40.3 Work experience No experience 8 10.4

Less than 1 year 17 22.1

1 to 5 years 24 31.2

5 to 10 years 14 18.2

10 years or more 14 18.2

Supervisory experience

No experience 36 46.8

Less than 1 year 17 22.1

1 to 5 years 18 23.4

5 to 10 years 2 2.6

10 years or more 4 5.2

a. Three participants stopped with the experiment before a condition was assigned. Therefore the respondents that started a condition and that did not finish a condition do not add up to the total respondents that started and did not finish.

(24)

and therefore the regional manager decided to look for the objective score later on.

Then a memo was shown on which briefly some notes were written about the administrative performance of the sales manager. The notes ranged from “office pretty messy, doesn’t seem to care for the paperwork…” to “budget reports are usually on time, budget estimates are pretty accurate…”. This memo was followed by the question about the dependent variable. The participants were asked to give a rating on a scale of 0 to 10 for the administrative performance of the sales manager based upon the memo.

A new page opened and then the experiment continued with some questions about the manipulations of the independent variables, to test whether the manipulations have worked. The participants could not go back to the case, so they had to remember the answers instead of just looking up the answers. There was one question that asked whether the target of the manager was easy or difficult and one question that asked whether the manager scored above or below this target. Furthermore it is important that the participants remembered that there was no relation between the objective and subjective performance measure. Therefor the next question was whether the score on sales is informative of the administrative performance. This question should have been answered with ‘no’, since the objective and subjective performance are completely independent of each other. It was necessary that the participants answered with ‘no’, because then it is possible to test whether the subjective score is higher because of the objective score. If participants answered that the scores were related with each other, it is logical that their subjective score is higher when the objective score is higher and vice versa. Thus, in that case there cannot be measured whether their exist an unintentional spillover effect.

After this question the following question was whether the participant took into account the score on the objective performance measure. The answer on this question makes clear whether the participants were aware of the fact that they took into account the objective score when determining the subjective score. If the participants answered ‘yes’ on this question another question was why they took into account the objective score. This was an open question and therefore the participants could explain why they gave this answer. The answer on this question gives more information about the reason why participants took into account the objective score when determining the subjective score, even if they knew that the objective and subjective performance measure were unrelated.

(25)

3.4 Validity concerns

In Qualtrics some standard options can be selected to make the results more valid. Some of them were used in the experiment. The first thing that was used is the save and continue option. This means that respondents could have saved their experiment and continued with it later on. Although this is not desirable, because respondents may have forgotten where the case was about if they stopped for a while and continued another moment, the manipulation checks make sure that the respondents knew where the case was about.

The second thing that was used is the option to prevent people to take the experiment more than once. If participants already filled in the experiment and again pushed the link they got a message that they already filled in the survey. This increases the validity of the data since all participants only read one case instead of participating more than once and thus reading more cases. If they participated in the case more than once they would have known after the first case which aspects of the case were manipulated and the second time they participated they probably would have given other answers. Therefore it is important that people did not have this option.

The last feature that was used is that the participants could not go back to other pages. After the cases and the rating of the sales manager the participants got two questions about the manipulations of the variables. It is necessary that the participants could not go back to the other pages, because then they could have searched for the right answers. If they looked back it is not certain that the rating they gave was based upon the information provided. When they answered the manipulation checks right, it is more certain that they have read the case and gave a rating based upon the right knowledge.

3.5 Manipulation checks

To check whether the manipulations have worked two manipulation questions were asked after the question about the dependent variable was asked. A third question that was asked was not a real manipulation question, because this situation was the same in all cases except for the control condition. This was the question about whether the objective score said something about the administrative performance of the sales manager. All obtained data in which the participants did not answer those three questions right were not used in the analysis. When a participant answered one question wrong it can be that he/she just guessed the answer on the other questions, since it are multiple choice questions. To prevent these participants to have an impact on the answers, their data is not used. Table 3 panel A shows that 43 participants answered one or more of the manipulation checks wrong and were therefore not used in the main analysis. In

(26)

condition 5, the control condition, no manipulations were available and therefore no manipulation questions were asked.

3.6 Expectations

This paragraph shows the relation between the five cases and the three hypotheses. In table 4 an overview is given of this. Based on H1 there is expected that in condition 1 and condition 2 a higher subjective evaluation will be given than condition 3 and 4, because H1 hypothesizes that the score on the objective performance measure will spill over to the subjective performance evaluation. Since the objective performance in condition 1 and 2 is higher than in condition 3 and 4, the subjective performance measure will be higher as well if there really exist a spillover effect.

Based on H2a there is predicted that the subjective performance evaluation will be higher in case 3 than in case 4. In both cases the subordinate scored below target on the objective performance measure. In case 3 the target on which the objective performance measure is based is difficult and in case 4 this target is easy. To find support for H2b, the subjective performance evaluation in case 1 should be higher than the evaluation in case 2. In both cases the score on the objective performance measure is above the ex-ante set performance target. In case 1 the target was difficult, while in case 2 the target on which the objective performance measure is based was easy. Taking this together the subjective performance evaluation in case 1 and 3 is expected to be higher than the subjective performance evaluation in case 2 and 4.

Table 4. Overview of hypotheses

Target difficulty Easy Difficult Score objective performance measure Below target [4] [3] Above target [2] [1] Control condition [5] Formulas H1 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 (3 + 4) < 𝑎𝑣𝑒𝑟𝑎𝑔𝑒(1 + 2) H2a 4 < 3 H2b 2 < 1 H2a+H2b 𝑎𝑣𝑒𝑟𝑎𝑔𝑒 (2 + 4) < 𝑎𝑣𝑒𝑟𝑎𝑔𝑒(1 + 3) H3 (1 − 3) < (2 − 4)

(27)

H3 finally is supported when the difference between case 1 and case 3 is smaller than the difference between case 2 and case 4. This can be calculated by subtracting the average score on the subjective performance evaluation of case 3 from case 1 and of case 4 from case 2. There should be a significant difference between both scores to support H3.

(28)

4 Results

In this section the results are described. First some preliminary analyses are done. Then the hypotheses are tested by doing a two-way ANOVA. This section ends with a supplemental analysis that examines whether there is a difference if all participants were included in the experiment instead of only the participants that answered the manipulation checks right.

4.1 Preliminary analyses 4.1.1 Descriptive statistics

Table 5 provides a clear overview of the mean, standard deviation and number of participants for each condition. It shows that the subjective performance evaluation in the condition in which the manager scored below target (Mean = 6.07, SD = 0.980) is lower than in the condition where the manager scored above target (Mean = 6.71, SD = 1.101) which is in line with H1. Furthermore the subjective evaluation is higher in the difficult target situation (Mean = 6.13, SD = 0.990) than in the easy target situation (Mean = 6.00, SD = 1.000) when the manager scored below target. This is conform H2a. The score on the subjective performance measure is lower when the target is difficult (Mean = 6.40, SD = 0.986) than when the target is easy (Mean = 7.00, SD = 1.155) if the manager scored above target on the objective performance measure. This is opposed to H2b.

The difference between the average score below and above target is bigger for the easy target conditions (1.00) than for the difficult target conditions (0.27). This is conform H3. Thus, table 5 shows that the scores on the subjective performance measure are in line with H1, H2a and H3 and the scores are in a different direction than expected in H2b.

Table 5. Mean, SD and number of participants per condition

Easy target Difficult target Row means

Below target 6.00 (1.000) N=15 6.13 (0.990) N=15 6.07 (0.980) N=30 Above target 7.00 (1.155) N=16 6.40 (0.986) N=15 6.71 (1.101) N=31 Column mean 6.52 (1.180) N=31 6.27 (0.980) N=30 Control condition: 6.81 (1.223) N=16

(29)

Table 5 also shows that the subjective performance evaluation in the control condition is higher than the subjective performance evaluation in the below target and above target condition. An independent sample T-test is done to test whether the control condition really differs from the above target and the below target condition. The subjective performance evaluation is significantly lower in the below target condition than in the control condition (T = -2.253, p < 0.05). In the above target condition however, the subjective performance evaluation is not significantly higher than the evaluation in the control condition (T = -0.292, p > 0.05). The score in the above target condition is even lower than in the control condition, but this is not significant. This means that the subjective performance evaluation in the above target conditions does not differ from the subjective performance evaluation in the control condition. This could indicate that there exist a ceiling effect (Bol & Smith, 2011), which means that managers do not give a higher score in the above target condition because the ‘normal’ score is already high. This can lead to an incorrect conclusion that the score on the objective performance measure and the level of target difficulty do not have an impact on the subjective performance evaluation.

During the experiment the characteristics of the participants were also asked. An overview of these characteristics are provided in table 6 for each condition separately and in total. 63% of all participants is man (Mean = 0.63, SD = 0.500). In two conditions more women participated than men (condition 1: 47% man, condition 2: 44% man), while in the other conditions more man participated. The average age is 28 years and circa 2 months (Mean = 28.19, SD = 9.527). In all conditions the average age is between 25 and 28 expect for condition one where the age is 35 on average. For the five cases the score for the subjective performance evaluation is on average 6.48 (Mean = 6.48, SD = 1.119). It is important that the characteristics of the participants are roughly equally distributed around the 5 conditions or that the characteristics do not have an impact on the dependent variable. Otherwise it could be possible that the subjective evaluation differs across the conditions not because of the independent variables, but due to the characteristics of the participants. Therefore in the next paragraph the correlations of the control variables with the dependent variable are given, to test whether there is a relation between those variables.

(30)

Table 6. Descriptive statistics

Variable N Minimum Maximum Mean Median SD

Condition 1

Subjective performance evaluation 15 4 8 6.40 7.00 0.986

Gender 15 0 1 0.47 0.00 0.516 Age 15 22 60 35.07 27.00 13.885 Education 15 0 5 3.93 4.00 1.483 Work experience 15 0 4 2.80 3.00 1.474 Supervisory experience 15 0 4 1.27 0.00 1.534 Condition 2

Gender 16 0 1 0.63 1.00 0.500 Age 16 21 56 28.00 24.00 10.244 Education 16 1 5 3.75 4.00 1.238 Work experience 16 0 4 2.19 2.00 1.223 Supervisory experience 16 0 4 1.50 1.00 1.211 All conditions

Gender 77 0 1 0.56 1.00 0.500

Age 77 21 60 28.19 25.00 9.527

Education 77 0 5 4.09 4.00 1.102

Work experience 77 0 4 2.12 2.00 1.246 Supervisory experience 77 0 4 0.97 1.00 1.135

(31)

4.1.2 Correlation

In table 7 both the Pearson correlation and the Spearman rank-order correlations are shown, as the sample size is small. It seems that none of the control variables is significantly correlated with the independent variable, the subjective performance evaluation. This reduces the chance that the results are influenced by the characteristics of the participants. Therefore it does not matter if the characteristics of the participants are not completely equally distributed across the conditions.

Some of the control variables are correlated with each other, as expected. Age is significantly positively correlated with work experience (Pearson r = 0.640, p < 0.05; Spearman ρ = 0.528, p < 0.05) and supervisory experience (Pearson r = 0.490, p < 0.05; Spearman ρ = 0.351, p < 0.05). This is as expected for the fact that mostly the younger people just finished college and just started working while the older people have worked already for years. Since they worked for years, they often have a higher position in the organization and it is more likely that they have had a promotion to a supervisory position. This could explain why work experience is positively significantly correlated with supervisory experience (Pearson r = 0.560, p < 0.05; Spearman ρ = 0.522, p < 0.05). Age is significantly negative correlated with education (Pearson r =-0.227, p < 0.05; Spearman ρ = -0.045, p > 0.05). The last variables that are correlated are supervisory experience and gender (Pearson r = 0.235, p < 0.05; Spearman ρ = 0.210, p > 0.05).

Table 7. Correlation matrix (p-values between the brackets)

Subjective

evaluation

Gender Age Education Work experience Supervisory experience Subjective evaluation 1 0.105 (0.364) -0.093 (0.419) 0.037 (0.750) -0.054 (0.640) 0.099 (0.392) Gender 0.102 (0.377) 1 -0.091 (0.433) 0.065 (0.577) -0.100 (0.385) 0.210 (0.067) Age -0.006 (0.956) -0.112 (0.334) 1 -0.045 (0.697) 0.528** (0.000) 0.351** (0.002) Education -0.015 (0.900) 0.145 (0.207) -0.227* (0.047) 1 -0.289 (0.011) 0.003 (0.976) Work experience -0.041 (0.725) -0.106 (0.358) 0.640** (0.000) -0.314** (0.005) 1 0.522** (0.000) Supervisory experience 0.082 (0.476) 0.235* (0.040) 0.490** 0.000 -0.030 (0.798) 0.560** (0.000) 1

Spearman rank-order correlations are provided above the diagonal, Pearson correlations are provided below the diagonal. The P-values are provided between the brackets.

** correlation is significant at the 0.01 level (2-tailed) * correlation is significant at the 0.05 level (2-tailed)

The spillover effect and the impact of target difficulty