The effect of an objective performance measure’s weight on subjective performance evaluations

Amsterdam Business School

The effect of an objective performance measure’s weight on

subjective performance evaluations

Name: Dennis Karacan Student number: 10730680

Thesis supervisor: Prof. Dr. V.S. Maas Date: May 20h, 2016

MSc Accountancy & Control, specialization Control

Statement of Originality

This document is written by student Dennis Karacan who declares to take full responsibility for the contents of this document. I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it. The Faculty of Economics and Business is responsible solely for the

Abstract

This study explores the question whether the relative weight of an objective performance measure influences the discretion in the subjective performance evaluation. In an experimental setting, professionals in the financial service industry with solid experience in performance evaluation processes assessed a fictional case in which they were asked to provide their subjective evaluation of a manager. Four groups were defined, varying between a high or low objective performance and a high or low weight on the objective performance measure. An expected pattern of a higher applied discretion when a manager scores high on a low weighted objective measure was not supported by the results. Neither did the results support our expectation that when the score of the objective performance measure was low, no differences between the weight groups would be notified. Contrary to our expectations, when an employee scored low on an objective performance measure, the relative weight placed significantly affects the subjective evaluation given. Furthermore, when an employee scored high on an objective performance measure, the relative weight placed (high or low) does not affect the subjective evaluation given. Even though the results are contrary to our predictions and this study is bound to several limitations, it can be stated that the relative weight of an objective performance measure influences the discretion in the subjective performance evaluation in certain conditions. However, based on the findings of this study, a conclusive verdict concerning the effects of prescribed relative weights on discretion in performance evaluations remains somewhat premature.

Contents

2.1.3 Objective and subjective performance measures: A guidance on placing relative weights .. 9

2.1.4 Psychology based behavioral research ... 11

2.1.5 Favoritism ... 11 2.1.6 Evaluative disposition ... 12 2.1.7 Perceived fairness ... 12 2.2 Hypotheses development ... 13 3 Methodology ... 16 3.1 Independent variables ... 16 3.2 Dependent Variables ... 17 3.3 Case ... 17 3.4 Participants ... 18 3.5 Procedure ... 18 3.6 Post-experimental questions ... 18 4 Results ... 19 4.1 Sample selection ... 19 4.2 Sample descriptives ... 19 4.3 Hypothesis verification ... 20 5 Discussion ... 23

6 Limitations and suggestions for future research ... 25

7 Conclusion ... 26

Appendices... 28

1

Introduction

An important characteristic of most incentive contracts is the use of subjectivity in evaluating and rewarding employees (Murphy and Cleveland, 1995; Prendergast, 1999). In performance evaluations subjectivity and objectivity are often used in combination. Current literature elaborately describes the effect of subjective evaluations on objective performance measures.

Recently, moderations of these effects have been researched by Bol & Smith (2011). Specifically, they find that a lower controllability of the objective performance results in more objective adjustments to be made by supervisors, and, contrary to intuition, the opposite effect does not hold. Ittner Larcker & Meyer (2003) studied how different types of performance measures (e.g., financial versus nonfinancial, quantitative versus qualitative, drivers versus results) were weighted in subjective bonus computations. They find that the use of subjectivity in weighting the measures result in a reduction in the balance of the overall performance, that managers change criteria from quarter to quarter, ignore measures that were predictive of future performance, and weight measures that were not predictive of desired result. Even though studies have proven undesirable effects when incorporating subjectivity, the use of subjectivity allow evaluators to exploit any additional relevant information that arises during the measurement period to the benefit of both the firm and employee. The firm can benefit through improved incentive alignment and the employee can benefit through reduced risk (Gibbs et al, 2002).

One critical implementation issue that arises when incorporating multiple performance measures in reward systems is determining the relative weights to place on the various measures (Ittner et al., 2003). Several studies provide minor guidance on applying weights to objective and subjective performance measures. However, the factors that determine relative weights have been researched to some extent. Murphy & Oyer (2001) show that the weight on subjective relative to objective measures for CEO compensation contracts increases with the importance of the actions that affect firm value but do not affect the objective measure, and that the relative weight on subjective measures decreases as agents can more effectively manipulate these measures. Kaplan and Norton (1996) discuss the desired types of measures in balanced scorecards (financial and nonfinancial, external and internal, input and outcome, and objective and subjective). Although Kaplan and Norton (1996, 2001) provide little guidance how to combine or "balance" these disparate measures when evaluating managerial performance, they conjecture that the balanced scorecard renders subjective reward systems "easier and more defensible to administer and also less susceptible to game playing.” (Kaplan and Norton, 1996).

Objective performance measure Subjective performance evaluation Weight of objective performance measure in overall evaluation

Relatively few psychology-based performance evaluation studies directly examine the relative weights placed on financial and nonfinancial measures for compensation purposes (Ittner et al., 2003). This study aims to provide insights in the interaction between the objective financial performance measure and the relative weight placed on this objective performance measure in the overall subjective performance evaluation. Moderating the relative weight placed on either performance measure will possibly elicit effects in the subjective performance evaluation. However, it is not the ambition of this study to provide guidance on the optimal weights of either objective or subjective performance measures.

Several studies examine moderation effects of objectives performance measures on subjective performance evaluation. As stated earlier, Bol & Smith (2011) moderate their study with controllability. Kelly et al. (2015) studied the interactive effects of ex post goal adjustment and goal difficulty on performance. They find that objective ex post goal adjustments have positive effects on performance under a moderate level goals, but no effects on difficult goals. Both studies elaborate on the objective aspect of performance measures by moderating effects, either with controllability, or with goal difficulty. In this thesis, earlier research of Bol & Smith (2011) and Ittner et al. (2003) will function as a starting point to examine whether the relative weight of the objective performance measure influences the discretion in the subjective performance evaluation.

The given discretion when conducting subjective evaluations may differ with organizational culture. Bol & Smith (2010) and Kelly et al. (2015) both conducted their research amongst a participant group of supervisors or students at a University. Performing such an experiment among a contrasting organization might affect outcomes of these earlier studies. As the perception of fairness possibly differs between organizational cultures, our experimental setting will include participants from the financial services industry. This study might therefore support or strengthen existing literature when certain effects hold under a different participant group.

At this point I am not aware of studies that combine a moderation of relative weight with the effect that objective performance measures have on subjective performance evaluation, and will therefore provide possible new insights. Results might support contract designers when embedding objective performance measures in compensation contracts. Also, managers might take into account several aspects on performance evaluations they conduct for their employees as result of this research.

From here, this thesis will continue with a literature research in section two, in which the variables used in this research will be explained and the current state of knowledge of objectivity, subjectivity, and weight placement will be discussed. Thereafter, the methodology used for the research will be covered in section three, followed by the results in section four. In section five the findings will be discussed, after which suggestions for further research will be provided in section six. This research will end with the conclusion in section seven.

2

Theory

The first part of this section will consist of a literature review. The literature review will place this study in the context of management control at first by examining the agency theory and the object-of-control framework. Further on, the literature review will cover two research areas, namely, psychological- or behavioral based research on one hand, and agency-based research on the other hand. In the context of agency based research, the current state of knowledge of objective and subjective performance measures or evaluations and guidance on their relative weights will be discussed. In the context of behavioral based research, we will discuss the psychological effects affecting subjective performance evaluations. In the second part, the knowledge gained will be brought to practice in the hypothesis development.

2.1

Literature review

One commonly used theory in the Management Control discipline is Jensen & Meckling’s (1979) agency theory. They define an agency relationship as a contract under which one or more individuals (the principal(s)) engage another individual (the agent) to perform some service on their behalf which involves delegating some decision making authority to the agent. It is imminent that when decision making authority is delegated, divergencies will occur between both parties. The agent might not always act in the best interest of the principal due to asymmetry in information. After all, the principal is not able to continuously monitor the agent. In their study, Jensen & Meckling (1979) focus on how to structure the contractual relation between the principal and agent to provide appropriate incentives for the agent to make choices which will maximize the principal’s welfare, given that uncertainty and imperfect monitoring exists. Indeed, the principal can limit the divergence of interests by establishing appropriate incentives for the agent to minimize undesired outcome. When placing the implications of this study in the context of this thesis, we define the principal as one who will perform a performance evaluation (the manager), and the agent as one who will be evaluated (the employee).

2.1.2 Result controls

Management Control Systems (MCS) are defined broadly to include everything managers do to help ensure that their organization’s strategies and plans are carried out or, if conditions warrant, that they are modified (Merchant & Van der Stede, 2012). While MCS’s are broadly studied,

studies in the field of performance measures, performance evaluations, and incentives cover a great part of MCS studies in general. Merchant & Van der Stede (2012) distinguish four control types in their object-of-control framework: result controls, action controls, personnel controls, and cultural controls. A powerful way to influence behavior in organizations is pay for performance. This is a proper example of a type of control that can be categorized under result controls as it involves rewarding agents for generating results (Merchant & Van der Stede, 2012). Result controls will cause employees to manage the consequences of actions they undertake to generate the result. The organization will not provide guidance on what actions to undertake as it is in the employees best interest to undertake the right actions to generate the required result.

2.1.3 Objective and subjective performance measures: A guidance on placing relative weights

It appears that accounting data is often used as a basis for determining salary increases, job transfers, and promotions (Indjejikian, 1999). However, compensations in these forms based merely on objective financial accounting data has been criticized for encouraging managers to lay focus on short-term results and in place, sacrifice the long run performance in interest of the organization (Ittner et al., 2003). This hurdle is often overcome by implementing different measures in compensation plans, from quantitative nonfinancial metrics (e.g. employee and customer survey results), to qualitative assessments of performance by the manager’s supervisor. Determining the weight of each measure when several are combined in compensation plans is a critical implementation issue (Ittner et al., 2003). In the upcoming paragraphs we will draw upon literature to provide guidance on the several aspects of placing relative weights on either objective or subjective performance measures.

Gosh & Lusch (2000) study a frequently observed principle in a field study: the outcome effect. The performance evaluations of managers in a retail chain show that subjective evaluations of store managers by their supervisors were negatively impacted by unfavorable outcome knowledge. That is, outcome determinants over which the managers have control influence their performance evaluations and environmental determinants of outcome over which they have no control do not influence their evaluations (Gosh & Lusch, 2000). Consistent with this study, it is assumable that greater weight will be placed on outcome measures than on the measures of the drivers of this outcome, even though these measures are informative of the manager’s actions. This is consistent with the definition of result controls which Merchant & Van der Stede (2012) define in their object-of-control framework.

The reliability of either performance measure is a feature that is often studied. Blum and Naylor (1968) for example,. state that greater weight should be placed on performance measures that are more reliable. Besides this intuitive statement, they argue that subjective qualitative performance evaluations are often less reliable than objective quantitative measures. The cause of a lesser reliability is the influence of the rater’s bias on the subjective, qualitative performance measure. The rater’s bias is the tendency of supervisors to judge an employee as either good or bad and then to seek out evidence supporting that earlier established opinion (Müller & Weinschenk, 2015).

Prendergast and Topel (1996) elaborate on bias-effects by examining how favoritism and bias in subjective performance evaluations affect the relative weights placed on objective versus subjective performance measures in compensation plans. They find that subjectivity enables bias and favoritism, and thus, supervisors act on personal preferences towards employees to favor some over others. Additionally, they indicate that due to favoritism, little weight is placed on subjective evaluations and too much weight on objective performance measures when multiple indicators are used in combination to constrain favoritism and reduce noise.

The dominance of the role of accounting data in performance evaluation and compensation plans is thoroughly explained in earlier research. The variety of controls and audits on accounting data enhances their reliability. Thus, when using a high weighted accounting measure as a performance measure, we meet the statement of Blum and Naylor (1968). Additionally, managers and employees understand the impact of their performance on accounting based measures much better than they understand how their actions affect other non-financial indicators of performance. (Indjejikian, 1999).

Even though literature elaborately describes positive consequences of placing more relative weight on objective performance measures, several drawbacks are worth mentioning. To continue on the discussion of the dominance of accounting data, greater understandability often implies greater opportunities for “gaming” performance measures. Additionally, measures based on accounting data are often backward-looking and will therefore not adequately reflect all economic consequences of current actions and decisions (Indjejikian, 1999).

Incentive contracts often include important subjective components that mitigate incentive distortions caused by imperfect objective measures. In extreme cases, sufficiently effective explicit contracts can even render all implicit contracts infeasible, even those that would otherwise yield the first-best (Baker et al, 1994). Baker et al (1994) provide additional findings worth mentioning when arguing why subjectivity must not be left unnoticed. In some

circumstances objective and subjective measures are complements: neither an explicit nor an implicit contract alone yields positive outcome, but a combination of the two does. An ideal performance measure would reflect an employee’s contribution to the firm value, not only on the short term, but especially on the dynamic effects of current actions on the long-term. Therefore, placing high weights on backward-looking objective measures will give rational agents a short-term focus which will likely destroy long-short-term company value (Bol, 2008).

The introduction of subjectivity can improve incentive contracting because it allows value-enhancing efforts that are not easily quantified to be included. This prevents the agent from directing his effort solely towards the objectively measured task. Furthermore, objective performance measures are often contaminated by uncontrollable random events such as a financial crisis. Subjectivity can play a beneficial role by allowing the principal to filter out the noise of the uncontrollable events that affect the objective performance measure. Also, as Indjejikian (1999) mentions, when understandability of objective performance measures increase, it will become more susceptible to manipulations in the form of “gaming”. Subjectivity can reduce the vulnerability to gaming because part of the performance is assessed ex post rather than measured according to measures that are set ex ante. This provides the principal with the possibility of punishing the agent when manipulation is detected (Bol, 2008). Overall, we can conclude that indeed placing high incentive weights on incomplete objective measures will make agents ignore unmeasured job dimensions and will so lead to distorted incentives (Bol, 2008).

2.1.4 Psychology based behavioral research

Ittner et al (2003) indicate that behavioral-based explanations may be equally or more predictive than agency-based explanations when studying the role of subjectivity in compensation contracts. Hence, we will discuss several behavioral effects that can occur when subjective evaluations are performed.

2.1.5 Favoritism

Prendergast and Topel's (1996) define favoritism as a supervisors act on personal preferences toward subordinates to favor some employees over others. In their study, they find that favoritism leads firms to place too little weight on supervisor appraisals and too much weight on objective performance measures when combining multiple indicators in order to constrain favoritism. This finding might imply that when indeed little weight is placed on subjective

performance measures (e.g. by organizational compensation policy), a supervisor is inclined to provide more extreme scores for the subjective evaluation in order to have a bigger influence on the overall performance evaluation.

2.1.6 Evaluative disposition

Bond et al (2007) discuss that subjective judgments can be unduly influenced by an individual’s knowledge of other, unrelated information. In their study they elaborate on the so called ‘evaluative disposition’. The evaluative disposition is, as older literature describes (Heider, 1946), a specific cognitive element, generated early in the decision process, to which other elements of the decision are aligned. So, information viewed early in the decision process evokes an initial evaluative disposition, and this disposition influences the interpretation of information viewed later. For example, in the eyes of an individual with a highly unfavorable initial disposition, attributes which are objectively “neutral” towards the option may instead seem quite negative, and even “positive” attributes will appear less favorable (Bond et al, 2007). Additionally, they argue about explanations of this evaluative disposition and find these in twofold. First, the biased search from memory elicits a recency effect. When judging from memory, one is likely to form the initial disposition based the value which is remembered from most recent memories (recency effect), and subsequent judgments are adjusted towards this value. Second, the biased search from online judgments elicits a primacy effect. When judging from information that is retrievable, one is likely to form the initial disposition based on the value which the evaluator is able to judge first. In our experimental design, we will make use of small notes and informal interviews which should represent an average performance of an office administration task. We expect that the average performance setting of this subjective measure will not be strong or distinct enough for the evaluator to form an initial disposition based on this subjective information. Furthermore, our participants are asked to provide the subjective evaluation on a point at which the information is no longer retrievable (by means of a modular survey).

2.1.7 Perceived fairness

An aspect that the traditional agency model has not captured, but that social scientists have long recognized, is the importance of justice in compensation contracting. Employee incentives are not solely determined by the relation between pay and performance but also on the perceived fairness. (Bol, 2008). In general, literature distinguishes two types of fairness: 1) the fairness of the outcome distributions (distributive justice) and 2) the fairness of the procedure used to

determine these outcome distributions (procedural justice). Lack of either of the two is likely to result in reduced employee morale. We predict that when the score on the objective measure is high, but the relative weight placed by organizational incentive policy on the objective measure is low, the upward adjustments made by the evaluator will be high. In other words, when the score on the objective performance measure is high, we expect the evaluator to provide a higher score on the subjective evaluation when the weight placed is low than when the weight placed is high as the evaluator might perceive the low weight of a high score as unfair.

2.2

Hypotheses development

The literature review on the agency-related research will provide a guidance on the manipulations of the relative weight that will be placed on the objective performance measures. In light of our hypotheses development we will mostly draw upon the psychological literature.

As stated by Bol (2008), employee incentives are not solely determined by the relation between pay and performance but also on the perceived fairness. In this light, we predict that when the score on the objective measure is high, but the relative weight placed by organizational incentive policy on the objective measure is low, the applied discretion by the evaluator will be higher than when a high weight is placed on the objective performance measure. The evaluator might perceive the low weight of a high score as unfair. In the same context, when both the score on the objective performance measure and the relative weight placed is high, we expect the evaluator to perceive the process as fair. Thus, the evaluator will expectedly not be elicited to apply discretion in this condition.

In line with the study of Prendergast and Topel (1996), we expect that when indeed little weight is placed on subjective performance measures (e.g. by organizational compensation policy), a supervisor is inclined to provide more extreme scores for the subjective evaluation in order to have a bigger influence on the overall performance evaluation.

Heider (1946) and Bond et al. (2007) both state that information viewed early in the decision process evokes an initial evaluative disposition, and this disposition influences the interpretation of information viewed later. In order to evoke such a disposition, cues of information throughout the process must deviate from each other. Processing a high objective performance score and a low weight of this score is contradictory and is therefore expected to evoke an initial disposition. We expect this to result in a higher applied discretion in the high objective score and low weight condition.

Summarized, we expect the following hypothesis to hold:

H1: When the employee’s performance on an objective measure is relatively high, the subjective evaluation score will be higher when there is a low weight placed, than when there is a high weight placed on the objective performance measure

When the weight of the objective performance measure and the performance on this actual measure are both low, it might be possible that the initial disposition of the evaluator towards a negative score can be weaker compared to a high weight. After all, both cues of information are negative of nature and do not deviate from one another. An initial disposition might therefore not occur. The evaluator could perceive the objective measure as less important and might therefore be less inclined to align his subjective score with the objective performance. We expect this to result in a subjective score that is slightly above the performance of the objective performance measure.

On the other hand, when the weight of the objective performance measure is high, and the performance on this measure is low, the evaluator might find that the weight placed on the other job dimensions, which are included in subjective information cues, is low. The evaluator might therefore be inclined to compensate for this low weight and can therefore provide a slightly higher subjective score than the objective performance measure. The motive of the evaluator can be to reward the evaluatee for the efforts that are not easily quantified, which will else remain unrewarded. This is in line with the study of Indjejikian (1999) where he states that subjectivity can improve incentive contracting because it allows value-enhancing efforts that are not easily quantified to be included

Comparing the high and low weight condition when the objective performance measure is scored low, we expect that both conditions that the subjective score will positively deviate from the objective performance, but we expect no significant difference on the height of this subjective score between the two conditions.

Summarized, we expect the following hypothesis to hold:

H2: When the employee’s performance on an objective measure is relatively low, the subjective evaluation score is not affected by the weight placed on the objective performance measure

Both hypotheses are visualized in the graphs below, where OP stands for Objective Performance.

3

Methodology

The research methodology will consist of a 2x2 between subjects design. I will draw upon the experimental instrument of Bol & Smith (2011) which they used in their study to elicit a moderation effect of controllability of the objective performance measure on subjective performance evaluation. The instrument has been modified to suit the purpose of this study. The hypothetical case setting requires participants to complete a performance evaluation task, assuming the role of a regional director for an industrial pipe and fitting company. The regional director’s job is described as overseeing a number of district managers who are responsible for both individual sales and district office administration. The experimental task is to perform a subjective evaluation of the office administration performance of one of the district managers, while the manager’s score on an objective individual sales measure is known, and its weight in the overall performance evaluation system is manipulated.

3.1

Independent variables

The objective measure used in the performance-evaluation system is the district manager’s individual sales during the most recent twelve-month period. The measure is scored on a 0–10 scale, with each score connected to a particular range of sales. The level of the target district manager’s performance on the sales dimension is manipulated by varying his individual sales score. Specifically, the district manager’s individual sales score is stated as 2 in the low, and 9 in the high sales condition. The manipulation on the performance of the objective measure as used in the study of Bol & Smith (2011) has been used unaltered for this study. However, the table in which the actual scores of the regional director and his peers are illustrated has been modified. By changing the names in the table it is attempted to illustrate a more representative view of gender and nationality in the financial services industry.

The relative weight of the objective performance measure is manipulated by varying the percentage of the part in the overall performance-evaluation system that consists of the sales measure. Thus, in the hypothetical case setting, participants are not only informed of the objective performance measurement score, but also of the relative weight of the score as part of the overall performance, stated by the organizations evaluation policy. Evaluators are provided with information about their organization’s evaluation policy, which informs them to place either 30% (low relative weight condition) or 70% (high relative weight condition) on the objective financial performance measure (sales).

3.2

Dependent Variables

The dependent variable is the subjective evaluation of the district manager’s office administration performance. In order for participants to evaluate the district manager, the case information includes a collection of personal notes and interview responses from the manager’s office staff. This information will remain unaltered among all treatment groups. After reading the case information, participants provide a performance-evaluation score, on a scale from 0 to 10, reflecting their assessment of the district manager’s office administration performance.

3.3

Case

Contact has been made with associate professor J. Bol, one of the authors of the research paper “Spillover Effects in Subjective Performance Evaluations”, in order to receive the instrument which they utilized to conduct their research. As stated earlier, several adaptions have been made in order to make the instrument fit for the purpose of this study. The objective financial performance measure (sales) as designed in the instrument of Bol & Smith (2011) remains unchanged for this study as manipulations considering the sales score were successful. The moderation effect will encompass the manipulation of a high or low relative weight on the objective performance measure. As this manipulation was not part of the original instrument, an additional section has been added, explaining the contents of the overall performance evaluation consisting of a high/low weight of either the objective or subjective score. The study of Bol & Smith (2011) included information relating the controllability of the objective performance measure. As this is not in any form related to this study, these sections have been excluded. Furthermore, the case has been translated from English to Dutch as it is expected that the majority of the participants will have Dutch as their native language. This enables the participants to focus on the contents.

I pre-tested the case with four native Dutch speakers in order to investigate whether they understand what is asked of them, if manipulations have worked, and whether they understand the consequences of the manipulations. Each tester represented an experimental group and were therefore assigned to a different case representing the experimental group (for instance: low sales, high weight). The pre-test resulted in manipulations being perceived as expected and were therefore deemed successful. However, slight textual modifications have been made to the case in order to increase the understandability. The final experimental instrument used for this study is included in appendix I.

3.4

Participants

The participants of this experiment will consist of professionals in the financial services industry who fulfill a supervisory role or have fulfilled a supervisory role in the past 5 years. Business unit managers and their subordinate managers within several Our experimental design will consist of four groups, with a fitting case in each of the following conditions:

• Low objective performance measure, high relative weight placed

• Low objective performance measure, low relative weight placed

• High objective performance measure, high relative weight placed

• High objective performance measure, low relative weight placed

3.5

Procedure

Each case will be made available by online survey tooling. Each participant is provided the same URL, after which the tooling redirects the participant to one of the four cases which has the least responses at that time. Thus, participants will be randomly assigned to each of the conditional groups in such manner that n will be equally divided.

3.6

Post-experimental questions

For descriptive and validation purposes, participants will be asked to answer a number of post experimental questions. Several questions will be asked to ensure the participants represent the audience as desired in this study: professionals in the financial services industry who fulfill a supervisory role or have fulfilled a supervisory role in the past 5 years. Furthermore, two questions will be asked to ensure the desired manipulations have been perceived as intended. One question is added to ensure whether the provided subjective information was sufficient to provide an evaluation. To prevent participants reconsidering their answers given for the experiment after reading the post-experimental questions, it is made impossible to return to the experiment section after the section of post-experimental questions has been reached.

4

Results

4.1

Sample selection

The post-experimental questionnaire included two manipulation check questions. The manipulation checks confirmed the manipulations in all four conditions, and thus, the weight of the objective performance measure (sales), and the performance on the objective performance measure were perceived correctly as high or low, dependent on their assigned condition. 6 out of 72 respondents perceived either one of the manipulations incorrectly. These respondents have been excluded from our sample.

From all the remaining respondents, two had to be excluded due to an unlikely registration of age and/or the years of experience in performance evaluations. Furthermore, one respondent had to be excluded due to an unfinished survey. So, from a total of 72 respondents, 9 had to be excluded from further analysis in order to obtain a clean dataset of 63 respondents.

4.2

Sample descriptives

The participants in the sample are 63 professionals working in the financial service industry, varying between several branches like banking, financial consulting, and accounting. Ideally, the size of the sample among the four conditions is equal, and this is confirmed by n in Table 1. Table 1 provides the sample’s descriptive statistics. On average, participants were 42,62 years old, varying between the ages of 22 and 64. No significant differences in age exist between the four conditions. Of the total sample, 17 out of 63 (27%) of the respondents are female. The educational level of the participants shows a mean of 7,14, in which 7 encompasses the level of University of Applied Sciences, and an 8 encompasses an education at University level (either bachelors or master’s degree). An average of 17,51 years of experience in the financial service industry is reported, varying between an average of 15,88 in the High Sales, Low Weight condition, and 20,56 in the Low Sales, Low Weight condition.

The participants have an average of 5,1 years of experience in giving performance evaluations to subordinates. This confirms our requirement of the targeted audience of this study. The average varies between 4,31 in the High Sales, High Weight condition, and 6,93 in the Low Sales, Low Weight condition. However, these differences are not expected to create a direct difference in the mean of the subjective performance evaluation.

We asked participants to rate their agreement with a statement that the personal notes and interview responses about the target manager allowed them to make a reasonable judgment about the manager’s office administration performance on a six-point scale. The mean of this measure was 4,92, significantly higher than the scale midpoint of 3.5. These results suggest that participants felt they had sufficient information on the manager’s office administration performance to make a reasonable assessment.

4.3

Hypothesis verification

Hypothesis one assumes a high performance on sales. Under this condition, it is expected that the subjective evaluation score will be higher when there is a low weight placed on the sales measure, than when there is a high weight placed. Table 2, which provides the means of the sample, reveals that the expected effect does not occur. With n = 16 in both high sales groups, the means are identical. Using these means as input for an independent samples t-test, the p-value logically results in a p-value of 1.000. These p-values imply that the null hypothesis, which assumes that there is no significant difference between both groups, cannot be rejected. In summary, there is no difference between the observed means of both groups. Consequently, the data does neither support H1 nor a potential contrary direction.

Table 2: Sample means

Hypothesis two assumes a low performance on sales. Under this condition, it is expected that the subjective evaluation score will not differ regardless of a high or low weight placed on the sales measure. The means in the low sales condition in Table 2 reveal a difference in the high and low weight condition. A mean of 4,33 is reported for the high weight condition, and 5,63 in the low weight condition. These means form the input for an independent samples t-test. Under the conditions of equal variances (p = 0,578) and a two-tailed significance level of 5%, a p-value of 0,025 is reported. This confirms a significant difference in means between the low and high weight condition. These values imply that H2, which assumes that there is no significant difference between both groups, does not hold.

In order to complement the separate t-tests the ANOVA is able to provide a test of whether the interaction between sales score and sales weight shows statistical significance. The results are displayed in table 3. The within group analysis of sales score reports a significant difference (p = 0,000) at a confidence interval of 90%. Both the within group analysis of sales weight, as well as the between group analysis of sales score and sales weight report a significant difference with a p-value of 0,054 at a confidence interval of 90%. It should be noted that for most analyses a confidence interval of 95% is used. However, the significance of the latter result is that close to a 95% threshold that I find it worth reporting as significant.

Figure 3 summarizes the results as found in our earlier tests. Contrary to H1, there is no reported difference in subjective evaluation means between a high or low weight within the high sales group. H2 does not hold either, as it assumes equal subjective evaluation means between a high or low weight within the low sales group. However, an interaction effect between sales score and sales weight is found at a confidence interval of 90%. The results imply that when a manager performs well on an objective performance measure, the subjective evaluation given is not affected by the weight of the objective performance measure. However, when a manager performs low on an objective performance measure, the subjective evaluation given is higher when the weight of the objective measure is low, compared to a high weight.

5

Discussion

The hypotheses of this research are based on the current state of knowledge of objective and subjective performance in the context of agency based research, combined with the psychological effects affecting subjective performance evaluations in the context of behavioral based research.

In that light, I predicted that when the employee’s performance on an objective measure is relatively high, the subjective evaluation score will be higher when there is a low weight placed, than when there is a high weight placed on the objective performance measure. When organizational policy prescribes that the score of the objective performance measure consist of a small portion and the subjective evaluation of a high portion in the overall performance evaluation, but the employee being evaluated scored high on the objective performance measure, the manager is tempted to compensate for this low appreciation by providing a higher subjective evaluation score, resulting in a higher overall performance evaluation. We predicted that this would occur as this situation might be perceived as unfair by the manager. The incentive for the manager to compensate on the subjective evaluation is that else a lack of motivation among the employee may arise.

Contrary to our prediction (H1), we find no statistical difference in the subjective evaluations given between the high or low weight conditions of the objective performance measure when the employee’s performance on this objective performance measure is high. A possible explanation for this finding might also be found in a study of Wilks (2002). He finds that individuals tend to unknowingly process information in a way that favors a previously held belief, desired outcome, or developing preference. In context of our research, this might imply that the high objective performance measure is the first point of information that is perceived by the evaluator. In our fictional case, the sales measure is the first assessable piece of information, which may cause the evaluator to process information thereafter as less important. This is also supported by Baily (2011). They argue that supervisors process information on a piecemeal approach, considering the implications of each information cue separately. Managers will rely on the anchoring heuristic, thereby choosing some piece of information as a starting point for the allocation, and processing additional information cues as qualitative adjustments from that starting point. An interesting addition to this notion is that managers using a piecemeal approach to process information, tend to incorporate noncontractible information to a lesser degree. This might be a plausible explanation that in our findings, the weight aspect and the subjective

information is overruled by the high performance of the objective performance measure, causing the means of the subjective evaluation to be equal between the high and low weight group.

For the second hypothesis of this study, we predict that when the employee’s performance on an objective measure is relatively low, the subjective evaluation score is not affected by the weight placed on the objective performance measure.

In contradiction to our hypothesis (H2) we find a statistical difference in means of subjective evaluations given in the low sales group between the high and low weight condition. An intuitive explanation could be that the effect of the initial disposition is weakened at a low score on the objective performance, resulting in the weight aspect and the subjective information to be taken more into consideration by the evaluator. This would ultimately result in the evaluator to apply discretion in the overall score in the low weight condition by providing a higher subjective evaluation score as the evaluator will perceive the low weight condition as opportunity to reward the evaluatee for the efforts that are not easily quantified. After all, the applied the discretion in the form of a higher subjective evaluation score, will have a greater impact due to the higher weight placed on the subjective score.

Additional to this, Bol & Smith (2010) argue that a low objective performance score elicits supervisors to take the controllability of the measure into consideration. Our fictional case did not include information indicating the level of controllability of the objective performance measure. However, it might be possible the organizational policy, prescribing the weights of the objective and subjective aspects, and some subjective information, is perceived as an uncontrollable factor. It might have occurred that participants applied discretion with the intention to correct for these uncontrollable factors. Blakely (1993) adds additional arguments to this statement. He argues that when the low performance of an employee is attributed to external factors by the supervisor, employee’s will react less negatively opposed to when the low performance is attributed to internal factors such as a lack of ability and effort. This indicates the increased likeliness of the supervisor to apply discretion in the subjective performance evaluation.

6

Limitations and suggestions for future research

This study is bound to several limitations. First of all, the experimental instrument consists of subjective pieces of information based on fictional notes and conversations between supervisor and employees. Even though the instrument has been pre-tested, post experimental questions verified the usability, and the subjective part being equal in all four groups, room for individual interpretation remains. Second, due to a fictional setting of the case, participants attribute evaluation scores, but the (possible confrontational) conversation with the employee is left out. This might cause participants to attribute a lower evaluation score more easily opposed to a working environment where commonly the employee is provided feedback, enabling the supervisor to explain or defend the attributed score. Third, the sample size is not as large as anticipated which slightly reduces the reliability of the statistical analysis. In order to increase the reliability of the results, and enable a more in depth analysis of demographic differences, future research should ensure a greater sample size. Finally, as stated by Wilks (2002), Baily (2011), and Bond et al (2007), individuals tend to unknowingly process information in a way that favors previously held belief, and when judging from information that is retrievable, one is likely to form the initial disposition based on the value which the evaluator is able to judge first. These studies emphasize the possible effects of the order in which the information is placed. Focusing on assessable pieces of information, our case start with the weights placed, followed by the objective score. Thereafter, the subjective information is provided. Having the assessable information ordered differently might have affected the results of the study. Therefore, interesting for future research is to elaborate on the effect of the ordering of assessable information on subjective performance evaluations.

7

Conclusion

In performance evaluations subjectivity and objectivity are often used in combination. Current literature elaborately describes the effect of subjective evaluations on objective performance measures. This study adds a moderating effect of the relative weight applied, and answers the question whether the relative weight of an objective performance measure influences the discretion in the subjective performance evaluation. The contribution of this paper is in twofold. Not earlier has the moderating effect of the relative weight been incorporated in an experimental research setting combined with objective and subjective performance evaluations. Second, a significant amount of studies in this line of research use students as participants for their experiments. Our study includes participants in a managerial function in the financial services industry, having an average of 5 years of experience in providing performance evaluations.

It was argued that when an employee scored high on an objective performance measure, but this high score was being weighted low in the overall performance evaluation, the supervisor would apply discretion by providing a higher subjective evaluation than in the high weight condition. Furthermore, we expected the relative weight not to affect the subjective evaluation when the employee scored low on an objective performance measure.

The results do not support these expectations. When an employee scored high on an objective performance measure, the relative weight placed (high or low) does not affect the subjective evaluation given. Contrary to our expectations, when an employee scored low on an objective performance measure, the relative weight placed does significantly affect the subjective evaluation given. When the low objective performance is weighted low in the overall evaluation, more discretion is applied by the supervisor by giving a higher subjective evaluation. That is, more discretion than when the low objective performance is weighted high in the overall evaluation. Research supports this finding by arguing the controllability of the performance measures, as to where the prescribed relative weight decreases the controllability, causing an increase in applied discretion.

Based on the findings of this study, a conclusive verdict concerning the effects of prescribed relative weights on discretion in performance evaluations remains somewhat premature. It might be promising for future studies to incorporate post-evaluation elements where supervisor is able to provide feedback and explanation on the attributed score towards subordinate. Furthermore, a statistical analysis based on a larger sample would strengthen the reliability of the results. The ordering in which information is processed also proves to be an unavoidable element to incorporate in future studies. Even though the results are contrary to our predictions, and this

study is bound to the aforementioned limitations, it can be stated that the relative weight of an objective performance measure influences the discretion in the subjective performance evaluation in certain conditions. One should therefore take this into account when implementing performance evaluation systems. Since evaluation practices remain on topic today, questions that remain in this line of study will thrive further academic research in this area.

Appendices

Regio-directeur Manager Administratief personeel Sales personeel

Openingstekst

Dank voor uw deelname aan dit onderzoek!

Voor dit onderzoek wordt u gevraagd een fictieve casus te lezen en daarover een aantal vragen te beantwoorden. Er zijn geen juiste of onjuiste antwoorden, wij zijn geïnteresseerd in uw individuele professionele oordeel. Deelname aan het onderzoek zal ongeveer 10 minuten van uw tijd in beslag nemen.

Uw antwoorden worden vertrouwelijk behandeld en zullen niet naar individuele deelnemers getraceerd kunnen worden. De gegevens worden uitsluitend gebruikt voor een afstudeeronderzoek.

Casus

Verplaats u in de volgende situatie: De organisatie

U bent een regiodirecteur bij Dexter Buizen & Fittingen BV, een bedrijf dat buizen en fittingen fabriceert en verkoopt. Onder uw regio vallen 10 managers die allemaal aan u rapporteren. Iedere manager heeft verschillende verantwoordelijkheden in twee hoofdgebieden:

• Sales

Van managers wordt verwacht dat zij ongeveer 50% van hun tijd besteden aan het genereren van sales.

• Administratieve werkzaamheden

Van managers wordt verwacht dat zij ongeveer 50% van hun tijd besteden aan het beheren van het regionale kantoor, wat inhoudt dat budgetten gemonitord moeten worden, administratie verwerkt moet worden, rapportages en andere documenten opgesteld moeten worden, en administratief personeel gemanaged moet worden.

De Beoordeling

Ieder jaar worden de managers beoordeeld. De totale beoordeling bestaat uit twee onderdelen: een objectieve beoordeling gebaseerd op sales, en een subjectieve beoordeling gebaseerd op de administratieve werkzaamheden.

De organisatie heeft besloten dat twee beoordelingsonderdelen nodig zijn omdat de twee hoofdgebieden (sales en administratieve werkzaamheden) compleet onafhankelijk van elkaar zijn. Met andere woorden, de prestatie op één gebied is niet indicatief voor de prestatie op een ander gebied.

Voor beide beoordelingsonderdelen worden de prestaties gescoord op een schaal van 0 tot 10. Het organisatiebeleid schrijft voor hoeveel procent de objectieve en subjectieve beoordeling gewogen worden in de totale beoordeling. De objectieve beoordeling gebaseerd op sales weegt voor 70% (30%) mee, en de subjectieve beoordeling gebaseerd op de administratieve werkzaamheden weegt voor 30% (70%) mee.

Aan u de taak om de administratieve performance van één van de managers, Jordy van Oosanen, te beoordelen.

De Prestatie

Het eerste beoordelingsonderdeel van de totale beoordeling is gebaseerd op de individuele sales van de manager van het afgelopen jaar. Iedere manager krijgt één punt voor iedere €50,000 aan verkopen die hij of zij genereert. Onderstaande tabel toont de sales scores voor de tien managers in uw regio van het afgelopen jaar. Iedere manager bedient ongeveer een even groot marktgebied met een vergelijkbaar aantal potentiele klanten.

Individuele Sales

Manager Sales Score

Stian Thoresen € 539.562 10

Jordy van Oosanen / Daniel Wesseling € 476.553 9

Rob van der Loo € 375.144 7

Mustafa Çalinalti € 346.848 6

Michael Hal € 266.018 5

Anwar Almedani € 240.757 4

Willemijn van den Berg € 249.059 4

Coen Janssen € 150.321 3

Jordy van Oosanen / Daniel Wesseling € 115.758 2

Ferdy Akerboom € 46.715 0

Het tweede beoordelingsonderdeel van de totale beoordeling is een score gebaseerd op de administratieve performance van Jordy van Oosanen. Gebaseerd op interacties die u heeft gehad met Jordy heeft u enkele aantekening van het afgelopen jaar verzameld. Deze aantekeningen vindt u hieronder.

Als regiodirecteur bent u niet in staat om dagelijks het directe gedrag te observeren. Daarom heeft u ook enkele collega’s in Jordy’s team van administratief personeel ondervraagd. De reacties over de prestatie van de administratieve werkzaamheden van Jordy zijn hieronder te lezen.

Aantekeningen over Jordy van Oosanen, Manager

Was niet op tijd op kantoor de eerste ochtend dat ik langs kwam... Is altijd vrolijk, lijkt in een goede stemming te zijn….Het personeel mag hem en respecteert hem….. Zijn kantoor is redelijk rommelig, schijnt niet te veel te geven om administratief werk….Budgetrapporten zijn vaak op tijd, budget schattingen zijn vrij accuraat….Wat kleinere issues met accuraatheid en volledigheid van andere rapporten en papierwerk, maar over het algemeen OK….kan prima overweg met andere managers, maar is niet een hele sociale jongen op bijeenkomsten.

Reacties administratief personeel van Jordy van Oosanen

Vraag: Kan je commentaar geven op Jordy’s administratieve performance van het afgelopen jaar? Marjolein van Loon, administratief medewerkster:

Jordy levert vrij goed werk als het gaat om hoe hij omgaat met de dingen hier op kantoor. Ik denk niet dat administratieve werkzaamheden echt zijn ding zijn, maar hij is in staat om te doen wat gedaan moet

worden. Hij maakt snel beslissingen, dat is soms goed en soms niet zo goed. Hij communiceert duidelijk - dat

is zeker een van zijn echte sterke punten. Maar hij houdt er niet van om te gaan met conflicten en meningsverschillen tussen het kantoorpersoneel - hij geeft de voorkeur aan hen met rust te laten in de hoop dat het zichzelf oplost. We hebben regelmatig vergaderingen die hij voorzit. Maar hij heeft de voorkeur om anderen het woord te laten doen. Als we praten over iets waar hij om geeft, betrekt hij zich bij de discussie en heeft hij goede inzichten. Maar als dat niet zo is trekt hij zich terug van de discussie en wacht hij op het volgende onderwerp.

Nicole de Jong, administratief medewerkster:

Jordy is zeer aanspreekbaar, en dat helpt veel. Via e-mails of telefoontjes is hij niet heel goed te bereiken, dus ik loop meestal langs om met hem te praten. Maar hij is gemakkelijk in de omgang als je ‘m eenmaal te pakken hebt. Het is duidelijk dat Jordy extreem veel kennis heeft over de business en zo’n beetje alles kan verklaren. Hij is erg vriendelijk, iedereen houdt van hem, en ik denk dat iedereen hem vertrouwt. Voor het grootste gedeelte laat hij ons onze gang gaan, en dat helpt ons productiever te zijn.

Jasper Stam, administratief medewerker:

Jordy is een aardige vent en een leuke baas, maar ik zou niet zeggen dat administratieve werkzaamheden echt zijn ding zijn. Hij is niet echt dol op de computer – ik help hem vaak met relatief eenvoudige dingen. Ook is hij niet erg georganiseerd. Ik weet niet hoe hij de dingen gedaan krijgt in dat kantoor van ‘m – Ik ben meestal bang om daar naar binnen te lopen. In principe heeft hij goede mensen voor hem werken, dus er wordt goed werk verricht. Aan de andere kant, hij is degene die ons aangenomen heeft. En hij erkent dat wij beter weten hoe dingen zouden moeten gaan dan hij, en hij laat ons vaak die beslissingen nemen. Dat is een mooie eigenschap.

Uw Beoordeling

Hier geeft u uw beoordeling over Jordy’s administratieve performance. Deze score wordt voor 30% (70%) meegewogen in zijn totale beoordeling van het afgelopen jaar. De andere 70% (30%) is gebaseerd op de sales score.

Geef hieronder uw beoordeling over Jordy’s administratieve performance. Het cijfer dat u invoert moet tussen de 0 (laagste) en 10 (hoogste) zijn.

Manager Jordy van Oosanen

Beoordelingsonderdeel 1: Sales score [0-10] 9 (2)

Beoordelingsonderdeel 2: Score administatieve werkzaamheden [0-10] Uw beoordeling Bereken de overall gewogen score van Jordy middels de volgende berekening:

( 0,7 x 9 ) + ( 0,3 x [Uw beoordeling] )

Gewogen score

Aanvullende vragen (Post-experimental questions)

1. Wat is uw leeftijd? 2. Wat is uw geslacht?

o Man

o Vrouw

3. Wat is uw hoogst genoten opleiding?

o Geen o VMBO o HAVO o VWO o MBO o HBO o WO o Anders, namelijk…

4. Hoeveel jaar werkervaring heeft u in de financiële dienstverlening?

5. Hoeveel jaar ervaring heeft u in het beoordelen van prestaties van medewerkers (eventueel vanuit een leidinggevende functie)?

Aanvullende vragen (Manipulation Checks)

6. Ik vond de objectieve score (sales):

• Laag

• Hoog

7. Ik vond de weging van de objectieve score (sales):

• Laag

• Hoog

8. Op basis van de aantekeningen en gespreksnotities kon ik een redelijke beoordeling geven over de prestaties van Jordy zijn administratieve werkzaamheden

References

Bailey, W. J., G. Hecht, and K. L. Towry. “Dividing the pie: The influence of managerial discretion extent on bonus pool allocation”. Contemporary Accounting Research, (2011). Baker, George, Robert Gibbons, and Kevin J. Murphy. “Subjective performance measures in

optimal incentive contracts”. Quarterly Journal of Economics 109, nr. 4 (1994): 1125.

Blakely, G. L. .“The effects of performance rating discrepancies on supervisors and subordinates”. Organizational Behavior and Human Decision Processes, (1993): 57–80.

Blum, M. L., and J. C. Naylor. “Industrial Psychology”. New York, NY: Harper & Row, (1968) Bol, Jasmijn C. “Subjectivity in compensation contracting”. In AAA Management Accounting

Section (MAS) 2006 Meeting Paper, 2008.

Bol, Jasmijn C., and Steven D. Smith. “Spillover Effects in Subjective Performance Evaluation: Bias and the Asymmetric Influence of Controllability”. Accounting Review 86, nr. 4 (2011) Bond, Samuel D., Kurt A. Carlson, Margaret G. Meloy, J. Edward Russo, and Robin J. Tanner.

“Information Distortion in the Evaluation of a Single Option”. Organizational Behavior and Human Decision Processes 102, nr. 2 (2007)

Chenhall, Robert H. “Management Control Systems Design within Its Organizational Context: Findings from Contingency-Based Research and Directions for the future.(Author Abstract)”. Accounting, Organizations and Society, 2003, 127.

Ghosh, Dipankar, and Robert F. Lusch. “Outcome Effect, Controllability and Performance Evaluation of Managers: Some Field Evidence from Multi-Outlet Businesses”. Accounting, Organizations and Society 25, nr. 4 (2000)

Gibbs, Michael, Kenneth A. Merchant, Wim A. Van der Stede, and Mark E. Vargus. “Determinants and Effects of Subjectivity in Incentives”. The Accounting Review 79, nr. 2 (2004): 409–36.

Heider, F. . “Attitudes and cognitive organization”. The Journal of Psychology, (1946): 21(1), 107–112.

Indjejikian, Raffi. “Performance Evaluation and Compensation Research: An Agency Perspective”. Accounting Horizons 13, nr. 2 (1999): 147.

Ittner, Christopher D., David F. Larcker, and Marshall W. Meyer. “Subjectivity and the weighting of performance measures: Evidence from a balanced scorecard”. The Accounting Review 78, nr. 3 (2003): 725–758.

Jensen, Michael C., and William H. Meckling. “Theory of the firm: Managerial behavior, agency costs and ownership structure”. Journal of financial economics 3, nr. 4 (1976): 305–360. Kaplan, Robert S., and David P. Norton. “Building a strategy-focused organization”. Ivey Business

Journal 65, nr. 5 (2001): 12.

———. “The ‘Balanced Scorecard’ is more than just a new measurement system.(adapted from ‘The Balanced Scorecard: Translating Strategy into Action’)”. Harvard Business Review 74, nr. 3 (1996): S4.

Kelly, Khim Ong, R. Alan Webb, and Thomas Vance. “The Interactive Effects of Ex Post Goal Adjustment and Goal Difficulty on Performance”. Journal of Management Accounting Research 27, nr. 1 (2015)

Merchant, K. A. and Van der Stede, W. A. . “Management control systems: Performance measurement, evaluation and incentives”. 3rd ed. Harlow: Pearson, 2012.

Economics & Management Strategy 24, nr. 4 (2015)

Murphy, K. R., and J. N. Cleveland. “Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives”. Thousand Oaks, CA: Sage, 1995

Murphy, Kevin J., and Paul Oyer. “Discretion in executive incentive contracts: Theory and evidence”. Available at SSRN 294829, 2001.

Prendergast, Canice, and Robert Topel. “Discretion and Bias in Performance Evaluation”. European Economic Review 37, nr. 2 (1993): 355–365.

———. “Favoritism in Organizations”. The Journal of Political Economy 104, nr. 5 (1996): 958. ———. “The provision of incentives in firms”. Journal of Economic Literature, nr 37 (1999): 7–63. Wilks, T. J. “Predecisional distortion of evidence as a consequence of real-time audit review”. The