Performance evaluation
across continent divisions
Author: Karlijn van der Hoeff
Student number: 6157939
Master thesis: 15 ECTS
Date of final version: 4 August 2014
Master’s programme: Business Economics
Specialization: Organisation Economics
Supervisor: Thomas Buser
Abstract
The aim of this study is to investigate if there is a significant difference in the subjective evaluation of employees across different (continent) divisions of a company. Considering the increase in globalization, the international mobility of companies becomes more important. Because of this globalization more people will work as expats and therefore, the comparability of performance ratings becomes more important. By using field data, information is collected on the objective and subjective evaluation of 780 employees of one multinational company which is active in 5 divisions across the world. It turns out that there is a significant difference between the subjective and objective evaluation of employees. Furthermore, it turns out that there is a difference in the subjective evaluation across certain divisions. These differences arise between Asia and Latin America, Asia and the Netherlands, Asia and North America and between EMEA and North America.
Table of contents
1. Introduction 4
2. Literature review 7
2.1 Principal-agent problem . . . 7
2.2 Pay for performance . . . 7
2.3 Objective and subjective performance evaluation . . . 8
2.4 Country differences in HR strategy . . . 11
3. Research methodology 15 3.1 Data . . . 15 3.2 Dependent variable . . . 16 3.3 Independent variable . . . 17 3.4 Control variables . . . 17 4. Empirical results 19 4.1 Descriptive statistics . . . 19
4.2 Difference between subjective and objective evaluation . . . 19
4.3 Difference between subjective evaluation across divisions . . . 22
4.4 Robustness checks . . . 25
5. Discussion and limitations 28
6. Conclusions 30
References 32
1. Introduction
The performance of an employee is a widely discussed and researched element. This started with determining the principal-agent problem: the principal and agent have different goals. Eisenhardt (1989) mentions that in order to solve this problem some sort of incentive (outcome-based or behavior-based) is needed to encourage the agent to do what the principal wants him to do. Research shows that providing the right incentives to employees can increase the productivity of an employee. These incentives are often based on a pay for performance system, which increases the productivity of employees (Lazear, 2000). This topic is discussed in the literature from different points of view. There are quite some studies on how bonus payments should depend on performance (Baker, 1992; Lazear, 2000) and how performance and competence should be measured in order to make the bonus payment dependent on this performance (Baker, Gibbons & Murphy, 1994; Kaplan & Norton; 1996). It turns out that many companies have difficulties providing the right incentives to their employees and subsequently rating their performance and competence objectively (Bentley, 2003; Prendergast 1999). Research has shown that when the performance is not rated objectively and the subjectivity of the manager can play a role, this influences the performance rating. It turns out that leniency, favoritism of employees and fairness are problems when evaluating employees (Moers, 2005; Prendergast & Topel, 1996; Roch, Sternburgh & Caputo, 2007). Also, when the bonus payment is subject to more noise, the incentive which is given by using the bonus payment is less powerful (Murnane & Cohen, 1986). However, it is not yet proven that these problems differ across different countries. Considering this, it can also be possible that the subjective performance evaluation across different countries might differ due to cultural and/or geographical differences.
The performance ratings often form the basis for promotion, succession and international mobility (Gibbons & Murphy, 1992; Hofstede, 1994). A problem arises in case these performance ratings are not objectively conducted and similar performance levels cause different ratings across divisions, countries, or continents within a company. The performance rating usually consists of an individual and a company-based component. In making one overall performance score for each individual employee, the subjectivity of the manager comes into play compromising the comparability of ratings. The case that will be investigated in this research is divided in two parts of scoring: an objective way of scoring on particular competences (the scoring is dependent on reaching certain targets) and a final overall score
given by the manager. This latter score is where the subjectivity of the manager comes into play.
A difference in performance ratings is expected because different countries have different cultures, which might have an effect on certain actions in companies, especially related to HR practices (Aycan et al., 2000; Hofstede, 1994; Kopp, 1994). Also, subjective performance measures are related to more compressed and more lenient performance ratings (Moers, 2005). This compression could be stronger in countries were it is normal to blame a certain, f.e., economic situation for failure of business results instead of the performance of employees. If the economic situation is seen as the reason for the failure of business performance, this means that the employees did not do a bad job. Therefore, the average rating will be more in the middle of the rating scale.
If performance ratings are not comparable across divisions, the foundation of a companies’ decision making is flawed (Prendergast, 2002). In bonus decisions, this would lead to a misallocation of money to employees. In promotional or international mobility decisions, this would lead to a misfit between employees and positions. For example, consider companies that apply one international bonus plan based on a generic performance rating, or when employees are transferred internally across divisions or countries based on these ratings (119.000 employees out of 288 multinationals worked as expats, Mercer 2012). Considering the increase in globalization (United Nations, 2000), the international mobility of companies becomes more important. Because of this globalization more people will work as expats and therefore, the comparability of performance ratings becomes more important. Based on the above, the research question is as follows: Is there a difference in how performance is evaluated and rated in a multinational company across different (continent) divisions?
This research contributes to the literature in several ways. First of all, there is no research yet combining the literature on performance measurement and the literature on the effect of culture and/or country differences on management practices. Furthermore, Hawthorne effects do not influence this research since the data is collected by the company with no intention to use it for a scientific research. Finally, this study could create new insights on how to evaluate employees across different country divisions.
The results show that there is a significant difference between the subjective evaluation and objective evaluation of an employee. Furthermore, it turns out that there is a difference in the subjective evaluation across certain divisions. These differences arise between Asia and Latin America, Asia and the Netherlands, Asia and North America and between EMEA and North America.
The remainder of this paper is organized as follows. In chapter 2 the theoretical background is discussed and the hypotheses linked to the research question are introduced. Chapter 3 describes the dataset and the methodology that are used. Chapter 4 presents the empirical results. In chapter 5 the discussion and limitations of this research are stated. Finally, the conclusion is put forward in chapter 6.
2. Literature review
2.1 Principal-agent problemWithin a company employees have different tasks and responsibilities. This creates different layers in an organization which results in a principal-agent relationship. This relationship can be described as follows: “one party (the principal) delegates work to another (the agent), who performs that work” (Eisenhardt, 1989, p. 58). However, a problem arises in this relationship when the principal and the agent have different goals to achieve. This difference in goals is based on self-interest. The two most common problems in this relationship are that the desires or goals of both parties are in conflict and that the willingness to bear risk is different. This latter will cause the parties to prefer different actions with different risk levels (Eisenhardt, 1989). The principal-agent literature is therefore focused on determining the optimal contract between the principal and the agent in order to incentivize the agent in a way that the agent exerts the actions which are aligned with the goals of the principal. Eisenhardt (1989) summarizes researches which have shown that this can be achieved by using a behavior-oriented contract or an outcome-based contract. For example, if it is difficult to measure the outcome a behaviour-oriented contract is preferred. Eisenhardt (1989) makes it clear that to solve the principal-agent problem some sort of incentive (outcome-based or behavior-based) is needed to encourage the agent to do what the principal wants him to do.
2.2 Pay for performance
Nowadays, it is already widely known that performance pay is essential in a company to make your employees work hard and to let them perform the work you want them to do. Lazear (2000) shows in his empirical research what the effect of pay for performance is on the output of a company. By introducing a piece rate bonus scheme instead of a fixed rate it turns out that productivity increases by 44% and that the firm attracts more able employees. The performance of an employee was measured as number of glass units installed in an eight-hour day. The payment of employees depended on a piece rate, so more installed glass units leads to more income for an employee. In this way the employees have an incentive to work hard. Dohmen and Falk (2011) found the same evidence in a controlled laboratory experiment. They found that the output of subjects, in this case the task was multiplying one-digit
numbers with two-digit numbers, was higher when a piece rate was offered instead of a fixed compensation. The reason they give for this difference is productivity sorting. This means that more productive subjects choose to work for a piece rate instead of for a fixed compensation. However, if the level of pay for performance is really high, it might become ineffective. This will give the employee to many incentives to only complete the action which is taken into account when determining their piece rate payment (Baker, Jensen & Murphy, 1988). In this way, other important actions are left out and might be neglected. Therefore, when performance measures are used it is important to determine these performance measures in a specific and correct way. If the performance measures are not determined correctly or not evaluated in a correct way, the conclusions and decisions that follow from these measures may not be the right conclusions and decisions. In the worst case, these decisions could even harm the company. This problem arises also when the performance cannot be monitored perfectly or when evaluations are subjective.
2.3 Objective and subjective performance evaluation
There are different ways to evaluate the performance of employees. This can be divided in two parts: objective performance evaluation and subjective performance evaluation. Objective measures, used for objective performance evaluation, are defined as “direct measures of countable behaviors or outcomes” (Bommer et al., 1995, p.588). Performance evaluation started out with only financial measures, such as productivity and profit (Ghalayini & Noble, 1996). These kinds of measures can be measured objectively. However, there are also some problems that arise when a firm only uses financial measures. One of these problems is that these measures often do not have incorporated the firm’s strategy. This gives the employees the wrong incentives, which will lead to a bad implementation of the strategy the firm wants to follow. Furthermore, when the performance measure is misspecified, it gives employees the opportunity to start “gaming the system”. Baker, Jensen and Murphy (1988) explain in their theoretical paper that this leads to employees optimizing actual measures instead of the intended measures.
Financial measures contribute to an objective performance evaluation. However, since financial measures are not optimal to use as the only measure to evaluate performance, other measures should also be used. After a while, Ghalayini and Noble (1996) state, that operational measures were included in the performance evaluation. This includes measures like quality, cost and delivery. These measures can also be evaluated objectively. Eventually,
the balanced scorecard was introduced. This scorecard introduced multiple measures to evaluate performance from the perspectives of customers, internal business processes and learning and growth (Kaplan & Norton, 1996). This balanced scorecard gives companies the possibility to track financial results, but also to monitor other important features that focus on the company’s strategy. However, introducing this scorecard also meant that performance evaluation became more subjective.
Baker, Gibbons and Murphy (1994) also show that combining objective with subjective performance measures is optimal in some circumstances. Subjective performance measures, used for subjective performance evaluation, are defined as “superior’s subjective judgments about qualitative performance indicators” (Moers, 2005, p.68). Since this subjective performance evaluation has no clear performance indicators it becomes easy for the supervisor who performs the evaluation of the employees to manipulate the performance outcomes in a positive or negative direction. This bias causes problems such as the possibility to determine the right personnel decisions (Moers, 2005). Also, problems arise due to a biased evaluation when employees think the evaluation is unfair or when this bias leads to a compression of ratings which makes it harder to distinguish the talented workers from the less talented workers (Prendergast & Topel, 1993). This latter is empirically proven by Moers (2005). He shows in his research that the use of multiple objective performance measures and the use of subjective performance measures leads to more compressed ratings and also to more lenient performance ratings. Moers (2005) uses multiple subjective performance measures, however in this paper only one subjective measure is used: an overall performance evaluation score. If this score differs from the objective score it becomes easy to determine the subjective influence of the manager. Also, in the research of Moers (2005), the managers are explicitly told that the evaluation of an employee influences the bonus payment of the employee. This direct influence could lead to managers evaluating their friends in the company way better than others. However, the dataset used in this research is probably not biased in that way, since the managers who performed the evaluation were not told that the evaluation they give would have a direct influence on the bonus payment of the employees. Therefore, the focus of a manager is less on favoring his or her friends in order to let them earn more money by influencing the subjective evaluation. This makes it easier to determine the possible influence of country differences more precisely.
Problems which arise when using performance measures is that these measures cannot always be monitored precisely and as mentioned before, the subjectivity of a manager can play a role in the evaluation of an employee. Prendergast (2002) explains that when the
incentive pay of a worker is tied to the evaluation which a supervisor gives, the supervisor will distort this evaluation by favoring his friends so they earn more money. Prendergast (2002) then follows by saying that if the performance evaluation of a worker is used to allocate the worker to tasks in which they are talented, more mistakes are made in this process when supervisors lie more in the evaluation of workers. Also, if the firm uses the evaluation for the bonus payment to workers and the performance measuring is noisy the firm might pay to much bonus to the worker (Prendergast, 2002). Finally, if employees are aware of these noisy aspects (imperfect monitoring, subjectivity) in their performance pay, the employees have lower incentives and therefore the bonus payment is less powerful (Murnane & Cohen, 1986). Murnane and Cohen (1986) demonstrate that merit pay plans often not survive when they are used to incentivize teachers. By interviewing teachers and administrators of several school districts they got insights in how merit pay plans work in the field of teaching. They state that “piece-rate contracts work well when the actual contribution of the individual worker to the firm’s output can be measured at relatively low cost” (p.4). If this cannot be measured precisely due to f.e. imperfect monitoring, this way of incentivizing employees loses power and therefore the goal of why a piece-rate contract is offered in the first place will not be reached. Murnane and Cohen (1986) also mention that if the subjectivity of a supervisor comes into play when evaluating the employees, the employees will often not understand why some employees earn more bonus payment compared to themselves. This will results in a decreasing motivation of the employee who receives less bonus payment and therefore the incentive plan is less powerful.
In this research subjective and objective performance measures will be compared. The objective measures are rated with a number from 1 to 5, with 5 as the highest score. The subjective performance evaluation is also measured with a number from 1 to 5. With this subjective evaluation the discretion of the supervisor becomes visible. Taking into account that objective measures do not always cover all points of interest, it is expected that the subjective measure covers other things compared to the objective measure. This will cause a discrepancy between the objective and subjective performance evaluation. Therefore the first hypothesis is as follows: There is a difference in the objective rating of performance and
2.4 Country differences in HR strategy
Considering the subjective performance evaluation managers use, it is conceivable that this subjective performance rating differs across countries due to differences between countries such as culture differences. There is a reasonable amount of research that shows a difference in management practices across countries (Aycan et al., 2000; Hofstede, 1994; Kopp, 1994). Hofstede (1994) makes a clear distinction between national culture differences and organizational culture differences. He defines culture as “the collective programming of the mind which distinguishes the members of one category of people from another” (p.1). National culture differences are divided into five dimensions: power distance, individualism versus collectivism, masculinity versus femininity, uncertainty avoidance and long versus short-term orientation. Hofstede (1994) states that a difference in culture across countries affects all kinds of people: parents and children, politicians and citizens, but also managers and subordinates. This results in a difference in the effect of management practices in different countries. Also, management theories are written by people who are constrained by their own culture and knowledge. Therefore, these theories cannot be directly adopted across the world. In order to do so, some kind of adjustment is often needed to make the theories applicable and work in the way they should. The main difference between national and organizational culture differences Hofstede (1994, p.9) mentions, is that “national cultures differ mostly at the level of basic values while organizational cultures differ mostly at the level of the more superficial practices: symbols, heroes, and rituals”. Considering this difference, the possible disparity in subjective performance evaluation across countries is probably due to national culture differences instead of organizational culture differences. Hofstede (1994) also explains how culture differences in multinationals should be managed. He explains that when there are more nationalities working together in one business line or division, common practices is a tool to overcome national culture differences in values between organization members. This will help keeping multinationals together. Again, considering the performance evaluation across divisions, it is best to have one common way of performance rating and therefore the difference in subjective performance evaluation due to culture differences should be as small as possible. Hence, if the influence of national culture differences on a specific management task is great, it should be considered if the coordination should be on geographical level or on business level. Hofstede (1994) mentions this as a choice which only has to be taken once. However, if the environment changes, for example increasing globalization, this choice might need to be adjusted.
Aycan et al. (2000) did an empirical research on the impact of culture on management practices, more specifically the impact on human resource management practices. The reason they mention for the importance of their research is the change in business environment. Since the environment has become more globalized and liberalized, it has become increasingly important to do more research on culture effects. Also, people have started to realize that HRM theories developed by using Western cultural values may not be effective in environments with other socio-cultural values. Aycan et al. (2000) use in essence the same definition for culture as Hofstede (1994): “common patterns of beliefs, assumptions, values, and norms of behavior of human groups (represented by societies, institutions, and organizations)” (p.194). The research of Aycan et al. (2000) is based on the Model of Culture Fit (Aycan, Kanungo & Sinha, 1999). This model states that the socio-cultural environment as well as the organization environment affects the internal work culture and HRM practices. They state that the assumptions managers have about what their employees are like and how they should act, depend on the socio-cultural environment. Therefore, this environment influences the actions of a manager. As in Hofstede (1994), Aycan et al. (2000) mention that there is a difference between internal work culture and the socio-cultural environment. Since this paper uses data from one company with business divisions across the world, the assumption is that the internal work culture is the same but that the socio-cultural environment might differ across different business divisions in different countries. The research shows indirect evidence of socio-cultural differences via managerial assumptions (based on the managers perception of the environment) on HRM practices like job enrichment, empowering supervision and performance-based reward allocation. However, the effect differs across countries. Aycan et al. (2000) use a questionnaire in order to collect the necessary information on the socio-cultural environment, the internal work culture and HRM practices. The downside of this questionnaire is that the respondents are directly confronted with different environments and the effect on HRM practices. This could lead to a difference in the real situation and how the respondents fill out the questionnaire. The dataset, which is used for this research, is not only collected because it is needed for this research. The company collects this information every year in order to evaluate the employees.
Considering the results of the above explained literature the second and main hypothesis of this research is: There is a difference in the subjective performance rating of
employees across divisions.
To make a raw prediction on this difference, the analysis of Hofstede (1994) is used. He mentions a table in which different countries are ranked on every dimension of national
culture differences (power distance, individualism, masculinity, uncertainty avoidance and long-term orientation). I expect for these five dimensions that if they are stronger in a country, the subjective evaluation will be lower. For power distance, I expect that managers will want to show their power so they will rate the performance of their employees lower. This also counts for a masculine society. Men are more competitive and they will therefore rate the performance lower. Also, for individualism this argument counts. If people are more individualistic they will give other people a lower rating on their performance. If uncertainty avoidance is strong in a country, I expect that the subjective evaluation will be lower since there is more need for rules, formalization and standardization. Therefore, managers will not want to use the power in the subjective evaluation and the subjective evaluation will be more in line with the objective evaluation. Long-term orientation is linked to thrift and perseverance, this will cause the subjective evaluation to be smaller, since a manager wants employees to work hard and if the subjective evaluation is high the employees will think they are already good enough. If the countries used in Hofstede (1994) are grouped into the divisions used in this research (Asia, EMEA, Latin America, the Netherlands and North America) the following things stand out. EMEA, the Netherlands and North America have a lower rank on every dimension compared to Asia, except for individualism. Therefore, I expect that the subjective evaluation in Asia will be lower compared to EMEA, the Netherlands and North America. Also, EMEA is ranked lower on every dimension compared to North America. Therefore, I expect the subjective evaluation in North America to be lower compared to EMEA. For Latin America it is difficult to make a prediction, because Hofstede (1994) only has information on two countries in this division and these two counties are ranked quite different in the overview.
This paper contributes to the literature in several ways. First, there is no research yet which combines the literature on performance measurement (Baker, Gibbons & Murphy, 1994; Moers, 2005; Prendergast & Topel, 1993) and the literature on the effect of culture and/or country differences on management practices (Aycan et al., 2000; Hofstede 1994). Combining the two results which follows from this previous research can give new insights into the effects of performance evaluation across countries. However, it should be taken into account that a difference in ratings across divisions is not necessarily caused by the country and/or culture difference. This will be further explained in the discussion. Second, by using a dataset which is already collected by the company the data is not biased in a way that the managers who are responsible for the evaluation are not influenced by knowing that the data will be used for research purposes. In this way, the evaluation of the employees is not
influenced by prior knowledge and therefore the dataset reflects the reality. A downside however is that this narrows the possibilities to gather data on variables which could be interesting to take into account. Finally, this research could create new insights for companies how to evaluate employees across different country divisions and how this performance evaluation should influence decisions like bonus payment and succession planning.
3. Research methodology
As discussed before, the main research of this paper is focused on a possible difference in performance evaluation across countries. In order to test the hypotheses and answer the research question a dataset of a certain multinational company is used. This multinational company is headquartered in the Netherlands. It has nearly 6000 employees who are active in five divisions all over the world. The company has a policy in which managers perform an evaluation on the performance of the employees. This evaluation is performed every year and collects information on objective performance measures, which differ between employees, and it is completed with one overall (subjective) score of the manager. This information on performance might be used for bonus payments and succession planning.
The data on performance evaluation is used to test the hypotheses with several regressions. This chapter will further explain the specific data, the dependent variable, the independent variable and the control variables.
3.1 Data
As said, this empirical research uses a dataset of a multinational company. The dataset is collected in 2012 but also contains information on the overall performance in previous years. However, since the objective performance evaluation is only available for 2012, the variables refer to the 2012 data.
The original dataset contained more variables than needed for this research. Therefore, the dataset is narrowed down to the variables needed for this research. The first step was to determine the necessary variables and to consider for which subjects all these variables were determined. After this, a dataset is composed of the subjects and corresponding variables. The final dataset used for this empirical research contains the necessary information of 780 employees. There is still some missing information from a couple of subjects, but this will be taken into account when performing the various statistical tests and regressions. Also, because the different hypotheses are linked to different variables, it could be the case that the sample number differs a bit from the 780 stated above.
3.2 Dependent variable
From the final dataset the dependent, independent and control variables can be determined. In order to answer the research question and test the hypotheses it is important to understand which dependent, independent and control variables should be used. Therefore, these variables are explained in the next three parts. To connect these variables with the hypotheses stated in part 2 in a clear way, the hypotheses are listed below:
I. There is a difference in the objective rating of performance and competences and the final (subjective) score of employees.
II. There is a difference in the subjective performance rating of employees across divisions.
The dependent variable in this study is the overall (subjective) evaluation score a manager gives to an employee. This score is given by the manager and no further explanation about this score has to be given by the manager. Therefore, it is assumed that this score is subjective and does not have to depend on the objective score. The overall evaluation score is categorized into six groups. These groups are “does not meet expectations”, “not fully meets expectations”, “meets-”, “meets expectations”, “meets+”, “exceeds expectations” and “overachieves expectations by far (example for others)”. However, in order to make the overall score comparable with the objective score “meets-” is grouped with “not fully meets expectations” and “meets+” is grouped with “exceeds expectations”. These categories are coded as follows: 1 is given to “does not meet expectations”, “not fully meets expectations and meets-” is given a 2, “meets expectations” is a 3, “meets+ and exceeds expectations” is given a 4, and finally a 5 is given to “overachieves expectations by far (example for others)”. This variable is called “subjective”. This variable is tested by using an OLS and an ordered probit regression. The variable “subjective” is used as a dependent variable in the regressions for both hypotheses.
3.3 Independent variable
For the regressions, which relate to hypothesis II the independent variable is “division”. As mentioned before, the multinational company where the data comes from operates in five different divisions. These divisions are the Netherlands, EMEA (Europe, Middle East and Africa), Asia, North America and Latin America. The division in which an employee works is given for each employee, however also China and Global were mentioned. Therefore, China is coded as Asia and since the company is headquartered in the Netherlands, Global is re-coded as the Netherlands. The divisions are given the following numbers as variable names: Asia is 1, EMEA is 2, Latin America is 3, the Netherlands is 4 and North America is 5. Division will be the independent variable in the regressions and these divisions will all be dummy variables with a 1 if the employee works in the specific division and it is coded a 0 if the employee does not work in that specific division. In order to prevent the dummy variable trap, one of these division dummy variables is omitted in the regressions. This omitted dummy variable is the reference group.
3.4 Control variables
Besides the independent variable, control variables can also influence the dependent variable. Therefore, adding control variables to the regressions will make the results of the regression more powerful.
The dataset provided by the company contains information on salary grade, layer in the organization, position grade and the objective evaluation scores of employees. All these variables can be used as control variables, since these contain extra information which should be taken into account when determining the effect of the division on the subjective evaluation. The most important control variable is the objective evaluation score. Besides the overall (subjective) evaluation, the manager also performs the objective evaluation of the employee. The objective evaluation consists of different performance measures. These measures are dependent on the function in which the employee operates. There is a total of 121 different objective measures. Each employee is rated on average on 18 measures. The possibilities of the rating are “does not meet expectations”, “not fully meets expectations”, “meets expectations”, “exceeds expectations” and “overachieves expectations by far (example for other)”. Respectively, these ratings are coded from 1 to 5. Since the objective rating and the subjective rating have the same distribution amongst the ratings, these are now
comparable. Because the employees are not rated on every objective measure, one objective variable is constructed which is the mean of the objective measures on which the employee is measured and missing ratings are not taken into account when determining this mean. The assumption by constructing this objective evaluation variable is that the company’s weighting of importance on each measure is the same.
The salary grade is also given for each employee. This salary grade is measured in Hay-grades. These Hay-grades are based on “a set of common factors that measure inputs (required knowledge, skills, and capabilities), throughputs (processing of inputs to achieve results), and outputs (end results expectations from applying inputs constructively)” (Hay Group, 2005, p.2). This is a worldwide-accepted way of grading and the multinational company where the dataset is retrieved from uses this way to assign salaries to the employees. Therefore, this salary grade becomes comparable across the divisions.
Furthermore, it is known in which layer of the organization the employee is working. If the number of this variable is low (e.g. 1), this means that the employee is working at the top of the organization. A higher number (e.g. 8) refers to a lower level in the organization. It is important to take this “layer in the organization” as a control variable, since it could be that employees at a lower level are evaluated differently compared to higher levels.
For the robustness checks, position grade will be used as a control variable instead of salary grade. This position grade is a number the company links to certain jobs. This is dependent on the responsibility and tasks the employee has. This can therefore be used as a control variable instead of the salary grade as control variable.
4. Empirical results
In this chapter the empirical results of this study are presented. The descriptive statistics will be presented first. Second, the main results for testing the hypotheses will be discussed. The third part shows the results of the robustness checks.
4.1 Descriptive statistics
The dataset consists of 780 observations which are useful for this study. One observation concerns one employee for who a performance evaluation is given by a manager.
Of these 780 employees, 239 employees (30.6%) work in Asia. Furthermore, 116 employees (14.9%) work in EMEA, 79 employees (10.1%) in Latin America, 266 employees (34.1%) in the Netherlands and 80 employees (10.3%) in North America. The overall average of the subjective evaluation is 3.46, with the lowest score being a 2 and the highest score being a 5. The overall average objective evaluation score is smaller compared to the subjective evaluation score: 3.20. The lowest score is 1 and the highest score is 5. The spread of the salary grade of all employees is between 13 and 25, with an average of 16.94. The spread of the layers in the organization is between 3 and 8, with an average exactly between layer 5 and 6.
In table 1.1 (appendix I) the descriptive statistics of the main variables used in this study can be found. These tables describe the averages, the standard deviations, the range and number of observations for each variable.
4.2 Difference between subjective and objective evaluation
The first hypothesis stated that there is a significant difference between the subjective evaluation score and the objective evaluation score of an employee. By looking at the raw numbers the following graph can be computed with regard to the subjective and objective evaluation scores per division.
Figure 1. The objective and subjective performance evaluation means for all divisions.
As can be seen in figure 1, the average subjective performance evaluation is a bit higher for each division compared to the average objective performance evaluation. This indicates that there could be a significant difference between the overall subjective and objective evaluation. To determine if there really is a significant difference various tests are conducted. First of all, subjective evaluation and objective evaluation are ranked. The correlation between the subjective evaluation variable as a ranking variable and the objective evaluation as a ranking variable is 0.569, which is relatively low. This means they are not fully dependent on one another. Furthermore, the variance of the subjective rating is 0.486 and the variance of the objective rating is 0.119. Comparing these two variances it can be said that the subjective rating is much more spread out compared to the objective rating where the ratings are closer to the mean. This means that a lot of people probably have a subjective rating that varies substantially from the objective rating. Also, when computing a regression with the dependent variable being the subjective performance evaluation and the independent variable being the objective evaluation and when no other control variables are included, the adjusted R2 turns out to be almost 30%. This means that 70% of the variation in the subjective rating is unexplained by the objective rating of an employee. These results can be found in table 2.1 (appendix II).
Knowing that the subjective rating and objective rating differ from each other, it is interesting to see if there is more alignment between these two ratings in certain divisions. To analyse this question a dependent variable “absdiff” is created which is the absolute
0 1 2 3 4 5
Performance evaluation
Objective performance evaluation Subjective performance evaluationdifference between the subjective and objective rating. The following table shows the results of the regression.
Dependent variable: absdiff (subj - obj)
(1) Independent variables Division: 2 -.0849** (.0411) 3 .0024 (.0473) 4 -.0843*** (.0324) 5 -.0493 (.0471) R2 .0122 Adjusted R2 .0071 Observations 776 *p<.10. **p<.05. ***p<.01.
Table 2.2. Result of the OLS regression for measuring the influence of a certain division on the
alignment between the subjective and objective evaluation.
As said, the dependent variable is the absolute difference between the subjective and objective evaluation. The independent variables are the different divisions. In order to avoid the dummy variable trap, division 1 (Asia) is the reference group in the regression. For this regression 776 observations are taken into account. The regression shows significant results for division 2 and 4, respectively EMEA and the Netherlands. This means that the absolute difference between the two ratings in EMEA is 0.085 lower compared to the absolute difference in Asia. For the Netherlands, the absolute difference is 0.084 lower compared to the absolute
difference in Asia. These results mean that the subjective and objective evaluation in EMEA and the Netherlands are more aligned compared to the evaluation of subjective and objective performance in Asia.
The regression in table 2.2 only takes into account the differences between Asia and each division. To see the results of the alignment between the other divisions, the coefficients of the divisions are tested amongst each other. These results are presented in table 2.3 in appendix II. It turns out that besides the significant results for EMEA and the Netherlands compared to Asia, there is also a significant result (at a 10% level) when the Netherlands is compared to Latin America. The difference between these two divisions is measured by
calculating the difference between the coefficients of the original regression with Asia as reference group. For the Netherlands and Latin America this results in a difference of 0.088, so the absolute difference between the subjective and objective rating in the Netherlands is 0.088 lower compared to Latin America. This means that the subjective and objective evaluation in the Netherlands is more aligned compared to these evaluations in Latin America.
4.3 Difference between subjective evaluation across divisions
In chapter 2, the second hypothesis was introduced which stated that there is a difference in the subjective evaluation of employees across divisions. In order to test this hypothesis multiple statistical tests are conducted and coefficients are compared.
First of all, multiple unpaired t-tests are conducted to see if the mean subjective ratings are statistically different across divisions. Table 3.1 (appendix II) shows the results of the tests. These tests do not take the objective rating of an employee into account; the only variable of interest with these t-tests is the subjective rating. The results show that none of the differences across the five divisions is significant. It could be that this result is due to the fact that the objective evaluation is not taken into account when trying to estimate the influence of the division on the subjective rating. In order to take this into account, multiple OLS regressions are conducted. These results are shown in the table below.
Dependent variable: Subjective evaluation
(1) (2) Independent variables Division: 2 -.0499 -.0637 (.0666) (.0670) 3 -.0602 -.1553** (.0766) (.0785) 4 -.1208** -.1646*** (.0528) (.0538) 5 -.1776** -.2130*** (.0767) (.0765) Control variables Objective evaluation 1.1357*** 1.1329*** (.0626) (.0634) Salary grade -.0226 (.0153) Layer in organization -.1064*** (.0261) R2 .3008 .3261 Adjusted R2 .2962 .3199 Observations 776 761 *p<.10. **p<.05. ***p<.01.
Table 3.2. Results of the OLS regressions for measuring the influence of a certain division on the
subjective evaluation of an employee.
The dependent variable in both regressions is the subjective evaluation of an employee. The independent variable is the division. In order to avoid the dummy variable trap, division 1 (Asia) is the reference group in both regressions. For regression 1, 776 observations are taken into account and for regression 2, 761 observations are taken into account. Regression 1 shows that the subjective evaluation of employees working in division 4 and 5, respectively the Netherlands and North America, differs significantly from the subjective evaluation of an employee working in Asia. More specific, the subjective evaluation of an employee working in the Asian division is 0.121 higher compared to an employee working in the Dutch division and 0.178 higher compared to an employee working in North America. Also, the objective evaluation has a significant influence on the subjective evaluation of an employee. So, if the objective rating is increased by 1, the subjective rating increases by 1.14.
In regression 2, multiple control variables are added besides objective evaluation. These results still show a (even more) significant difference for division 4 and 5, but now the influence of division 3, Latin America, is also significant. The effect for division 3, 4 and 5 and the difference in subjective rating, results in a higher subjective rating for employees working in Asia of respectively 0.155, 0.165 and 0.213. The difference between division 4 and Asia and division 5 and Asia is greater compared to regression 1. As in regression 1, the objective rating is significant and has a coefficient of 1.13. Also, the layer in the organization in which the employee works has a significant influence, which results in a lower subjective rating if the employee works lower in the organization. If the variable layer in organization increases by 1 (thus the employee works in one layer lower in the organization) the subjective evaluation decreases by 0.106.
The regressions described above are executed as OLS regressions. However, since the dependent variable subjective evaluation is a categorized variable the regression should be executed by doing an ordered probit regression. The results of the ordered probit regressions can be found in table 3.3 (appendix II). It turns out that this makes no difference for the significance of the regressions. Therefore, OLS regressions are the standard regressions used in this study.
The regressions in table 3.2 only take into account the differences between Asia and each division. To see what the result is between all divisions, the coefficients of the divisions are tested amongst each other. For regression 1 these results can be found in table 3.4 (appendix II). For regression 2 these results are presented in the following table.
1 (Asia) 2 (EMEA) 3 (Latin America) 4 (Netherlands) 5 (North America) 1 (Asia) x - - - - 2 (EMEA) .3419 x - - - 3 (Latin America) .0483** .2969 x - - 4 (Netherlands) .0023*** .1241 .9031 x - 5 (North America) .0055*** .0808* .5394 .5175 x *p<.10. **p<.05. ***p<.01.
Table 3.5. Results of comparing the coefficients for each division, which resulted from the OLS
regression 2 in table 3.2 (including additional control variables). The values in this table are p-values.
As already discussed with the regression results above, there are significant influences between Asia and Latin America, Asia and the Netherlands and Asia and North America.
However, it turns out that there also is a significant difference in the subjective rating between EMEA and North America. The difference between these two divisions is measured by calculating the difference between the coefficients of the original regression with Asia as reference group. This results in a difference of 0.149, so the subjective evaluation of an employee working in EMEA is 0.149 higher compared to an employee working in North America. When comparing the results of the above matrix with the matrix belonging to regression 1 (without the additional control variables, table 3.4 in appendix II) it can be seen that without the control variables, there is no significant difference between EMEA and North America. Again, when the ordered probit results, which can be found in table 3.6 and 3.7 (appendix II), are compared with the above-described results no difference can be found in the significance.
These results support the hypothesis composed in chapter 2. However, not all divisions differ significantly from each other.
4.4 Robustness checks
In this subsection robustness checks are conducted by running the regressions with a different control variable, a different definition for the dependent variable and by using a different construction for the variable which measures the objective rating.
As explained in chapter 3, the variable salary grade can be replaced by position grade. In order to test if this indeed does not change the results, the regressions are done in the exact same way as the main regressions except now using position grade as control variable instead of salary grade. The result of this regression is presented in table 4.1 in appendix III. The results show that when position grade is used as control variable this does not change anything to the significance of the variables, however the coefficients increase slightly. If the adjusted R2 is compared, the adjusted R2 when using position grade as a control variable is 0.5% lower. This is a small decrease, but it indicates that salary grade is a slightly better control variable.
For the second robustness check, the dependent variable is determined differently. Instead of controlling for the objective rating of employees, the dependent variable already takes the objective rating into account. To account for this objective rating a new variable is conducted: diff. This diff variable is computed by taking the subjective rating of an employee minus the objective rating of the employee. The assumption hereby is that the objective rating is part of the subjective rating. The results of the regressions using this dependent variable are
presented in table 4.2 (appendix III). These regressions show a small difference in the significance level of the independent variables (divisions). Another big difference is the adjusted R2 of the regressions. The adjusted R2 for the regressions excluding and including control variables are respectively 0.0029 and 0.0281. These are really small, which means that the model of the main regressions (the dependent variable being the subjective rating and controlling for the objective rating) is a better fit to measure the impact of working in a certain division on the subjective evaluation.
The third robustness check concerns the objective rating variable. As explained in chapter 3, the objective rating variable is the mean of the objective measures on which the employee is rated. However, the assumption for this way of constructing the objective measure is that the company’s weighting of importance on each measure is the same. In order to test if this is a good way of determining the objective rating measure, a different variable for objective rating is constructed. The company determined five categories within the evaluation. These five categories each consist of three to five objective measures. The following table shows the categories and the associated measures.
Category Objective measures
Core Customer orientation, Quality orientation, Results orientation
Thinking ability Business context awareness, Continuous improvement, Eagerness to learn, Problem analysis & judgement, Vision
Interaction with others Behavioral flexibility, Communication & listening, Organizational awareness, Persuasiveness & negotiation, Sensitivity
Goal achievement Discipline, Initiative & entrepreneurship, Managing change, Persistence & resilience, Planning & organizing
Personal leadership Coaching & feedback, Directing performance, Networking, Ownership, Teamwork
Table 4.3. Overview of the objective measure categories.
These five categories are used as variables for the objective rating. These do not include all objective measures, however since the focus of the company goes to these measures (based on the information received), the other measures are left out. Two new regressions are conducted with the dependent variable being the subjective measure, the independent variable the divisions and as control variables the five objective rating categories, salary grade and layer in the organization. Table 4.4 in appendix III shows the results. The significance of the effect of
certain divisions changes. In regression 1, division 5 (North America) is not significant anymore and in regression 2, division 3 (Latin America) is not significant anymore. In both regressions the significance of division 4 (the Netherlands) is smaller. These results occur because not all objective measures are included. All five objective rating categories are significant and the layer in the organization is also significant. This is not different from the main regressions with one overall objective rating variable. The adjusted R2 is 0.3098 in regression 1, this does not differ much from the adjusted R2 in regression 1 from table 3.2 (adjusted R2 = 0.2962). This means the overall objective rating variable is a quite good way of measuring the overall objective rating.
5. Discussion and limitations
This study investigated two hypotheses which were introduced in chapter 2. The first hypothesis stated that there is a difference between the subjective and objective rating of employees. The results show that this hypothesis is supported and thus that the subjective evaluation includes other factors compared to only the objective measure. The second hypothesis, that there is a difference in the subjective evaluation across the different divisions, is also supported for a couple of divisions. The results showed a significant difference between Asia and Latin America, Asia and the Netherlands, Asia and North America and between EMEA and North America. These differences could arise because of significant cultural differences between these divisions. However, in order to correctly interpret the results a couple of limitations of this research should be taken into account.
The first limitation concerns an assumption which was also mentioned in the robustness checks: the assumption of equal weight of importance on the objective measures. This assumption had to be made otherwise an objective rating variable could not be conducted. However, this could influence the results in a way that if some objective measures are more important for the company, the relation between the subjective and objective rating could be different. When controlling for the objective rating in the regressions, this weighting of importance could also change the results.
The results show that working for some divisions, working in that division influences the subjective rating of an employee. A common problem with estimating a model is omitting important variables which will lead to a bias in the estimated coefficients. In this research, this problem also arises. In order to have a good estimation of the effect of working in a certain division on the subjective rating, it would be ideally if there were more control variables to include. For example, the education level of the manager who judges the performance of an employee could influence the subjective rating. If the omitted variables do have an effect on the independent variable, the coefficient of the division variable will be smaller. Also, in order to really determine if the effect of working in a certain division on the subjective rating of an employee is caused by cultural differences in the division more control variables should be added. Unfortunately for this research these variables of interest were not available, so it cannot be determined if the difference in subjective rating across divisions is due the differences in management behavior or due to differences in work force.
Another problem that arises is the possibility of reverse causality. It is determined that there is an effect of the division where an employee is working and the subjective rating, however it is difficult to determine if this is a causal relationship or only a correlation. Reverse causality might be playing a role. In this study it is assumed that the division where an employee is working affects the subjective rating. However, it could also be the reverse; the subjective rating might affect in which division an employee will work. For example, if an employee knows that in a certain division the rating is more lenient, he or she could choose to go work in that division. If this reverse causality is in place, it cannot be said that there is a causal relationship of division affecting the subjective rating.
The last thing that should be mentioned is the generalizability of this research. The data which is used in this study is only from one single firm. Although this controls for company specific factors which could influence the subjective rating, it reduces the generalizability of the results.
Considering the limitations mentioned above, it is necessary to do extra research in this field. Future research should contain more control variables in order to investigate if it is really cultural difference what causes the difference in subjective rating across divisions. Furthermore, for the generalizability of the results, future research should use datasets of multiple companies.
6. Conclusions
There is not much research done on the performance evaluation across countries. To increase the insights into the effect of certain performance evaluations, this research is conducted. The aim of this study was to investigate if there is a significant difference in the subjective evaluation of employees across different (continent) divisions of a multinational company which is headquartered in the Netherlands.
First of all, a comparison between the subjective and objective performance evaluation of employees was made. The results showed that subjective performance evaluation significantly differs from the objective rating. The subjective performance evaluation is higher compared to objective performance evaluation. Relating this to the previous described literature (Baker et al., 1994; Moers, 2005), this result is in line with the expectation from previous research; it turns out that the subjective evaluation includes other factors then only the objective evaluation, which could be favoritism, fairness and culture. The regression showed that if the objective evaluation increases by 1, the subjective evaluation increases by 1.1. The economic significance is quite high since the evaluation score is rated within a range of 1 to 5. In other words, the subjective evaluation differs a lot compared to the objective evaluation. This indicates that it is important to understand what causes this difference and if it is then necessary to change policies regarding the evaluations of employees. For example, if this difference is only caused by the manager favoring certain employees, this should be prevented because this is an unfair way of evaluating the employees.
Second, the relationship between the subjective rating and working in a certain division, and hereby controlling for objective rating, was investigated. It turns out that there is a significant difference in the subjective ratings between Asia and Latin America, Asia and the Netherlands, Asia and North America and between EMEA and North America. For the first three differences, the subjective rating in Asia is higher compared to the three divisions. For the difference between EMEA and North America, the subjective rating in EMEA is higher. These results support the previously discussed literature on differences in management practices between countries (Hofstede, 1994; Aycan et al., 2000). However, the differences between the specific divisions are not all the same as the predictions which were made in chapter 2. The difference between EMEA and North America was the only prediction which was correct. However, the difference between Asia and the Netherlands and Asia and North America turns out to be the opposite of what was expected. This could be due to different
factors. First of all, the countries used in Hofstede (1994) were grouped to compare the divisions. However, these groups do not contain all countries within one division. Therefore, the rank of a division in a certain dimension (power distance, individualism, masculinity, uncertainty avoidance and long-term orientation) could be different if all countries of a division are taken into account. Since the ranks of the divisions are used for the predictions, this different ranking could change the predictions. Another explanation for the difference in the results and the predictions might be that the expected relationship between the dimensions and the subjective evaluation is determined wrong. However, research should be needed to determine the actual effect of the dimensions on the subjective evaluation. The differences in the subjective evaluation between divisions lie between 0.149 and 0.213. Even though there are statistically significant differences between certain divisions, these differences are quite small (based on a range between 1 and 5) and therefore not really economically significant. More research should be done in order to see if in other cases the results are more economically significant.
The results of this research show that there is a difference in the subjective evaluation across certain divisions, which indicates that there is a factor which differs across the divisions which influences the subjective rating. This could be caused by the culture differences between the divisions. If this difference is indeed caused by culture differences, it is necessary for companies and managers to rethink the evaluation system and to determine if depending succession planning and international bonus payments on this evaluation system are fair things to do.
There are two important steps which should be logical to do in this field of research. The first step is to determine if in other cases the difference between subjective evaluations is more economically significant. The second step, which is necessary to undertake, is to determine what creates this difference in subjective evaluation, since this could have an important influence on the management practices concerning the evaluation of employees.
References
Aycan, Z., Kanungo, R., Mendonca, M., Yu, K., Deller, J., Stahl, G., & Kurshid, A. (2000). Impact of Culture on Human Resource Management Practices: A 10-Country
Comparison. Applied Psychology, 49(1), 192-221.
Aycan, Z., Kanungo, R. N., & Sinha, J. B. P. (1999). Organizational Culture and Human Resource Management Practices: The Model of Culture Fit. Journal of Cross-Cultural
Psychology, 30(4), 501-526.
Baker, G. P. (1992). Incentive Contracts and Performance Measurement. Journal of Political
Economy, 100(3), 598-614.
Baker, G., Gibbons, R., & Murphy, K. J. (1994). Subjective Performance Measures in Optimal Incentive Contracts. The Quarterly Journal of Economics, 109(4), 1125-1156. Baker, G. P., Jensen, M. C., & Murphy, K. J. (1988). Compensation and Incentives: Practice
vs. Theory. The Journal of Finance, 43(3), 593-616.
Bentley, W. (2003). Optimal Contracting with Subjective Evaluation. American Economic
Review, 93 (1), 216-240.
Bommer, W. H., Johnson, J. L., Rich, G. A., Podsakoff, P. M., & Kenzie, S. B. M. (1995). On the Interchangeability of Objective and Subjective Measures of Employee Performance.
Personnel Psychology, 48(3), 587-605.
Dohmen, T., & Falk, A. (2011). Performance Pay and Multidimensional Sorting:
Productivity, Preferences, and Gender. American Economic Review, 101(2), 556-590. Eisenhardt, K.M. (1989). Agency Theory: An Assessment and Review. Academy of
Management Review, 14(1), 57-74.
Ghalayini, A. M., & Noble, J. S. (1996). The Changing Basis of Performance Measurement.
International Journal of Operations & Production Management, 16(8), 63-80.
Gibbons, R., & Murphy, K. J. (1992). Optimal Incentive Contracts in the Presence of Career Concerns: Theory and Evidence. The Journal of Political Economy, 100(3), 468-505. Hay Group (2005). Hay Job Evaluation. Retrieved June 11, 2014, from
http://www.haygroup.com/downloads/ww/wp-Job_Evaluation.pdf.
Hofstede, G. (1994). The Business of International Business is Culture. International
Business Review, 3(1), 1-14.
Kaplan, R. S., & Norton, D. P. (1996). Using the Balanced Scorecard as a Strategic Management System. Harvard Business Review, 74(1), 75-85.
Kopp, R. (1994). International Human Resource Policies and Practices in Japanese, European, and United States Multinationals. Human Resource Management, 33 (4), 581-599.
Lazear, E. P. (2000). Performance Pay and Productivity. American Economic Review, 90(5), 1346-1361.
Mercer, (2012). Pensioen gat dreigt voor expats. Retrieved March 18, 2014, from http://www.mercer.nl/press-releases/pensioengat-dreigt-voor-expats.
Moers, F. (2005). Discretion and Bias in Performance Evaluation: The Impact of Diversity and Subjectivity. Accounting, Organizations and Society, 30(1), 67-80.
Murnane, R.J., & Cohen, D.K. (1986). Merit Pay and the Evaluation Problem: Why Most Merit Pay Plans Fail and a Few Survive. Harvard Educational Review, 56(1), 1-18. Prendergast, C. (1999). The Provision of Incentives in Firms. Journal of Economic Literature,
37(1), 7-63.
Prendergast, C. (2002). Uncertainty and Incentives. Journal of Labor Economics, 20(S2), S115-S137.
Prendergast, C., & Topel, R. (1993). Discretion and Bias in Performance Evaluation.
European Economic Review, 37(2-3), 355-365.
Prendergast, C., & Topel, R. (1996). Favoritism in organizations. The Journal of Political
Economy, 104(5), 958-978.
Roch, S., Sternburgh, A., and Caputo, P. (2007). Absolute vs Relative Performance Rating Formats: Implications for fairness and organizational justice. International Journal of
Selection Assessment, 15(3), 302-316.
Appendix I
Descriptive table
Table 1.1. Descriptive table for the main variables used in the statistical tests.
Mean (SD) Min Max Observations
Dependent variable Subjective evaluation 3.46 (.70) 2 5 776 Independent variable Division 2.78 (1.44) 1 5 780 Control variables Objective evaluation 3.20 (.34) 1.3 4.89 780 Salary grade 16.94 (1.89) 13 25 771 Position grade 16.96 (1.85) 15 25 780 Layer in organization 5.51 (1.12) 3 8 773
Appendix II
Tables of the analyses
Table 2.1. Result of the OLS regression for measuring the effect of the objective evaluation on the
subjective evaluation.
Dependent variable: Subjective evaluation
(1) Independent variable Objective evaluation 1.1067*** (.0617) R2 .2935 Adjusted R2 .2926 Observations 776 *p<.10. **p<.05. ***p<.01.
Table 2.3. Results of comparing the coefficients for each division, which resulted from the OLS
regression 1 in table 2.2. The values in this table are p-values.
1 (Asia) 2 (EMEA) 3 (Latin America) 4 (Netherlands) 5 (North America) 1 (Asia) x - - - - 2 (EMEA) .0390** x - - - 3 (Latin America) .9598 .1008 x - - 4 (Netherlands) .0095*** .9877 .0642* x - 5 (North America) .2955 .5012 .3725 .4523 x *p<.10. **p<.05. ***p<.01.
