The effect of the level of automation on performance: The mediating role of workload in a mirror-tracing task 1

The effect of the level of automation on performance: The mediating role of workload in a mirror-tracing task

S2343274 – Wendy Olsder Master Thesis

Abstract

The purpose of this study was to examine how performance is affected by the level of automation, through its effect on workload. To investigate this mediation effect, 78 participants conducted an experiment consisting of a tracing task and a questionnaire. In the mirror-tracing task, the level of automation is manipulated in three different levels. The results suggest that workload is no mediator of the relation between the level of automation on performance. In addition, the results show that performance is influenced by time, which might compensate the mediating effect of workload. When participants normally would have perceived a different workload, they now compensate the performance with time and the workload remains stable. In addition, the results showed that the level of automation is a significant predictor of performance. Furthermore, situation awareness significantly moderates the relation between the level of automation and performance. The results of this study help in giving a better understanding of the role of human behavior and their consequences in automated systems.

1. Introduction

In industrial companies, more and more systems are becoming automated (Parasuraman & Wickens, 2008; Sjobakk, Thomassen & Alfnes, 2014). These automated systems aim to increase the efficiency, often by limiting the role of the operator (Dekker, 2004). However, full automation lacks the flexibility of human operators when it comes to detecting failures. Therefore, humans are remaining vital to automated systems and many levels of automation (LOA) have been proposed as a mean to support the human operator (Endsley & Kaber, 1997; Endsley & Kaber, 1999).

It is generally agreed that providing the appropriate LOA can optimize operator’s performance (Endsley and Kaber, 1999; Kaber and Endsley, 2004). However, it is not always agreed on why this increases the performance of the operator. One of the identified factors that influences this relationship is workload (Endsley & Kaber, 1999; Kaber & Endsley, 2007). In this study, workload means the workload rated by the individuals carrying out the tasks; the perceived workload (Hart & Staveland, 1988; Parasuraman & Sheridan, 2008). To investigate the effects of workload on the LOA and performance, numerous studies were conducted.

One of the first studies on the effect of workload is conducted by Endsley and Kiris (1995). They observed that there is no difference in perceived workload across LOA. This conclusion is supported by other studies conducted by Jou, Yenn and Lin (2009), Wiener and Curry (1980) and Woods (1996).

performance. In addition, the effect of a lower workload is also observed by Endsley and Kaber (1999), Kaber and Endsley (2007) and Kaber, Riley and Zhou (2000). These studies show contradicting findings on the effect of workload compared to Endsley and Kiris (1995) and Jou et al. (2009).

The contradicting findings on the effect of workload propose the need to evaluate the effect of the LOA on workload and further investigate its effect on performance. An explanation of the different conclusions of the effect of workload might be the characteristics of the task (Jou et al., 2009). For instance, Endsley and Kaber (1999) suggest that the measured workload in their study might differ when subjects are required to perform the task over extended time periods. Furthermore, instead of workload affecting the relation between LOA and performance, there may be by other dependent variables influencing the relationship. Two other relevant identified factors that influence the effect of LOA on performance are situation awareness and trust (Endsley & Kaber, 1999; Kaber & Endsley, 2007; Parasuraman & Wickens, 2008). These two factors might explain the positive relationship between the LOA and performance in the studies without a decreasing effect of workload.

The aim of this research is to examine how performance is affected by the LOA, through its effect on workload. It will provide an answer to the following research question:

To address this question, an experiment will be conducted in which the effects of the LOA on performance, through its effect on workload will be tested. This experiment will be performed in an online study which uses the mirror-tracing task (Cusack, Vezenkova & Gottschalk, 2015). The experiment will consist of three groups, each with a different LOA. During the experiment, both performance and workload will be measured.

This study is contributing to theory by proving more understanding of the relation between the LOA and performance, indirectly influenced by workload. The study will test the indirect effect of workload on performance by a different type of task. It will explore the contradicting interactions of Endsley and Kiris (1995) and Endsley and Kaber (1999) and try to explain the effect of workload on LOA and performance. In practice, the study will give more understanding about the role of human behavior and their consequences in automated systems. The results can help managers to make better decisions about their automation design and the corresponding human-machine interaction.

2. Background

This chapter will review the available literature on the effect of the level of automation (LOA) on human performance, through its effect on workload. The chapter is divided in five sections. The first section will discuss the current developments of automated systems. Thereafter, the second section will present the available literature on the effect of the LOA on human performance. In addition, the third will present the available literature of the effect of workload on LOA and performance and the fourth section will present the influences of other factors on this relationship. Finally, the last section will discuss the hypotheses of this study and present the conceptual model.

2.1 Increase of interaction between automated systems and humans

Automation is defined by Parasuraman and Riley (1997) as the performance of tasks by machines rather than human operators. Various industrial settings, such as the healthcare industry, are increasingly deploying automation in their production systems (Parasuraman & Wickens, 2008; Sjobakk, Thomassen & Alfnes, 2014). Some of the automated systems are there to replace humans; others are there to work together with humans to reduce the workload and errors of the operator. The aim of deploying these automated systems is to increase the efficiency and thereby the outcomes of the system (Dekker, 2004).

2.2 Effect of the level of automation on performance

automation is conducted by Endsley (1987). This study developed a LOA hierarchy to support human decision making. This hierarchy stated that a task could be performed in five ways of control: manual control, decision support, consensual artificial intelligence, monitored artificial intelligence and full automation with no operator interaction (Endsley, 1987). Building on this work, several papers are describing other hierarchies for the LOA (Endsley & Kaber, 1997, 1999; Ntuen, 1998; Parasuraman, Sheridan & Wickens 2000; Sheridan & Verplank, 1978). Endsley and Kaber (1997, 1999) developed a 10-level hierarchy for the LOA. They expanded the hierarchy of Endsley (1987) with a wider applicable range of cognitive and psychomotor tasks. The ten levels of LOA of Endsley and Kaber (1997, 1999) are manual control, action support, batch processing, shared control, decision support, blended decision making, rigid system, automated decision making, supervisory control and full automation. Endsley and Kaber’s LOA hierarchy has many advantages over the previously defined hierarchies, due to the more generic applicability of the task types. Furthermore, the hierarchy provides greater detail on the distinction between who (either the human or the machine) is doing what at each LOA. For this study the three levels of human-machine interaction of Endsley and Kaber are considered, which are manual control, action support and decision support.

the effect of LOA and performance and concluded that a higher LOA results in a higher performance (Lee & Moray, 1992; Lorenz, Di Nocera & Röttger, 2002; Metzger & Parasuraman, 2005). Furthermore, the study of Metzger and Parasuraman (2005) found that automation did improve performance, but only when the automation algorithm worked perfect and did not have any failures or breakdowns. In conclusion, most studies agree that a higher LOA results in a higher performance (Lee and Moray, 1992; Lorenz et al., 2002; Metzger and Parasuraman, 2005).

2.3 Indirect effect of workload

As mentioned, most studies agree that a higher LOA results in a higher performance (Lee and Moray, 1992; Lorenz et al., 2002; Metzger and Parasuraman, 2005). However, it is not always agreed on the influences of factors on this relationship. One of the identified factors that influences this relationship is workload (Endsley & Kaber, 1999; Kaber & Endsley, 2007). Workload can be partitioned in the physical workload, the workload as imposed by the system, and the perceived workload, the workload rated by those individuals carrying out the tasks (Hart & Staveland, 1988; Parasuraman & Sheridan, 2008). In this study, workload will refer to the perceived workload of individuals. To investigate the effects of workload on the LOA and performance, numerous studies were conducted.

perceived workload across the LOA is supported by other studies conducted by Wiener and Curry (1980) and Woods (1996). In a more recent study conducted by Jou et al. (2009) operators’ mental workload and system performance are evaluated during a human-system interface automation in an advanced nuclear power plant. The experiment consisted of two type of tasks, reactor shutdown task and alarm reset task, with each two LOA, high automation and low automation. The results of the study of Jou et al. (2009) indicate that the degree of automation does not show a significant difference to the operators’ mental workload. They concluded that it is the type of the task that has powerful influences on workload, instead of the degree of automation.

(2000) and Balfe et al. (2015) show contradicting findings on the effect of workload compared to Endsley and Kiris (1995) and Jou et al. (2009).

2.4 Alternative factors on the level of automation and performance

There are various factors that might explain the positive relationship between the LOA and performance in the studies without a decreasing effect of workload. An explanation of the different conclusions of Endsley and Kiris (1995) and Jou et al. (2009) compared to Endsley and Kaber (1997, 1999), Kaber et al. (2000) and Balfe et al. (2015) might be the characteristics of the task. The results of Jou et al. (2009) suggest it are the characteristics of the task that has powerful influences on workload, instead of the degree of automation. This is supported by Endsley and Kaber (1999) who suggest that the measured workload in their study might differ when subjects are required to perform the task over extended time periods. The characteristics of the task might give an explanation to the differences in workload. When participants normally would have perceived a different workload, they now compensated the performance with other factors and the workload remains stable. This is supported by other studies, which stated that individuals keep workload constant by compensating it with other factors, such as time (Tan & Netessine, 2014; Bertrand & Van Ooijen, 2002).

found that an increasing LOA leads to a decreasing situation awareness. In addition, studies on trust found that operators are not using reliable automated systems if they believe it to be untrustworthy (Muir, 1987, Parasuraman & Wickens, 2008). These two factors might also explain the positive relationship between the LOA and performance in the studies without a decreasing effect of workload.

2.5 Hypotheses and conceptual model

Following from the existing findings in literature, the studies of Endsley and Kaber (1997, 1999), Kaber et al. (2000) and Balfe et al. (2015) show contradicting findings on the effect of workload compared to Endsley and Kiris (1995) and Jou et al. (2009). Whereas the studies of Endsley and Kaber (1997,1999), Kaber et al. (2000) and Balfe et al. (2015) show a mediating effect of workload, the studies of Endsley and Kiris (1995) and Jou et al. (2009) found no difference in workload across LOA. The contradicting findings propose that the results of Endsley and Kiris (1995) and Jou et al. (2009) are influenced by other factors, such as the characteristics of the task and other dependent variables such as situation awareness and trust. Following these existing findings in literature, the following hypothesis is proposed:

The level of automation [X] leads to decreased workload [M], which in turn leads to increased performance [Y].

Figure 1: Conceptual model

mediating relationship of workload in a different context and explore the various task characteristics and variables that might influence the relationship between the LOA and performance. This mediation study is conducted to help explain how and why LOA influences performance. It will gain insight in the mechanism between level of automation and performance and it may identify other variables that play a role in this relationship.

3. Methodology

In this study, a convenience sample is used. Thus, the participants of this study are students. The choice for a convenience sample is based on the relative high number of participants that can be reached in short time. Furthermore, the sampling technique is easy and inexpensive. The participants were recruited by social media, email and websites. A sample of 78 respondents was taken for the survey. Out of the 78 participants, 62.8% were male and 37.2% were female. The age of the participants ranges from 18 to 30 years, with a mean of 24.1 years. Out of the 78 participants, 83.3% were Dutch, 7.7% German and the remaining 9.0% were of 8 other nationalities. In the sample, both undergraduate, graduate and post-graduate students were participating. Most of the participants completed a bachelor degree 64.1%, 24.4% completed only high school, 10.3% completed a master’s degree and 1.3% completed another degree.

3.2 Experimental design

3.3 Experimental task

For the measurement of performance, an online mirror-tracing task is developed. This mirror-tracing task is developed using JavaScript and HTML5 and distributed using Qualtrics

(http://qualtrics.com). The task showed a display with two rectangular canvases; a top canvas

that shows the mirror and a bottom canvas that shows the drawing canvas. The participants had to draw on the canvas by moving their cursor in the drawing pad, as shown in Figure 2. The development of the mirror-tracing task is based on Cusack, Vezenkova and Gottschalk (2015).

The developed experiment has three versions, in which a different LOA is presented in each experiment. The three LOA are based on the hierarchy of Endsley and Kaber (1999), which included three LOA on the interaction between humans and machines. The goal for the participant was to trace the drawing of the mirror as precisely as possible. The trial began as soon as the participants moved the cursor on the drawing canvas and ended if the end box was marked. The three groups each had four trials, in which the drawing was varied. The first image

was mirrored in three other ways such that the drawings have the same line thickness, number of edges and total drawing distance, as shown in Figure 3.

3.4 Measures

The dependent variable, performance [Y], is measured with the drawing performed on the drawing pad. For each trial, a score of the performance on the drawing pad is calculated. This score is based on the percentage that the drawing is within the outline and the percentage that the drawing is outside the outline. Out of the four trials, the best two percentages are used for measuring the final performance.

The independent variable, level of automation [X], is manipulated in this experiment. The LOA consisted of three LOA based on the hierarchy of Endsley and Kaber (1999). These three LOA are chosen, because they are three levels were the implementation is done by humans or by the interaction between humans and machines. The first level is manual control, where the human performs the task and there is no automation. Thus, the participants need to perform the drawing without any help of an automated system. The second LOA is action support, where the system assists the operator with the action. In this level, there is a small row of black squares on the drawing panel that shows the path where the participants have to draw. The third and final

LOA is shared control, where the actions are carried out by both the human and the system. In this level, there is also a row of black squares; however, in this level the number of black squares is increased. Furthermore, the black squares stay on the screen, through which it does not interfere with the speed of the participants. The three LOA are shown in Figure 4.

The mediating variable, perceived workload [M], is measured using a survey after each trial. This survey measured the workload using a subjective rating instrument: the NASA task load index (NASA TLX) (Hart & Staveland, 1988). The NASA TLX is an often used tool for measuring and conducting a subjective mental workload assessment. It determines the mental workload across six dimensions to determine the overall workload rate. These six dimensions are: mental demand, physical demand, temporal demand, performance, effort and frustration level. The participants rated statements on their perceived workload from 1 (strongly disagree) to 7 (strongly agree). The statements are shown in Appendix A.I.

Besides the mediating variable, there are other dependent variables that were measured. These variables are situation awareness and trust. The participants rated statements using Likert

scales, from 1 (strongly disagree) to 7 (strongly agree). Situation awareness is measured using the situation awareness rating technique (SART) (Taylor, 1990), as shown in Appendix A.II. In the same way, general trust in automation, trust in automation, motivation and experience are measured by often used statements in literature. The items of these four constructs are shown in Appendix A.III, A.IV, A.V and A.VI, respectively. Finally, age, gender, level of education, nationality and input device (mouse vs. touchscreen) were measured.

3.5 Procedure

Before the start of the experiment, participants needed to agree with the consent form. Then the participants answered general questions on their general trust in automation, motivation and experience. After these questions, the participants were informed about the goal and steps of the experimental rounds and performed a practice round. For these practice rounds, each participant was assigned to one of the three levels. After this step, the actual experiment started and the participant performed four trials. After these four trials, there was a questionnaire with the proposed statements discussed in the previous sections. After the trials and questionnaires, there was a debriefing of the experiment.

3.6 Validity

confounding variables are also measured. These variables are age, gender, level of education and nationality. By measuring these confounding variables, an alternative explanation for the findings of the experiment could be made.

Other forms of validity that need to be addressed are concurrent validity and construct validity. To validate these two forms of validity, variables are manipulated and measured according to existing measuring instruments defined in the literature. For instance, the LOA is manipulated according to the hierarchy of Endsley and Kaber (1999). In this way, the operational definition accurately represents the measured construct.

4. Results

Out of 105 participants, 8 individuals did not finish the experiment and 2 did not meet the requirements to participate. All other participants (95) correctly finished the experiment. To identify outliers in the data on workload and performance, two boxplots are generated. This resulted in the identification of 9 outliers within workload and performance. In addition, a boxplot is generated for situation awareness. This resulted in the identification of 8 outliers. The corresponding boxplots are shown in Appendix B. The resulted dataset has 78 participants that successfully finished the experiment.

4.2 Randomization check

For the experiment, participants are randomized across three different groups, each with a different LOA. To test the mediating effect of workload, it is important that the participants of the three experimental groups do not significantly differ from each other. Therefore, a randomization check is included in the experiment.

Experience Motivation One-way ANOVA Group 1 M 4.54 5.92 SD 1.48 0.934 Range 1.00-7.00 4.00-7.00 Group 2 M 4.42 5.63 SD 1.92 1.09 Range 1.00-7.00 3.00-7.00 Group 3 M 4.59 6.00 SD 1.74 0.760 Range 1.00-7.00 4.50-7.00 Post-hoc comparisons 1 vs. 2 p 0.888 0.270 1 vs. 3 p 0.909 0.766 2 vs. 3 p 0.996 0.160 4.3 Manipulation check

In addition to the randomization check, a manipulation check is necessary to verify whether the participants use the automation. To measure the use of the automation, participants had to rate four items on a scale from 1 (strongly disagree) to 7 (strongly agree) on the use of automation (α = .925). The participants experienced the automation as useful (M=5.64, SD=1.46). Furthermore, there is no significant difference in the experience of automation between the groups (F(1,51)=.329, p=.569). It can be concluded that the manipulation check is successful.

4.4 Control variables

In this study, additional variables are included to assess the vulnerability of the outcomes of the experiment. Consistent with other studies, these include gender, education and nationality. In addition, experience, motivation and input device are measured. For these six control variables, regression analyses are conducted. All the control variables have no significant influence on the dependent variable performance, as shown in Table 2. For instance, the mean performance of

men (M=.964, SD=.036) does not significant differ (t=.839, p=.404) from women (M=.957, SD=.028). Furthermore, using an external mouse instead of a touchpad as input device, shows no significant differences in performance (t=-1.315, p=.193). The mean performance of the external mouse (M=.956, SD=.035) does not significant differ from the performance of the touchpad (M=.966, SD=.031). n Mean SD α p (1) Gender 79 1.37 .485 ─ 0.404 (2) Education 79 1.25 .966 ─ 0.947 (3) Nationality 79 2.61 .609 ─ 0.684 (4) Experience 79 4.19 1.784 ─ 0.948 (5) Motivation 79 5.85 .938 0.701 0.142 (6) Input device 79 1.48 .528 ─ 0.193 4.5 Hypothesis testing

The hypothesis of this study is that the LOA leads to decreased workload, which in turn leads to increased performance. This hypothesis will be tested in this section. The correlations between the LOA and performance (r = .249, p = .031) suggest that the LOA has a positive relationship on performance. In addition, the correlations between workload and the LOA (r = -.057, p = .616) and between workload and performance (r = -.120, p = .306) suggest that there is no relation between workload and both the LOA and performance. The correlations are shown in Table 3. n Mean SD α (1) (2) (3) (1) Level of automation 79 2.01 .824 ─ 1 (2) Performance 75 .961 .033 ─ .031** 1 (3) Workload 79 3.24 .811 .711 .616 .306 1

Table 2: Results control variables

Note: * Correlation is significant at the .10 level (2-tailed). ** Correlation is significant at the .05 level (2-tailed). *** Correlation is significant at the .01 level (2-tailed).

To test this negative mediating effect, a partial mediating model is examined. Path A of the mediation model, which tested the regression of the LOA on workload, was not significant, B=-.056, t=-.504, p=.616. In addition, Path B showed that the regression of the mediator workload on the dependent variable performance, while controlling for LOA, was not significant B=-.005, t=-1.031, p=.306. As such, there is no partial mediation, as shown in Table 4. There is no evidence to support the mediating effect of workload and the hypothesis is rejected. However, the analysis showed that the LOA is a significant predictor of performance, B=.010, t=.249, p=.031, as shown in Figure 5 and 6.

Mediation steps Level of

automation

Workload Level of

automation

Step 1 Step 2 Step 3

Independent variables

Performance 0.010** -0.005 0.010**

Workload -0.056

R Square 0.249 0.120 0.057

Note: * Correlation is significant at the .10 level (2-tailed). ** Correlation is significant at the .05 level (2-tailed). *** Correlation is significant at the .01 level (2-tailed). Table 4: Results mediation analysis

0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99 1

Manual control Action support Shared control

Per for m anc e Level of automation

Figure 6: Results hypothesis testing

4.6 Alternative hypothesis testing

The hypothesis of this study is rejected, because there is no evidence that workload is mediating the relationship between the LOA and performance. To explain the rejection of this hypothesis, alternative hypotheses are explored and tested. Table 5 shows the correlation between various variables included in this study. There are multiple significant correlations that could help explain why there is no significant effect of workload. These significant correlations are further elaborated in this section.

n Mean SD α (1) (2) (3) (4) (5) (6) (7) (8) (9) (1) Level of automation 79 2.01 .824 ─ 1 (2) Performance 75 .961 .033 ─ .031** 1 (3) Workload 79 3.24 .811 .711 .616 .306 1 (4) Time 75 42.393 17.955 ─ .121 .041** .282 1 (5) Situation awareness 79 6.15 .491 .664 .689 .018** .104 .073* 1 (6) Experience 79 4.19 1.784 ─ .890 .478 .106 .249 .045** 1 (7) Motivation 79 5.85 .938 .701 .755 .464 .811 .178 .001*** .018** 1 (8) General trust in automation 79 5.35 .801 ─ .098* .246 .059* .007*** .926 .247 .198 1 (9) Trust in automation 53 5.64 1.461 .925 .569 .646 .785 .143 .934 .042** .482 .294 1 Note: * Correlation is significant at the .10 level (2-tailed).

** Correlation is significant at the .05 level (2-tailed). *** Correlation is significant at the .01 level (2-tailed). Table 5: Correlation matrix

Level of automation Performance

.616 _.306

The first significant correlation that could explain no mediation effect of workload, is the correlation between performance and time. Time significant influences performance (B=.000, p=.041), as shown in Figure 7. Therefore, workload could be stable by the fact that participants are compensating the workload with time. Thus, when participants normally would have felt a higher workload, they now compensated the workload by taking more time to complete the task. In addition, a regression analysis is conducted on the LOA and time. This regression analysis showed no significant influence of the LOA on time (B=-3945.91, p=.121). A higher LOA does not lead to a lower time for the task.

Another interesting correlation is between situation awareness and performance (p=.018), shown in Figure 8. A moderation analysis is conducted to test the effect of SA on the LOA and performance. There is a significant moderating interaction of SA on the LOA and performance (B=-.008, p=.025). This means that an increase in the LOA will lead to an increase in performance and this relation will become even stronger when the SA is low. Thus, when there is a low SA, a higher LOA helps to get a higher performance.

0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99 1 20 30 40 50 60 70 >70 P er fo rm a nce Time (seconds)

Figure 8: The effect of time on performance

The final correlation that is elaborated is between performance and trust in automation. Trust in automation is measured both before and after the task with a questionnaire. The questions before the task are questions about their general trust in automation. General trust showed a marginally significant relationship with either the LOA (B=.182, p=.098) or performance (B=.991, p=.246), as shown in Figure 9. However, there is a highly significant correlation with time (B=-6.745, p=.007). The higher the general trust in automation, the lower the total time spend on the task. Besides the general trust, the trust in automation is also measured afterwards. The trust in automation is both analyzed for a mediation and moderation effect. Both regression analyses showed no significant results (respectively B=-.232, p=.569 and B=.009, p=.151), as shown in Figure 10. As such, there is no partial mediation or moderation of trust in automation. 0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99 1 4,5 5 5,5 6 6,5 7 P er fo rm a nce Situation awareness

The hypothesis of this study is rejected, since there is no evidence that workload is mediating the relationship between the LOA and performance. No mediating effect of workload could be explained by the time, since time significant influences performance (B=.000, p=.041). In addition, there is a moderating effect of situation awareness (B=-.008, p=.025). Furthermore, both general trust in automation and trust in automation after the experiment show no significant relations with performance. However, there is a highly significant correlation with time (B=-6.745, p=.007). The higher the general trust in automation, the lower the total time spend on the task. 0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99 1 3 4 5 6 7 P er fo rm a nce Trust in automation

Figure 9: The effect of general trust in automation on performance

0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99 1 3 4 5 6 7 P er fo rm a nce

General trust in automation

5. Discussion

The aim of this research was to examine how performance is affected by LOA, through its effect on workload. This indirect effect is measured by performing a mirror-tracing task. In this task, the LOA is manipulated in three levels. These three levels are according to the levels of human-machine interaction of the hierarchy of Endsley and Kaber (1999). The desired manipulation of the LOA was successful, participants experienced the automation as useful (M=5.64, SD=1.46).

The results of the experiment reject the hypothesis of the mediation effect of workload. There is no evidence that the increase in performance by manipulating the LOA is caused by a decrease in workload (B=-.056, p=.616), as measured in this experiment. These results support the findings of Endsley and Kiris (1995) and Jou et al. (2009), who found no difference in workload across the LOA. In addition, the results of this experiment showed that the LOA is a significant predictor of performance (B=.010, p=.031). This is in line with the studies of Lee and Moray (1992) and Metzger and Parasuraman (2005), who found that a higher LOA results in a higher performance.

of the task (Jou et al., 2009). The results of Jou et al. (2009) suggest that it are the task characteristics that have powerful influences on workload, instead of the degree of automation. This is supported by Endsley and Kaber (1999) who suggest that the measured workload in their study might differ when subjects are required to perform the task over extended time periods. To observe this differences, the time spend on the task is tested on the performance. There is a significant influence of time on performance (B=.000, p=.041). Therefore, the effect of workload could be influenced by the fact that participants are compensating the workload with time. When participants normally would have perceived a higher workload, they now compensated the performance with taking more time and the workload remains stable. This is in line with previous studies, which stated that individuals keep workload constant by compensating it with other factors, such as time (Tan & Netessine, 2014; Bertrand & Van Ooijen, 2002).

performance (B=.009, p=.151). This is not in line with previous studies (Muir, 1987; Parasuraman & Wickens, 2008), who found that trust in automation influenced the performance. This difference can be explained through the fact that trust in automation was only measured in two LOA during this study. Thus, the positive relationship between the LOA and human performance is moderated by situation awareness, whereas trust in automation has no significant influence.

In conclusion, the results show no significant effect of workload on the relation of the LOA and performance. The results do show that the performance is influenced by time, which (at least partially) might compensate the mediating effect of workload. This is in line with previous studies, which stated that individuals keep workload constant by compensating it with other factors, such as time (Bertrand & Van Ooijen, 2002; Endsley & Kaber, 1999; Tan & Netessine, 2014). Resulting from this study, the relation between the LOA and performance appears to be moderated by situation awareness. However, the trust in automation has no significant influence.

studies (Bertrand & Van Ooijen, 2002; Endsley & Kaber, 1999; Tan & Netessine, 2014), it would be interesting to conduct the same experiment with a time restriction. This could support the result of this study that workload is compensated by time to improve the performance.

6. Conclusion

References

Balfe, N., Sharples, S. and Wilson, J.R. (2015). Impact of automation: Measurement of performance, workload and behavior in a complex control environment. Applied Ergonomics, Vol. 47, 52-64.

Bertrand, J.W.M. and Van Ooijen, H.P.G. (2002). Workload based order release and productivity: a missing link, Production Planning & Control, 13(7), 665-678.

Billings, C.E. (1991). Human-centered Aircraft Automation: A Concept and Guidelines,

National Technical Information Service, Springfield.

Cuzack, M., Vezenkoza, N., Gottschalk, C. and Calin-Jageman, R.J. (2015). Direct and conceptual replications of Burgmer & Englich (2012); Power may have little to no effect on motor performance. PLoS ONE, 10(11).

Donmez, B., Pina, P.E. and Cummings, M.L. (2009). Evaluation criteria for human automation performance metrics. Performance Evaluation and Benchmarking of Intelligent

Systems, 21-40.

Endsley, M.R. (1987). The application of human factor to the development of expert systems for advanced cockpits. Proceedings of the Human Factors Society 31st Annual Meeting,

1388-1392.

Endsley, M.R. and Kaber, D.B. (1997). The use of level of automation as a means of alleviating out-of-the-loop performance problems: a taxonomy and empirical analysis. 13th

Triennial Congress of the International Ergonomics Association, Vol. 1 (Helsinki: Finnish

Institute of Occupational Health), 168-170.

Endsley, M.R. and Kiris, E.O. (1995). The out-of-the-loop performance problem and level of control in automation, Human factors, 37, 381-394.

Fransoo, J.C. and Wiers, V.C.S. (2006). Action variety of planners: Cognitive load and requisite variety. Journal of Operations Management, Vol. 24(6), 813–821.

Hart, S.G. and Staveland, L.E. (1988). Development of NASA-TLX (task load index): results of experimental and theoretical research. Human Mental Workload, 139–178.

Jordan, N. (1963). Allocation of functions between man and machines in automated systems. Journal of Applied Psychology, Vol. 47(3), 161-165.

Jou, Y., Yenn, T., Lin, C.J., Yang, C and Chiang, C. (2009). Evaluation of operators’ mental workload of human-system interface automation in the advanced nuclear power plants.

Kaber, D.B. and Endsley, M.R. (2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task.

Theoretical Issues in Ergonomics Science, Vol. 5, 113-153.

Kaber, D.B., Onal, E. and Endsley, M.R. (2000). Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload. Human

Factors Ergonomics Manufacturing, Vol. 10(4), 409-430.

Kaber, D.B., Perry, C.M., Segall, N., McClernon, C.K. and Prinzel, L.J. (2006). Situation awareness implications of adaptive automation for information processing in an air traffic control control-related task. International Journal Industrial Ergonomics, Vol. 36(5), 447-462.

Kaber, D.B., Riley, J.M., Zhou. R. and Draper, J.V. (2000). Effects of visual interface design, control interface type, and control latency on performance, telepresence, and workload in a teleoperation task. Proceedings of the XIVth Triennial Congress of the International

Kumar, N. and Siddharthan, N.S. (1994). Technology, firm size and export behavior in developing countries: The case of Indian enterprises. The Journal of Development

Studies, Volume (31), 1994 - Issue 2.

Lee, J.D. and Moray, N. (1992). Trust, control strategies and allocation of function in human-machine systems. Ergonomics, Vol. 35(10), 1243-1270.

Lorenz, B., Di Nocera, F., Röttger, S. and Parasuraman, R. (2002). Automated fault-management in simulated spacelight micro-world. Aviation, Space and Environmental Medicine, Vol. 73(9), 886-897.

Megaw, T. (2005). The definition of measurement of mental workload. Evaluation of

Human Work, Vol. 3, 525-551.

Metzger, U. and Parasuraman, R. (2005). Automation in future air traffic management: effects of decision aid reliability on controller performance and mental workload. Human

Factors, Vol. 47(1), 35-49.

Muir, B. M. (1987). Trust between humans and machines, and the design of decision aides. International Journal of Machine Studies, 27, 527–539.

Parasuraman, R. and Riley, V. (1997). Humans and automation: use, misuse, disuse, abuse. Human Factors, Vol. 39(2), 230-253.

Parasuraman, R., Sheridan, T.B. and Wickens, C.D. (2000). A model for types and levels of human interaction with automation. IEE Transaction Systematic Management Cybernetics, Vol. 30(3), 286-297.

Parasuraman, R. and Wickens, C.D. (2008). Humans: Still Vital After All These Years of Automation. Human Factors: The Journal of the Human Factors and Ergonomics Society. Vol. 50(3), 511-520.

Pikosz, P. and Malmqvist, J. (1998). A comparative study of engineering change management in three Swedish companies. Proceedings of ASME DETC’98, 9006.

Sheridan, T.B. (1997). Supervisory control. Handbook of human factors, 1295-1327. Sheridan, T.B. and Verplank, W.L. (1978). Human and Computer Control of Undersea Teleoperators, Massachusetts Institute of Technology, Cambridge.

Sjobakk, B., Thomassen, M.K. and Alfnes, E. (2014). Implications of automation in engineer-to-order production: a case study. Advances in Manufacturing, Vol. 2, 141-149.

Tan, T.F. and Netessine, S. (2014). When Does the Devil Make Work? An Empirical Study of the Impact of Workload on Worker Productivity. Management Science, 60(6), 1574– 1593.

Vidulich, M.A. and Wickens, C.D. (1986). Causes of dissociation between subjective workload measures and performance: Caveats for the use of subjective assessments. Applied

Ergonomics, Vol. 17, 291-296.

Warm, J.S., Dember, W.N. and Hancock, P.A. (1996). Vigilance and workload in automated systems. Automation and Human Performance: Theory and Applications, 183-200.

Wiener, E.L. and Curry, R.E. (1980). Flight-deck automation: promises and problems.

Ergonomics, Vol. 23 (10), 995-1011.

Woods, D.D. (1996). Decomposing automation: apparent simplicity, real complexity.

Appendices

Item

Direction

W1 The task was mentally demanding.

_-

W2 The task was difficult.

_-

W3 The task was physically demanding.

_-

W4 I was successful in performing the task.

₊

W5 I am satisfied with my performance.

₊

W6 I worked hard to accomplish this level of performance.

_-

W7 I felt insecure in performing the task.

_-

W8 I experienced stress during the task.

_-

Appendix A.II - Situation awareness

Item

Direction

S1 I concentrated during the task.

₊

S2 I understood the goal of the task.

₊

S3 I wanted to perform well.

₊

S4 I felt in control of the outcome of the drawing.

₊

₊

S6 The task was important.

₊

Appendix A.III – General trust in automation

Item

Direction

AA1 Automated systems can help humans to improve their performance.

₊

AA2 I trust automated systems.

₊

AA3 Automated systems can make better decisions than humans.

₊

Appendix A.IV - Trust in automation

Item

Direction

T1 I trusted the black squares that it performed the right drawing.

₊

₊

T3 The black squares were helpful.

₊

T4 I looked at the bottom panel the majority of the time.

₊

Appendix A.V - Motivation

Item

Direction

M1 I want to perform well in any task I do.

₊

M2 I am highly motivated to succeed.

₊

Appendix A.VI - Experience

Item

Direction