Faculty of Behavioural, Management, and Social Sciences (BMS)
The Modern Way of Training Situation Awareness:
Virtual Reality for Firefighters
Chen Yu Chiu M.Sc. Thesis
09 03 2020
Supervisors:
Dr. Friso van den Bos Dr. Kolloffel
Faculty of BMS Educational Science and Technology University of Twente 5, Drienerlolaan 7522 NB Enschede The Netherlands
Author Note
Chen Yu Chiu, Faculty of Behavioral Management and Social Sciences, University of Twente Correspondence concerning this article should be addressed to Chen Yu Chiu, Educational
Science and Technology, c.chiu@student.utwente.nl
Table of Contents
Abstract ...3
Introduction ...4
Theoretical Framework ...5
Virtual reality ...5
VR Learning effects ...7
Situation Awareness ...8
The application of SA ... 11
Individual Differences in SA Development ... 13
Research question and hypotheses ... 17
Method ... 18
Research design ... 18
Participants ... 19
Instruments ... 19
X-VR ... 19
Situation Awareness questionnaire ... 20
Years of Experience ... 21
Technology Acceptance ... 21
Interview questions... 22
Procedure ... 25
Data analysis ... 25
Quantitative data ... 25
Qualitative data ... 30
Results ... 31
Descriptive statistics ... 31
Spearman’s rho correlation tests ... 31
Trainer difference ... 32
Regression analysis ... 32
Moderation analysis ... 33
Interview analysis ... 36
Discussion ... 41
The learning effects of VR training ... 41
Theoretical and practical implications ... 46
Limitations ... 47
Future research... 49
Conclusion ... 50
Reference ... 52
Appendices ... 61
Appendix A Background questionnaire ... 61
Appendix B Situation Awareness questionnaire ... 63
Appendix C Interview transcription ... 66
Abstract
Topic and Research question- Firefighters training usually takes a great amount of time to equip the firefighters with the skills needed. Thus, the firefighters’ department hope to find another effective way to train their firefighters. The current study aims at investigating the effectiveness of virtual reality in the context of firefighters training. The main research question is ‘Can training with virtual reality increase firefighters’ situational awareness when coping with traffic accidents?’ The research question was answered using two predictors, ‘number of observed scenarios’ and ‘number of action scenarios’ as independent variables and three levels of situational awareness as dependent variables. Two moderators ‘Technology Acceptance’ and
‘years of experience’ are included to investigate whether the growth rate of Situation Awareness can be explained by personal characteristics. Method - The research design includes both a quantitative design and a qualitative design. Ten firefighters from the region of Twente airport participated in answering the questionnaires while three participants from the same group
participated in a semi-structured interview. The questionnaire data was analyzed using SPSS and tested by linear regression. The interview was collected at the firehouse where the interviewees worked. The data was analyzed using the thematic network analysis. Results and Conclusion- The results show that observation in VR training has a positive influence on firefighters’ SA growth while being in action has partial positive influence on firefighters’ SA growth. The study contributes to further understanding of VR training effectiveness. Meanwhile, a positive view toward VR training is provided to the firefighter department.
Keywords: virtual reality, firefighters training, Situational Awareness, Technology Acceptance, professional experience
The Modern Way of Training Situation Awareness: Virtual Reality for Firefighters When it comes to fire accidents, the fewer casualties, the better. Fire accidents can be quite dangerous for closed places with little ventilation and proper evacuation routes (Cha, Han, Lee,
& Choi, 2012). Firefighters training always aims to equip firefighters with the ability to make the optimal choices for the most urgent situations by undergoing a full-scale training. However, in order to fully equip firefighters with the knowledge of all kinds of situations, the economical expenses can be quite heavy (Cha, Han, Lee, & Choi, 2012). According to Bliss, Tidwell, and Guest (1997), traditional firefighters training may waste precious time to rescue victims by looking at the blueprint of a specific building and still fail to find the location of the strategic fixtures, for example, a gas cut-off. These difficulties are worrying the firefighters’ institutions and cannot be neglected because of the high stakes fire accidents can contain. They are trying to train their firefighters with lower costs but cannot sacrifice any aspect of the training. To provide an equally effective but less expensive solution, Virtual Reality (VR) technology has been introduced to this area.
Researchers as well as practitioners have recognized the need for an alternative training method and have tried to investigate how VR can improve the traditional firefighters training and minimize the casualties by creating virtual scenarios related to firefighters training (Bliss,
Tidwell, & Guest, 1997; Cha et al., 2012; Querrec & Chevaillier, 2001). By using VR, the training scenarios can provide virtual experiences to observe how firefighters respond to a car on fire, or wounded passengers, which stand as valuable data to predict the outcome of an actual traffic accident (Cha et al., 2012). Moreover, it is possible to rebuild and repeat the accidents.
This can already be the first step to reducing the heavy expenses that traditional trainings may
impose. As such, the aim of this study is to gain insights into whether virtual reality has a positive influence on firefighters training outcomes.
In this study, the training scenarios covered were mostly car accidents that can happen in real-life. These training scenarios were mainly about vehicle collisions. In these scenarios, firefighters learn to take on the role of officer-in-charge and divide tasks to their teammates.
They also need to familiarize themselves with the procedure when a car accident happens. These skills not only require multiple practice sessions to develop, but also require firefighters to pay close attention to what is happening in each virtual incident. Situation Awareness can help a person be aware of the things happening around them as well as retrieving the correct information from an incident.
Theoretical Framework Virtual reality
Virtual reality (VR) is an emerging technology that offers rich sensory perception under a virtual scenario (Burdea & Coiffet, 2017). It is also defined as a way for people to visualize and interact with computers that allows them to feel more deeply involved in the virtual scenarios (Schultheis & Rizzo, 2001). Some technologies share similarities with VR technology, for instance, 3D computer simulation (Carrozzino & Bergamasco, 2010). The characteristics of VR are similar to those in a video game or the 3D reconstruction of a building (Bruno, Bruno, De Sensi, Luchi, Mancuso, & Muzzupappa, 2010). These technologies all feature a virtual
representation of subjects or objects. VR technology aims at reinforcing such an effect further.
There are two main important qualities when it comes to designing VR technology. The first one is the level of immersion, and the other one is the level of interaction (Carrozzino &
Bergamasco, 2010). The first important quality is psychological immersion. The term
‘immersion’ is defined in a dictionary as ‘dipping something or someone into a liquid’. To transfer this meaning to the context of VR technology, psychological immersion represents that someone has deep mental involvement in the virtual scenarios presented in front of them. VR technology is expected to have a high level of technological immersion because it has three dimensions rather than two (Schultheis & Rizzo, 2001). It is crucial for VR to provide a large virtual space because it can have a high level of immersion (Mirhosseini, Sun, Gurijala, Laha, &
Kaufman, 2014). There are different factors that might influence the level of psychological immersion one can experience. For example, larger screens can provide players with a greater sense of presence and heavier mood change than those with smaller screens while playing video games (Hou, Nam, Peng, & Lee, 2012). This study was in line with why VR simulation was changed to projection display on blank spaces or walls instead of screens in later development.
(Stone, 2001).
The other important quality of VR technology is the interaction between the technology itself and the users. When people use technology, it is often that they interact with it. If the design of technology fails to include this quality, the technology will likely be considered difficult to use. For example, if the technology does not give response when users click on the wrong button, it can be hard for them to know what goes wrong during the process. Furthermore, low usability can lead to negative emotional experiences, which also influence the user’s
perception of the technology (Thüring & Mahlke, 2007). For example, technology such as web- based systems support reciprocal communication with the learners, which allows the learners to gain control over their learning process by actively knowing if they understand everything before moving on to the next step (Liaw & Huang, 2002). This can provide learners with the
opportunity to develop their own way of learning that may result in better learning outcomes. In
addition, using tablets to learn mathematics has proven to be effective if interaction such as timely feedback was included during the learning process (Galligan, Hobohm, Loch, 2012).
These studies showed that it is crucial for technology to be interactive with its learners. Hence, it is beneficial to facilitate and exploit its merits in the development of technologies. It is also important to note that the more interaction between humans and technology, the higher level of acceptance for VR (Shin, 2007).
VR Learning Effects
With the growing interest in VR technology, the question of whether it is effective for improving job performance has become a popular topic. Some researchers focused on exploring the effectiveness of VR in the context of medical training or students’ learning (Aggarwal, Ward, Balasundaram, Sains, Athanasiou, & Darzi, 2007; Merchant, Goetz, Cifuentes, Keeney-
Kennicutt, & Davis, 2014; Schultheis & Rizzo, 2001; Van der Meijden & Schijven, 2009). The learning effects are reported to be positive for both technical skills and non-technical skills. For example, Aggarwal et al. (2007) studied the effectiveness of VR simulation in surgical training.
Their study provided evidence for the effectiveness of VR as a training method: the trainees who trained with VR performed better in dexterity, a technical skill, during surgery than those who did not. Trained surgeons need to obtain the technical skills which involve specific surgical knowledge to perform successful surgeries after their training. This study supported that VR training can improve technical skills. Besides medical training, VR simulation has also been reported to be effective for other types of job training. In manufacturing, VR simulation can increase worker safety and decrease the time needed to accomplish job goals (Mujber, Szecsi, &
Hashmi, 2004). In this context, VR can also train the workers on the manufacturing lines to obtain technical skills, which they need to use for machining, assembly, and inspection.
Additionally, VR technology can allow the users to engage in active learning because they can choose which subject interests them the most (Mantovani, Castelnuovo, Gaggioli, & Riva, 2003).
Apart from training hard skills, VR can also train people’s non-technical skills, including communication skills or problem-solving skills. In VR scenarios, people can be trained to communicate with other people in a fast and unmistakable manner under disastrous situations (Haferkamp, Kraemer, Linehan, & Schembri, 2011). Thus, VR training is not just a tool for training technical skills but also the non-technical ones. With SA being a non-technical skill, it is hypothesized that VR can train SA too. From these studies, VR technology seems to provide a promising path for the future development of training programs.
As VR technology may become a new tool for training programs, it is expected that users will encounter difficulties when using it for the first time. In most training sessions, trainees are trained in groups. The benefits of training in groups is that people can observe others. Observing others is considered a way of learning because it invites people to evaluate the information spilled in advance (Delong & Deyoung, 2007). The role of observation often allows learners to compare their understanding and ideas with others (Okita & Schwartz, 2006). In training settings, trainees usually take turns to be the observers and the participants. Therefore, being observers gives them the chance to learn from others’ mistakes or places the participants have overlooked. Thus, it is likely that VR training can benefit from learning in groups by using the concept ‘learning-by-observing’.
Situation Awareness
The earliest concept of Situation Awareness (SA) was brought in during the stage of World War I. It was connected to the importance of gaining awareness of the enemy before the enemy develops similar awareness (Gilson, 1995). However, SA did not receive much attention
either in the technical or academic field until the 1980s. But ever since then, SA has become a hot topic and has been widely used in training settings such as aviation pilot training (Endsley 2000).
SA is generally defined as a person being aware of the things that happen around him or her (Endsley, 1999). To further elaborate on that concept, Endsley (1995) created a model that broke down the concept of SA into three sublevels: perceiving the elements in the environment, understanding the current situation, and predicting the future status (Endsley, 1995; Stanton, Chambers, & Piggott, 2001).
The three-level model provided by Endsley (1995) is arranged in a hierarchical order, meaning that in order to evolve into the next level, the previous level must be developed. The first level is the perception level. This is considered the lowest level of SA because it requires one to have the ability to perceive the elements in the environment (Stanton, Chambers, &
Piggott, 2001). Since this is a perception level, no interpretation of the collected data is involved (Stanton, Chambers, & Piggott, 2001). An example of this level is that a pilot is aware of the flying altitude of the plane.
The second level is comprehension. This is a level in which one needs to start interpreting and integrating the data they gathered from level one. The ability to comprehend the information becomes crucial because it helps create a mental picture of what is going on. For example, a firefighter can recognize that gas leaks can cause great danger to the people around the area.
During this stage, people learn to detect patterns and relate these patterns to specific scenarios so that the information they gather from level 1 can be integrated into more meaningful data. In psychological terms, schemas mean the ‘matching behavior’. Schemas allow people to focus and can help direct their attention to the appropriate information (Endlsey, 1996). Schemas are
described as a pattern of thoughts or behavior that one organizes based on their previous
experiences or information (Mcvee, Dunsmore, & Gavelek, 2005). Once schemas are formed, it is expected that people can recognize patterns faster under similar scenarios because the
connection is already formed. For example, the firefighters can quickly connect severe car accidents with the possibility of fire explosions because of the leaking gas. Furthermore, if activated properly from long-term memory, schemas can be used to aid performance in domain- specific tasks (Endsley, 1995). Nevertheless, it is possible that people are tricked by the bias they come to develop during this process. This is called the false habitual schemas (Endsley, 1995).
Schemas can inhibit processing irrelevant objects, but it can also blind people from seeing things from a new angle. That being said, the merits of schemas cannot be overlooked. Having
appropriate schemas can speed up the information process, which is considered crucial to forming meaningful understanding of the relevant objects. SA level 2 aims at achieving this process and is the steppingstone to the third level.
The third level is the projection on the future. This is the highest level in the model because it consists of the ability to foresee future status (Stanton, Chambers, & Piggott, 2001). The accuracy of the prediction is highly dependent on the completeness of the collected data from level one and level two (Stanton, Chambers, & Piggott, 2001). For example, a fireman can see that there is a car accident and that there are wounded subjects, which is the observation from SA level 1. Then, he understands that the wounded subjects need to be treated because heavy
bleeding can put the subject's life in danger, which is the comprehension from SA level 2.
Finally, the firefighters decide to call the ambulance because he knows his patient might die if he does not do anything, which is the projection from SA level 3.
The application of SA
SA is so important because it can help cope with complex situations. Quite a few studies have investigated the benefits SA can bring to professional training. Companion (1989)
emphasized that SA is a skill, which includes having the ability to make sense of the surrounding environment using cognitive skills such as perception or attention. This statement corresponds to the first level of SA, which emphasizes the significant role of perception in early SA
development. In another study done by Endsley (1999), she pointed out that SA is important to aviation training because pilots need to make the optimal decisions under complex situations.
Besides, trained pilots not only need to know how to operate the aircraft but also need to understand the up-to-date data fast to make proper decisions. This reflects on both the second and third level of SA (Endsley, 1999). Therefore, including SA in aviation training has been viewed as a crucial application to successful pilot performance. Another complex job training, police officers training programs, provided supporting evidence that people who trained using SA showed a lower level of mental workload compared to those who did not (Saus, Johnsen, Eid, Riisem, Andersen, and Thaye, 2006). Lower mental workload can lead to lower mental stress, which may increase job performance. Additionally, SA is also important in prehospital emergency care. Medical staff need to have the ability to obtain the correct and critical
information to prevent job failures (Norri et al., 2015). Based on these facts, it is evident that SA is both a concept and a skill that can strengthen one’s ability to handle matters, especially the complex ones.
To learn how to develop the three levels of SA fully, many practice sessions are required since SA is not a skill that one can pick up in a day. VR can be very useful in that sense. One of the benefits of training with VR is that it is a cost-effective method when it comes to repeating
training (Cha, Han, Lee, & Choi, 2012). Unlike traditional methods, VR can rebuild and repeat the scenarios as many times as needed. This can shorten the amount of time needed to train SA.
Apart from that, VR can create various virtual scenarios that can either focus on training a specific level of SA or a combination of it. Consequently, it is hypothesized that VR is expected to be a good tool to help people learn to develop SA by creating customized scenarios that fit the need.
The benefits of training SA have been widely mentioned in previous studies. These benefits include better job performance and lower mental effort. However, few studies have investigated how SA can benefit firefighters training. The reason why SA is so relevant for the firefighters is that their job is complex. Not only do they need to receive proper procedure training, but they also need to make optimal choices under limited time. They need to minimize the casualties (Bliss, Tidwell, & Guest, 1997; Cha et al., 2012). The level of job complexity of firefighters can resemble that of the pilots. Both jobs require them to cope with complicated situations. Since Endsley (1999) claimed that pilot training could benefit from training with SA because of its high level of job complexity, firefighters’ training should be no different. Having SA can assist the firefighters in decision making when they need to process large volumes of information. Furthermore, each level of SA plays a significant role in firefighters’ training programs. Firefighters first start with learning how to recognize relevant hazards and eventually possessing the ability to predict the future correctly. Thus, including SA in the firefighters training programs can help them perform their job better. Considering that SA is crucial to firefighters’ training, this study aims at examining the effectiveness of firefighters’ simulation training with the construct of SA.
Individual Differences in SA Development
Although SA is often used in training settings, it is worth noting that there are factors that can influence forming SA (Endsley, 2000). A model provided by Endsley (2000) mentioned these factors. She reported that individual differences, for instance, personal experiences and innate abilities such as fast learning or high level of sensitivity, can generate various outcomes though given the same input data. Other external or internal factors can also jeopardize the development of SA during the training process (Perry, Sheik-Nainar, Segall, Ma, & Kaber, 2007). External factors such as noise or temperature can create stress, which can be distracting factors (Perry et al (2007); internal factors such as fear or self-esteem can have a negative impact on developing SA. Dealing with internal factors can be difficult because they mostly come from a rather personal aspect. Personal factors such as previous experiences and individual abilities can already create a different level of comprehension of the same information, which can
generate different objectives and expectations on the subjects they observe (Lee, Suh, & Whang, 2003). To summarize, it should be remarked that the outcome of SA development might differ because of the personal factors.
SA development represents the process of which one learns to observe, comprehend, and project future status based on the relevant subjects. Studies pointed out that previous experiences can show significant differences in acquiring SA during trainings. For instance, pilots who had previous flight experiences acquired SA skills better during the training compared to those who did not (Waag & Houck, 1994). Similarly, experienced learners are likely to have better SA and thus make less mistakes compared to novice counterparts under conversational distractions (Kass, Cole, & Stanny, 2007). Studies have also stated that previous job-related experiences have a positive influence on job motivation and job performance (Klassen, & Chiu, 2010; Li, &
Zhang, 2000). Experience can help a person form schema, which are useful for developing SA.
With these studies demonstrating the positive influence of job experience on SA development, it is worth investigating if such influence will also reflect on the SA growth rate. In the context of firefighters training, the ones with more experience in firefighting may acquire SA faster. Thus, it is relevant to find out whether ‘years of experience’ can also make these firefighters output a steeper learning curve in SA development during the training process.
Another factor that may influence the effect of VR training on SA is ‘Technology
Acceptance’. While studies provided evidence for the learning effectiveness of VR technology, others provided insight into how human perception, for example, motivation can influence the acceptance of VR technology (Shin, 2017). The definition of TA is mostly known through the UTAUT (Unified Theory of Acceptance and Use of Technology) model. It is known for identifying the four determinants of TA. The four determinants are performance expectancy, effort expectancy, social influence, and facilitating conditions (Venkatesh, Morris, Davis &
Davis, 2003) (see Figure 1.).Performance expectancy is defined as people believe using the technology can help them in their job performance. This factor can influence TA because people tend to use technology when they can perceive its usefulness or their personal gains. Effort expectancy is defined as the level of ease of use of technology. This factor can influence TA because technologies that are too complicated can lower users’ motivation in the beginning of the learning process. People tend to give up easily when they fail to see immediate progress.
Social influence is defined as the degree to which a person perceives that their important others believe they should use the technology. This factor can influence TA because people tend to be persuaded by the ones they trust. If their important others hold positive attitudes toward the new technologies, it is likely that they will think alike. Facilitating conditions are defined as people
believing that an organization or a technical infrastructure can support them in using the
technology. This factor can influence TA because people need to be technically supported to use the technology. A higher level of TA was reported to have a positive effect on both user’s learnability and the perception toward the usability of VR (Shin, 2017). Since the current study includes training with VR technology, it is likely that TA can also influence the growth rate of SA development.
Figure 1. The UTAUT model without the four moderators (gender, age, experience, and voluntariness of use)
Many studies have provided information on the application of the UTAUT model. For instance, a study found that people show low intention of using internet banking because it is difficult to learn (Sok Foon & Chan Yin Fah, 2011). Because the performance expectancy is too high, a negative attitude is likely to form. Another study reported that the four determinants from the UTAUT model successfully predicted the students’ acceptance of using tablets in educational settings (El-Gayar, & Moran, 2006). Two factors in the UTAUT model, perceived usefulness and perceived ease of use are known to affect the learners’ satisfaction in e-learning (Lim, Lee,
& Nam, 2007). Furthermore, it was reported that the easier it is to use the e-learning environment, the better the learning performance (Sun, Tsai, Finger, Chen, & Yeh, 2008).
Consequently, the UTAUT model is selected as a guideline to design the TA questionnaire in this study. However, Venkatesh et al. (2003) reported that behavioral intentions, which represents the facilitating conditions moderator, is fully mediated by effort expectancy. Thus, facilitating conditions is not included in the questionnaire.
To conclude, individual factors such as ‘years of experience’ may influence the growth rate of SA. Meanwhile, TA may influence the training effects of VR technology and can also
influence the growth rate of SA. Therefore, these factors are chosen as moderators in this study to present a clearer view of the training effects.
The research can contribute both in research and in practice. For the research part, the current study is expected to contribute to the field of technology, specifically in the field of virtual reality development because the research aims at investigating the effectiveness of VR.
Apart from that, this study can provide empirical data to assist and support further research on understanding the effectiveness of VR. For the practice part, since the target group of this study is the firefighter department from the region of Twente airport, it is expected that the results will be able to provide the firefighter department with the necessary information to consider
alternative methods when training their firefighters. If the effectiveness of VR is proven through this research, it is possible that firefighters training can cost less. Furthermore, if a positive outcome can be reported after the firefighter department from the region of Twente airport has applied such technology to their training program, it can be a great opportunity for the
technology to spread nationwide and worldwide.
Research question and hypotheses
The current study first focuses on investigating the effectiveness of firefighters training with virtual reality. The training effects are measured using the construct of Situation Awareness divided into three levels. The first level of SA is to recognize relevant objects. The second level of SA is to make proper interpretation of the information gathered from level one. The third level of SA is to have the ability to predict the future based on the information collected and
interpreted from level one and level two. Therefore, the first research question is: Can training with virtual reality increase firefighters’ training effects in the context of Situation Awareness when coping with traffic accidents? Secondly, this study focuses on the moderating effects of the two moderators in this study: ‘years of experience’ and ‘Technology Acceptance’. Therefore, the second research question is: Do years of experience and Technology Acceptance have
moderating effects on VR training effectiveness?
The following hypotheses guide this study.
Hypothesis 1: Training with virtual reality has a positive effect on aspiring officers-in- charge’s number of perceived key objects (SA level 1).
Hypothesis 2: Training with virtual reality has a positive effect on aspiring officers-in- charge’s understanding of the situations (SA level 2).
Hypothesis 3: Training with virtual reality has a positive effect on aspiring officers-in- charge’s prediction of the near future (SA level 3).
Hypothesis 4: The more years of experience a fireman has, the higher the growth rate one will have on perceiving the key objects.
Hypothesis 5: The more years of experience a fireman has, the higher the growth rate one will have on understanding the meaning of the key objects.
Hypothesis 6: The more years of experience a fireman has, the higher the growth rate one will have on predicting the near future.
Hypothesis 7: The higher the Technology Acceptance, the higher the growth rate one will have on perceiving the key objects
Hypothesis 8: The higher the Technology Acceptance, the higher the growth rate one will have on understanding the meaning of the key objects.
Hypothesis 9: The higher the Technology Acceptance, the higher the growth rate one will have on predicting the near future.
The aim of the interviews is to gain insight into an overall perception of how the
interviewees perceive the X-VR training. During the interview, the interviewees will be asked to give inputs regarding the process of the training and to express their likes and dislikes about the training. They will also be guided to discuss whether they think the training is effective or not. It is expected that the interviewees think the training is effective, but the limitations of the
technology still remain.
Method
Research design
The research design of this study was a mixed methods design which included a quantitative design and a qualitative design. The quantitative design was conducted using
questionnaires. It was a quasi-experiment study which aimed at investigating the SA growth rate of the firefighters. The qualitative design was conducted using interviews, which aimed at gathering individual perspectives of the X-VR training. The study had two purposes. First, the study focused on investigating whether participants showed growth in Situation Awareness during simulation training, which allowed the researchers to gain insight into whether such
training was effective or not. Therefore, three levels of Situation Awareness were measured using ‘aspiring officers-in-charge’s number of perceived key objects’, ‘aspiring officers-in- charge’s understanding of the key objects’, and ‘aspiring officers-in-charge’s prediction of the near future’ as dependent variables. Two predictors ‘number of observed scenarios’ and ‘number of action scenarios’ were used as independent variables, which represented the amount of X-VR training each participant had received. “Years of experience’ and ‘Technology Acceptance’ were selected as the moderators.
Participants
There were 10 people from the region of Twente airport who participated in the study.
The overall response rate for the quantitative design was 83%. They were selected based on the participation of the existing simulation training class from the firefighter department training program for the role ‘officer-in-charge’. All of them were male, aged between 18 and 50 years old. Two trainers participated as informants. The first trainer had approximately five years of experience with VR training. The other trainer has two to three sessions of VR training prior to the X-VR training sessions. Both trainers had a great amount of experience as firefighters. The 3 participants for the interviews were collected from the existing participants from the quantitative group. The overall response rate for the qualitative design was 30%.
Instruments
X-VR. X-VR is the virtual reality tool which was used in the experiments and was a tool that allowed users to control the virtual scenarios with a joystick at hand. The virtual scenarios designed were made for the training program named ‘bevelvoerder’, which meant commander training in English translation. The scenarios were projected on a white wall in front. All
directions (straight, left, right, back) and head-turning could be controlled using the joystick. The
joystick also had buttons on the front to zoom-in or zoom-out. All scenarios practiced in the training sessions were vehicle accidents. Each scenario started with a film clip. The film clip was a pre-recorded footage which was used to visualize how the fire truck arrived at the scene of the accident. The fire truck siren was included in the film clip to bring in the real-life feeling. After that, the collision of the cars, trucks or vans were presented in the scenarios. There were injured drivers and passengers. The scenarios took place in diverse locations. Each scenario lasted for around 45 minutes, including the film clip in the beginning. Additionally, there was a trainer manual which was provided to the trainers to check which information was made available for the trainees to ask for each scenario. During the training, the virtual scenarios were controlled by another person in the VR room. This person could insert live changes to the scenarios, for example, the arrival of the ambulances or the police. This person would also respond as the driver of the fire truck during the film clip. The number of observed scenarios and action
scenarios were provided by the organizer of the training. Each scenario had one person in action, while the others were registered as observers. They responded as a team of firefighters.
Situation Awareness questionnaire. A questionnaire consisting of nine questions was used to assess the level of Situation Awareness of each participant after each training session ended, and it was filled out by the trainer. The questions were divided into three parts according to the three levels of Situation Awareness (see Appendix B). Each part consisted of 3 questions.
Each question was answered using the Likert-scale with 1 as strongly disagree to 7 as strongly agree. The first part was focused on whether the participant could perceive the relevant objects, hazards, and location which needed to be recognized by the participants, for example, ‘The trainee could accurately perceive relevant information such as location, status, and hazardous objects in the selected scenario(s)”. The second part was focused on whether the participant
understood the meaning of the recognized objects, hazards, and location from the first part; for example, ‘The trainee had the ability to make an integrated analysis based on the information perceived from the selected scenario(s)’, for example, ‘The trainee could make a realistic
assessment of when and how hazards will become immediate threats’. The third part was focused on whether the participant could project the future based on the subjects they perceived and the understanding of the subjects; for example, ‘The trainee can make a realistic assessment of when and how hazards will become immediate threats.’
The questions were designed based on the soft-copy booklet provided by the firefighter department. In the booklet, information on how the trainer would mark the performance of each participant after a scenario was included. From the information, the researchers combined the professional knowledge provided in the booklet and their own understanding of Situation Awareness from literature such as Endsley (2000) to generate the optimal questions for the study. Additionally, the questions were formulated broadly on purpose so they could apply to all scenarios. For the detailed content of the questionnaire, please see Appendix B.
Years of Experience. A single question in the background questionnaire was used to collect the data of a firefighter's years of experience. Seven options were provided. The years can range from 0 to more than 20 years. For the detailed content of the background questionnaire, please see Appendix A.
Technology Acceptance. Nine questions were used to measure TA. The aim of the questions was to evaluate the level of TA for each participant. Three constructs were taken from the model and were incorporated into the sub-questions, namely performance expectancy, effort expectancy, and social influence. Performance expectancy was measured with three questions, for example, ‘I think VR technology is useful for my job.’. Effort expectancy was measured with
three questions, for example, ‘I think VR technology is easy to use.’. Social influence was measured with three questions, for example, ‘I think VR technology is necessary because my colleagues said so.’ (See Appendix A). All questions were answered using the 7-point Likert scale with 1 as strongly disagree to 7 as strongly agree. The questions designed in the
questionnaire were adapted from Madigan, Louw, Dziennus, Graindorge, Ortega, Graindorge, &
Merat (2016) and the UTAUT model.
Interview questions. The questions aimed at gaining knowledge of the participants’
perception of the X-VR training process and their perception toward the technology. The interview questions were generated based on the researcher's knowledge on learning effectiveness and SA. There were six open-ended questions. Three main concepts, the
usefulness, the practicality, and the training process, were investigated based on the questions.
Follow up questions could be asked if needed. For detailed content of the questions, please see Appendix C.
Procedure
The research method of this study has been approved by the Ethical Committee of the University of Twente. The location of the data collection took place in the firefighter department from the region of Twente airport. In the training room, the joystick was placed on a table. The table was placed in front of the walls, of which the virtual projection would appear. There was another table for the observers to stand or sit around and take notes. The trainers sat behind the computers to monitor the whole training process. There were four training sessions in total. One session took place per week. Each of it lasted around 3.5 hours.
The researchers arrived before the start of each training session. Upon arrival, they explained why they were present and what they would do during the session. Before the first
session began, two questionnaires and the active informed consent forms were handed out to the participants (trainers and trainees). The first questionnaire was the background questionnaire.
This questionnaire was used to collect the background information of each participant. The second questionnaire was used to assess the trainee’s level of Situation Awareness, which was filled out by the trainer after the scenario ended. Participants were asked to sign after reading the informed consent form. The participants had the chance to ask questions about any aspect of the study. All experiments were carried out in the same manner and took place with the presence of the researchers. All participation of the experiments was voluntary.
In each training session, two virtual scenarios were practiced. All virtual scenarios were already selected by the trainer from the firefighter department before the training sessions began.
For each training session, 10 trainees attended. Most trainees remained the same throughout the four training sessions. They were divided into two groups for training purposes. One group for 6 people and the other one for 4 people. However, there was no difference between the groups in terms of training methods or procedures.
In each group, one person would take control of the joystick while the others stand behind him to observe. The observers were allowed to take notes. They could also interact with the trainer and the person in action by giving feedback or providing reminding comments.
During the simulation training, each participant freely controlled how they wanted to explore the scenario. They also needed to follow the training procedure set by the firefighter department. The training was divided into two parts for each scenario. The first part contained a ride scene that lasted for exactly 6 minutes. During this time, the trainee needed to select and filter information and communicate with the rest of his team, the observers, who were standing or sitting behind
him. The trainee also needed to communicate with his trainer. The communication between the trainers and the trainees resembled that of the radio communication.
After the first part ended, the trainer, the trainee, and the observers evaluated the
performance. During this evaluation, the trainer asked the trainees if there were any information they forgot to discuss during the film clip. After that, they moved on to the second part.
The second part of the scenario was the scene where an accident took place. In this part, the trainee had to give commands, think aloud, say the subjects or objects he saw, the actions he planned to do, and communicate with his commanders and his team. When the incident master signal was given, the second part would end. Then, the trainer would give the trainer his
feedback on what would be a good performance for each scenario. The observers also provided feedback and discussed what would happen if the scenario was real-life. Each scenario was carried out in the same manner. 15 scenarios in total were practiced throughout the training sessions. Each session contained four scenarios except for the third one. The third training contained three scenarios.
After four training sessions were finished, three interviews were done to collect qualitative data. Since the interviews were an additional request to the participants, another informed consent form was given out to the three participants who agreed to do the interview.
The additional data collection was also approved by the Ethical Committee of the University of Twente. The interviews were scheduled based on the available time slots of the participants. The location of the interview was at the firehouse in which each participant worked as a firefighter.
Each interview took around 15 minutes. Each interview was done individually to avoid answers being influenced by other participants. The interview was audio recorded. During the interview, the interviewer made encouraging remarks to help the interviewees elaborate their answers.
Data analysis Quantitative data
Quantitative data was collected using the SA questionnaire and the background
questionnaire. The data was inserted into SPSS manually. To investigate whether the questions in the SA questionnaire addressed the relevant variables, a factor analysis was carried out (see Table 1). Based on the factor analysis results, only the questions aimed for SA level 1 were grouped into a single factor. The possible explanations for this was that SA level 2 and 3
included more diverse skills than SA level 1. It was possible that the questions in SA level 2 and 3 did test the comprehension and the projection skills. However, observation skills might still be included in those questions. Therefore, those questions might not be restricted to only
comprehension or projection skills. Due to this hypothesis, SA was still investigated as three separate variables. To test the reliability, Cronbach's’ ɑ was: 0.98 for the entire SA questionnaire, 0.95 for SA level 1, 0.94 for SA level 2, and 0.94 for SA level 3. The number of Cronbach’s ɑ indicated that the reliability for the SA questionnaire was excellent.
SA growth was investigated based on the mark on SA questionnaire. If the mark of each SA questionnaire went up together with the number of observed and action scenarios, then there was SA growth during the training process. Then, linear regression was conducted to report the effect of X-VR training on SA growth. Each SA score was entered separately into the regression analysis. Three levels of SA and two predictors were tested separately. The predictors tested in the models would be ‘number of observed scenarios’ and ‘number of action scenarios’. Due to time constraints, not every participant has the data for SA questionnaire in every session.
To analyze the effect of the two moderators ‘Technology Acceptance’ and’ years of experience’ on the SA growth rate, the correlation between the moderators and SA was first
investigated using Spearman’s rho in SPSS. Spearman’s’ rho was used because the data in this study violated some assumptions of the Pearson’s correlation while no assumptions needed to be met when using Spearman’s rho. After that, linear regression was used to further investigate the relation between the moderators and the SA growth. Variables TA, years of experience, number of observed scenarios, and number of action scenarios were first centered to reduce the changes of multicollinearity before the analysis. The interaction terms, however, were computed before the variables were centered. Each interaction term was created with one moderator (TA or years of experience) and one predictor (‘number of observed scenarios’ or ‘number of action
scenarios’). The three levels of SA were tested separately for the interaction effect. This allowed the researcher to see if the interaction effect of the particular moderator on a specific SA level was significantly stronger than the one without the interaction effect. To guarantee the questions in TA questionnaire answered the relevant variable, a factor analysis table was made (See Table 1). Based on the results, effort expectancy and social influence did not meet the expected factor loadings. Although only the questions aimed for performance expectancy met the expected loadings, other questions did successfully investigate at least one of the constructs of TA. This indicated that the questions still maintained its value based on their factor loadings on TA.
To test the reliability of TA questionnaire, Cronbach's’ ɑ was: 0.86 for the entire TA
questionnaire, 0.94 for performance expectancy, 0.59 for effort expectancy, and 0.36 for social influence. Due to the low reliability of effort expectancy and social influence, the idea of merging TA into one variable was supported. Merging TA into one variable reframed the focus of the value of Cronbach’s ɑ to 0.86. This meets the standard view of high reliability, which is 0.70 for Cronbach’s ɑ (Shelby, 2011). To further support the decision of merging TA into one variable, pre-analysis was made on the difference between analyzing the factors of TA separately
and analyzing them as one single variable. The pre-analysis revealed that the results did not generate different interpretations even if factors were to be analyzed separately. This conclusion applied for both the correlation test and the moderation analysis. Thus, it was decided that TA continued to be analyzed as a single variable throughout the study.
The variable trainer who filled in the questionnaire was scrutinized in several exploratory analyses to investigate if different trainers graded the SA questionnaire differently. If so, the learning curve might yield different results. All data was analyzed anonymously.
Table 1
SAFactor Loadings Resulting from a Principal Components Using Oblique Rotation (N=10) Factor Loadings
Item SA level 1 SA level 2 SA level 3
The trainee can accurately perceive relevant information such as location, status, and hazardous objects in the selected scenarios
.68 -.15 .16
The trainee is aware of the location and status of tools that may be needed for effective execution of tasks
.75 -.33 -.09
The trainee is aware of the activities of people in the selected scenarios, such as first-aid helpers, victims, and bystanders
1.07 .14 -.01
The trainee can make an integrated analysis based on the information perceived from the selected scenario(s).
.06 -.90 .05
The trainee can evaluate the locations and activities of people present in light of planning and task division and assess progress
.72 -.19 .09
The trainee understands how and why hazards may pose a threat and is able to use the knowledge and resources while making decisions
.03 -.27 .72
The trainee can make a realistic assessment of when and how hazards will become immediate threats.
.73 -.03 .24
The trainee can make accurate predictions based on the current activities and observation of the people on the scene
.12 -.78 .13
The trainee can foresee the possible damage of the surrounding environment and can act accordingly
.07
.06 .97
Eigenvalues
% of explained Variance
7.33 81.49
.49 5.48
.37 4.16 Note. factor loadings over .40 appear in bold.
Table 2
TA Factor Loadings Resulting from a Principal components Using Oblique Rotation (N=10)
Factor Loadings
Item
Performance expectancy
Effort expectancy Social influence
I think VR technology is useful for my job
.87 .05 .20
I think VR technology will become an important method of job training in the future
.99 -.12 .15
I think VR technology is/will be an effective and efficient
technology when it comes to fulfilling my job
1.02 -.12 .01
I think VR technology is easy to use
.59 .05 -.05
It is not difficult for me to understand how to use VR technology
.17 .61 .43
I think it will not take me long to learn how to use VR technology
-.12 .95 .13
I think VR technology is necessary because my colleagues said so
.67 .31 -.37
The people around me think I should learn how to use VR technology
.09 .93 -.26
I am more likely to use VR technology if I know my friends or family are using it
.16
.06 .94
Eigenvalues
% of explained Variance
4.50 50.03
1.65 18.37
1.26 14.01
Note. factor loadings over .40 appear in bold.
Qualitative Data
The interview data was analyzed using the thematic network analysis approach adopted from Attride-Stirling (2001). This approach aims at understanding an issue or an idea by pinning down the topics based on the interview data in an organized manner. In this study, exploring the understanding of participants’ perception of X-VR training was the issue. The Thematic network analysis could help identify the themes which the content of the interviews was centered around, and draw a conclusion based on the interpretation of the conversational texts (Attride-Stirling, 2001). The full process of the thematic network analysis included three steps: breakdown of the text, exploration of the text, and integration of the exploration. Due to the limited number of interviewees, the researcher decided to skip the coding scheme in the first step because a
complete coding scheme worked better when there were more interviewees. However, the initial concept of the coding scheme was still used. The raw data such as similar answers or word choices would be marked as meaningful chunks of texts. To better analyze the content of the interviews, the conversations were recorded and transcribed (See Appendix D). The transcripts were used to compare the similarities and the differences between the answers of the
interviewees. After that, abstract themes from the meaningful texts were organized and refined so that the data was reduced to a more manageable set of specific themes. This step contains careful interpretation of answers given by the interviewees. The three levels of SA and the overall attitude of X-VR training would be analyzed based on the answers generated by the six main questions in the interview. The last step was to arrange all the themes derived from the previous steps. All themes would be compared to form groups of similar themes. Eventually, this would result in a global theme, which could summarize the main inputs gathered from the
interviewees.
Results Descriptive statistics
The mean and standard deviation of Situation Awareness and Technology Acceptance were presented (see table 3). For SA, three levels were reported separately. For TA, three determinants were reported separately.
The moderator ‘years of experience’ was an interval variable. Therefore, the mean and standard deviation were not possible to present. Among all the participants, 4 participants have 4-6 years of experience in firefighting; 2 participants have 7-10 years; 1 participant has 11-13 years; 1 participant has 14-16 years; 1 participant has more than 20 years.
Spearman’s rho correlation tests
Based on the results, there was no significant relationship found between the three levels of SA and TA (SA1, r = .26; SA2, r = .35, SA 3, r = .42). Similarly, no significant relationship between ‘years of experience’ and the 3 levels SA was found (SA1, r = .26; SA2, r = .16, SA 3, r
= -.01). The correlation table was presented in Table 4.
Trainer difference
To investigate whether different trainers could make a difference in SA growth, trainer difference was analyzed as a moderator in the linear regression. The mean of the SA
questionnaire was different. Trainer 1 had a lower mean while trainer 2 had a higher mean (M1 = 4.32, SD1 = 1.40; M2 = 5.23, SD2 = .49). Also, based on the results from regression analysis, significant effect was found between trainer differences on SA growth on both predictors.
(‘number of observed scenarios’, R² change = .22, p = <.01; ‘number of action scenarios’, R² change = .20, p = .01).
Regression analysis
The regression analysis was used because it can identify if there is a casual relationship between the predictors and the SA scores. Three out of five assumptions for running a linear regression analysis were not met. The assumptions are normality, homoscedasticity, and
codependency. The normality assumptions were not entirely met because variable SA is slightly skewed to the left while both predictors are normal or close to normal distribution. The simple linear regression test was conducted to investigate the SA growth from the first session to the last one. Two predictors, ‘number of observed scenarios’ and ‘number of action scenarios’ were tested separately. For ‘number of observed scenarios’, it revealed that there was a statistically significant effect found on SA growth for all levels. SA level 1, F(1,19) = 13.34, p = < .01; SA
level 2, F(1,19) = 6.81, p = .02; SA level 3, F(1,19) = 11.80, p = < .01. Investigation of the parameters showed that ‘number of observed scenarios’ had a significant effect on SA growth on all levels. For ‘number of action scenarios’, it revealed that there was a statistically significant effect found only on SA level 1, F(1,19) = 8.76, p = < .01 and level 3, F(1,19) = 5.00, p = .04, but not SA level 2, F(1,19) = 2.38, p = .14. The results indicated that the SA growth increased if one observed more scenarios. On the other hand, high number of action scenarios partially led to higher SA growth rate.
Moderation Analysis
Moderation models were built to investigate the interaction effect between the SA training progress and the moderator variables TA and ‘years of experience’. The results showed that TA has partial interaction effect between VR training and SA growth rate with the predictor,
‘number of observed scenarios’. Such interaction effect was only found on SA level 2 (R² change
= .14, p =. 02). There was no interaction effect found on SA level 1 (R² change = .06, p = .10) or level 3 (R² change = .06, p = .13). The same results were found for the predictor, ‘number of action scenarios’. The interaction effect was only found on SA level 2 (R² change = .19, p = .01).
There was no interaction effect found on SA level 1 (R² change = .05, p = .17) or level 3 (R²
change =. 09, p = .09). To conclude, both predictors indicated that TA does not fully moderate VR training and SA growth rate.
The results for moderator ‘years of experience’ showed no interaction effect between VR training and the SA growth rate. The results applied for both predictors, which were ‘number of observed scenarios’ and ‘number of action scenarios’. The results are presented below (See Table 6).
Table 6
Regression moderation models of study variables
SA level B β p R² R²
change
Exp -.02 -.01 .90
O Level 1 .79 .47 <.01*
Exp*O -.54 -.05 .36 .80 .02
Exp -.10 -.06 .62
O Level 2 .73 .38 <.01*
Exp*O -.55 -.04 .42 .72 .02
Exp -.27 -.18 .12
O Level 3 .80 .44 <.01*
Exp*O -.64 -.05 .29 .79 .03
Exp .13 .10 .49
A Level 1 .63 1.00 .03*
Exp*A .04 .01 .96 .65 <.01
Exp .06 .04 .18
A Level 2 .43 .60 <.01*
Exp*A .03 <.01* .96 .44 <.01
Exp -.10 -.07 .64
A Level 3 .51 .76 .03*
Exp*A .06 .02 .94 .51 <.01
TA .12 .16 .47
O Level 1 .75 .45 <.01*
TA*O -1.45 -.18 <.10 .84 .06
TA .29 .35 .12
O Level 2 .60 .31 <.01*
TA*O -2.18 -.24 .02* .84 .14
TA .23 .30 .20
O Level 3 .65 .36 <.01*
TA*O -1.43 -.17 .13 .80 .06
TA .41 .57 .02*
A Level 1 .67 1.06 <.01*
TA*A -1.00 -.40 .17 .79 .05
TA .52 .63 .01*
A Level 2 .46 .65 .02*
TA*A -1.92 -.67 .01* .80 .19
TA .48 .48 .02*
A Level 3 .53 .53 .01*
TA*A -1.33 -1.33 .09 .76 .09
*p < .05
TA, Technology Acceptance; O, number of observed scenarios; A, number of action scenarios;
Exp, years of experience
Interview analysis
The inputs from the participants confirmed the initial hypothesis of their perception toward the X-VR training. The overall perception of the XVR training was good. Interviewees reported that XVR was not hard to use and was very useful for training specific skills, which were the procedures when car accidents happened. The following quotes provided the evidence for the perceived ease of use of the X-VR training.
Interviewee 1:
At first, the exercises were very difficult, also from my experience because you really have to know how it works. In the first exercises, it played a role for me because you really focus on the stick because you have to handle it. But the last exercises, I didn’t even [notice] the things are there. (Interviewee 1, 2020)
Interviewee 2:
At first, it’s difficult. It’s an extra obstacle. In the procedure, when you repeat it, it gets easier. (Interviewee 2, 2020)
Interviewee 3:
...the first time is not easy because it’s new and you are very busy with the joystick.
But after two, three times, it gets there automatically and then you can use your mind for other things that are within the simulation. (Interviewee 3, 2020)
The following quotes provided evidence for the perceived usefulness of the X-VR training.
Interviewee 1:
I learned different things starting with I think that’s for myself, the most important thing is you get structure with the incident. Second, to observe things in the screen and in the system which can help you with solving the problem. And the first thing is
that you learn to manage the incident. And the system, in that case, is very useful because the system brings you some structure. (Interviewee 1. 2020)
And the system, in my opinion, is the first step. So when we have achieved that, we get the structure of the incident, and we make the next step to […] dealing with other people…(Interviewee 1, 2020)
Interviewee 2:
To set priorities to the setting, for that, it’s good. Yeah, prioritizing and procedure, that’s the most important I think....partly I think it helps. (Interviewee 2, 2020) I think...XVR is the future. It’s unstoppable. (Interviewee 2, 2020)
Interviewee 3:
...we are practicing the procedures. And that’s a skill that we are developing. It’s getting better and better…(Interviewee 3, 2020)
What did I learn? Hmm, like I said, the procedures. You know them in theory but now you have do it...hmmm...you can learn them from paper, but now it takes time for someone to do the job. So you have to wait and think again ‘Oh wait, what is the next step in the procedure. Oh, that’s that.' So we learned that. Yeah and thinking the incident before you arrive. ‘What kind of scenarios can you expect?’ And that’s also a learning point. (Interviewee 3, 2020)
Apart from that, they gave credits to the merits of X-VR. Two interviewees mentioned that training with X-VR saved time, and it gave them chances to think in a more organized way.
Therefore, they considered X-VR as an effective tool to train skills.
Interviewee 2:
I write things down so I have a structure and the procedure...it’s got a standard in my head…(Interviewee 2, 2020)
Interviewee 3:
I like it because they can do everything within the small amount of time. Normally, when we go practice in the real world, you can...it’s a lot of work to put the vehicles.
...everyone participates, and we can learn within one evening, you can do four incidents. Well, normally, you do one incident and it takes about one and a half hour to two hours. So your learning curve is much better. Interviewee 3, 2020)
However, all interviewees agreed that X-VR was not realistic enough as the system did not include any human factors that could interfere with the training process. They mentioned that this was a major limitation they saw from the X-VR. Phrases such as ‘not practical enough’, ‘X- VR stays the same’, ‘not realistic’, ‘X-VR is limited’, or ‘missing the dynamics’ were mentioned by the interviewees. For example, people with bumpy personalities in an incident could influence the rescuing process of the injured people. But this quality could not be represented in the X-VR scenarios. Also, there were few obstacles to communications and therefore was too ideal
compared to real case incidents.
The following quotes provided evidence for such views.
Interviewee 1:
The dislike of the system is it’s, sometimes, I miss the dynamics. ...in real incident, when people achieve task, they get back to you and say ‘I have completed the task.
What’s the next task?” And here is, I give them a task, they do it, and they walk back... in real incident, when they get on scene, they will come to you, and they will ask what to do. (Interviewee 1, 2020)
Interviewee 2:
I think in reality, when you train in reality, it’s better. There are much more
obstacles ... In real-world, sometimes things go wrong. So when you are the leader of that incident. You have to participate on that so that’s not the XVR. Everything stays the same. (Interviewee 2, 2020)
No, I think it’s not practical enough. So in the real world, when you free a victim out of the car, you have to watch what your men are doing. So that’s...not you don’t have to when you say ‘remove the door’, there’s one way to remove the door and you don’t have to look at it if it’s good or not. (Interviewee 2, 2020)
I think, personality of people affects most of the incidents. And you can’t let it see on VR. So when someone is tired, that goes...the quality of the incident goes down. So you have to [anticipate] on that. And [in] VR, it’s [fixed], it’s always the same.
(Interviewee 2, 2020) Interviewee 3:
I find it difficult that the last time we trained, there was some glass on the ground but I didn’t see it because we are focused on the ‘Okay, did the individual in the video was doing some kind of things?" And normally, you would see real glass on the ground and you can act on it. But that’s difficult in reality. (Interviewee 3, 2020)
Additional remarks about X-VR training were that the trainers in the training sessions were helpful because they urged the trainees to think critically. Trainers stood as a necessary component to effective training in procedures and standards.
Interviewee 1:
