Using a Virtual Patient to Improve Communication Skills:
The Correlation Between Engagement and Learning to Apply
Shared Decision Making
Kristy Timmers
Master's Thesis: Graduate School of Communication
Student ID: 10557695 Supervisor: Gert-Jan de Bruijn
March 29th, 2019
Master's Program Communication Science, Persuasive Communication
Abstract
Effective communication between health care professionals and patients has a positive effect on medication adherence (Oshima Lee & Emmanuel, 2013), consult satisfaction (Kunneman et al., 2015), and can reduce conflicts (Walsh et al., 2014) as well as medical costs (O'Conner et al., 2014). One way for health care professionals to establish good communication is by including patients in the process of making medical decisions through Shared Decision Making (SDM). This present study explored the potential of a computer-simulated virtual patient (VP) and its system-generated feedback to teach medical students communication skills of SDM. Moreover, the correlation between SDM learning and levels of engagement was investigated. The research design consisted of two twelve-minute virtual consults with a VP and corresponding feedback after each session, based on how well SDM was applied. Additionally, the engagement experienced during the virtual consult was reported after each session. The results revealed that the ability to apply SDM during the consult with the VP improved significantly. However, this improvement was not correlated to levels of engagement. Overall it can thus be concluded that the VP provided a valuable learning opportunity for training SDM communication skills, but that this learning effect could not be explained by a variance in levels of engagement. These findings therefore leave questions about the learning effect of SDM but they do provide valuable knowledge for medical curricula in offering a promising innovative tool that can enhance communication skills through VPs.
Introduction
Effective communication between a medical professional and a patient is an important aspect that influences various health-related outcomes such as increased medication adherence (Oshima Lee & Emmanuel, 2013), less conflict (Walsh et al., 2014), more recall of medical information (Stiggelbout, Pieterse, de Haes, 2015), enhanced patient satisfaction (Kunneman et al., 2015), and a reduction of medical costs (O'Conner et al., 2014). For these reasons, it is desirable that health care professionals master medical communication skills and, consequently, optimize the care they can provide for their patients (Aspegren & Lønberg-Madsen, 2005; Légaré et al., 2014).
A crucial aspect of good communication during a medical consult is that a patient's wishes and needs are attentively discussed and well understood by the health care professional. Although health care professionals have frequently claimed to value this way of communicating with their patients (Pieterse, Baas-Thijssen, Marijnen, & Stiggelbout, 2008), in practice, patients are only infrequently involved in the medical decision-making processes (Stacey et al., 2017). As a result, the care that is provided often does not reflect their specific needs and wishes (Couët et al., 2013). This notion is supported by a study which revealed that of 3500 medical decisions, less than ten percent met the minimum standard of a well-informed, collaborative decision-making process (Oshima Lee & Emmanuel, 2013). Additionally, this study reported that only 41 percent of the patients felt as though the care they received aligned with their preference for palliative care over more aggressive interventions. For these reasons, communication trainings for health care professionals should be considered an essential aspect of their medical education. In the long-term, this will also benefit the overall health care system and the countless people involved.
To enhance these beneficial communication skills, digital tools currently play an important role (Pertaub et al., 2001; Garau et al., 2001). One such tool makes use of computer-simulated virtual patients (VP). Using VPs seems especially promising in the context of learning communication skills because they can be tailored to the current preferences, needs, and capabilities of the learner. Moreover, VPs provide a solution to the impracticalities of patient-actors that are used in traditional communication training programs and allow for a structured and safe environment for students to practise. For these reasons, they being implemented more extensively in medical education.
However, the usefulness of VPs for training communication abilities is still a matter of debate (Poulton & Balasubramaniam, 2011). For example, a literate review about VPs could only find very few studies that provided supporting evidence for their effectiveness (Cook &
Triola, 2009). It was noted that the majority articles were descriptive and solely measured user-satisfaction with the VPs rather than its potential to improve communication abilities. Although challenged by the small number of available literature, another meta-analysis concluded that VPs were limited in their usefulness for educational outcomes in the specific domain of communication skills (Consorti, Mancuso, & Piccolo, 2012). Yet another literature review emphasized that, despite the rapid technical advances of educational VPs, the current majority is not very sophisticated and therefore seems insufficient for promoting learning (Kononowicz, 2015). So, although in theory VPs seem to be a promising tool, the currently available evidence remains hesitant about the effectiveness of VPs for medical communication education. For this reason, it is essential to conduct further research to extend the scientific knowledge about the usability of VPs and their potential to be integrated into medical curricula.
To better understand the learning process that takes place from communicating with a VP, engagement is a crucial variable to consider. Whitton (2011) emphasized the importance of understanding engagement when attempting to understand learning. Previous studies on engagement with computer games have already contributed to a greater understanding of the nature of learning (Dickey, 2005; Garris, Ahlers, & Driskell, 2002). However, current theories about the relation between learning and engagement typically fail to consider virtual aspects that may influence levels of engagement. This missing aspect is a serious deficiency in the current theoretical literature since one of the benefits of virtual learning is the enhanced engagement and motivation in learners (Oblinger, 2004). Therefore, this study will include engagement and place it into relation with learning communication skills. This will provide insight into the connection between engagement and virtual learning.
This study aims to offer numerous important contributions. Firstly, potential new insights will extend the current understanding of a VPs effectiveness as a teaching tool for communication skills and can clarify the ambiguities of the previous empirical findings. Secondly, it will contribute to a greater understanding of how learning communication skills in a digital context (i.e. medical decision making with a VP) are related to levels of engagement experienced during learning. This will be a valuable topic of research because current theoretical knowledge lacks evidence about the relation between engagement and learning with VPs. Thirdly, this study can be used to provide valuable advice for future medical curricula in deciding whether to use VPs for educational purposes. Indirectly, this can help improve the health care system by enhancing the communication skills of future medical professionals. For these reasons, it is essential to conduct further research to the value of VPs in teaching medical
communication skills and the role of engagement during this learning process. Therefore, the following research question was formulated:
RQ: Can a virtual patient help medical students improve their SDM communication skills? And can this improvement of skills be explained by a correlation with levels of engagement?
Theoretical Background Learning
Shared Decision Making
The present study focuses on the training of medical communication skills. A way for health care professionals to establish effective communication is by including patients in the process of making medical choices through Shared Decision Making (SDM). SDM occurs when a health-related choice is made by a healthcare professional and a patient collaboratively. The SDM process can be categorized into four separate steps (Stiggelbout, Pieterse, de Haes, 2015). Firstly, the professional informs the patient that a decision needs to be made and that his or her opinion matters during this process (step one). Then, the professional explains the feasible options along with the pros and cons of each one (step two). Next, the professional and the patient discuss the patient's preferences and the professional supports the deliberations of the patient (step three). Finally, the professional discusses the patient's preferred choice and the patient and professional verify their agreement (step four).
Légaré et al. (2010) found that current SDM training modules vary enormously in how and what they deliver. This variability in SDM training makes it challenging to investigate the most effective types of programs and, consequently, the available evidence about SDM-training effectiveness is sparse. The traditional group-based teaching is only infrequently offered and has considerable limitations in terms of the ability to adapt to personal skill levels, learning needs, and in providing appropriate feedback (Légaré et al., 2010). Because of these drawbacks, VPs must be further explored as a potential solution to more effective SDM teaching.
However, since empirical studies on the traditional SDM training methods are already sparse, it comes as no surprise that prior research into the potential of VPs for SDM training is non-existent. As a result, theoretical knowledge and empirical evidence about VPs in the specific context of SDM trainings remain obscure. Since the VP that is used in this study will provide customized system-generated feedback, empirical evidence about the effect of feedback on learning will be studied next.
Learning from Feedback
Feedback has repeatedly been acknowledged for its effectiveness in promoting learning (Mory, 2003; Hattie & Timperley, 2007). To better understand the effect of feedback, various learning theories from psychology help to provide insight (Thurlings, Vermeulen, Bastiaens, & Stijnen, 2013). For example, from a behaviourism stance, simply providing feedback can enhance learning significantly (see Figure 1). According to this theory, the valence of the feedback and the timing are crucial elements towards the learning effect that can be obtained (Atkinson, Atkinson, & Hilgard, 1983; Skinner, 1968). Feedback can be positive or negative depending on whether aspects of the current performance are encouraged or discouraged. Numerous studies have reported how positive feedback can lead to an increase of the specific behavior while negative feedback leads to a decrease of the behavior (Duchaine et al., 2011). Furthermore, behavioural learning theories emphasize how immediacy and correctness of the feedback are essential elements to achieve effectiveness. Similar predictions about feedback on learning are made by cognitive learning theories but here emphasis is put on the receiver's information processing of the feedback content (see Figure 2) (Newell & Simon, 1972; Shuell, 1986).
Figure 1. Learning from feedback according to the behavioural learning theory
Figure 2. Learning from feedback according to the cognitive learning theory
Although there are differences between the specific learning theories from psychology, Thurlings, Vermeulen, Bastiaens, & Stijnen (2013) found that they all recognize how effective feedback is characterized as task- or goal-directed and specific. From a behaviouristic and cognitive stance, feedback should elaborate on the mistakes that were made because simply providing an indication of whether an error was made is not sufficient. Also, feedback should expand on how to improve future performance. Finally, feedback should be provided
immediately, frequently, and when it is still relevant for the learners. Based on these learning theories, effective feedback is characterized by the following aspects:
Tailored System-Generated Feedback
A major advantage of this present study's VP program is its capacity to instantaneously deliver tailored system-generated feedback. Tailored communication is aimed at a specific person based on their unique characteristics and can be derived from individual assessments (Rimer & Kreuter, 2006). For this reason, tailoring feedback can provide students the opportunity to make improvements from their current level to the desired reference. Receiving appropriate feedback on one's skills and understanding is a crucial aspect of the learning process and benefits learners considerably more than simply receiving praises or punishments (Hattie & Timperley, 2007). Moreover, learners have repeatedly claimed that they are more likely to follow the content of feedback that was specifically created for them rather than general feedback content that is not tailored (Bloxham & Campbell, 2010).
Based on the abovementioned literature of learning theories and tailored feedback, it is expected that SDM communication skills will improve throughout training sessions with a VP. This improvement will stem from the feedback that is provided, which meets the five characteristics that were proposed by the cognitive and behavioural learning theories. Therefore, the following hypothesis derived:
H1: There will be an increase in SDM communication scores throughout the virtual trainings,
such that scores of the second session will be higher than the scores of the first session.
Engagement
Engagement is a psychological construct that relates to a person's active involvement in a task or activity (Reeve et al., 2004). A major benefit of a virtual environment is the ability to engage and motivate learners (Whitton, 2011). Therefore, engagement is an interesting variable as it provides insight into the involvement with the VP and, consequently, potential learning effects (Whitton, 2011). It has already been recognized that engagement is a key factor in understanding general user behaviour and overall efficacy of task-oriented behaviour within computer-based environments (Boyle, Connolly, & Hainey, 2011). Therefore, existing
1. Immediate
2. Goal-/task-directed 3. Elaborate
literature on engagement during virtual gaming supports the value of including the variable of engagement in the present study (O’Brien, & Toms, 2008).
However, current theories of learning and engagement typically fail to consider the gaming engagement theory (Koster, 2005). This theory states that well-designed virtual games can promote engagement and that they have the potential to be very effective learning environments. Additionally, the flow theory (Csikszentmihalyi, 1992) can be used to better understand the engaging nature of virtual environments. The flow theory claims that the inclusion of a challenge, clear goals, and immediate feedback determine how engaging a (learning) experience will be.
Based on the gaming engagement theory and the flow theory, it is expected that VPs have the potential to encourage high levels of engagement in its users. Because of the virtual aspect of VPs and their capacity to provide tailored feedback, engagement is expected to increase over the training sessions with the VP. For these reasons, the following hypothesis was established:
H2: There will be an increase in levels of engagement from training session 1 to training session
2, such that engagement in the second sessions will be higher than in the first sessions.
Engagement and SDM Learning
Engagement is an essential aspect in understanding learning. Therefore, research on engagement with virtual learning can also increase the available theoretical knowledge about the nature of learning (Whitton, 2011). In the specific context of virtual learning, it has been reported that sophisticated digital environments can stimulate engagement and, more importantly, increase the potential to learn (Dede, 2009). This is particularly interesting because it can suggest that engagement plays a crucial role on the effectiveness of virtual learning environments. Whitton (2011) created a theory about engagement and learning in which it is a central idea that learners must be able to notice improvements in their skills and be able to excel at the learning activity. It further highlights the need to include appropriate challenges and the importance of feedback mechanisms in order to amplify learning.
Previous studies have been able to link engagement in virtual environments to engagement in learning (Koster, 2005). Here, it has been substantiated that digital contexts can enhance participants' engagement and, consequently, their learning. Indeed, earlier research has found a link between high levels of student engagement with educational outcomes in literacy and math (Dede, 2009). However, the promotion of social and conversational skills in
a digital context with avatars may require further detailing of whether engagement can enhance the learning of conversational skills. For the specific purpose of (medical) communication skill-training, virtual learning needs more empirical evidence to support its effectiveness.
Altogether, it is hypothesized that in levels of engagements and the learning of SDM communication skills are correlated to one another. More specifically, this correlation is expected to be linear and positive; indicating that higher levels of engagement correspond to more SDM learning. Therefore, the following hypothesis was formulated:
H3: There will be a significant linear-positive correlation between SDM communication skill
learning and engagement, such that higher scores on learning are linked to higher scores on engagement while lower scores on learning are linked to lower scores on engagement.
It is important to note, however, that correlational studies are not able to draw a causal conclusion and therefore it will not be known whether engagement leads to more learning or learning leads to more engagement. Altogether, this study includes the learning of SDM communication skills as an outcome measure and engagement as a correlating variable. A visualization of this study's hypotheses is presented in Figure 3.
Figure 3. A conceptual model of the hypothesized relationships.
Methods Participants
A total of 26 participants took part in this study that was conducted in September and October of 2018. Since one participant failed to complete the study due to technical difficulties, the sample was left with 25 participants (age, M = 20.08, SD = 1.62). Nineteen of these participants were female and the other six were male. All the participants were second-year medical students. This was a selection criteria to be able to participate in the study and was justified by the assumption that this group of participants would have considerable medical
knowledge yet no prior experience and skills in SDM. The recruitment of participants took place at seminars from the University's Medical Centre (UMC), which is part of the University of Amsterdam. As a compensation for their participation, participants received a financial incentive of €40. The specific sample size and the relatively high financial reward were decisions beyond the control of this current study because it was part of a collaboration with a Research Priority Area (RPA) financed research project.
Materials
Virtual Program
The virtual consult of this study regarded the decision about an appropriate breast cancer treatment for the VP. The medical students and VPs were requested to collaboratively select a breast cancer treatment. The options for treatments in this study consisted of chemotherapy or hormone therapy. Here, the most suitable treatment was meant to be chosen which implied that the therapy was congruent with the values and preferences of the VP. The virtual program contained over a hundred VPs with varying backgrounds, values and preferences. An example of the VP that was used in this study is displayed in Image 1.
Image 1. The set-up of the virtual patient in this study.
System-Generated Feedback
After each of the two sessions, system-generated feedback was provided based on how well the participant applied SDM communication skills during the virtual consult. The feedback was based on the SDM protocol and appeared in text format on the computer screen immediately after each session was completed. This meant that per session, the system provided new and relevant feedback based on the participant's performance. Therefore, the feedback varied both between participants and between sessions. Examples of feedback points
are "You did not mention that the purpose of the consult was to make a shared decision" and "You did not verify the definite decision for a therapy with the patient".
It is important to note here that the feedback varied but that differences in feedback were not controlled for. Potential consequences of this tailored feedback on the learning effects of SDM will be considered in the discussion section of this thesis.
Measures
SDM Learning. To assess the learning of SDM skills, an assessment protocol and procedure was developed specifically for the purposes of this study. Firstly, the audio recordings of the virtual consults were transcribed into written text documents. Then, the transcripts were analysed using the self-created systematic assessment protocol (Appendix 1). For each session with the virtual patient, the participant received a value between one and four, depending on how well they used SDM communication during the virtual consult. These values were based on the four steps of SDM as suggested by Stiggelbout, Pieterse & de Haes (2015).
To ensure reliability of the assessments, two coders independently scrutinized and assessed the first transcript with two sessions and compared their results. A third person removed the session numbers and switched them to unknown codes before the assessments started. This was to avoid the expectancy bias of improved SDM skills in later sessions. Consequently, both coders were blind to the session numbers while they were assessing the transcripts. After the analysis of the first transcript, some minor adjustments were made in the assessment criteria to assure higher inter-coder reliability. Next, six more sessions were assessed independently and the results of both coders were compared. After all the transcripts of the virtual consults were assessed, the true session numbers were revealed for the purposes of the data analysis.
Engagement. The Video Engagement Scale (VES) was used in this study to better understand the extent to which individuals were engaged in the virtual world during the consults. This scale has been found to reliably and validly measure viewers’ engagement in health communication research that makes use of video vignettes (α .94) (Visser et al., 2016). The VES was used after each training sessions, making it a total of two measurement moments within each participant. The scale included nine items (Appendix 2). An example of an item from the VES was 'During the virtual consult I felt as though I was truly present in the situation', which could be answered on a Likert-scale ranging from 1 (strongly disagree) to 7 (strongly agree).
Procedure
After the recruitment of participants who met the inclusion criterion of being a second-year medical student, they were invited to the University's Medical Centre (UMC) in Amsterdam. First, participants received an information sheet about the research project that included a brief explanation of the upcoming procedure (Appendix 3) and then signed the informed-consent form (Appendix 4). This was followed by an online questionnaire in Qualtrics, which contained items about demographic characteristics.
Next, participants viewed a slideshow with instructions. Here, a thorough explanation was provided on how to operate the system of the VP. Additionally, participants had the opportunity to go through a brief trial session with the VP. To avoid prior learning effects, this trial session was unrelated to the topic of oncology, and instead regarded a VP who had broken a bone. This trial session was useful for participants to better understand how to interact with the VP and to get a clear impression of what the upcoming sessions would be like. The introduction, the survey and the trial session took approximately forty minutes in total. After participants understood what the consults were going to be like, the real sessions with the VP could begin. Before the start of each session, QuickTime Player was switched on to record the audio information during the virtual consults. Each session had a maximum duration of twelve minutes. After nine minutes, participants received a reminder that the consults only had three remaining minutes.
After each session, participants received the customized system-generated feedback based on their performance of applying the four SDM steps. Next, they answered a brief online questionnaire which included the scales for engagement. Overall, each participant completed two interactive virtual consults with the VP. The total duration of these sessions and the corresponding reflection questionnaires took approximately 50 minutes.
Finally, participants received a short debriefing in which they were thanked for their contribution to the research with the VP. Altogether, the entire procedure took approximately ninety minutes and is visualized in Figure 4.
Figure 4. The procedure of the study, including the two measurement moments.
Design
The variable of SDM learning included two values: one for each session. The variable of engagement also consisted of two values: one reported after the first session and one reported
after the second session. Altogether, this was a correlational study with learning SDM communication skills as an outcome measure and engagement as the independent variable.
Data Analysis Plan
Assessing Learning
To test the first hypothesis, which stated that there would be an increase in SDM performance scores from session 1 to session 2, the mean scores of the two sessions were compared to each other using a Paired Samples t Test. A maximum score of 4 could be obtained on SDM performance and a minimum score of 0. The value of this variable consisted of a participant's score of the second session subtracted by the score of the first session. Due to the small sample size of the study, a difference score of SDM learning from session 1 to session 2 was created. This allowed for a greater insight into the variance. The SDM learning was calculated as follows: (SDM score session 2 – SDM score session 1) = Learning SDM.If this value was positive, learning had occurred, while if this value was neutral, no changes in SDM skills were established and if this value was negative, the SDM skills worsened.
Engagement and Correlation
To test the second hypothesis, which stated that there would be an increase in levels of engagement from session 1 to session 2, a Paired Samples t Test was used to compare the differences between the average engagement scores of session 1 and session 2. To test the third hypothesis, which stated that there would be a significant positive correlation between learning SDM communication skills and engagement, a Pearson's correlation was computed from the scores on the two variables.
Results Prior SDM Knowledge
To rule out that participants' prior SDM knowledge would cause variance in learning, this variable was measured at the start of the study. All participants reported to have limited SDM knowledge prior to the experiment (M = 1.91, SD = 0.11). The answering options ranged on a scale from 1 (no knowledge at all) to 7 (a lot of knowledge) and the maximum reported score was 4.
SDM Performance Scores
The first hypothesis stated that there would be an increase in SDM performance scores from session 1 to session 2. A Paired Samples t Test compared the average SDM scores of each session and showed a significant difference in SDM scores between session 1 and session 2, (t(24) = 15.51, p < .001, d = .82 ). Thus, H1 was supported because participants did indeed
perform better on session 2 (M = 2.84, SD = 0.70) than on session 1 (M = 2.10, SD = 0.68). Overall, the learning performance difference score was 0.74 on a scale ranging from 0 to 4 (S2-S1 = 2.84 – 2.10 = 0.74). These results are presented in Table 1.
Table 1. Descriptives of SDM Learning Scores
Engagement Scores
To test whether levels of engagement increased from session 1 to session 2 (H2), a Paired Samples t Test was performed to compare the average engagement scores of each session. The test revealed no significant difference in engagement scores between session 1 and session 2, (t(24) = .18, p = .857, d = .04). Thus, H2 could not be supported based on these results because participants did not report different levels of engagement between session 1 (M = 3.81, SD = 0.72) and session 2 (M = 3.83, SD = 0.85). These results are presented in Table 2.
Table 2. Descriptives of Engagement Scores
Engagement Correlations
To test whether engagement would be higher for participants who scored higher on SDM learning compared to those who scored lower (H3), a Pearson's correlation analysis was performed. There was evidence for a very weak, non-significant correlation between SDM learning and engagement for both session 1: r(25) = .11, p = .30 and session 2: r(25) = .09, p = .35. Consequently, there was no clear correlation between engagement and SDM performance. Increases in engagement were therefore not correlated to increases in SDM learning. For this reason, H3 could not be supported based on the evidence from this study.
Conclusion and Discussion
This study examined the effectiveness of VPs for training medical communication skills. The first aim of the study was to investigate whether the VP could help medical students improve their SDM communication skills. This was expected because the VPs provided the students a learning opportunity with corresponding system-generated feedback. Indeed, improvements in SDM skills were substantiated during the sessions with the VP. The observed learning effect is in line with the behavioural and cognitive learning theories from psychology which emphasize the effectiveness of feedback on learning. This is also congruent with previous literature that highlights the ability of feedback to enhance educational outcomes (Mory, 2003; Hattie & Timperley, 2007; Thurlings, Vermeulen, Bastiaens, & Stijnen, 2013). Additionally, this study further examined the potential correlation with SDM learning and levels of engagement. The results revealed no correlation between learning and engagement. This non-existent correlation contrasts earlier findings that do report a relationship between engagement and learning in virtual environments (Whitton, 2011; Boyle, Connolly, & Hainey, 2011; O’Brien, & Toms, 2008).
Theoretical Implications SDM Learning
The first aspect of the research question focused on whether a virtual patient could help medical students improve their communication skills. To answer this, the first
hypothesis sought to determine whether the virtual consults and the VP's feedback could help medical students improve their SDM communication skills. The results revealed that, indeed, SDM communication skills improved significantly from the first session to the second
session. As a result, the first hypothesis was supported. These findings are congruent with the predictions based on cognitive and behavioural learning theories from psychology. The cognitive and behavioural learning theories state that the effectiveness of feedback on learning is determined by the following four characteristics: immediate, goal-/task-oriented, elaborate, and frequent. The system-generated feedback in this present study meets these criteria since it was provided directly after each consult (immediate and frequent) and
explained which parts of the SDM protocol succeeded or failed (goal-oriented and elaborate).
Engagement
The second aspect of the research question focused on the underlying relationship of engagement with learning SDM skills. The second hypothesis stated that there would be an increase in levels of engagement over the sessions so that engagement in the second sessions would be higher than in the first session. For this reason, engagement during each virtual
consult was investigated. However, the present study found no evidence for an increase in levels of engagement throughout the sessions. As a result, the second hypothesis was
rejected. It must be acknowledged, however, that these findings do not necessarily contradict the theoretical knowledge about engagement in virtual environments such as the gaming engagement theory and the flow theory. The gaming engagement theory states that well-designed virtual games can promote engagement and that they can be very effective learning environments (Koster, 2005). The flow theory claims that the inclusion of a challenge, clear goals, and immediate feedback determine how engaging an experience will be
Csikszentmihalyi, 1992). In this sense, the present research was conducted whilst following the suggestions of the theories, but to experimentally test the effect of virtual environments on engagement, a non-virtual control study would be required.
SDM Learning and Engagement
The third hypotheses of this present study stated that there would be a correlation between SDM learning and engagement. In contrast to the hypothesis however, no correlation was found between the learning of SDM skills with a VP and engagement. As a result, learning effects could not be explained based on the scores of engagement.
If learning is truly not correlated to engagement, this would contradict the current academic theory and literature. Whitton (2011) theorized that learners must be able to notice improvements in their skills and be able to excel at the learning activity. Here, the need to include appropriate challenges and the importance of feedback mechanisms for learning are crucial. Although the VP used in this study does meet these suggestions, it is important to consider that the scores on engagement varied only minimally between the sessions, as indicated by the rejection of the second hypothesis. Consequently, it is difficult to find an effect with such limited variance. Therefore, the previous studies that have linked engagement in virtual environments to learning performance should not be discredited based on the findings of this present study (Koster, 2005; Dede, 2009). Instead, more empirical investigations should be conducted in order to gain a better understanding of the relationship between virtual learning and engagement.
Practical Implications
Overall, this study can be used to support the notion that the VP and its corresponding system-generated feedback was an effective tool for teaching SDM communication skills. However, to explain the SDM learning, the findings of this present study cannot link the scores of engagement to learning. The interesting notion is that SDM skills seemed to improve (H1) but this finding could not be linked to levels of engagement (H3). Therefore, this study leaves
numerous questions about the learning effects and it is strongly encouraged that further research investigates other variables that may be related to learning effects in virtual environments such as immersion, perceived realism, and experienced presence (Bowman & McMahan, 2007).
The evidence for improved SDM communication skills can be used to support the value of VR as a teaching tool in medical communication curricula. Notably, the optimistic feedback that the students provided about their virtual experience also seems promising. Numerous students reported to enjoy this way of teaching more than class-based teaching methods and found it an interesting and favourable tool for future means. Therefore, the interest and enjoyment of this learning activity can be considered a positive characteristic of VR as a teaching tool. However, more research is needed to clarify the specific role and value of VPs in relation to other SDM teaching methods. Further research should also explore the different formats and cases that can vary within VPs to optimize their teaching. Additionally, other evaluation systems can be further investigated. For example, comparing the direct system-generated feedback to feedback from a tutor or summative assessments.
Limitations and Future Research
One main limitation of this study is the sample size that was used. The size of the sample was related to the time-consuming procedure of the study, the specific selection criteria of participants (second year medical students), and mainly, the external deadlines and budget. Since this current study was part of a greater RPA research project, the deadlines, budget, and sample size were set externally. However, it is important to note that with only 25 participants, it is difficult to find great effect sizes and strong correlations. To compensate for this and to have more variance to explain, difference scores of SDM performance between the sessions were computed. Nonetheless, the power of the sample was bound to be low and it is therefore suggested to continue follow-up research with a greater pool of participants.
Another notable limitation of this study relates to the nature of the feedback that participants received. As mentioned earlier, the feedback was customized based on how well SDM communication was implemented during the virtual consult. However, due to tailoring, the feedback also differed per participant and per session. Perhaps some of the feedback points were found more useful and aided learning more than others. However, these differences were not controlled for nor measured, and therefore it remains unclear how feedback may vary in effectiveness of stimulating learning. Additionally, it was noted that the attention that participants paid to read through the feedback varied. Some participants carefully read through each of the feedback points while others quickly scanned through it. Interestingly, prior
research has highlighted that the quality of engagement with the feedback is a critical determinant on effectiveness of learning (Winstone et al., 2016; Black & Wiliam, 1998; Hattie & Timperley, 2007). As a result, it has become clear that the learners' engagement with the feedback plays a crucial role in the learning process. However, studies that focus on this role of engagement are scarce in the current academic literature and further research is therefore encouraged (Bounds et al., 2013). For this reason, it is suggested that another study is conducted that focuses specifically on the role of the learner's engagement when receiving feedback and how this may affect learning outcomes.
Overall, effective medical communication that can be established by the SDM protocol is a very valuable tool for future health care professionals to master. This can increase the alignment between patient preferences and the care that is provided; as a result, patient satisfaction will increase (Stiggelbout, Pieterse, de Haes, 2015). Therefore, creating a training program that efficiently teaches health care professionals SDM skills can become a long-term investment in a better communication with patients and achieving higher patient satisfaction. For this reason, researchers should be encouraged to further develop these computer-based teaching programs and to investigate VPs' capacity as educational tools for improving medical communication skills.
References
Aspegren, K., & LØnberg-Madsen, P. (2005). Which basic communication skills in medicine are learnt spontaneously and which need to be taught and trained?. Medical Teacher, 27(6), 539-543.
Atkinson, R. L, Atkinson, R. C, & Hilgard, R. E (1983). Introduction to psychology. San Diego: Harcourt, Brace & Jovanovich.
Bloxham, S., & Campbell, L. (2010). Generating dialogue in assessment feedback: Exploring the use of interactive cover sheets. Assessment & Evaluation in Higher Education, 35(3), 291-300.
Boyle, E., Connolly, T. M., & Hainey, T. (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2(2), 69-74.
Bowman, D. A., & McMahan, R. P. (2007). Virtual reality: how much immersion is enough?. Computer, 40(7), 36-43.
Consorti F, Mancuso R, Nocioni M, Piccolo A. Efficacy of virtual patients in medical education: a meta-analysis of randomized studies. Comput Educ. 2012;59(3):1001–8.
Cook, D. A., & Triola, M. M. (2009). Virtual patients: a critical literature review and proposed next steps. Medical education, 43(4), 303-311.
Csikszentmihalyi, M., & Csikszentmihalyi, I. S. (Eds.). (1992). Optimal experience: Psychological studies of flow in consciousness. Cambridge university press.
Couët, N., Desroches, S., Robitaille, H., Vaillancourt, H., Leblanc, A., Turcotte, S., & Légaré, F. (2015). Assessments of the extent to which health‐care providers involve patients in decision making: a systematic review of studies using the OPTION instrument. Health Expectations, 18(4), 542-561.
Dede, C. (2009). Immersive interfaces for engagement and learning. science, 323(5910), 66-69.
Dickey, M. (2005). Engaging by design: How engagement strategies in popular computer and video games can inform instructional design. Educational Technology Research and Development, 53(2), 67-83.
Duchaine, E. L., Jolivette, K., & Fredrick, L. D. (2011). The effect of teacher coaching with performance feedback on behavior-specific praise in inclusion classrooms. Education and Treatment of Children, 34(2), 209-227.
Garau, M., Slater, M., Bee, S., & Sasse, M. A. (2001). The impact of eye gaze on
communication using humanoid avatars. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 309-316). ACM.
Garris, R., Ahlers, R., & Driskell, J. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming: An Interdisciplinary Journal, 33, 441-467. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research,
77(1), 81–112.
Kononowicz, A.A., Zary, N., Edelbring, S., Corral, J., & Hege, I. (2015). Virtual patients – what are we talking about? A framework to classify the meanings of the term in healthcare education. BMC Medical Education, 15, 11.
Kunneman, M., Engelhardt, E. G., Ten Hove, F. L., Marijnen, C. A., Portielje, J. E., Smets, E. M., & Pieterse, A. H. (2016). Deciding about (neo-) adjuvant rectal and breast cancer treatment: missed opportunities for shared decision making. Acta Oncologica, 55(2), 134-139.
Légaré, F., Ratté, S., Stacey, D., Kryworuchko, J., Gravel, K., Graham, I. D., & Turcotte, S. d (2010). Interventions for improving the adoption of shared decision making by healthcare professionals. Cochrane database of systematic reviews, (5).
Mory, E. H. (2003). Feedback research revisited. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 745–783). New York: MacMillan Library Reference.
Newell, A., & Simon, H. A. (1972). Human problem solving(Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-Hall.
Oblinger, D. (2004). The next generation of educational engagement. Journal of Interactive Media in Education, 8.
O'Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American society for Information Science and Technology, 59(6), 938-955.
O'Connor, A. M., Bennett, C. L., Stacey, D., Barry, M., Col, N. F., Eden, K. B., & Llewellyn‐ Thomas, H. (2009). Decision aids for people facing health treatment or screening decisions. Cochrane database of systematic reviews, (3).
Oshima Lee, E., & Emanuel, E. J. (2013). Shared decision making to improve care and reduce costs. New England Journal of Medicine, 368(1), 6-8.
Pertaub, D. P., Slater, M., & Barker, C. (2001). An experiment on fear of public speaking in virtual reality. Studies in health technology and informatics, 372-378.
Pieterse, A. H., Baas-Thijssen, M. C. M., Marijnen, C. A. M., & Stiggelbout, A. M. (2008). Clinician and cancer patient views on patient participation in treatment decision-making: a quantitative and qualitative exploration. British journal of cancer, 99(6), 875.
Poulton, T., & Balasubramaniam, C. (2011). Virtual patients: a year of change. Medical teacher, 33(11), 933-937.
Reeve, J., Jang, H., Carrell, D., Jeon, S., & Barch, J. (2004). Enhancing students' engagement by increasing teachers' autonomy support. Motivation and emotion, 28(2), 147-169. Rimer, B. K., & Kreuter, M. W. (2006). Advancing tailored health communication: A
persuasion and message effects perspective. Journal of communication, 56, S184-S201.
Skinner, B. F. (1968). The technology of teaching. Prentice-Hall, NJ: Englewoods-Cliffs. Shuell, T. J. (1986). Cognitive conceptions of learning. Review of educational research, 56(4),
Stacey, D., Légaré, F., Lewis, K., Barry, M. J., Bennett, C. L., Eden, K. B., ... & Trevena, L. (2017). Decision aids for people facing health treatment or screening decisions. Cochrane database of systematic reviews, (4).
Stiggelbout, A. M., Pieterse, A. H., & De Haes, J. C. J. M. (2015). Shared decision making: concepts, evidence, and practice. Patient education and counseling, 98(10), 1172-1179.
Thurlings, M., Vermeulen, M., Bastiaens, T., & Stijnen, S. (2013). Understanding feedback: A learning theory perspective. Educational Research Review, 9, 1-15.
Visser, L. N., Hillen, M. A., Verdam, M. G., Bol, N., de Haes, H. C., & Smets, E. M. (2016). Assessing engagement while viewing video vignettes; validation of the Video Engagement Scale (VES). Patient education and counseling, 99(2), 227-235.
Walsh, T., Barr, P. J., Thompson, R., Ozanne, E., O’neill, C., & Elwyn, G. (2014). Undetermined impact of patient decision support interventions on healthcare costs and savings: systematic review. Bmj, 348, g188.
Whitton, N. (2011). Game engagement theory and adult learning. Simulation & Gaming, 42(5), 596-609.
Appendix I: Assessment Protocol for SDM Analysis Step 1:
A (it is mentioned that a shared decision is to be made = 0.5 points) B (the importance of the needs of the patient are emphasized = 0.5 points) Step 2:
A (at least one advantage of hormone therapy is mentioned = 0.25 points)
B (at least one disadvantage of hormone therapy is mentioned = 0.25 points) C (at least one advantage of chemotherapy is mentioned = 0.25 points) D (at least one disadvantage of chemotherapy is mentioned = 0.25 points) Step 3:
full points (1) = multiple deliberations of the patient are discussed, half points (0.5) = just one deliberation of the patient is discussed no points (0) = no deliberations of the patient are discussed Step 4:
Full points (1) = the final preference of the patient is literally stated and the doctor agrees half points (0.5) = the final preference of the patient is not literally stated but the doctor decides no points (0) = the final preference of the patient for a desired therapy is not discussed and no
Appendix I Continued
Beoordelingsformulier Shared-Decision-Making (SDM) Proefpersoon nummer:
Sessie code:
Stap: Wat? Punten: Behaald e punten per stap: 1 (hoeft niet aan het begin van het consult) De student benoemt dat er een gezamenlijke beslissing gemaakt dient te worden en dat de mening van de patiënt hierbij belangrijk is
Het wordt benoemd dat er een gezamenlijke
beslissing gemaakt dient te worden
Ja = 0.5 pt
Het belang van de mening/behoeften van
de patiënt worden benadrukt Ja= 0.5 pt
Er wordt niet gesproken over het maken van een gezamenlijke beslissing waarbij de mening van de patiënt belangrijk is
Nee = 0 pt
2 De student legt uit wat de
voor- en nadelen zijn van beide therapieën Minimaal 1 voordeel van hormoonther apie wordt besproken Ja = 0.25 pt Minimaal 1 nadeel van hormoonther apie wordt besproken Ja = 0.25 pt Minimaal 1 voordeel van chemotherap ie wordt besproken Ja = 0.25 pt Minimaal 1 nadeel van chemotherap ie wordt besproken Ja = 0.25 pt Voor beide therapieën worden geen voor- en nadelen besproken Ja = 0 pt 3 De student en de patiënt bespreken de overwegingen van de patiënt en de student ondersteunt deze overwegingen van de patiënt Meerdere overwegingen van de patiënt worden
besproken Ja = 1 pt
Slechts één overweging van de patiënt wordt
besproken Ja = 0.50 pt Er worden geen overwegingen van de patiënt besproken Ja = 0 pt 4 De student en patiënt bespreken de uiteindelijke voorkeur/besli ssing van patiënt voor de gewenste therapie De uiteindelijke voorkeur van de patiënt
wordt letterlijk benoemd of instemming
met voorstel arts vindt plaats Ja = 1 pt
De uiteindelijke voorkeur wordt niet
letterlijk benoemd door
arts. Maar arts neemt een besluit waar overwegingen van
patiënt wel in meegenomen zijn
Ja = 0.50 pt
Er wordt niet over de uiteindelijke voorkeur
van patiënt voor een gewenste therapie gesproken dus er vindt
geen besluit plaats Ja = 0 pt
Totaal aantal behaalde
Appendix III: Information Form
Beste deelnemer,
Hierbij wil ik u uitnodigen deel te nemen aan een onderzoek dat wordt uitgevoerd onder
verantwoordelijkheid van onderzoeksinstituut ASCoR, onderdeel van de Universiteit van Amsterdam. ASCoR doet wetenschappelijk onderzoek naar media en communicatie in de samenleving. Dit onderzoek richt zich op de bruikbaarheid van een virtuele patiënt voor onderwijs over shared decision making (gezamenlijke besluitvorming). Deelname aan dit onderzoek bestaat uit drie interacties, d.w.z. gesprekken tussen u als arts (in opleiding) en een virtuele patiënt, steeds gevolgd door het invullen van een vragenlijst over uw ervaring met de virtuele patiënt. Voorafgaand aan de eerste interactie vult u ook een vragenlijst in. Interacties met de virtuele patiënt zullen elk ongeveer 20 minuten duren, inclusief het invullen van de vragenlijst.
Dit onderzoek richt zich bij uitstek op het aanleren van shared decision making vaardigheden en niet op medische kennis. Voorafgaand aan de interacties krijgt u als de deelnemer daarom medische informatie over de aandoening van de virtuele patiënt en (mogelijke) medische behandelingen op schrift. Na elke interactie ontvangt u door het systeem gegenereerde feedback. Tijdens elke interactie zullen audio-opnames worden gemaakt. Daarnaast worden de interacties zelf door het systeem opgeslagen. Dit betreft arts-virtuele patiënt gespreksdata en feedback data.
Alle data zal anoniem worden gemaakt, inclusied de data in de transcripties van de audio-opnames en in de analyses. Resultaten zullen zodanig worden gepresenteerd en gepubliceerd dat deze niet te herleiden zijn tot individuele deelnemers. Ruwe audiodata zal maximaal zes maanden worden bewaard. De totale duur van dit onderzoek is ongeveer 2 uur. Deelname aan dit onderzoek wordt beloond met €40.- (inclusief reiskosten).
Omdat dit onderzoek wordt uitgevoerd onder de verantwoordelijkheid van ASCoR, Universiteit van Amsterdam, heeft u de garantie dat:
1. Uw anonimiteit is gewaarborgd en dat uw antwoorden of gegevens onder geen enkele voorwaarde aan derden zullen worden verstrekt, tenzij u hiervoor van tevoren uitdrukkelijke toestemming hebt verleend.
2. U zonder opgaaf van redenen kunt weigeren mee te doen aan het onderzoek of uw deelname voortijdig kunt afbreken. Ook kunt u achteraf (binnen 7 dagen na deelname) uw toestemming intrekken voor het gebruik van uw antwoorden of andere gegevens voor het onderzoek.
3. Deelname aan het onderzoek geen noemenswaardige risico’s of ongemakken voor u met zich meebrengt, geen moedwillige misleiding plaatsvindt, en u niet met expliciet aanstootgevend materiaal zult worden geconfronteerd.
Voor meer informatie over dit onderzoek en de uitnodiging tot deelname kunt u te allen tijde contact opnemen met de verantwoordelijke onderzoeker prof. dr. Julia van Weert:
E: J.C.M.vanWeert@uva.nl) T: 020-5252091.
Mochten er naar aanleiding van uw deelname aan dit onderzoek bij u klachten of opmerkingen zijn over het verloop van het onderzoek en de daarbij gevolgde procedure, dan kunt u contact opnemen met het lid van de Commissie Ethiek namens ASCoR, per adres: ASCoR secretariaat, Commissie Ethiek, Universiteit van Amsterdam, Postbus 15793, 1001 NG, te Amsterdam; 020- 525 3680; ascor-secr-fmg@uva.nl. Een vertrouwelijke behandeling van uw klacht of opmerking is daarbij
gewaarborgd.
Wij hopen u hiermee voldoende te hebben geïnformeerd en danken u bij voorbaat hartelijk voor uw deelname aan dit onderzoek dat voor ons van grote waarde is.
Appendix IIII: Informed Consent
Ik verklaar hierbij op voor mij duidelijke wijze te zijn ingelicht over de aard en methode van het onderzoek, zoals uiteengezet in de uitnodigingsmail voor dit onderzoek. Ik stem geheel vrijwillig in met deelname aan dit onderzoek. Ik behoud daarbij het recht deze instemming weer in te trekken zonder dat ik daarvoor een reden hoef op te geven. Ik besef dat ik op elk moment mag stoppen met het onderzoek. Als mijn onderzoeksresultaten worden gebruikt in wetenschappelijke publicaties, of op een andere manier openbaar worden gemaakt, dan zal dit volledig geanonimiseerd gebeuren. Mijn persoonsgegevens worden niet door derden ingezien zonder mijn uitdrukkelijke toestemming. Als ik meer informatie wil, nu of in de toekomst, dan kan ik me wenden tot Julia van Weert. Voor eventuele klachten over dit onderzoek kan ik me wenden tot het lid van de Commissie Ethiek namens ASCoR, per adres: ASCoR secretariaat, Commissie Ethiek, Universiteit van Amsterdam, Postbus 15793, 1001 NG, te Amsterdam; 020- 525 3680; ascor-secr-fmg@uva.nl.
