Evaluation of augmented reality-based remote family visits in nursing
homes: A proof of concept
Author: Eva A.M. Abels
University:Master program: Student number: Date:
University of Amsterdam, the Netherlands Brain and Cognitive Sciences (research) 12098396
January 31st, 2021
Assessor: Examiner:
Dr. Alexander Toet Dr. Zeph M.C. van Berlo
ABSTRACT
Visitation restrictions in nursing homes due to COVID-19-related measures have a major negative impact on residents and their family members. In this study, an augmented reality (AR)-based solution was developed to realize high-quality virtual contact between residents and their family members, as an alternative communication means during the COVID-19 pandemic. The aim of this work was to investigate the suitability of this new technique for remote family visits in nursing homes. This project was divided into two parts. Firstly, we performed a user needs study to develop an evaluation methodology tailored to remote family visits. We conducted focus groups with residents, family members and caretakers. This led to the establishment of user experience indicators of enjoyment, spatial presence and social presence. Secondly, we performed an evaluation study involving 10 participant pairs of residents and family members. We compared the user experience of the AR solution to that of regular video calling based on the user experience indicators as well as attitudes, behavior and duration. Although the difference between conditions was only visible for the residents, no differences were found in their user experience. Family members only reported some marginal differences in user experience, including higher spatial presence for the AR solution compared to regular video calling. However, we also found that durations with the AR solution were significantly longer than with regular video calling. This might indicate that residents experienced a stronger sense of comfort and privacy for the AR solution compared to regular video calling. Therefore, we tentatively suggest that there may be (unconscious) differences in user experience while using the AR solution for remote family visits.
KEYWORDS • Augmented Reality • Social XR • User Experience • Social Presence • Spatial Presence • Nursing Home 1 INTRODUCTION
Loneliness is suggested to be one of the most predominant feelings among residents of nursing homes and is associated with negative health outcomes and reduced quality of life [38]. Estimates of loneliness levels among nursing home residents vary widely in the literature, with rates as high as 50 to 60% in some cases [8,36]. Receiving visitors, such as family members, can decrease the residents’ feelings of loneliness [38]. However, family members often fail to pay sufficient visits to the residents, due to a lack of time and long travel distances [13,41].
Due to the COVID-19 pandemic, stringent safety measures have been implemented in nursing homes, including a strict limitation on family visits. These visitation restrictions have an enormous negative impact on residents as well as their family members [35]. For instance, reduced contact with family is likely to contribute to the residents’ sense of loneliness [9], which in turn is a major concern of family members [52].
Contemporary means to maintain communication between residents and their family members include phone calling and video calling. Video calling offers a suitable and practical solution for residents to stay in touch with their family members [44,55] and may be more useful in reducing loneliness than phone calls or written communication [48–50]. However, video calling was indicated to be not nearly as fulfilling as face-to-face interaction due to a lack of feelings of togetherness [13].
Virtual visits via extended reality (XR, which include virtual reality and augmented reality; VR and AR, respectively) communication systems could offer a better and more sophisticated solution. Such technologies are typically highly immersive, which refers to the technological capabilities of the system to deliver an illusion of reality to the senses of the user [46]. In the context of family visits, XR systems could provide high-quality virtual contact between residents and their
family members [7,28]. As such, this could lead to increased feelings of togetherness and ultimately improved mental health [7,18].
In this research project, we developed a social XR solution based on AR to facilitate life-like remote family visits in nursing homes, as an alternative communication means during the COVID-19 crisis. This AR solution could also have potential beyond the pandemic, for example for family members who live far away. The goal of this work was to investigate the suitability of this new technique in the context of remote family visits in nursing homes. However, before we could test the AR solution, an evaluation methodology needed to be developed. This needed to be tailored to AR settings as well as nursing home residents and their family members, such that their user experience would be adequately measured.
Therefore, this project was divided into two studies. Study 1 was a user needs study. We conducted focus groups to assess the needs and wishes of the involved parties (i.e., residents, family members and caretakers). The aim of this study was to gain deeper insight into the user needs regarding remote family visits, by means of regular video calling as well as the AR solution. These findings would provide the foundation for the development of an evaluation methodology. Study 2 was an evaluation study involving residents and their family members. We compared the user experience of the AR solution to the user experience of regular video calling, using an evaluation methodology based on the identified needs from Study 1. It is important to note that difference between conditions was only visible on the resident’s side. For the AR solution the visiting family member was projected in three dimensions (3D) onto a chair in the resident’s own environment, whereas during regular video calling the family member was presented as a 2D presentation with a background of the family member’s room. On the family member’s side, the resident was visible in 2D with a background of the resident’s room in both conditions. Therefore, we hypothesized that in the context of remote family visits, residents would evaluate their user experience more positively when using the AR solution compared to during regular video calling, while family members would not report differences between conditions.
2 RELATED WORK
2.1 Assessing user experience
To successfully design a social XR application, it is necessary to acquire insight into its user experience [33]. For virtual environments, user experience research has been primarily focused on the technology’s capability to promote a sense of (spatial) presence (“being there” [3,46]) and social presence (“being together with an intelligent other” [2,37,45]). This subjective experience of being together with a ‘real’ intelligent being is considered a crucial component of virtual interactions [37]. In addition, measures of presence have been used as quality parameters for (comparisons between) different communication systems [3,6,15].
Measuring and understanding user experience in social virtual environments also involves concepts as meaning, affect and value [27,51]. For example, affective states such as enjoyment are believed to be of great importance to user involvement when using XR technologies [28]. Li and colleagues have developed a questionnaire with such a comprehensive approach to evaluate social experiences in VR settings [29]. Their questionnaire incorporates interaction quality and presence as well as social meaning by including experiences such as the naturalness of the interaction, conversational engagement, comfortableness of the environment, feelings of social presence and enjoyment. Considering the social nature of family visits in nursing homes, a similar approach would be highly suitable to evaluate social interactions between residents and family members in AR settings.
2.2 Evaluating remote family visits in AR
To assess the user experience of our newly developed AR solution, two matters needed to be acknowledged. Firstly, while presence-related concepts have been investigated extensively in VR environments [37], less is known in the context of AR [32] and it is unclear to which extent (social) interaction research in VR translates to AR. Secondly, no evaluation methodology currently exists to assess (differences in) user experience during social interactions between residents and family members that take place in AR [23,28].
Regarding the first matter, technical characteristics of VR and AR must be recognized, as their affordances are fundamentally different [22]. AR environments allow for co-existence of real and virtual elements and do not replace the
user’s surroundings, as is typical in VR [28,43]. Instead, AR-enabled systems supplement the real world with 3D-registered virtual objects, which the user can interact with in real-time [1,26,32].
This leads to different interpretations of immersion and spatial presence. In the context of VR, these concepts refer to the medium’s capabilities of delivering an illusion of reality of the virtual world and the user’s perception of physically being there [3,46]. However, in AR environments this illusion of reality applies more to the plausibility and consistency of the virtual content in the user’s own surroundings, such that it seems as if the virtual objects really belong to the physical world [21,47].
As for the evaluation methodology to assess user experience in social AR settings, methodologies from VR research may be adapted for AR applications. Moreover, since the AR solution is applied in nursing homes in this study, the involved user groups consisted of residents, family members and caretakers. This required an evaluation methodology tailored to these groups and specifically to the residents, because they would be the primary users of the AR solution. Hence, their experiences were prioritized in this study.
3 STUDY 1: USER NEEDS 3.1 Method
3.1.1 Participants and procedure
We assessed the needs and wishes of the involved user groups (i.e., residents, family members and caretakers) during three separate focus groups. A focus group is a qualitative research method used for collecting and analyzing attitudes, reactions and opinions and is typically set up as a moderated group discussion with a facilitator who provides the discussion guideline [12]. The use of this method to discuss video calling technology with older adults has proven beneficial in earlier research [4]. During the focus groups, we discussed experiences and (dis)advantages of regular video calling, after which we introduced the AR solution and talked about its expected positive and negative aspects. The AR solution was explained in terms of a hologram, where the family member would not physically be present but visit as a virtual 3D projection in the resident’s own environment.
The participants for the focus groups were recruited by caretakers working at the nursing home. The focus group with the residents (N = 5, all female, Mage = 87.80, SDage = 8.53) was held physically at the nursing home where the residents
lived. In addition to the discussion topics of regular video calling and the AR solution, we conducted an interest survey with the residents (N = 4 due to a dropout halfway through the focus group). The survey consisted of 11 items (Appendix A.1, Supplementary table 1A) that were formulated based on presence and social VR literature [16,29]. For each item, the residents could indicate their interests by means of voting on a five-point scale using a self-developed intuitive system with colors and smiley faces (Figure 1). Such methods are typically used in research populations who have difficulty in expressing their feelings and/or experiences [14].
The focus group with family members (N = 9) took place online due to the COVID-19-related measures. We used an online interactive tool called Mentimeter (https://www.mentimeter.com/) to receive answers to predefined questions (Appendix A.1, Supplementary table 1B).
The focus group with caretakers was performed partly online and partly physical at the nursing home (N = 9; 5 online and 4 physical). It was structured as a group interview where the physical and online participants took turns in responding (Appendix A.1, Supplementary table 1C).
The three focus groups were transcribed and analyzed based on thematic content. By examining common and frequently mentioned themes across the focus groups, user experience indicators were established. Since the residents would be the primary users of the AR solution, the emphasis was on their perspective.
Figure 1: Example of the residents voting for an item on the interest survey that was conducted during the focus group. Residents were able to indicate their interests by means of voting on a five-point scale, where the dark green color with a happy smiley was the most positive and the
red color with an angry smiley was the most negative. In this example, the residents expressed the most positive vote.
3.2 Results
3.2.1 Regular video calling
Regular video calling was indicated to have become increasingly popular in the nursing home since the COVID-19 outbreak. Video calling sessions between residents and family members typically lasted between 10 and 30 minutes and were in most cases supported by caretakers. During the focus groups, various (dis)advantages of video calling were mentioned, as presented in Table 1A.
All user groups indicated that video calling was a suitable communication means to stay connected with family members and in general, these experiences were enjoyable and positive. One of the family members indicated: “I video call with my
father every day and this goes really well. I notice that my father enjoys having contact like this. This way he can also see his grandchildren more often". In the context of the COVID-19 pandemic, a caretaker claimed: "You could see that it did the residents good to see their family members again, even though it was digitally".
Residents and family members appreciated the addition of the visual aspect, which allows for new possibilities such as seeing facial expressions. For example, a resident stated: “With video calling, you can see the expression of the faces. I enjoy
that”. However, some residents had impaired vision and therefore a detailed representation of the family member was less
important to them. Additionally, residents and family members indicated to perceive more feelings of togetherness and connectedness than during phone calling. A resident explained: “You are closer to each other, like [the family member] is
sitting in front of you. Or we can exchange other things, like [the family member] showing me new things in the house”.
Residents frequently experienced difficulties with aiming and holding the video calling device: “I often find myself fiddling
with the [device] and then I lose the face”. This was also inconvenient for family members: "Let me put it this way, I saw a lot of nostril and forehead". It is important to note that these issues were caused by the specific implementation at the nursing
home. Besides, a few residents eventually lost interest in video calling with family, which the caretakers suggested could be due to too much effort and/or stimulation during the activity.
3.2.2 The AR solution
Several expected (dis)advantages of the AR solution were mentioned during the focus groups, as shown in Table 1B. The initial reactions after the explanation of the AR solution were mixed. Some residents were directly enthusiastic: “Wonderful.
Yes, I think it is amazing. So you really get the feeling that [the family member] is with you”. Others were more reluctant: “I would say, how did [the family member] get here? That overwhelms me. Soon [the family member] will be gone again. I find it a bit scary and I think I have to get used to it first”. In addition, most family members indicated that they wanted to see the
AR solution in real life before they could decide if it would be suitable for the residents. For example: "It looks good, but I
Most residents and family members believed that using the AR solution would yield stronger feelings of togetherness compared to regular video calling. One of the residents declared: “Yes, I think it is a very good idea. I do think it has added
value compared to the other solutions that we have had. […] That you really have the feeling that you are together in the same space”. At the end of the focus groups, nearly all participants were at least willing to try out the AR solution. Residents
stated: “I did find it very interesting I must say” and “I think I would enjoy it”.
Caretakers were interested to be involved during the implementation process of the AR solution. They suggested that they could contribute by transferring information to the residents and family members as well as assessing the residents’ state of mind: "We are very familiar with the residents and can often tell from their body language whether they like something
or not".
Table 1: (Dis)advantages of regular video calling and the AR solution A. Regular video calling
Advantages Disadvantages
Residents
• More feelings of togetherness and connectedness than during phone calling
• Being able to see facial expressions
• Being able to notice and comment on appearances • Family can show things, e.g. in the house • Better remembrance afterwards • Seeing other family members more often
• Issues with correct camera aiming
• Issues with holding due to heaviness and/or shaking • Being able to see only one face
• Losing interest
Family members
• More feelings of togetherness and connectedness than during phone calling
• Being able to see each other
• Being able to ask questions to the supporting caretaker about the resident’s health and behavior • Resident can see other family members more often
• Confusion among some residents
• Residents having issues with aiming and holding
Caretakers
• Consider it their duty to support the residents in their desire to see their family in any possible way • No additional workload
• Increased happiness among residents
• Some residents losing interest
• Frustration due to failing internet connection • Invasion of privacy
B. AR solution
Advantages Disadvantages
Residents
• More feelings of togetherness than during regular video calling
• Added value • Stable construction
• Need to familiarize with technology
Family members
• More feelings of togetherness than during regular video calling
• Stable construction
• Unsure about suitability for and understanding of the residents
• Need to familiarize with technology
Caretakers
• Feelings of competence in using this solution • Interested to be involved during implementation,
such as by assessing the residents’ state of mind
(Dis)advantages of (A) regular video calling and (B) the AR solution, as mentioned by residents, family members and caretakers during the focus groups.
3.2.3 Interest survey
In the interest survey, the residents indicated their interests regarding remote family visits (Appendix A.1, Supplementary table 1A). Residents attributed the highest score to the item of feeling connected with their family members during remote family visits. This item was also selected when the residents were asked to choose the most important item as a group.
Furthermore, residents felt that it was equally important that their family members understood them as that they understood their family members. However, they found it more important to hear and see their family members clearly than whether their family members could see and hear them. Additionally, residents indicated that being able to sense their family members' moods and emotions was more important than the other way around. Having the impression of the family member being in the same room was fairly important to the residents, although some indicated that they found this enjoyable rather than important. During family visits, residents indicated to generally have fun and like doing something together with their family member.
3.3 Discussion
The focus groups with residents, family members and caretakers have resulted in a deeper understanding of their needs and wishes. From the responses on the discussion topics of regular video calling and the AR solution as well as on the interest survey, it became apparent that having an enjoyable experience and experiencing feelings of togetherness and connectedness were highly valued by the user groups. Based on these findings, we established three user experience indicators: enjoyment, spatial presence and social presence. The latter was mentioned the most often during the focus groups and is thus considered of greatest importance. These indicators will provide the foundation for the development of an evaluation methodology for Study 2.
Based on the outcomes of Study 1, three recommendations were derived for the design of Study 2: (1) allow the residents to familiarize themselves with the AR solution prior to the experiment, (2) implement an extensive evaluation methodology, and (3) involve caretakers during the experiment. Before addressing these recommendations, an important difference between the previous video calling implementation at the nursing home and the future setup in Study 2 needs to be discussed.
During the focus groups, it became evident that the use and opinion of technological systems strongly depend on their implementation. For regular video calling, the issues of aiming and holding the device negatively influenced the use and the experience among the users. However, these issues were a disadvantage of the specific implementation that was used at the nursing home and therefore not of video calling in general. Moreover, the setup that will be used in Study 2 will involve a tablet mounted on a standard, for both regular video calling and the AR solution. Hence, the stable construction is not a specific advantage for the AR solution and this construction could solve the abovementioned issues for regular video calling as well.
To resume, our first recommendation was about to familiarization with the AR solution. Among elderly populations, there appears to be a stereotypical resistance towards novel ideas and advances in technology: they feel too old to adopt new (interactive) technologies and have less confidence in their abilities to successfully use these [10,28]. This might be due to them not having been exposed to the same level of technology during their lives compared to the younger generation [28]. Consequently, their behavior regarding (new) technologies might incline towards tentativeness and caution. In our sample, this could be reflected by the need for familiarization with the AR solution. We therefore recommend to allow the residents to see and try out the AR solution prior to the experiment, as a preparatory step for Study 2. This would reduce feelings of anxiety and/or uncertainty towards the AR solution and its setup. At the same time, this would decrease the chance of having biased results due to novelty effects. This is a phenomenon associated with first-time use of a new technology, where outcomes do not reflect the actual experience and/or performance but apply to the initial response to the novelty of the technology.
Additionally, we advise to implement an extensive evaluation methodology. A limitation of this study is that the focus group with the residents consisted exclusively of female participants. Thus, the findings might not appropriately reflect the needs of the entire population of nursing home residents, because the needs and wishes of male residents might be (partially) different. While designing an evaluation methodology for Study 2, a more extensive approach should be
implemented since male residents will be included as well. For instance, factors such as interest, response and initiative could be incorporated to ensure that the user experience is adequately assessed.
Lastly, caretakers should be involved during the experiment. The importance of including caretakers during the implementation of a technological innovation in nursing homes has been recognized in prior research [55]. In this study, the caretakers displayed great interest in the AR solution and indicated that they would like to be involved during the entire implementation process. They also suggested that they could make valuable contributions due to their close relationships with the residents and their family members. Hence, caretakers could fulfill central roles within such technological user experience experiments.
4 STUDY 2: EVALUATION 4.1 Method
4.1.1 Participants
For the evaluation study, 10 pairs of participants were included, with each pair consisting of one resident and one family member. The participants were recruited by caretakers working at the nursing home, whereby participants from Study 1 were excluded. Other exclusion criteria were poor vision (after correction by glasses/lenses), poor hearing (after correction by hearing aids) and inability to give informed consent e.g. due to (extreme) dementia.
Demographics of the residents and their family members are presented in Table 2. Residents were recruited from the psychogeriatric (n = 4) and somatic (n = 6) departments of the nursing home (that is, the same nursing home as in Study 1). This division between departments is based on the residents’ complaints and associated care: psychogeriatric residents typically suffer from mental disorders such as dementia, while somatic residents are characterized by physical complaints. In our sample, the psychogeriatric residents had significantly worse cognitive functioning than the somatic residents, t(8) = -3.722, p < 0.01, 95% CI [-13.90, -3.27]. This was assessed using a Dutch adaptation of the Mini-Mental State Examination, a screening instrument for cognitive impairments in the elderly where lower scores correspond to lower cognitive levels [11,25]. These findings match the typical symptomatology of the residents’ departments. Despite these group differences, further analyses were conducted for all participants because of the small sample size.
Table 2: Demographics of the residents and their family members Residents (N = 10) Family members (N = 10) Sex (M/F) 2/8 4/6 Age (years) 87.90 ± 5.69 59.00 ± 6.88 Technology experience* 4.10 ± 1.60 N/A
MMSE score** 20.90 ± 5.57 N/A
Relationship
Child (son/daughter) N/A 8 (2/6)
Other 2
Data are mean ± SD or N. Abbreviations: MMSE = Mini-Mental State Examination. *Assessed by the caretaker on a scale from 1 to 10, where 0 is no experience at all and 10 is expert (see also Appendix A.2). **Assessed by the caretaker, maximal score is 30.
4.1.2 Technical setup
A schematic overview of the AR solution is depicted in Figure 2 and the rooms and setup of the experiment are presented in Figure 3. The resident was seated in front of the AR-enabled tablet. This was a 12.9” iPad Pro (Apple Inc., Cupertino, California, United States). The iPad was mounted vertically on a stand, which was positioned next to a small table and in between the resident’s chair and an empty chair (Figure 3, left). The resident was recorded using a Logitech webcam that was attached on top of the iPad. The device was decorated to appear user friendly, as advised by Zamir and colleagues [55], and other (Christmas) decorations were placed in the room as well. When looking at the screen of the iPad, the resident could see the family member as a 3D projection on the chair. This made it seem as if the family member was physically present and sitting opposite the resident.
On the other side, the family member was seated in front of a 43” TV screen on which the resident was displayed (Figure 2 and 3, right). A 3D RGB-D camera (Azure Kinect) was positioned in front of the TV to register the family member (Figure 3, right). Expert opinions indicated that placing the camera in front of the TV did not form a distraction. The family member was then virtually displayed in the resident's environment as a photorealistic live capture.
Two computers were used to connect the technology on both sides via the internet (Figure 2). To deliver the video frames from the resident to the family member and the audio signals in both directions, we used a connection via Microsoft Teams, an online video conferencing service of Microsoft 365 (Microsoft Corporation, Redmond, Washington, United States). Audio was recorded and played out using Jabra 750 speakerphones. This is a high-end device known for good audio quality and echo cancelling which allows for dual-talk (i.e., both conversation partners talking at the same time). A dedicated local area network was used for video and audio data transmission, ensuring a high-performance connection.
Figure 2: Schematic overview of the AR solution. The resident (left) could view the family member as a 3D projection in AR on the 12.9” iPad Pro and was recorded by a Logitech webcam attached on top of the iPad. The family member (right) could view the resident as a 2D presentation on a 43” TV screen and was recorded by the Kinect camera. Two computers were used to connect the technology on both sides via the internet.
Figure 3: Rooms and set up of the experiment. On the resident’s side (left): a = chair onto which the family member would be projected; b = Logitech webcam; c = iPad; d = Jabra 750 speakerphone; e = position of the resident. On the family member’s side (right): f = Azure Kinect
camera; g = TV screen; h = Jabra 750 speakerphone; i = position of the family member.
To create a highly detailed 3D point cloud from a 2D image, depth information is required. The quality of the depth image is significantly influenced by the color space (i.e., how colors are stored) and bit-depth (i.e., how much information is stored per pixel). To project depth information on the video frames, the depth capture of the family member by the Kinect camera was processed in multiple steps. These included a foreground-background removal, mapping and stitching of depth and color, conversion into the hue, saturation and brightness color model, and dilation and erosion to smoothen edges and remove noise. Subsequently, the frames were transmitted to the resident’s side by a Web Real-Time Communication transmission module and received by a Web Real-Time Communication reception module. Lastly, the frames were rendered in AR on the iPad, such that the 3D depth content of the family member was displayed on the resident’s side.
In the regular video calling condition, a live video connection was realized using Microsoft Teams. During regular video calling, the same setup was used as for the AR solution, except now the family member was presented in 2D on the iPad and the resident was filmed by the iPad front camera.
Examples of the two family visiting conditions are shown in Figure 4. It is important to note that the difference between both conditions was only visible for the residents. For the AR solution, the family member was projected in 3D onto a chair in the resident’s own environment (Figure 4, left), whereas during regular video calling, the family member was depicted as a 2D presentation with a background of the family member’s room (Figure 4, right). For the family member, the resident was visible in 2D with a background of the resident’s room in both conditions. The mode of communication was only manipulated on the side of the resident, because the AR solution was primarily intended for them. Due to limited time and budget, we were not able to develop the AR solution for the family member’s side as well.
Several steps were taken to ensure that all other circumstances were constant. In both conditions, the same screens were used in the same orientations. The audio signals were similar due to using the same Jabra 750 speakerphones and Microsoft Teams connections for audio transmission in both conditions. The user’s self-view, which is normally visible while using Microsoft Teams, was disabled during regular video calling, since the AR solution also lacked a self-view. Delays were very similar as well due to using Microsoft Teams connection in both conditions. This offered good lip synchronization, which was also confirmed by expert opinions.
Figure 4: Examples of the AR solution (left) and regular video calling (right) on the resident’s side, with close-ups of the iPad for each condition.
4.1.3 Procedure
The experiment was structured in a within-subjects design, as shown in Figure 5. Each participant pair underwent the two family visiting conditions (i.e., the AR solution and regular video calling) in counterbalanced order within a single testing day.
Figure 5: Within-subjects study design, 10 participant pairs underwent two sessions (i.e. the AR solution and regular video calling) in counterbalanced order. After each session, we evaluated the user experience.
A few days prior to experiment, the residents were given the opportunity to familiarize themselves with the technology. This was recommended in Study 1. However, this was not done for the psychogeriatric residents due to their impaired memory and for some of the somatic residents, because of their last-minute recruitment after COVID-19-related cancellations.
Participant pairs were not allowed to see each other before and in between the sessions. During the experiment, the resident was seated in a room of the nursing home, while the family member was seated in another room at the same
location. At the start of each session, the family member was already seated and visible before the resident took a seat behind the iPad. Therefore, we were able to prevent potential confusion among the residents due to a sudden virtual appearance of the family member. Also, some conversation starters were provided at both sides, such as newspapers and Christmas decorations. Participants did not receive instructions about the content of conversation, except for the instruction to the family members not to bring any big news such as pregnancies, deaths and weddings. This was done to prevent the conversation from having (too) much emotional significance. Besides, the participants did not undertake any specific activities during the sessions. As such, the user experience would only pertain to the experience itself and not the activity.
We evaluated the user experience of the residents and their family members directly after each session (Figure 5). An overview of the evaluation methodology is presented in Table 3. The evaluation methodology included the user experience indicators from Study 1 as well as attitudes, behavior and use. This extensive approach was recommended in Study 1. For the residents, the evaluation methodology consisted of a semi structured interview, observations and global impressions. The user experience of the family members was assessed using questionnaires and global impressions. Furthermore, we recorded the duration of each session as a measure of use.
Table 3: Evaluation methodology
Residents Family members
User experience indicators • Semi structured interview • Questionnaire
• Networked Minds Questionnaire Attitudes • Semi structured interview • Questionnaire
Behavior • Observations
• Global impressions • Global impressions
Use • Duration
Advantages and disadvantages of (A) regular video calling and (B) the AR solution, as mentioned by residents, family members and caretakers during the focus groups.
Firstly, we evaluated user experience using the user experience indicators of enjoyment, spatial presence and social presence, which were derived from Study 1. The residents and family members were asked to grade their experiences on a scale from 0 to 10 (see also Appendix A.2). We selected this approach because of the similarity to school grades, a concept that the residents and family members were typically familiar with. This approach was inspired by the single-item style of Bouchard and colleagues, whose formulations were well understood, reliable and sensitive [5]. In addition, single-item questionnaires have shown to be a valid method to measure feelings of presence, especially for certain groups [53]. We asked the residents to assign grades for the user experience indicators during the semi structured interview. For family members, this was done in a questionnaire.
Besides, the family members filled in a Dutch adaptation of the Networked Minds Questionnaire [16]. This questionnaire was selected because it examines self-perception as well as perception of the other with regard to (social) presence and psycho-behavioral interaction. The items of the questionnaire were assessed on a seven-point Likert scale and were grouped into six subscales (i.e., copresence, attentional allocation, perceived message understanding, perceived emotional understanding, perceived emotional interdependence and perceived behavioral interdependence). These subscales were identified by a factor analysis and Cronbach alpha tests indicated that these subscales were internally consistent, with alpha reliabilities between 0.81 and 0.87 [16].
Secondly, we investigated the participants’ attitudes towards the AR solution and regular video calling (see also Appendix A.2). During the semi structured interview, we asked the residents if they were interested to use the technology again in the future. For the family members, questionnaire items were included about the technology’s suitability for repeated use for the residents as well as for themselves.
Thirdly, we studied the behavior of the residents and the family members. Two caretakers, one for the psychogeriatric and one for the somatic residents, observed (changes in) the residents’ behavior based on an adapted Dutch version of the Music in Dementia Assessment Scales [30]. This adapted version was accommodated to examine (changes in) the well-being of a person (with dementia) during remote family visits (see also Appendix A.2). Caretakers were deemed capable to
examine (changes in) behavior due to their familiarity with the residents, in accordance with the recommendation from Study 1. The Music in Dementia Assessment Scales included visual analog scales to score levels of interest, response, initiation, involvement and enjoyment of the resident during each session. These items showed high internal consistency with a Cronbach alpha of 0.97 [31]. The highest scores on these items were set as the maximal level an individual can achieve, such that each resident had their own unique set of highest levels [30]. Additionally, we reported global impressions, for example with regard to focus and naturalness, on the side of the resident and the family member (see also Appendix A.2).
Lastly, as a measure of use, we recorded the duration of each session. The maximum duration was 20 minutes. This experimental protocol was approved by the local review board and written informed consent was obtained from all participants.
4.1.4 Statistical analysis
Statistical analyses were performed using the Statistical Package for Social Sciences (IBM SPSS Statistics, version 26, SPSS Inc.). A significance level of 0.05 was used for all hypothesis testing.
We tested the normality of the data with Shapiro-Wilk tests. Within-group differences were tested using paired samples t tests for normally distributed data and Wilcoxon signed rank tests for skewed data. Between-group differences were tested using independent samples t tests for normally distributed data and Mann-Whitney U tests for skewed data.
One-way repeated-measures analysis of variance (ANOVA) were conducted due to the within-subjects design of this study. We tested for interaction effects by adding the variable of interest to the model as a between-subjects factor. For user experience indicator scores, we used two-way repeated measures ANOVA to test the interaction of group and condition.
4.2 Results
4.2.1 User experience indicators
Table 4 shows the mean scores of the user experience indicators of enjoyment, spatial presence and social presence, as assigned by the residents and their family members for the AR solution and regular video calling. For all user experience indicator scores, there were no significant within-group or between-group differences and we did not find significant interaction effects of group and condition. However, the difference in spatial presence for family members was marginally significant (z = -1.90, p = 0.06), with higher scores for the AR solution compared to regular video calling.
Table 4: Mean scores of the user experience indicators
AR solution Regular video calling p value
Enjoyment Residents 8.50 ± 1.27 8.50 ± 1.27 1.00
Family members 8.80 ± 1.14 8.20 ± 0.92 0.11*
p value 0.56* 0.55 0.26
Spatial presence Residents 7.60 ± 2.01 7.50 ± 1.96 0.34 Family members 6.60 ± 3.75 5.10 ± 3.67 0.06*
p value 0.91* 0.08 0.11
Social presence Residents 7.60 ± 2.07 7.60 ± 1.96 1.00 Family members 8.00 ± 2.45 7.90 ± 1.66 0.85*
p value 0.51* 0.72 0.86
Data are mean ± SD. Abbreviations: AR = augmented reality. *Nonparametric tests were used due to skewed data.
The results of the family members’ responses on the Networked Minds Questionnaire are presented in Table 5 (for separate scores the items of self-perception and perception of other, see Appendix A.3, Supplementary table 2). We did not find significant differences between conditions for total score as well as the subscales. However, the difference in perceived message understanding was marginally significant (z = - 1.91, p = 0.06), with higher scores for the AR solution compared to regular video calling.
Table 5: Networked Minds Questionnaire scores
AR solution Regular video calling p value
Total score 182.60 ± 17.28 178.90 ± 22.64 0.52
Subscales
Co-presence 6.65 ± 0.73 6.30 ± 0.87 0.40*
Attentional allocation 5.67 ± 0.92 5.23 ± 0.81 0.17* Perceived message understanding 6.33 ± 0.88 6.08 ± 1.11 0.06* Perceived affective understanding 5.87 ± 1.07 5.80 ± 1.03 0.86* Perceived emotional interdependence 3.15 ± 1.90 3.08 ± 1.60 0.88 Perceived behavioral interdependence 4.88 ± 1.74 5.35 ± 1.46 0.24
Data are mean ± SD. For the subscales, total scores are divided by the number of items belonging to that scale due to the number of items per subscale being uneven. All subscales included six items, except perceived message understanding, which included four. *Nonparametric tests
were used due to skewed data.
4.2.2 Attitudes
For both the AR solution and regular video calling, 70% of the residents indicated that they were interested in using the technology again. One resident indicated not being interested in using the technologies again because all family members lived close by. Therefore, this resident did not see the benefit of remote family visits.
For the AR solution, 90% of the family members indicated that repeated use of this technology could be suitable for the residents as well as for themselves. For regular video calling, this was 70% for the residents and for themselves.
4.2.3 Behavior
Mean behavioral observation scores on the Music in Dementia Assessment Scales are presented in Table 6. There were no significant differences between conditions for all items.
Table 6: Behavioral observation scores on the Music in Dementia Assessment Scales AR solution Regular video calling p value Interest 75.00 ± 17.16 71.50 ± 11.07 0.34* Response 72.50 ± 17.20 73.00 ± 12.74 0.86 Initiation 71.50 ± 15.28 73.00 ± 7.89 0.59* Involvement 76.50 ± 14.15 73.50 ± 14.15 0.28 Enjoyment 74.00 ± 15.06 69.00 ± 9.94 0.32 Data are mean ± SD. *Nonparametric tests were used due to skewed data.
Global impressions were highly similar between conditions. We observed natural conversations with frequent non-verbal communication, such as waving to each other. Nearly all participants displayed a high level of interest, had an inquisitive approach and were primarily focused on their screens during the sessions. In a few cases, residents were distracted, looked at their surroundings or sought support by the caretaker. Occasionally, residents were somewhat uncomfortable or tense at the beginning of the experiment, but this usually disappeared after a few minutes. Furthermore, some residents did not seem to understand the video calling technology.
During the majority of the sessions, a high degree of interaction was observed. Residents and family members seemed to enjoy the experience as they often smiled and made jokes. In most cases, communication derived from both sides, while sometimes it was the family member who mostly kept the conversation going. Conversation topics included talking about how things were going, what they had been up to and how other family members were doing. Also, a few participant pairs commented on each other’s appearances. At the end, some family members and residents suggested to video call with family members who live far away. The conversations were usually terminated by the family members, but also sometimes by the residents. All residents expressed after the experiment that they perceived the conditions as similar and did not notice any differences in background or presentation.
4.2.4 Use
Mean durations of the use of the AR solution and regular video calling are depicted in Figure 6. On average, the sessions with the AR solution lasted significantly longer than those with regular video calling, t(9) = 2.52, p = 0.03. The mean difference in duration was 2.50 minutes (SD = 3.14; 95% CI = 0.26, 4.74).
Figure 6: Mean session durations with the AR solution (left, dark blue) and regular video calling (right, light blue). Error bars denote 95% confidence intervals. *p < 0.05.
4.3 Discussion
This evaluation study involved nursing home residents and their family members. In the context of remote family visits, their user experience of the AR solution was compared to their user experience of regular video calling. This was done by assessing user experience indicators, attitudes, behavior and use. We found that family members reported marginally higher levels of spatial presence and perceived message understanding for the AR solution than for regular video calling. Family members also seemed to prefer the AR solution for future use, while residents had a positive attitude towards both technologies. Lastly, the average session duration with the AR solution was significantly longer than with regular video calling.
Contrary to our hypotheses, we did not find differences between conditions for the residents’ user experience. In fact, only family members reported some marginal differences in user experience. Besides, we did not find differences in social presence between conditions. This was remarkable because this user experience indicator was deemed the most important in Study 1. We have three possible explanations for these unexpected findings.
First of all, the residents were less able to verbalize their user experience. The difficulty in understanding of and responding to questionnaire items has previously been identified in presence research [15] and also specifically among nursing home residents in the context of video calling [55]. In our study, it became indeed apparent that understanding and answering the user experience indicator questions was a fairly difficult task for most residents. Despite saying that they enjoyed the sessions and experienced feelings of togetherness and connectedness, the residents had difficulties in articulating their experiences in a ‘school grade’. As such, some residents attributed the same score to all user experience indicators for both conditions, even when their verbal expressions indicated otherwise. These difficulties may have been caused by the residents’ varying degrees of cognitive impairment and/or their susceptibility to response bias. This bias has been recognized in the field of (social) presence [15,24]. In our study, some residents tended to give socially desirable scores. For example, one resident indicated being very content with the received care at the nursing home and would therefore only assign the most positive scores in the experiment.
Secondly, another methodological issue within presence research is the (over-)reliance on self-evaluation methods [15,24]. Self-report questionnaires are the most commonly used method to study feelings of (social) presence [19], which is justified by the fact that it is a highly subjective experience [34]. In our study, residents found it difficult to accurately verbalize their experiences and express them in a ‘school grade’. Thus, relying on self-evaluation methodology for assessment of the residents’ user experience might not have been the best approach. Consequently, we may have missed effects due to the evaluation methodology being less suitable for this population.
Thirdly, previous research has shown that stereotypes and expectations greatly influence the usage of new technologies [39]. As described in Study 1, elderly populations seem to have a stereotypical resistance towards technological innovations [10,28]. In addition, prior information and expectations have been suggested to influence the user’s rating of presence [12]. In our study, these factors may have played a (subliminal) role in the residents’ assessment of the user experience indicators.
Taken together, it seems that residents did not (consciously) experience differences in user experience between conditions. After all, each resident indicated to perceive the conditions as the same. Admittedly, the conditions might actually have looked identical to the residents, in the sense that they were looking at a screen in front of them on which their family member was visible. This is endorsed by the residents’ equally positive attitude towards both technologies.
Nevertheless, the remote family visits were always a shared experience between residents and the family members. In our study, we found that one party, in this case the family members, perceived marginally higher spatial presence and message understanding for the AR solution compared to during regular video calling. Considering that the other party (i.e., the residents) did not perceive the AR solution as worse, indicates that this still is a practical and relevant finding. Possibly, while the residents used the AR solution, there may have been subtle changes in the conversation and their behavior. In turn, this might have affected the user experience of the family member, such that they perceived more spatial presence and better message understanding. These speculations can be linked to the longer session durations of the AR solution, as they jointly suggest that the residents may have unconsciously experienced differences between conditions. We propose four explanations for this.
To begin with, during regular video calling, the user is constantly aware of the ‘other side’. In this study, the residents could see their family members with a background of the family members’ room, displaying this ‘other side’. However, in the AR solution the family members were projected in the residents’ own surroundings, which may have decreased their awareness of the ‘other side’. This might have provided the residents with a greater sense of comfort, security and privacy. For instance, the residents might have had less feelings of needing to pay attention to what they say. It is possible that this contributed to longer session durations and/or different user experience for the family members for the AR solution.
Also, nursing home residents belong to a generation that is typically concerned and familiar with the costs of phone calling [17,40]. If the experience is less similar to making a phone call, residents may be less concerned about the costs of a long session. Instead, if the AR solution diminished the impression of the ‘other side’ and was more like a real-life visit, residents might not think in terms of duration and thereby costs at all. This could have led to longer durations for the sessions with the AR solution.
On the contrary, it could also be that the sessions with the AR solution were longer because there was more time needed for effective communication. For instance, the AR solution could have been more difficult to use or the AR solution itself formed an additional conversational topic. However, this is not supported by our observations and global impressions. There were no differences in behavioral observation scores and natural communication with non-verbal cues were observed during both conditions. Also, conversation topics did not seem to differ between conditions. Thus, this explanation is not very likely.
Lastly, the marginal finding that family members experienced higher spatial presence while using the AR solution may be associated with the longer session durations. Research in the consumer research area suggest that spatial presence is an important mediator when interacting in AR environments [20,21,47]. These studies have found that a heightened feeling of spatial presence is often followed by more enjoyment or longer engagement in a service. Future studies should further investigate this possible mediating role of spatial presence in the context of remote family visits in nursing homes.
4.3.1 Strengths and limitations
A strength of this research project is that the characteristics of the residents have been taken into account while designing the evaluation methodology as well as while developing the AR solution. The comfort of the residents was prioritized in this study. Therefore, a screen-based AR solution was chosen over a glasses-based AR or a VR system. This was advantageous from the technical point of view (a part of the space to be presented already existed and did not require a computer-generated model) as well as for the resident’s comfort (a detachment form the real world can lead to physical
and mental discomfort) [42,53]. For instance, the AR solution allowed for direct contact between the resident and the caretaker during the remote family visits.
We also used an extensive evaluation methodology by covering subjective and observational measures on both the residents’ and the family members’ side. Our evaluation methodology was tailored to assess social interactions between residents and family members that took place in AR. Moreover, it allowed for investigating differences in user experience between the AR solution and regular video calling in the context of remote family visits. Potentially, this study could contribute to the development of a general user experience methodology to evaluate social interactions in AR.
The main limitation of this study was its small sample size. Inclusion of participants was hampered because of the COVID-19 pandemic. Due to the limited sample size, we may have missed some effects. Future studies should further investigate spatial presence and perceived message understanding during AR-based remote family visits, as these findings were only marginally significant in our study.
Moreover, the family members’ reported differences in user experience may have been affected by their awareness of the conditions. During the experiment, the family members were aware of how they were shown to the residents, i.e. what condition was presented on the residents’ side. This could have resulted in response bias among the family members. As such, they possibly tended to evaluate the AR solution more positively simply because they knew when these sessions took place. This should be prevented in future studies by conducting the experiment blindly for the family members.
Another limitation with regard to the study sample is the highly variable group of residents. Prior research has acknowledged that older adults are a very diverse group with diverse characteristics, behaviors, and needs [28]. Moreover, the role of individual differences in the perception of presence and related concepts has been widely recognized in the literature [37,43,54]. The residents in our sample were diverse, for example because we included both psychogeriatric and somatic residents with varying levels of cognitive impairment. Remote family visits might not have been suitable for all of them. Hence, AR-based remote family visits may not be one-size-fits-all solution, however, residential care in general requires tailoring to the individual. Future studies should consider including nursing home residents who are deemed suitable for such technological solutions, for example as determined by caretakers.
The limitations of the AR system include screen size and visible edges. Due to the iPad’s screen size of 12.9”, the family member’s facial expressions were relatively hard to distinguish. Residents typically enjoy being able to see the facial expressions of their family members, as mentioned during the focus group of Study 1. Besides, the AR solution may have been unsuccessful in providing a convincing seamless experience since the edges of the screen were still visible. Thus, the residents might not have experienced the AR solution as a transparent surface, such that they were continuously aware of looking at a screen and not ‘through’ it. For future development, the effects of zooming in and removing edges could be explored.
Also, this study only involved a one-time use of the AR solution. A suggestion for future research is to study the long(er) term use. This would allow for gaining deeper insight into the actual suitability of the AR solution for remote family visits in nursing homes. For example, this could involve studying effects of repeated use of the AR solution on the residents’ (mental) well-being and loneliness.
Lastly, as mentioned above, the evaluation methodology may have been less suitable to adequately assess the user experience of the residents. This is an important limitation within our study. However, duration as a measure of use still provided insight to some extent. Indeed, indicators of the use itself could be more appropriate to assess user experience in research populations who are less able to self-evaluate their experiences, such as nursing home residents. We suggest that use indicators could pertain to duration as well as frequency and consistency.
5 CONCLUSION
In this research project, an AR solution was developed to enable life-like remote family visits in nursing homes, as an alternative communication means during the COVID-19 pandemic. The aim of this project was to investigate the suitability of the AR solution in this context. We compared the user experience of the AR solution to the user experience of regular video calling, with evaluation methodology based on identified user needs in relation to remote family visits.
Although the difference between conditions was only visible on the residents’ side, we did not find differences in their user experience. In fact, residents indicted to not (consciously) perceive differences between conditions. This may also be related to used self-report evaluation methodology, which was suggested to be less suitable for this research population. Moreover, family members reported some marginal differences in user experience. Considering the small sample size and the family members’ awareness of the conditions, these findings should be treated with caution, but could provide a starting point for future research. Nevertheless, we found that the average session durations with the AR solution were longer compared to regular video calling. Based on this finding, we tentatively suggest that there may be (unconscious) differences in user experience after all.
ACKNOWLEDGMENTS
This research was part of a collaboration between The Netherlands Organisation for applied scientific research (TNO) and MeanderGroep, a healthcare provider in South-Limburg, the Netherlands. This project was partially funded by TNO’s Early Research Project ‘Social eXtended Reality’.
REFERENCES
[1] Ronald T. Azuma. 1997. A survey of augmented reality. Presence Teleoperators Virtual Environ. 6, 4 (1997), 355–385. DOI:https://doi.org/10.1162/pres.1997.6.4.355
[2] Frank Biocca, Chad Harms, and Judee K. Burgoon. 2004. Toward a More Robust Theory and Measure of Social Presence: Review and Suggested Criteria. Presence Teleoperators Virtual Environ. 12, 5 (2004), 456–480. DOI:https://doi.org/10.1162/105474603322761270
[3] Frank Biocca, Chad Harms, and Jenn Gregg. 2001. The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity. In 4th Annual International Workshop on Presence, 1–9.
[4] I. L. Boman, L. Rosenberg, S. Lundberg, and L. Nygård. 2012. First steps in designing a videophone for people with dementia: Identification of users’ potentials and the requirements of communication technology. Disabil. Rehabil. Assist. Technol. 7, 5 (2012), 356–363.
DOI:https://doi.org/10.3109/17483107.2011.635750
[5] Stéphane Bouchard, Geneviève Robillard, Julie St-Jacques, Stéphanie Dumoulin, Marie Josée Patry, and Patrice Renaud. 2004. Reliability and validity of a single-item measure of presence in VR. In Proceedings - 3rd IEEE International Workshop on Haptic, Audio and Visual Environments
and their Applications - HAVE 2004, 59–61. DOI:https://doi.org/10.1109/have.2004.1391882
[6] Doug A. Bowman and Ryan P. McMahan. 2007. Virtual reality: How much immersion is enough? Computer (Long. Beach. Calif). 40, 7 (2007), 36– 43. DOI:https://doi.org/10.1109/MC.2007.257
[7] Nathan M. D’Cunha, Dung Nguyen, Nenad Naumovski, Andrew J. McKune, Jane Kellett, Ekavi N. Georgousopoulou, Jane Frost, and Stephen Isbel. 2019. A mini-review of virtual reality-based interventions to promote well-being for people living with dementia and mild cognitive impairment.
Gerontology 65, 4 (2019), 430–440. DOI:https://doi.org/10.1159/000500040
[8] Jorunn Drageset, Marit Kirkevold, and Birgitte Espehaug. 2011. Loneliness and social support among nursing home residents without cognitive impairment: A questionnaire survey. Int. J. Nurs. Stud. 48, 5 (2011), 611–619. DOI:https://doi.org/10.1016/j.ijnurstu.2010.09.008
[9] Laura I. van Dyck, Kirsten M. Wilkins, Jennifer Ouellet, Gregory M. Ouellet, and Michelle L. Conroy. 2020. Combating Heightened Social Isolation of Nursing Home Elders: The Telephone Outreach in the COVID-19 Outbreak Program. Am. J. Geriatr. Psychiatry 28, 9 (2020), 989–992.
DOI:https://doi.org/10.1016/j.jagp.2020.05.026
[10] R. Eisma, A. Dickinson, J. Goodman, A. Syme, L. Tiwari, and A. F. Newell. 2004. Early user involvement in the development of information technology-related products for older people. Univers. Access Inf. Soc. 3, 2 (2004), 131–140. DOI:https://doi.org/10.1007/s10209-004-0092-z [11] Marshal F. Folstein, Susan E. Folstein, and Paul R. McHugh. 1975. “Mini-mental state”: A practical method for grading the cognitive state of patients
for the clinician. J. Psychiatr. Res. 12, 3 (1975), 189–198. DOI:https://doi.org/https://doi.org/10.1016/0022-3956(75)90026-6
[12] Jonathan Freeman and S E Avons. 2000. Focus Group Exploration of Presence through Advanced Broadcast Services. Hum. Vis. Electron. Imaging V 3959, June (2000), 530–539.
[13] Ryoko Furukawa and Martha Driessnack. 2013. Video-Mediated Communication to Support Distant Family Connectedness. Clin. Nurs. Res. 22, 1 (2013), 82–94. DOI:https://doi.org/10.1177/1054773812446150
[14] Gregory Garra, Adam J. Singer, Breena R. Taira, Jasmin Chohan, Hiran Cardoz, Ernest Chisena, and Henry C. Thode. 2010. Validation of the Wong-Baker FACES pain rating scale in pediatric emergency department patients. Acad. Emerg. Med. 17, 1 (2010), 50–54.
DOI:https://doi.org/10.1111/j.1553-2712.2009.00620.x
[15] Simone Grassini and Karin Laumann. 2020. Questionnaire Measures and Physiological Correlates of Presence: A Systematic Review. Front. Psychol. 11, March (2020), 1–21. DOI:https://doi.org/10.3389/fpsyg.2020.00349
[16] Chad Harms and Frank Biocca. 2004. Internal Consistency and Reliability of the Networked Minds Measure of Social Presence. In Seventh Annual
International Workshop: Presence 2004, 246–251.
[17] Wesley Hartkamp. 2021. Personal communication with the author.
[18] Theresa Heidinger and Lukas Richter. 2020. The Effect of COVID-19 on Loneliness in the Elderly. An Empirical Comparison of Pre-and Peri-Pandemic Loneliness in Community-Dwelling Elderly. Front. Psychol. 11, September (2020), 1–5.
DOI:https://doi.org/10.3389/fpsyg.2020.585308
[19] Dimitri Hein, Christian Mai, and Heinrich Hußmann. 2018. The usage of presence measurements in research: a review. In Proceedings of the
International Society for Presence Research Annual Conference (Presence’18). The International Society for Presence Research, 1–28.
[20] Jonas Heller, Mathew Chylinski, Ko de Ruyter, Debbie I. Keeling, Tim Hilken, and Dominik Mahr. 2021. Tangible Service Automation: Decomposing the Technology-Enabled Engagement Process (TEEP) for Augmented Reality. J. Serv. Res. 24, 1 (2021), 84–103.
DOI:https://doi.org/10.1177/1094670520933692
[21] Tim Hilken, Ko de Ruyter, Mathew Chylinski, Dominik Mahr, and Debbie I. Keeling. 2017. Augmenting the eye of the beholder: exploring the strategic potential of augmented reality to enhance online service experiences. J. Acad. Mark. Sci. 45, 6 (2017), 884–905.
DOI:https://doi.org/10.1007/s11747-017-0541-x
[22] Kuo Ting Huang, Christopher Ball, Jessica Francis, Rabindra Ratan, Josephine Boumis, and Joseph Fordham. 2019. Augmented versus virtual reality in education: An exploratory study examining science knowledge retention when using augmented reality/virtual reality mobile applications. Cyberpsychology, Behav. Soc. Netw. 22, 2 (2019), 105–110. DOI:https://doi.org/10.1089/cyber.2018.0150
[23] Kangsoo Kim, Mark Billinghurst, Gerd Bruder, Henry Been Lirn Duh, and Gregory F. Welch. 2018. Revisiting trends in augmented reality research: A review of the 2nd Decade of ISMAR (2008-2017). IEEE Trans. Vis. Comput. Graph. 24, 11 (2018), 2947–2962.
