The Efficacy of Virtual Reality Biofeedback Game DEEP:
Psychological Need Satisfaction, Motivation, and Anxiety
Reduction
Liesbeth A. van den Berg
10763066
M
ASTER
I
NFORMATION
S
TUDIES
G
AME
S
TUDIES
F
ACULTY OF
S
CIENCE
U
NIVERSITY OF
A
MSTERDAM
July 9, 20161st supervisor
Dr. ir. Marieke M. J. W. van Rooij Games for Emotional and Mental Health Lab
Behavioural Science Institute Radboud University
Nijmegen
2nd supervisor
Dr. Frank Nack
Intelligent System Lab Amsterdam Institute for Informatics University of Amsterdam
1
The Efficacy of Virtual Reality Biofeedback Game DEEP:
Psychological Need Satisfaction, Motivation, and Anxiety
Reduction
L. A. van den Berg
University of Amsterdam Science Park 406 1098 XH Amsterdam
liesbeth.vandenberg@amc.uva.nl
ABSTRACT
This study assesses the efficacy of the virtual reality biofeedback game DEEP, an explorative game; controlled by breathing, designed to calm and soothe its players. The game’s efficacy was assessed on three categories: psychological need satisfaction, motivation, and anxiety reduction. Additionally, research was conducted on how a health focused introduction of the game would influence these outcomes when compared to a neutral introduction. Two randomly allocated groups were shown either a serious trailer (N = 20) which empathised the anxiety reducing effect of DEEP or a non-serious trailer (N = 21) which introduced the game as a fun game to play. Both groups played a ten minute DEEP game session. Participants were tested on self-reported state-trait anxiety, intrinsic motivation, player experience need satisfaction, and qualitative findings through means of short semi-structured interviews and gameplay observations.
Results indicated that a ten minute DEEP game session can result in a significant decrease in self-reported state-anxiety. Participants who viewed the serious trailer showcased a lower motivation to play DEEP when compared to those who had seen the non-serious trailer. Post-hoc research indicated that in-game autonomy could potentially be a predictor for reduced anxiety outcomes. No differences were found between the two groups with regard to psychological experience need satisfaction, intrinsic motivation nor their evaluation of the game. Qualitative results revealed that a large group of players had difficulty mastering the breathing controls and one third spent one fourth of their gameplay in a non-interactive area.
DEEP has potential as a tool for people with anxiety problems but its effects on players need to be addressed further in future research. Specifically, increasing player’s in-game autonomy may strengthen DEEP’s beneficial effects on anxiety-levels. Finally, user experience testing is needed to get more insight into the way the game should be further developed and presented.
CCS Concepts
• Applied computing ~ Law, social and behavioral sciences • Applied computing ~ Health informatics • Applied computing ~ Psychology • Software and its engineering ~ Virtual worlds
software • Software and its engineering ~ Interactive games • Computing methodologies ~ Virtual reality
Keywords
Serious Games; Games for Health; Virtual Reality; Biofeedback; Intrinsic Motivation; Intervention; Anxiety.
1. INTRODUCTION
In the last decade the interest in, quantity, and research of serious games has grown rapidly [1], among others in the health care
domain. Games for health have been proven to induce positive effects in players [2] such as; increased cognitive functions, high interest/enjoyment of gamified procedures, higher awareness of health behaviour, and treatment adherence [3-8]. Several studies claim that serious games should be utilised more often because of their alleged motivational appeal [8-10]. However, a meta-analysis conducted by Wouters et al. [11] found that serious games are not more motivating than conventional instructional methods. Whether serious games have a motivational appeal and subsequently whether this benefits the outcomes of serious games remains a little explored field of research. However, aside from the potential health outcomes of serious games, ensuring an intrinsically motivated player will improve overall enjoyment and could potentially enhance health outcomes [12-14]. Intrinsic motivation is the motivation to do or act for the person’s own internal rewards [15]. Whereas extrinsic motivation occurs when rewards are external, such as money or grades. Tests for efficacy, intrinsic motivation to play, and experiencing a serious game as intrinsically motivating are compelling features to assess for in any serious game for health.
1.1 Intrinsic Motivation
Over 155 million people in the United States play videogames [16], convincingly demonstrating games are engaging to a large body of the population. One can argue this to be the results of intrinsic motivation. But what makes a game intrinsically motivating, and how can we use this when designing serious games? The self-determination theory (SDT) by Ryan and Deci addresses factors that either facilitate or undermine both intrinsic and extrinsic motivation [15]. The SDT expresses three psychological needs; autonomy, competence, and relatedness. Autonomy is the sense of freedom that people experience. Competence describes how a person can acquire new skills. And finally, relatedness describes how well a person connects and cares for others.
Additionally, the SDT describes a distinction between so-called autonomous and controlled motivation [15, 17]. Autonomous motivation entails motivation of both intrinsic and (some) extrinsic nature, where a person experiences volition. For example, someone playing a videogame because it is fun and challenging. Controlled motivation entails behaviour conducted for external reasons, such as sense of pressure or punishment avoidance. For example, following certain classes because your parents expect you to and pay for your tuition. A sub-theory within SDT is the cognitive evaluation theory (CET), which specifies contextual factors of intrinsic motivation [15]. CET states that meeting autonomy and competence needs during an action leads to intrinsic motivation. For example, deadlines or imposed goals reduce intrinsic motivation, because the person does not experience autonomy. The authors of the SDT and CET performed multiple studies to assess how psychological needs are satisfied in-game and how they can be measured [13]. For example, the Player Experience of Need
2 Satisfaction (PENS) measures amongst others; in-game autonomy, in-game competence, and in-game relatedness [13], which comprise the elementary parts of the SDT framework. These in-game psychological needs account for the player’s in-game enjoyment. Additionally, the PENS allows for measurements of in-game presence/immersion and intuitive controls. It was found that psychological need satisfaction accounts for preference for future play and game enjoyment.
Currently, a lot of emphasis is put on how serious games are a tool to educate or help its players. When players are made aware of the ‘serious’ part of a serious game, their intrinsic or autonomous motivation might be affected. Specifically, the perception of imposed goals or instructions could reduce autonomy, which could result in lower intrinsic motivation and controlled autonomy. The effect of the aforementioned psychological need satisfaction accounts for preference for future play and game enjoyment. One goal of this study is to test whether enforcing imposed goals indeed result in reduced psychological need satisfaction and subsequently intrinsic motivation. Designing a serious game and its instruction, which provides intrinsic or autonomous motivation could be a powerful tool for therapy engagement and could potentially result in better outcomes over time [12, 14].
1.2 The Virtual Reality Game DEEP
The virtual reality (VR) game DEEP is a biofeedback game controlled through players’ breathing [18]. The game is co-developed by O. Harris and N. Smit. Players of DEEP wear a VR headset and a custom DEEP controller. Players are situated in an abstract underwater world, which they can explore at their leisure (Figure 1). The game provides a relaxing and immersive experience; there are no in-game objectives other than to enjoy the aesthetics of the virtual world.
The DEEP controller is a belt fastened around the player’s abdomen. Through slow, diaphragmatic breathing the player is able to move in the virtual world. The belt registers diaphragm expansions. It registers how full the player’s lungs are (0%-100%) and whether the player is breathing in or out. There are four ‘breathing states’ that the belt registers, which control in-game movement. These four states and their corresponding in-game movements are:
<50% lung capacity and not breathing in: player sinks; <50% lung capacity and breathing in: player rises slowly; >50% lung capacity and not breathing in: player float; >50% lung capacity and breathing in: player rises quickly. A small semi-transparent circle is depicted in the middle of the player’s visual field. This circle expands with inhalation and shrinks with exhalation (Figure1a-b). This serves as a visual feedback mechanism for the player.
Figure 1a-b. Screenshots of the VR game DEEP illustrating the biofeedback circle in inhalation (1a) and exhalation (1b).
Researchers of the Behavioural Science Institute (BSI) at the Radboud University in Nijmegen and developers of DEEP are collaborating to further develop the game as a potential tool for prevention and treatment of anxiety using biofeedback, relaxation, and other existing therapeutic elements [19]. This study marks the beginning of a number of studies that will be conducted to further study and develop DEEP as a potential intervention for people at-risk for anxiety.
The goal of this thesis is to analyse the effects of DEEP on the players’ self-reported state-anxiety, the intrinsic motivation of the players, in-game psychological needs fulfilment, and serves as an explorative study for future study designs.
2. RELATED WORK
A pilot study with regard to the efficacy of DEEP demonstrated initial potential of DEEP as an intervention for children at-risk of anxiety [20]. A total of 86 children (aged 8-12) played DEEP for seven minutes. Before and after playing the game the researchers measured self-reported state-anxiety (utilising the State-Trait Anxiety Inventory for Children, STAIC [21]), self-reported positive and negative affect (utilising the Positive and Negative Affect Schedule, PANAS [22]), and experience of playing DEEP. Additionally, the researchers collected diaphragm expansion data of the players. Results showed that after playing DEEP there was a significant decrease in self-reported state-anxiety (compared with the pre intervention STAIC).
M. Poppelaars et al. [23] hypothesized that the usage of a mental health introduction of a videogame will negatively affect autonomy and intrinsic motivation. An experiment was conducted with 146 first-year college students who played a cooperative videogame, WAY [24]. Participants were randomly shown one of two similar trailers about the game, a mental health trailer and a regular trailer. The mental health trailer portrayed the game as being beneficial to the player’s mental health, whereas the other regular trailer emphasized that the game was highly acclaimed. Results showed that intrinsic motivation was high and similar in both groups. Additionally, no differences were found in experienced competence and relatedness between the two groups. The mental health introduction caused reduced in-game autonomy in comparison to the regular trailer.
3. RESEARCH QUESTIONS
Based on the findings of the literature study, related work, and the aims of the DEEP research project, the research questions and its underlying motivations are the following:
1a
3
1. Does playing DEEP result in a decrease in self-reported state-anxiety levels?
The main goal of the DEEP research project is to develop a tool that can prevent or treat anxiety. Breathing exercises are widely used to control and regulate anxiety [25-27]. In particular, exercises focused on diaphragmatic breathing are utilised in evidence-based physiological therapies [28, 29]. Slow inhalation and exhalation as a mechanism to move forward in the game could therefore be a powerful tool to reduce anxiety in players.
2. How does DEEP affect the player’s intrinsic motivation?
The extent to which players experience intrinsic motivation is an essential measurement needed for future development of DEEP. A game that elicits a low intrinsic motivation from players could lead to less therapy adherence, preference for future play, and gameplay enjoyment. Moreover, a game that is introduced as a serious tool could potentially diminish the player’s intrinsic motivation.
3. How does the utilization of a serious trailer, compared to a non-serious trailer, affect players?
This research question is subdivided by the following sub questions:
Does the utilisation of a serious trailer, compared to a non-serious trailer, negatively affect the player’s perceived in-game autonomy?
Perceived in-game autonomy is the psychological need primarily researched for this study. It can be hypothesised that imposed goals or instructions will reduce in-game autonomy (autonomous motivation) and subsequently intrinsic motivation.
Does the utilisation of a serious trailer negatively affect in-game autonomy and decrease the effect of self-reported state-anxiety outcomes?
The usage of a serious trailer could result in a lower in-game autonomy score when compared to the score of a non-serious trailer, due to the creation of perceived imposed constraints. Less autonomy can result in less intrinsic and autonomous motivation, which in turn could negatively influence the potential anxiety reduction via DEEP. If such an effect indeed takes place, future instructions should be carefully designed to maximize the effect of the serious game.
Does the utilisation of a serious trailer, compared to a non-serious trailer, negatively affect the player’s intrinsic motivation to play DEEP?
If DEEP will indeed be a future prevention or therapeutic tool, the instruction of the game to its players could influence the player’s intrinsic motivation to play the game. Presenting the game as beneficial for player’s health could diminish their interest and willingness to play, which in turn could result in an extrinsically motivated players and a decrease in the overall enjoyment of the game.
Does the utilisation of a serious trailer, compared to a non-serious trailer, negatively affect the other player’s experience needs satisfaction?
How a serious trailer could affect other in-game player experience of needs such as in-game competence, presence/immersion, and intuitive controls is also worth exploring. If there are significant differences between the two trailers with regards to psychological or experience of needs satisfaction, future presentation and gameplay of DEEP might need adjustment.
4. METHODS
4.1 Participants
Posters and flyers were distributed on social media and multiple (applied sciences) universities to recruit eligible participants. The inclusion criteria were: Dutch native speakers with basic English proficiency and aged between 17 and 30. As a small token of gratitude all participants received a chocolate bar after completing the experiment. Additionally, one gift certificate worth €50 was raffled amongst all the participants. Participants were randomly allocated into two groups, a “serious trailer” group and a “non-serious” trailer group that differed in their presentation of the game. Participants were unaware that they were allocated into different groups. Both groups got the same questionnaires, instructions from the researcher, and ten minute DEEP game session. The serious game trailer group (N = 20) included 15 males and 5 females, aged between 18 and 26 (M = 22.5, SD = 2.37). The non-serious game trailer group (N = 21) included 10 males and 11 females, aged between 17 and 27 (M = 22.43, SD = 2.69).
4.2 Materials
4.2.1 Questionnaires
Dutch translations of questionnaires were used to prevent erroneous answers due to language barriers. All survey questions were imported into Qualtrics [30].
4.2.1.1 State-Trait Anxiety Inventory
The State-Trait Anxiety Inventory (STAI) [21] for adults was used to assess state-anxiety (i.e., state of anxiety during the moment of answering) before and after playing DEEP. The STAI presents the participant with 20 statements with regard to (non-) anxiety-related affect states (e.g., calm, tense, relaxed, worried, etc.). On a scale from one to four, the participant had to indicate to what extent these states matched their current affect.
4.2.1.2 Player Experience of Need Satisfaction
The Player Experience of Need Satisfaction (PENS) v1.6 [13] was used to assess the satisfaction of in-game psychological needs: autonomy and competence. The in-game psychological need relatedness was excluded, as the players did not interact with other (in-game) people. Additionally, the subscales: presence/immersion and intuitive controls were measured. The PENS provides the participant with statements to which they must rate on a scale from 1 (do not agree) to 7 (strongly agree) as to which extent they agree with that statement. Permission and access to use PENS Metrics and Methodologies was granted by Immersyve, Inc. [31].
4.2.1.3 Intrinsic Motivation Inventory
The interest/enjoyment subscale of the Intrinsic Motivation Inventory (IMI) [32] was used to assess self-reported intrinsic motivation before (i.e., willingness to play) and after (i.e., intrinsic motivation experienced while playing DEEP) playing the game. The interest/enjoyment IMI subscale consist of 7 statements where the participant has to rate on a scale from 1 (do not agree) to 7 (strongly agree) to which extent they agree with that statement. Lastly, at the end of the experiment all participants were asked to give the game a grade from 1 to 10.
Additional questionnaires and questions were conducted to further assist the DEEP research project. These measurements and its data did not contribute to the research aims of this study. The Positive and Negative Affect Schedule (PANAS) [22] was included to measure positive and negative mood states before the trailer, after the trailer, and after playing DEEP. Each participant was also asked whether they experienced (on a scale from 1-7) any nausea (i.e.,
4 VR sickness) during the game. The occurrence of nausea is regarded as one of the disadvantages of virtual reality [33, 34] and might exclude this intervention for being applicable in future practice.
4.2.1.4 Semi Structured Interview
Lastly, qualitative data was collected in the form of an open question right after the participant played DEEP. In order to capture a direct report of the experience, the researcher asked what the player thought of the game right after the VR headset was removed. This data was used to assess how players experienced the game that could not be answered with the pre-determined questionnaires. Additionally, this data could potentially be used to add a new dimension to the interpretation of the quantitative results (e.g., very low IMI score).
4.2.2 Trailers
Two trailers were made using video footage from the official DEEP trailer. The duration of both trailers was 1:48 minutes. The video material (i.e., gameplay footage) and music from the original trailer were directly copied to both trailers. Text in-between shots were cut out and replaced with text that either provided a serious or non-serious atmosphere. The non-serious game trailer, among others, declared: ‘an evidence-based serious game’, ‘recommended by Games for Health’, and ‘Start reducing your anxiety….’. The non-serious trailer showed slogans such as: ‘explore an underwater fantasy world’, ‘designed to calm and soothe’, and ‘Start exploring the underwater world…’.
Figure 2a-b. Screenshots of the serious trailer (2a) and non-serious trailer (2b).
Additional visual elements were added to strengthen the core message of the trailer. The serious trailer constantly visualised the Radboud University logo in the bottom left corner. The viewer is continually reminded that the Radboud University is involved in this study and/or development of the game. Furthermore, this trailer lists a number of acclaimed associations that published about DEEP. The trailer also showcases five quotes about the (beneficial)
effects DEEP had on players (Figure 2a). The non-serious trailer focusses on the game being featured on entertainment platforms. Additionally, instead of quotes from players, the non-serious trailer showcases the nominations and prizes the game has received (Figure 2b).
4.2.3 DEEP
The developers of DEEP provided a new version of the game to be used in this study. Both the belt and built of game differed from the materials used in the pilot study conducted by van Rooij et al. (2016) [20]. The map of the game was expanded and included different elevations, whereas the previous version was all on the same height level (e.g., no drops). The belt, placed around the abdomen, kept its hardware technology only the exterior look differed from the previous version. Within the belt there is an Arduino compatible FLORA wearable [35], opening Arduino software [36] allows reviewing real-time belt values.
The experiment took place at the dedicated game lab at the University of Amsterdam. DEEP ran on a desktop computer, in conjunction with an Oculus Rift DK2 headset [37].
4.2.4 Additional Measurements
During the gameplay sessions observations were written down in a logbook. These finding would include: pre- and post- play breathing belt values, technical malfunctions, glitches, and the player’s (in-game) behaviour. The logbook's original purpose was to serve as a tool to register technical findings or glitches. However, starting from the first participant the researcher noted whenever potentially interesting (in-game) behaviour would occur. Reasons for this decision were two-fold. First, the gameplay of the players was not recorded nor stored. Having an impression of what the player did or encountered during the game could provide explanations as to how certain quantitative scores came to be. Second, qualitative data such as gameplay behaviour can provide insights into how to further develop game mechanics.
The participant’s heart rate was also monitored during the game. A three-lead set-up with the Biopac MP30 [38] was used to measure the participant’s heart rate. Heart rate data collection was beyond the scope of this study. The data will be utilised in the DEEP research project to analyse potential physiological regulation during the game.
4.3 Procedure
The experiment comprised of a five phases with three data collection (via questionnaires) blocks (Figure 3).
*VR sickness & grading DEEP
Figure 3. Overview of study design.
Before the experiment took place the participants were informed about the content of the experiment and were asked to fill in an informed consent. Thereafter the participant would fill in a demographics survey, STAI questionnaire, and PANAS
DEEP Heart rate (ECG) Logbook Experience (N = 41) Block 1 Demographics State-Anxiety (STAI) Mood (PANAS) (N = 41) Block 2 Mood (PANAS) Motivation to play (IMI) (N = 41) Non-serious trailer (N = 21) Block 3 State-Anxiety (STAI) Mood (PANAS) Needs satisfaction (PENS) Intrinsic motivation (IMI) Extra questions* (N = 41) Time Serious trailer (N = 20) 2a 2b
5 questionnaire. A wall was placed between the researcher and participant (Figure 4) in order to reduce the Hawthorne effect [39].
Figure 4. The researcher was seated on the left, the desk on the right was utilised by the participant.
Participants were randomly allocated to either the serious or non-serious trailer group. The non-serious trailer group viewed a trailer, which empathised the beneficial (mental health) effects of the game. The non-serious trailer group watched a trailer, which empathised the fun aspects and the calming effect of the game. After watching the trailer the participant would fill in two more questionnaires: PANAS and IMI.
After block 2 the participant would be prepared for the ten minute play session. The three ECG leads were placed on the inside of the wrists and left ankle. The VR headset was adjusted accordingly to fit each player. The volume of the game’s music was checked and adjusted. The breathing belt was placed around the abdomen of the player and adjusted to reach appropriate in- and exhalation belt values. All participants received the same instruction with regard to how to play the game and what to do in case they encounter a glitch or bug. Participants were also told that they could, to some extent, look and move around in their chair. Lastly, the light was dimmed and the curtains closed to ensure that the participant was not distracted by light coming through the bottom of the VR headset. When the set-up was completed (Figure 5), a timer was started and simultaneously a signal would be given to the Biopac MP30 to mark the start of the ten minute gameplay.
Figure 5. Set-up DEEP gameplay session.
Throughout the participant’s game session wires were readjusted to allow the player as much freedom of movement as possible. Observations about significant technical details and gameplay behaviour were registered in the logbook. Additionally, technical
difficulties (e.g., player got stuck, disconnected wires) were remedied on the spot. After ten minutes of playing an audio recorder was started to record the participant’s reaction about their gameplay experience. Once all the wires, belts, and headsets were removed the participant would fill in the last questionnaires: STAI, PANAS, PENS, IMI, VR sickness, and grade DEEP.
4.4 Data Analysis
All the responses collected with Qualtrics were exported to a .csv file to perform pre-process statistical analyses. Shapiro-Wilk tests were run for all dependent variables to test for normality. The Shapiro-Wilk tests revealed that post STAI, STAI difference, and post IMI scores were not normally distributed. Appropriate statistical tests were used based on the results of the normality assessment. Non-parametric tests and a generalized linear mixed model were utilised to assess data that were not normally distributed. 95% confidence intervals were used for all statistical analyses.
Independent t-tests and Chi-squared tests found no significant differences at baseline between the two groups based on: age (t = -0.09, p = .93), gender (X2 = 3.23, p = .07), educational level (X2 =
0.41, p = .52), or STAI scores (t = 0.42, p = .68) before playing DEEP.
5. RESULTS
5.1 Effect DEEP On Anxiety
A Wilcoxon Signed Ranks test was conducted to assess if there was a decrease between self-reported state-anxiety (i.e., STAI-) scores before and after playing DEEP in the entire sample. The mean STAI score before playing DEEP was 32 (SD = 5.96) the mean STAI score after playing DEEP was 30.59 (SD = 7.87). The non-parametric Wilcoxon Signed Ranks test showed a marginally significant (Z = -1.92, p =.06) decrease in self-reported state-anxiety score after playing DEEP.
Furthermore, a Mann-Whitney U test was conducted to assess whether there was a significant difference in STAI scores (STAI score before playing minus the STAI score after playing DEEP) between the two groups. The non-serious trailer group had a mean STAI difference of -1.67 (SD = 7.50). The serious trailer group had a mean STAI difference of -1.15 (SD = 7.14). The Mann-Whitney U test indicated no significant dissimilarity (Z = -0.12, p = .91) in STAI difference between the two groups.
5.2 Intrinsic Motivation to Play
An independent samples t-test was conducted to assess potential differences in IMI scores between the two groups. The IMI score (1-7) before playing DEEP indicates the participant’s willingness to play the game after they have seen the trailer. The serious trailer group had a mean IMI score of 5.62 (SD = 0.86). The non-serious trailer group scored higher and had a mean IMI score of 6.08 (SD = 0.61). The t-test indicated a marginally significant difference (t = -1.94, p =.06) between the two groups.
5.3 Player Experience of Need Satisfaction
Independent samples t-tests were conducted to assess whether there were significant differences between the two groups with regard to in-game competence, in-game autonomy, in-game presence/immersion, and intuitive controls. As depicted in Table 1 there were no significant differences found between the two groups.
6
Table 1. Player experience of need satisfaction outcomes. Serious trailer (N = 20) Non-serious trailer (N = 21) Entire sample (N = 41) M (SD) M (SD) t p-value M (SD) Competence 4.22 (1.33) 4.14 (1.29) -.18 .86 4.18 (1.29) Autonomy 4.25 (1.19) 4.56 (1.28) .79 .44 4.41 (1.23) Presence / immersion 4.20 (1.24) 4.40 (1.17) .54 .59 4.30 (1.19) Intuitive controls 5.13 (1.31) 4.54 (1.25) -1.49 .15 4.83 (1.30)
5.4 Correlation Self-reported State-anxiety
and In-game Autonomy
A generalized linear mixed model (GLMM) was constructed in R [40] to assess potential correlation between self-reported state-anxiety outcomes, the two trailers, in-game autonomy, and the two moments in time which represent the time points when the STAI questionnaires were filled in.
gmler(anxiety ~ autonomy + trailer + moment + (1|StudyID), family = poisson)
Glmer is a function provided in the lme4 package in R [41], which performs the GLMM calculation. Anxiety represents the two STAI scores of all participants, trailer are the two trailers, moment indicates the two moments where the STAI questionnaires were filled in. (1|StudyID) accounts for the randomness, at baseline, of the different participants, with regard to STAI scores. As both STAI scores are considered count data the Poisson distribution was chosen for this function’s outcome.
A likelihood ratio test was conducted to assess whether there was an interaction between perceived in-game autonomy and the trailers. The full model with the interaction added was tested against the model without the interaction. The test indicated a p-value of .70 (X2 = 0.15), which states that the null hypothesis (no
interaction between autonomy and trailer) cannot be rejected. Therefore, the aforementioned function does not include an interaction between autonomy and trailer. Visual inspection of residual plots did not showcase obvious deviation from normality.
Table 2. GLMM fixed effects outcomes.
Fixed effects Estimator (β) Z-value p-value (z-test)
Autonomy -0.048 -2.04 0.04
Trailer -0.034 -0.60 0.55
Moment -0.045 -1.15 0.25
The GLMM outcomes (Table 2) show that autonomy was found to be a predictor (Z = -2.04, p = .04) of anxiety reduction. The results show that for every +1 autonomy score, anxiety scores reduce with -0.048. The trailers (Z = -0.59, p = .55) and moments (Z = -1.15, p = .25) are not predictive for anxiety (STAI) outcomes.
5.5 Intrinsic Motivation and Evaluation of
DEEP
The overall mean IMI score for DEEP was 5.41 (SD = 1.18). A Mann-Whitney U test was conducted to assess whether there was a difference in IMI scores between the two groups. The mean IMI score for the serious trailer group was 5.52 (SD = 0.91). The mean IMI score for the non-serious trailer group was 5.30 (SD = 1.41). The results of the Mann-Whitney U test indicated that there is no
significant difference (Z = -0.08, p = .94) between the two groups with regard to intrinsic motivation after playing DEEP.
The average grade that DEEP received from the participants was a 6.88 (SD = 1.50). The non-serious trailer group gave the game a mean grade of 7.10 (SD = 1.7), whereas the serious game trailer group gave a mean grade of 6.65 (SD = 1.27). A Chi-squared test was utilised to test for a significant difference between the two trailer groups with regard to their grading. No significant differences (X2 = 5.96, p = .43) were found between the two groups.
One participant reported experiencing nausea whilst playing DEEP. The other 40 participants (98%) did not experience any nausea during DEEP gameplay. The mean score of VR sickness was 1.41 (SD = 0.90) on a scale from 1 to 7.
5.5.1 Interviews
Directly after ten minutes of playing DEEP 36 (88%) out of the 41 participants described their experience with DEEP in a positive manner. Responses such as: ‘I found it special, it was different’, ‘Pretty cool, very immersive’, and ‘Pretty, it’s really a world to explore’ described the participants’ positive experiences. Four participants used terms such as relaxing or calming to describe their experience. Three participants explicitly stated that they found the experience not calming.
Movement through diaphragmatic breathing was described as difficult by 30 (73%) out of 41 participants. Participants gave responses such as: ‘I did have the feeling that I had to inhale deeply if I wanted to go up. That didn't always work out’ and ‘I kept sinking to the bottom. I made little hops every time. It didn't go as I wanted’. These responses correlated with the participant behavioural findings in the logbook (e.g., participants holding in their breath for an extended period of time). Additionally, 15 participants described how they felt that they had cheated or differed from instructions in order to move more freely. Participants described their ‘cheating’ among others as: ‘It felt a bit like cheating’, ‘I felt like I had to cheat a little to go up’, and ‘…in the end it was more a matter of tightening my belly and relaxing than actual breathing’.
5.5.2 Logbook Findings
The map of the game is relatively small. There is a starting point, a field, and platforms that lead to another field. In between the two fields, where the platforms are located, there is a vast open space where players can ‘fall’ into a deep pit. In this pit, the aptly appointed ‘deep sea’, there are no visual or interactive elements, other than small white specks (Figure 6).
Figure 6. Screenshot of ‘deep sea’ area.
Logbook observations revealed that 14 participants (34%) spent at least one fourth of their gameplay duration (≥ 2:30 minutes) in this environment. This ‘deep sea’ environment was not in the version utilised in the pilot study by van Rooij et al. (2016) [20].
7
6. DISCUSSION
6.1 Interpretation of the Results
Marginal significant differences were found when comparing the self-reported anxiety levels, in the entire sample, before and after playing DEEP and intrinsic motivation to play between the two groups. The GLMM indicated that perceived in-game autonomy in DEEP could be a significant predictor for anxiety reduction. This study found no significant differences between the two trailers groups with regard to self-reported anxiety levels before and after playing DEEP, need satisfaction, intrinsic motivation after playing DEEP, and overall assessment of DEEP.
6.1.1 DEEP and Its Effects on Anxiety
A marginal significant decrease in self-reported state-anxiety was found within the entire population. Albeit a minor mean self-reported state-anxiety reduction (-1.41, STAI scores can range from 20-80), the result does show indication that only a ten minute DEEP session can result in anxiety reduction. There was no difference found between the groups with regard to STAI outcomes, i.e. the trailers do not differ in their effect on anxiety outcomes.
6.1.2 Intrinsic Motivation to Play DEEP
A marginal significant difference was found between the two groups with regard to motivation to play. As expected, the non-serious trailer group was more intrinsically motivated (M = 6.08) to play the game in comparison to the serious trailer group (M = 5.62). This result indicates that trailers indeed have an effect on the player’s motivation to play. Heavily instructive or ‘serious’ trailers can result in less motivation which in turn could influence the game’s outcomes.
6.1.3 Player Experience Need Satisfaction in DEEP
In-game autonomy, in-game competence, presence/immersion, and intuitive control scores were all, as expected, lower in the serious trailer group than the scores of the non-serious trailer group. However, none of these differences were statistically significant.
6.1.3.1 In-game Autonomy
A potential reason for this lack of in-game autonomy difference could be that the non-serious trailer unintentionally provided instruction. The non-serious trailer states that the game is ‘designed to calm and soothe’. This slogan was from the original trailer and kept in because its message was not believed to be instructive or serious. Upon reflection this message does indeed provide some form of instruction or imposed goal to the player; the player is aware there is an aim. This reveals that both trailers provided (unintentional) imposed goals, which could be a cause of the lack of differences found between the two trailer groups.
Additionally, the player in this research setting was attached to multiple wires. Especially the ECG leads restricted the player’s freedom of movement. This could also have contributed to feeling less free (autonomous) in in-game decisions (i.e., moving and looking around freely).
6.1.3.2 Other Player Experience Needs
An interesting find was the somewhat larger difference between the two groups’ intuitive controls scores. The players in the serious trailer group found the controls more intuitive than those in the non-serious trailer group. It can be hypothesised that because the players in the serious trailer group are aware of the goals of the game, they are more aware of the underlying intention of the movement controls (i.e., diaphragmatic breathing). No direct emphasis was put on the importance of the breathing controls in the non-serious trailer.
6.1.4 Effect of In-game Autonomy on Anxiety
The GLMM revealed post-hoc that autonomy could serve as a predictor for anxiety reduction. This result should be treated with some caution. The relatively small sample did not allow for determination of the presence or absence of the heteroscedasticity assumption [42, 43]. Additionally, the -0.04781 decrease in anxiety for every autonomy point is not a large decrease given the fact that anxiety scores can range between 20 and 80. However, this outcome does corroborate with the belief that outcomes will improve as a result from autonomous motivation. This result also coincides with the marginal significant decrease found in anxiety levels in the entire sample.
No interaction was found between the trailers and autonomy outcomes. This was expected as the t-test result showed that there was no difference in in-game autonomy scores between the two trailer groups. The GLMM also showcased that the trailers did not have effect on anxiety outcomes. This was confirmed by the Mann-Whitney U test result, which showed that there was no significant difference in anxiety scores between the two groups.
6.1.5 Evaluation of DEEP
The mean overall IMI score (5.41) was relatively high, which would suggest that DEEP elicits high intrinsic motivation. No differences were found between the two groups. This would suggest that, regardless of which trailer was shown, playing DEEP results in approximately the same level of intrinsic motivation in both groups.
The grade given to DEEP did not significantly differ between the two groups. The overall mean grade of 6.88 suggests that the game is regarded as a positive experience, but there is room for improvement. Areas of improvement can easily be deduced, as 73% of the participants had difficulty with the breathing controls and 34% spent one fourth of their game session in an unstimulating environment.
6.1.5.1 Gameplay Experiences
Experience reports from the participants indicate that 88% of the participants found playing DEEP a positive experience. The overall mean score participants gave DEEP was a 5.41 on the enjoyment IMI subscale. Such numbers indicate that the game DEEP is found to be enjoyable. A 5.41 out of a scale of 7 indicates that enjoyment of the game is relatively high. Such a score can be evaluated as relatively high, as it is nearly impossible to create a game that appeals to everyone’s individual preferences.
VR sickness results show that DEEP can be easily be implemented as a future tool, as a vast majority (98%) did not experience any nausea. These results conform to studies that suggest that diaphragmatic breathing can be used to decrease motion sickness symptoms [44, 45].
6.1.5.2 Calming Effect
A remarkable finding was that only four participants explicitly said that they found the game either calming or relaxing. Additionally, three participants stated that they did not find that the game had a calming effect. Both trailers indicated (to a different extent) that the game had a calming effect, therefore finding these contradicting responses indicates that the current version of the game does not primarily provide a calming effect to its players. The small decrease in STAI scores before and after playing DEEP conforms to these findings. This lack in calming effects could be attributed to difficulty players had mastering the breathing controls.
8
6.1.5.3 Breathing Controls
A large group (73%) within the sample indicated that they found the movement through breathing difficult to master. Subsequently, 36% of the participants stated that they tried different methods to move more freely in the game (e.g., holding in breath, tensing abdominal muscles). Logbook observations also indicated that nearly all participants performed abnormal breathing patterns to control the game. All these findings cannot be traced back to the intuitive control scores, which were not particularly low (M = 4.83). As was elucidated in the introduction the belt controls movement via four ‘breathing states’. It appeared that the states: ‘<50% lung capacity and breathing in: player rises slowly’ and ‘>50% lung capacity and not breathing in: player float’ did not seem to properly function during the game sessions. It appeared that the gravitational pull on the player was too much. Participants would sink rapidly to the floor during exhalation and could only move forward and upwards via inhalation. This would explain the behaviour and responses observed; where participants held in the breath on purpose or tensed their abdominal muscles.
Movement through breathing is the most prominent feature of the game and is the core mechanism that could make the game a tool for anxiety prevention or treatment. The struggle that a large group of participants had with these controls could have influenced the in-game and post DEEP in-gameplay results reported in this study. For example, the ten minute game session did result in a small reduction in STAI outcomes, however it is plausible that if the breathing controls were easier to perform this reduction could have been more significant.
6.1.5.4 DEEP Environment
One third of the participants (34%) spent at least 2.30 minutes in the ‘deep sea’ area. Once a participant falls into the ‘deep sea’ a lot of strength is required to get back up on the platforms. Exploring the ‘deep sea’ is an unfulfilling task as there is only an invisible barrier to find which marks the edge of the map. Spending a significant amount of time in that space could lead to frustration and boredom. Provided that 34% of the participants spent a significant amount of time in that space, its potential effects on the player’s experience should not be ruled out.
6.2 Limitations
There are a number of limitations to this study. This study was conducted within a very limited timeframe of three months. Such a tight timeframe does not allow for elaborate testing of the game, set-up, and questionnaires. It also restricted the number of participants that were included in this study. A small sample size increases the chance of non-normally distributed data and allows the possibility that this population does not adequately present the general population. Consequently, non-parametric tests had to be utilised to answer a number of research questions. These tests are known to be less robust than parametric tests [46].
The study conducted for this thesis was also an appointed pilot study for the DEEP research project. This meant that a lot of data was gathered, but not everything was within the scope of this thesis. The additional data collection could have interfered with the data collection needed for this study. For example, the ECG leads attached to the participant’s wrists and ankles restricted the participants in his or her movements, which could have decreased their perceived in-game autonomy.
Given the timeframe, the scale and nature of the study, we did not search for participants diagnosed with (sub-clinical or clinical) anxiety. However, in order to fully assess the effects of DEEP as a
prevention or therapy tool, the intervention should be tested with participants who suffer from anxiety. Furthermore, the lack of a control group limits this study’s implications with regard to anxiety outcomes.
Lastly, this version of the game DEEP differed from the version that was used for the pilot study of van Rooij et al. (2016) [20]. It differed in environment, but also the ability for the player to manage the breathing controls. A large number of participants struggled with the breathing controls and preferably this had been tested more thoroughly before starting the data collection phase. The difference in the game control settings would explain the variance in results between the two studies and potentially the lack of distinct outcomes within this study.
6.3 Future Development of DEEP
There are a few bugs within the game that should be easily fixed in newer versions (e.g., moving through walls). Additional observations within this study resulted in a number of recommendations for further future development of the game DEEP.
6.3.1 Improvement Breathing Controls
The breathing control was found to be difficult by many and should be made easier to master. We recommend that the player should not be ‘punished’ by breathing out (i.e., sinking rapidly) and only awarded for breathing in and holding it for an extended period of time. Instead of sinking the player should stay afloat or not sink as rapidly. Such an adjustment would improve the players’ (relaxing) experience and potentially certain outcomes measured in this study.
6.3.2 No More ‘Deep Sea’
The drop between the platforms and the bottom of the ‘deep sea’ is, as or right now, too big. It takes strenuous effort from the player to get back on the platform, which does not improve the players’ calming experience. The ‘deep sea’ should not cease to exist. If the area of the ‘deep sea’ were to be removed, the game would become very small. Instead it should be enriched with more interactive elements such as other fish types, rock formations, and rings to swim through. This space should allow players, if desired, to continuously explore new areas. Some participants remarked that they expected certain game elements (e.g., in-game goals or points) during the game. Whether and how such game elements should be added cannot be extracted from this study.
6.3.3 Future trailers
This study has shown that trailers can have an impact on the player’s willingness to play the game. A trailer that showcases the ‘seriousness’ of the game results in a lower motivation to play as opposed to a non-serious trailer. Future instruction via means of trailers should stay clear of heavily instructive claims. However, the non-serious trailer also (unintentionally) provided an imposed goal (i.e., ‘designed to calm and soothe’). Moreover, no significant differences were found between the groups with regard to need satisfaction, intrinsic motivation, and anxiety outcomes. The non-serious trailer designed for this study could therefore be a stepping stone for constructing a new trailer that still manages to inform the player adequately, but does not heavily instruct the player about the game’s aims.
6.3.4 More Autonomy
As derived from the GLMM; in-game autonomy could be a predictor for anxiety reduction. In this study the mean perceived in-game autonomy was 4.41, which leaves room for improvement. More areas to explore, freedom of movement (e.g. no restrictive
9 wires), and easier breathing controls could have an impact on improving in-game autonomy. Increasing in-game autonomy could strengthen the anxiety reduction effects of DEEP.
6.4 Future Research
Future research should be conducted to further assess the marginally significant findings within this study. A future randomized controlled trial should be conducted to fully assess the efficacy of DEEP on anxiety reduction. Furthermore, future research should further assess the impact of in-game autonomy on anxiety reduction. Another study should be conducted in which the non-serious trailer is free of any imposed goal. If a significant difference were to be found between two trailer groups then this could support the hypothesis that imposed goals, via means of a trailer, reduce in-game autonomy.
All future study designs should include a larger sample size. Moreover, the effect DEEP has on anxiety should also be tested within a sample that suffers from (sub-clinical or clinical) anxiety. Additionally, future study designs should be more focussed on tackling independent research aims. In doing so, different data assessment methods cannot interfere with one another. Moreover, an updated version of DEEP with improved breathing control mechanism and no more ‘deep sea’ should be utilised in all future study designs
Lastly, additional studies should focus on user experience and test the game’s mechanics. User experience was not an aim within this study, ad-hoc data was gathered via observation and interview responses. This study cannot provide an overall image of the user’s experience however the data that was gathered was very useful to explain certain outcomes. That is why a future study should structurally asses user experience.
7. CONCLUSION
This research concludes that playing a ten minute DEEP session could result in anxiety reduction, demonstrating its potential as an intervention of people with anxiety problems. Second, there is an indication that a serious trailer results in players wanting to play the game less when compared to those who have seen a non-serious trailer. Third, post-hoc research via a generalized linear mixed model revealed that in-game autonomy could potentially be a predictor for reduced anxiety outcomes. Lastly, the utilization of a serious trailer compared to a non-serious trailer does not result in significant differences between in-game autonomy, in-game competence, in-game presence/immersion, intuitive controls, intrinsic motivation, and grades given to DEEP.
This study has served its purpose to provide insights into how to further develop and research DEEP. Future research and development of the game is needed to fully assess its implications with regard to intrinsic motivation and anxiety reduction. Future development should focus on improving the breathing controls which were deemed too difficult by the participants. Additionally, empty areas within the game should be filled with more interactive elements. An updated version of game where the previously mentioned improvements are integrated should be used in future study designs. Future research should take place in three domains: validation of DEEP’s effects on anxiety reduction, assessing psychological need assessment (with special interest in in-game autonomy) and intrinsic motivation, and user experience tests. This study marked the beginning of a research trajectory with regard to DEEP’s efficacy and further future development. DEEP showcases potential as a serious game for prevention and treatment of anxiety. Satisfying in-game psychological needs, specifically
autonomy, may increase DEEP’s beneficial effects. Continuous research, user testing, and improvements are facets that need to be addressed in order to realise a future version of DEEP as an appealing and valuable tool for anxiety problems.
8. ACKNOWLEDGEMENTS
A number of people deserve gratitude for their valuable contribution to this thesis: Marieke van Rooij, for guidance, believing in me, and offering me this opportunity. Joanneke Weerdmeester, for helping me every step along the way. You helped with practical issues, but you were also someone I could talk to in times of need. Frank Nack, for being reassuring in times when I needed just that. My parents, for helping me, despite all trials and tribulations, to become the best person I can be. Finally, the author of this thesis would like to thank everyone who participated in this study. Without you this study would not have been possible.
9. REFERENCES
[1] Laamarti, F., Eid, M. and Saddik, A. E. 2014. An overview of serious games. International Journal of Computer Games Technology, 2014 (2014), 11.
[2] Granic, I., Lobel, A. and Engels, R. C. 2014. The benefits of playing video games. American Psychologist, 69, 1 (2014), 66. [3] Kueider, A. M., Parisi, J. M., Gross, A. L. and Rebok, G. W. 2012. Computerized cognitive training with older adults: a systematic review. PloS one, 7, 7 (2012), e40588.
[4] Gamito, P., Oliveira, J., Lopes, P., Brito, R., Morais, D., Silva, D., Silva, A., Rebelo, S., Bastos, M. and Deus, A. 2014. Executive functioning in alcoholics following an mHealth cognitive stimulation program: randomized controlled trial. Journal of medical Internet research, 16, 4 (2014), e102.
[5] Prange, G. B., Kottink, A. I., Buurke, J. H., Eckhardt, M. M., van Keulen-Rouweler, B. J., Ribbers, G. M. and Rietman, J. S. 2015. The Effect of Arm Support Combined With Rehabilitation Games on Upper-Extremity Function in Subacute Stroke A Randomized Controlled Trial. Neurorehabilitation and neural repair, 29, 2 (2015), 174-182.
[6] Majumdar, D., Koch, P. A., Lee, H., Contento, I. R., Islas-Ramos, A. d. L. and Fu, D. 2013. “Creature-101”: A Serious Game to Promote Energy Balance-Related Behaviors Among Middle School Adolescents. GAMES FOR HEALTH: Research, Development, and Clinical Applications, 2, 5 (2013), 280-290. [7] Kato, P. M., Cole, S. W., Bradlyn, A. S. and Pollock, B. H. 2008. A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics, 122, 2 (2008), e305-e317.
[8] Forsberg, A., Nilsagård, Y. and Boström, K. 2015. Perceptions of using videogames in rehabilitation: a dual perspective of people with multiple sclerosis and physiotherapists. Disability and rehabilitation, 37, 4 (2015), 338-344.
[9] Garris, R., Ahlers, R. and Driskell, J. E. 2002. Games, motivation, and learning: A research and practice model. Simulation & gaming, 33, 4 (2002), 441-467.
[10] Malone, T. W. 1981. Toward a theory of intrinsically motivating instruction. Cognitive science, 5, 4 (1981), 333-369. [11] Wouters, P., Van Nimwegen, C., Van Oostendorp, H. and Van Der Spek, E. D. 2013. A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105, 2 (2013), 249.
10 [12] Ryan, R. M., Patrick, H., Deci, E. L. and Williams, G. C. 2008. Facilitating health behaviour change and its maintenance: Interventions based on self-determination theory. European Health Psychologist, 10, 1 (2008), 2-5.
[13] Ryan, R. M., Rigby, C. S. and Przybylski, A. 2006. The motivational pull of video games: A self-determination theory approach. Motivation and emotion, 30, 4 (2006), 344-360. [14] Zuroff, D. C., Koestner, R., Moskowitz, D., McBride, C., Marshall, M. and Bagby, M. R. 2007. Autonomous motivation for therapy: A new common factor in brief treatments for depression. Psychotherapy Research, 17, 2 (2007), 137-147.
[15] Ryan, R. M. and Deci, E. L. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55, 1 (2000), 68.
[16] Entertaint Software Association 2015. 2015 Sales, Demographics and Usage Data - Essential facts about the computer and video game industry. ESA, Washington, 2015. [17] Deci, E. L. and Ryan, R. M. 2008. Self-determination theory: A macrotheory of human motivation, development, and health. Canadian Psychology/Psychologie canadienne, 49, 3 (2008), 182. [18] Harris, O. and Smit, N. 2016. Explore DEEP. [01-03-2016]. http://www.exploredeep.com/#about-deep.
[19] Harris, O., Smit, N., Granic, I., van Rooij, M. and Weerdmeester, J. 2015. Stimuleringsfonds Open Call Applied Games - DEEP A game for Mental Health 2015.
[20] van Rooij, M., Lobel, A., Harris, O., Smit, N. and Granic, I. 2016. DEEP: A Biofeedback Virtual Reality Game for Children At-risk for Anxiety (2016). ACM.
[21] Spielberger, C. D. 2010. Test anxiety inventory. Wiley Online Library.
[22] Watson, D., Clark, L. A. and Tellegen, A. 1988. Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of personality and social psychology, 54, 6 (1988), 1063.
[23] Poppelaars, M., Lichtwarck-Aschoff, A., van Rooij, M. M. J. W., Dibeklioglu, H. and Granic, I. 2016. Let’s play together! Motivation, Emotion & Depressive Symptoms in a Cooperative Video Game. Behavioural Science Institute.
[24] Coco & Co 2016. Way - Coco & Co. [15-04-2016]. http://www.makeourway.com/.
[25] Han, J., Stegen, K., De Valck, C., Clement, J. and Van de Woestijne, K. 1996. Influence of breathing therapy on complaints, anxiety and breathing pattern in patients with hyperventilation syndrome and anxiety disorders. Journal of psychosomatic research, 41, 5 (1996), 481-493.
[26] Wetherell, J. L. 1998. Treatment of anxiety in older adults. Psychotherapy: Theory, Research, Practice, Training, 35, 4 (1998), 444.
[27] Stanley, M. A., Diefenbach, G. J. and Hopko, D. R. 2004. Cognitive Behavioral Treatment for Older Adults with Generalized Anxiety Disorder A Therapist Manual for Primary Care Settings. Behavior Modification, 28, 1 (2004), 73-117.
[28] Liza, V. 2011. Stress management techniques: evidence-based procedures that reduce stress and promote health. Health Science Journal (2011).
[29] McLean, P. D. and Woody, S. R. 2000.Anxiety disorders in adults: An evidence-based approach to psychological treatment. Oxford University Press.
[30] Qualtrics LLC 2016. The World's Leading Research & Insights Platform | Qualtrics. [21-04-2016]. https://www.qualtrics.com/.
[31] Immersyve, I. 2016. Immersyve - Science of Engagement. [06-06-2016]. http://immersyve.com/.
[32] Plant, R. W. and Ryan, R. M. 1985. Intrinsic motivation and the effects of self‐consciousness, self‐awareness, and ego‐ involvement: An investigation of internally controlling styles. Journal of Personality, 53, 3 (1985), 435-449.
[33] Polcar, J. and Horesji, P. 2013. Knowledge Acquisition And Cyber Sickness: A Comparison Of VR Devices In Virtual Tours. Science (2013).
[34] Davis, S., Nesbitt, K. and Nalivaiko, E. 2014. A systematic review of cybersickness (2014). ACM.
[35] Adafruit 2016. FLORA - Wearable electronic platform: Arduino-compatible - v2. [30-06-2016]. https://www.adafruit.com/product/659.
[36] Arduino 2016. Arduino Software. [14-05-2016]. https://www.arduino.cc/en/Main/Software.
[37] LLC, O. V. 2016. Oculus Rift | Oculus. [03-03-2016]. https://www.oculus.com/en-us/rift/.
[38] BIOPAC Systems Inc. 2016. Data Acquisition, Loggers, Amplifiers, Transducers, Electrodes | BIOPAC. [29-04-2016]. http://www.biopac.com/.
[39] Merrett, F. 2006. Reflections on the Hawthorne effect. Educational Psychology, 26, 1 (2006), 143-146.
[40] R Core Team 2013. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, 2013.
[41] Bates, D., Machler, M., Bolker, B. and Walker, S. 2015. Fitting Linear Mixed-Effects Models Using {lme4}. Journal of Statistical Software, 67, 1 (2015), 47.
[42] Winter, B. 2013. Linear models and linear mixed effects models in R: Tutorial 11. arXiv preprint arXiv:1308.5499 (2013). [43] Winter, B. 2013. A very basic tutorial for performing linear mixed effects analyses. arXiv preprint arXiv:1308.5499 (2013). [44] Stromberg, S. E., Russell, M. E. and Carlson, C. R. 2015. Diaphragmatic breathing and its effectiveness for the management of motion sickness. Aerospace medicine and human performance, 86, 5 (May 2015), 452-457.
[45] Russell, M. E., Hoffman, B., Stromberg, S. and Carlson, C. R. 2014. Use of controlled diaphragmatic breathing for the management of motion sickness in a virtual reality environment. Applied psychophysiology and biofeedback, 39, 3-4 (Dec 2014), 269-277.
[46] Petrie, A. and Sabin, C. 2013. Medical statistics at a glance. John Wiley & Sons.
