Master Thesis Clinical Neuropsychology
Faculty of Behavioural and Social Sciences – Leiden University (May, 2017)
Student number: 1377485
Daily Supervisor: dr. van der Ham, Department of Health, Medical and Neuropsychology; Leiden University
CNP-co-evaluator: dr. Habers, Department of Health, Medical and Neuropsychology; Leiden University
The effect of computer experience and performance
on appreciation of a serious game in brain-injured
patients
Contents Abstract……….. Preface………….……… Introduction………... Method...………. Participants……….. Materials………. Questionnaires………. Games………. Thinking aloud protocol……….. Design and procedure……… Statistical analysis………. Results……….. Participants………..………. Reliability………. Hypothesis 1: computer experience as predictor of appreciation of a serious game... Hypothesis 2: performance as predictor of appreciation of a serious game………….. Hypothesis 3: the effect of computer experience and performance on appreciation of a serious game………. Exploratory analysis……… Difference in time……… Controls……… Discussion……… References……… 3 4 5 9 9 9 10 11 12 12 13 15 15 15 15 17 17 19 19 19 19 23
Abstract
Brain-injured patients can suffer from navigation impairments, which decrease their quality of life. Currently, no standardized treatment for navigation impairment has been developed. In this study the usability of a serious game (Wayfinder) which is developed to improve navigation skills is investigated. This study investigated whether users’ computer experience and performance in a keyboard-controlled 3D movement task have an effect on the appreciation of the serious game. If computer experience and performance prove good
predictors for the appreciation of the serious game, the game should be adapted to these findings, to increase its usability. When individuals are more satisfied about the ease of use of the serious game, and their appreciation is high, the game could be more effective.
Patients with acquired brain injury completed a questionnaire to assess their level of computer experience. In addition, patients played a keyboard-controlled 3D movement task in Wayfinder at the beginning of the study and at the end, and the decrease in time between the pretest and posttest was calculated to measure performance. Furthermore, after the patients had played Wayfinder they completed a questionnaire about their appreciation of the serious game. Firstly, a positive relation between computer experience and appreciation of the serious game was found: the higher the level of computer experience, the more positive the
appreciation of the serious game. This means that computer experience might be a good predictor for the appreciation of the serious game. In follow-up studies and usage of the serious game in brain-injured patients it would be advisable to ensure a sufficient level of computer experience. Secondly, performance in a 3D movement task using the keyboard did not significantly affect the appreciation of the serious game. It is recommended that patients play the game using the computer mouse. The expectation is that the effectiveness of
Preface
This master thesis in Clinical Neuropsychology is performed internally within the Department of Health, Medical and Neuropsychology of Leiden University. The learning goal was to obtain experience with all the phases of research, such as studying scientific literature in the field of neuropsychology, selecting/constructing measures/instruments, data collection and data analysis, as well as writing an academic report. For me, a highlight of this thesis project was having contact with brain-injured patients. Some patients were more taxable than others, which makes the research fascinating. Furthermore, it was interesting to help with the development of a study. Unfortunately, my planning was not correct, mainly as a consequence of external factors. For example, the period of data collection took longer than expected, as a result of lack of participants.
The study of this thesis is based on a larger study from PhD-student Milan van der Kuil. Therefore, I would like to express my gratitude to Milan van der Kuil for his support and trust in me. In addition, I would like to thank my supervisor Ineke van der Ham for the useful help and engagement through the learning process of my thesis. Her door was always open. I would also like to thank Maria Sherwood-Smith for her help with academic writing in English. Finally, I express my gratitude to the participants of this study for their contribution to scientific research.
Introduction
An essential element in human survivalis finding the way in an environment. If an individual’s internal navigation system does not work sufficiently, this has severe
consequences, such as not being able to find the way home (Lemoncello, Sohlberg & Fickas, 2010). This can lead to a decrease in social relationships, and problems in occupational functioning (Hort et al., 2007; Lemoncello et al., 2010). Research has shown that almost a third (29%) of patients with mild stroke report impairment in navigation ability (van der Ham, Kant, Postma & Visser-Meily, 2013). As a result, these patients can no longer operate
independently and their quality of life decreases (van der Ham et al., 2013).
Patients with acquired brain injury, including stroke patients, can be helped by developing a rehabilitation training to improve their navigation skills (Lemoncello et al., 2010; Livingstone & Skelton, 2007). Currently, no standardized treatment for navigation impairment is available. A rehabilitation training which improves navigation skills may be provided using serious games. Serious games are computer games, which can improve skills and increase knowledge (DeSmet et al., 2014). Furthermore, serious games are used because they are cost-effective, and they are motivating in different target groups (Wouters, van Nimwegen, van Oostendorp & van der Spek, 2013). Certain key components determine the effectiveness of serious games: motivation, the target group, and the usability of the game. Motivation is an effective component, which can be increased because players typically enjoy playing serious games (Wouters et al., 2013). This is in contrast to learning for instance from a textbook, which is less motivating (Wouters et al., 2013). Research has shown that
motivation is positively related to performance (Hess & Gunter, 2013), what means that a serious game is effective when the user of the game is motivated. Furthermore, individuals learn faster from a serious game if their motivation is higher, and they consequently perform better (Orvis, Horn & Belanich, 2008). In addition, it is crucial that people enjoy the
performed game in the long term, so that gains can be maintained (Park & Bischof, 2013). Therefore, motivation is a key component that makes a serious game effective (Wouters et al., 2013). Another component is the target group, to which the serious game has to be adapted. The game should be challenging for the target group, but not too difficult (DeSmet et al., 2014). The degree of difficulty should be adapted to the estimated performance level of the target group. The serious game which serves as rehabilitation training in this study is intended for brain-injured patients who have been struggling with navigation. Thus, the serious game must have sufficient difficulty to challenge brain-injured patients and avoid ceiling effects,
but not be so difficult that floor effects occur in the frailest brain-injured patients (Park & Bischof, 2013). This is also explained by the flow theory, which suggests that a balance between task difficulty and skills is necessary (Moneta & Csikszentmihalyi, 1996; Ullen et al., 2012). In other words, the difficulty of the serious game has to be adapted to the skills of the brain-injured patients. Furthermore, research has demonstrated that serious games are more effective, when the usability of the game is easy (Virvou & Katsionis, 2008). Usability can be defined as the ability of the potential user to navigate easily through the game menu, and the users’ overall satisfaction or attitudes towards their experience (Reichlin et al., 2011). The main measure of usability is the self-reported perceptions of the user about the ease of use of the serious game (Hornbaeck, 2006; Reichlin et al., 2011). Usability is a key concept in human-computer interaction (Hornbaek, 2006), and can be increased when using
User-Centered Design (UCD) (Ebner & Holzinger, 2007; Verganti, 2008). UCD is a method that facilitates the development of a product, in this case the navigation serious game, using an analysis of user needs (Verganti, 2008). In short, serious games could be a suitable means for the development of a training which should improve the navigation skills of brain-injured patients. In this study, the usability of a navigation serious game will be evaluated, so that elements of the game could be adapted to the target group consisting of brain-injured patients. Consequently, the serious game could be more effective.
A serious game training which improves navigation skills is currently being developed, relying on recent scientific evidence. The name of the serious game is
‘Wayfinder’.Patients train their navigation skills in a virtual 3D environment, which is a safe learning environment (Claessen, van der Ham, Jagersma & Visser-Meily, 2016; Kober et al., 2013).
The aim of the present study is to evaluate Wayfinder and consequently to adapt game elements, so that brain-injured patients can train their navigation skills independently at home without assistance of a physician. The usability of the rehabilitation training should be high enough to make this possible. Multiple factors should be taken into account in investigating the usability of the serious game.For example, the interaction between the user and the game should be investigated, as explained by the International Classification of Functioning, Disability, and Health (ICF) model: a scheme that shows the interaction between the various components in the process of functioning and disability (Steiner et al., 2002). Personal factors, such as age, gender, and previous experience, are one of the components of the ICF model that affect the ease of use of the serious game (Sivan et al., 2014; Steiner et al., 2002). Orvis, Horn and Belanich (2008) suggest that people with more experience in the use of computers
are more satisfied with the usability of the game. In addition, individuals with a high level of computer experience are faster in learning from the serious game in comparison with
individuals with a lower level of computer experience (Orvis, Horn & Belanich, 2008). Consequently, the satisfaction about the game could increase and the effectiveness of the serious game will be affected (Orvis, Horn & Belanich, 2008). Therefore, this study will investigate the interaction between the level of computer experience and satisfaction of Wayfinder.
In addition, it is also important to consider the needs of the user in investigating the usability, as explained by UCD (Verganti, 2008). For example, when the user performs well during the game, they are more positive about the game then when they do not perform well (Fredrickson et al., 2010). Therefore, performance during the game could be an important element that should be taken into account in investigating the usability of the serious game.
In the study of Hou and Li (2014) multiple aspects of a serious game were evaluated; this included usability. The game they evaluated was Boom Room: a problem-solving-based educational adventure game, designed for students. Differences between the study of Hou and Li (2014) and the current study are the target group and the goal of the serious game. In the study of Hou and Li (2014) participants were healthy students in a serious game which improves computer skills, instead of brain-injured patients who play a serious game which improves navigation skills in the current study. One similarity between the two studies is that they both have the same purpose, namely to test the usability of a serious game. In the study of Hou and Li (2014) the usability was investigated using Likert scale questions about game acceptance and evaluations of game elements. Game acceptance was tested by research of perceived ease of use and perceived usefulness. The evaluated game elements were challenge, entertainment, and interactivity, as well as fantasy. Moreover, the participants provided evaluations of their flow experience of the game using a Likert scale questionnaire: the Flow State Scale (Jackson & March, 1996). Flow experience means a positive feeling, which occurs when the player of the game is totally connected to the game (Jackson & March, 1996). The questions in the study of Hou and Li (2014) derived from the different domains of the Flow State Scale. These domains include clearly defined goals, experiencing intrinsic reward, sense of control, concentration, and balance between the challenges of a situation and one’s skills (Jackson & March, 1996). The results in the study of Hou and Li (2014) demonstrate that game acceptance was sufficient and participants experienced dimensions of flow. Besides, the dimensions of flow were significantly correlated with game acceptance. A part of the
from the Flow State Scale (Jackson & March, 1996) are going to be used in the current study to measure the appreciation of the serious game. The selected questions from the
questionnaires are clearly and briefly formulated. This implies that they use everyday vocabulary and do not use long sentences (Maynard, Houtkoop-Steenstra, Schaeffer & van der Zouwen, 2002). Therefore, the questions are accessible for the target group consisting of brain-injured patients.
This study investigates effects on the appreciation of the serious game Wayfinder. When the appreciation, and therefore the usability, of the serious game is high, the game could be more effective. A high usability is also important so that brain-injured patients can train their navigation skills independently at home without assistance of a physician. Both the interaction between the user and the game, and the needs of the user are important factors that affect the appreciation. Computer experience and game performance may play a role in these aspects, and the serious game can be adapted to these components. This study investigates whether users’ computer experience and performance in a 3D movement task using the keyboard have an effect on the appreciation of the serious game Wayfinder. First, participants with a high level of computer experience are compared with participants with a lower level of experience on the appreciation of the serious game. Previous computer experience will be investigated on the basis of questions concerning computer skills and computer use. In the current study it is expected that participants with a higher level of experience will express a higher level of appreciation in comparison with participants with a lower level of computer experience (Orvis, Horn & Belanich, 2008). In addition, research has shown that game performance predicts the appreciation (Fredrickson et al., 2010; Hou & Li, 2014). In this study, game performance is measured by improving control of the arrow keys, on the basis of time to task completion (Hornbaek, 2006). The improvement in performance in a keyboard-controlled movement task is being measured using a pretest and posttest. The hypothesis is that participants who have a higher level of improvement in controlling the arrow keys during playing of the game give a better appreciation in comparison with participants who have a lower level of improvement.This hypothesis is supported by Fredrickson and his colleagues (2010) who demonstrated that the appreciation is higher as a result of a better performance of the participant.
In short, this study investigates whether users’ computer experience and performance in a movement task using the keyboard could be a good predictor for the appreciation of the serious game. Both the combination of the two predictors and the predictors independently of each other will be examined. The effects of the predictors are analyzed independently of each
other in two different sub-questions: ‘Does computer experience predict the appreciation of the serious game?’ and ‘Does performance in a keyboard-controlled 3D movement task predict the appreciation of the serious game?’ Finally, the main question includes the combination of the two predictors: ‘Has the combination of users’ computer experience and performance in a keyboard-controlled 3D movement task an effect on their appreciation of the serious game?’
Method Participants
The target group was patients with acquired brain injury, who were treated in a rehabilitation process in the UMC, a university hospital, in Utrecht. The inclusion criteria were that the participants were between 18 and 80 years old and they were able to use their left or right arm and hand. The participants should be sufficiently taxable, that was estimated by the treating physiatrist. Participants were excluded if they had severe visual impairments. The physiatrist asked the patients if they would like to participate in the examination of this study. The examination took place in the UMC in Utrecht.
One participant is excluded after participation in the study. As a consequence, this participant is not used in further data analysis. The participant is excluded because he had not completed the entire study as a result of lack of time. In that case, answers of the
questionnaire about the appreciation of the serious game are not completed and the posttest from the keyboard-controlled 3D movement task is not played. As a result, necessary data for the study is missing. Furthermore, three other participants are only excluded in the analyses related to the hypotheses in which performance is a predictor (Table 1). These participants did not complete the posttest from the keyboard-controlled 3D movement task.
In addition, all participants scored within three standard deviations of the mean in all of the three variables. This indicates that no outliers are detected. 14 participants are included in the dataset.
Materials
This study is a section of a larger study. Participants completed all questionnaires which are described below. Only parts of the questionnaires were used in the current study. The questionnaires which are used in this study are discussed in more depth, in comparison to questionnaires which are not used.
Questionnaires. The participants completed all questionnaires in Qualtrics: an online
method where questionnaires can be filled in on a computer. They completed a questionnaire consisting of nine questions about their computer skills and computer use, which measured computer experience. This questionnaire is important for the research question, in which computer experience is the predictor. Four questions are answered on a five-point scale from 1 (never) to 5 (daily) based on previous year, and five questions are answered on a five-point scale from 1 (totally disagree) to 5 (totally agree). The questionnaire was specially developed for this study. Certain questions were added relying on existing questionnaires and scientific literature. Some questions came from the Computer Attitude Scale (CAS) (Loyd & Gressard, 1984). Three types of attitudes are distinguished in the CAS, namely ‘anxiety of computers’, ‘enjoying working with computers’, and ‘confidence in ability to use computers’. The first type about computer anxiety is negative related to previous computer experience, whereas the other two types are significant positive correlated with computer experience (Coffin &
Maclntyre, 1999; Loyd & Gressard, 1984). The reliability of the total score of this
questionnaire is high with a coefficient alpha of .95 (Loyd & Gressard, 1984). The validity of the CAS is explained by significant positive correlations between computer self-efficacy and the three types of attitudes toward computer use described in the CAS (Compeau & Higgers, 1995) and significant negative correlation between age and computer experience using the CAS (Popedavis & Twing, 1991).
Aside from the discussed questionnaire about computer experience, participants also completed a questionnaire about controlling the arrow keys. The same questionnaire with the same questions about controlling the arrow keys was completed about controlling the
computer mouse. Furthermore, open questions were asked about these two opportunities of controlling. In addition, participants twice completed a questionnaire about the
comprehensibility of instructions. One time the participants were given instructions in text form and the other time they were given instructions in video form. Besides, participants were asked to answer open questions about these two kinds of instructions. Another questionnaire measured the usability of the menu from the serious game. Furthermore, a questionnaire was twice completed about motivation during the game. On one occasion participants completed the questionnaire after playing a game with direct feedback and the other occasion
participants completed the questionnaire after a game in which they received delayed
feedback. Finally, participants completed a questionnaire about the appreciation of the serious game. This questionnaire contains nine questions, which were answered on a five-point scale from 1 (totally disagree) to 5 (totally agree). Also in this case, the reliability and validity of
the questionnaire is not exactly known, because the questionnaire was specially developed for this study. The majority of the questions in the questionnaire of the current study is originally from different domains from the Flow State Scale. These domains include clearly defined goals, experiencing intrinsic reward, sense of control, concentration, and balance between the challenges of a situation and one’s skills. The Flow State Scale measures the experience of flow: a positive feeling, that occurs when the player of the game is totally connected to the game (Jackson & Marsh, 1996). The flow experience is related to game acceptance and in this way also to the appreciation of the serious game (Hou & Li, 2014). The internal reliability from the Flow State Scale is high with an alpha from .83. Beside, good support is found for the construct validity of Flow State Scale responses using factor analysis (Jackson & March, 1996). Summarized, the Flow State Scale seems to be a reliable and valid measuring
instrument. Aside from questions from the Flow State Scale, the developed questionnaire also includes questions about usability and motivating elements in the serious game to measure users’ appreciation about the game (Hou & Li, 2014; Sailer et al., 2017; Wouters et al., 2013).
Games. The serious game used in this study is Wayfinder. This game is developed to
improve navigation skills. There is no previous scientific research into Wayfinder. The serious game includes six short subgames. The current study used two subgames from Wayfinderthat the participants have actually played. The first game was ‘In the distance’, in which participants train their allocentric navigation with regard to distal landmarks.
Allocentric navigation means navigating to the destination using landmarks in the environment (Jordan et al., 2004; Mou et al., 2004). The other game that was played was ‘Pillars’. Just like ‘In the distance’, participants trained their allocentric navigation using ‘Pillars’, but this time with regard to local landmarks. Furthermore, this study used two subgames from Wayfinder. In this case only the instruction was used, and the games were therefore not played. These games were ‘Map’ and ‘Orientation’. ‘Map’ is a game in which the use of maps is trained during navigation. With ‘Orientation’ participants train egocentric bearing: preservation of orientation relative to a certain point. In addition, the participants played in connection with the current study ‘Input Controller Assessment’, a game in which the participants walk in a virtual 3D environment from a starting point to an end point using the arrow keys. As instruction they were told that the exercise must be performed as quickly as possible, without touching the walls in the game. The time in seconds between the starting point and end point is measured. Participants played this game three times: two times with the arrow keys and one time with the computer mouse.
Thinking aloud protocol.During this study the usability of the menu from Wayfinder was investigated. Participants were asked to think aloud when they navigate to the menu. This task is named ‘Thinking Aloud’. The purpose of the Thinking Aloud protocol is that the reasoning of the participant could be followed. In this way could be traced when some parts of the game menu is too difficult for the target group (Jaspers, 2009).
Design and procedure
In this study a correlational research design was used. One group of participants was compared on different variables.
Given the observational nature of this study, it was evaluated that the study does not fall under the Medical Research (Human Subjects) Act by the Medical Ethical Committee at Utrecht University.
Before the study started, participants signed the informed consent on paper to agree with participation. Then participants sat down at the computer and completed the
questionnaire about computer experience. The participants played ‘Input Controller
Assessment’: half of the participants controlled the game with the computer mouse and the other half with the arrow keys. Which participants were assigned to what condition was determined using pseudo-randomization to make sure all combinations of order were
included. Next, they completed the questionnaire about the controlling of the game. Then they played again ‘Input Controller Assessment’, but in a mirrored 3D environment with the other kind of control. Thus, participants who started with playing the game with the arrow keys controlled the game the second time with the computer mouse, and vice versa. After this, they completed the same questionnaire about the controlling of the game. Next, the participants were given text or video instructions for the subgame ‘Map’ or ‘Orientation’. A quarter of the included participants were first given text instructions for ‘Map’, another quarter were given text instructions for ‘Orientation’, again another quarter were first given video instructions for ‘Map’, and the residual quarter were given video instructions for ‘Orientation’. Which
participants became what condition was again determined using pseudo-randomization. After this, participants were given the other kind of instruction and the other game which they had not played. For example, when a participant was first given a video instruction for the subgame ‘Map’, the participant was then given a text instruction for ‘Orientation’. Next, the participants were instructed to seek a part of the game program, which the examiner asked for. Participants navigated through the game menu with the computer mouse. During the exercise, they had to think aloud. Next, the participants completed the questionnaire about the usability
of the game menu. After this, participants were asked to play the subgame ‘In the distance’ or ‘Pillars’. Pseudo-randomization was used to determine in what condition the participant was. Pseudo-randomization was also used to determine whether the participant received direct feedback during the game or delayed feedback at the end of the game they played. After this, the participants completed a questionnaire about motivation. Next, participants were given the other game they had not played before, and they received feedback at a different time. For example, when a participant played the first time ‘In the distance’ and received direct feedback, the second time the participants got ‘Pillars’ with delayed feedback. After this, participants completed a questionnaire about the appreciation of the serious game. Finally, the participants played again ‘Input Controller Assessment’ using the arrow keys. The total study session took about an hour.
Statistical analysis
The first sub-question about computer experience as possible predictor of the appreciation of the serious game is answered using a simple regression analysis. The mean score from the nine items from the questionnaire which measures the appreciation of the serious game is the dependent variable. In addition, the independent variable is the mean score of the nine items from the questionnaire which measures the level of computer
experience. The prediction is that a higher level of computer experience will result in a more positive appreciation of the serious game in comparison with a lower level of computer experience.
An assumption which is to be monitored is the linearity of the regression equation. Linearity may be controlled by making a plot from standardized predicted values on the x-axis and standardized residuals on the y-x-axis. Linearity and homoscedasticity are present when the points in the plot are totally randomly distributed. Another assumption is normality, in which the residuals should be normally distributed. A method to check this is to use a histogram. The assumption is met if there is a normal distribution. Finally, another
assumption is that the residuals are completely independent from each other. This assumption is difficult to check, but it can be assumed that the study is met this assumption automatically. Data of participants are collected independently from each other. For example, the
participants fill in their questionnaire alone and are isolated from each other during the experiment.
A simple regression analysis is also performed to answer the second sub-question about performance as possible predictor of appreciation of the serious game. Again, the mean
score from the nine items from the questionnaire about the appreciation of the serious game is the dependent variable. The independent variable is the measured time difference between playing ‘Input Controller Assessment’ with the arrow keys at the beginning of the study and at the end. Participants are going to play ‘Input Controller Assessment’ using the arrow keys at the beginning and end of the study. The time in seconds between the starting point and end point is measured. The difference between time in the pretest and time in the posttest is calculated. The calculated game performance is the independent variable of this second hypothesis. The expectation is that participants who have a higher level of performance in the keyboard-controlled movement task give a more positive appreciation in comparison with participants who have a lower level of performance. This means that the higher the decrease in time between the game at the beginning of the study and at the end, the better the
appreciation of the game.
The same assumptions apply as above, because the same statistical analysis is used. This means that the linearity of the regression equation must be checked, as well as the normality. The assumption about independence of errors is again difficult to check, but it can be assumed that the research is done independently.
The main question of this study investigates whether the combination of users’ computer experience and performance in a keyboard-controlled movement task has an effect on their appreciation of the serious game. Computer experience (measured with a
questionnaire) and performance (measured by calculating the decrease in time in seconds between pretest and posttest) are the independent variables, and the appreciation of the serious game (also measured with a questionnaire) is the dependent variable. The expectation is that the higher the level of computer experience and the higher the level of performance in time, the more positive the appreciation of the serious game. Furthermore, the expectation is also that this combination of predictors explained a higher variance of appreciation of the serious game, than the predictors independently of each other. A multiple regression analysis (MRA) should be used to test the combination of the used predictors and consequently to answer the whole main question. The two independent variables should be added at the same time, while doing an MRA.
Again, the same assumptions apply about the linearity, the normality and the independence. A linear relationship between the independent variables and the dependent variable should be present. Furthermore, the residuals have a normal distribution and are completely independent from each other.
The hypotheses from the sub-questions with one independent variable are unilaterally tested with an alpha from .05. The main hypothesis with the combination of predictors is bilaterally tested, also with an alpha from .05. IBM SPSS Statistics 21 is used to test the hypotheses.
Results Participants
14 participants (7 males and 7 females) were included in the dataset. The mean age was 47.29 years(SD = 13.78). The characteristics of the 14 participants are summarized in Table 1.
Reliability
The reliability of the dependent variable, namely the appreciation of the serious game, was measured using Cronbach’s alpha. Cronbach’s alpha is .872, which indicates a high level of internal consistency.
Furthermore, the reliability from the predictor computer experience is also measured. The questionnaire is highly reliable according to the data (α = .858).
Hypothesis 1: computer experience as predictor of appreciation of a serious game
In the study a simple regression analysis was used to investigate whether the level of computer experience predicts the level of appreciation of a serious game. The assumptions were checked. The connection between the dependent and independent variable is linear, and the points in the plot are totally randomly distributed. This means that the assumptions of linearity and homoscedasticity are met. Next, the residuals are normally distributed. Therefore, the assumption about normality is also met. Finally, it can be assumed that the residuals are completely independent from each other, which meets the last assumption. Data of participants are collected independently from each other.
The independent interval variable computer experience is compared to the dependent interval variable appreciation of the serious game. A significant regression equation was found (F (1, 12) = 12.158, p = .004, f² = 1.012). Furthermore, 50.3 percent of the variance of appreciation of the serious game in the sample is explained by the predictor computer experience (R² = .503). Beside, evidence for multicollinearity is not found (Tolerance = 1.000). In addition, the regression coefficients show that when the mean score of the
Table 1. Summary of Participant Characteristics about Gender, Current age in years, Age at onset in years, Etiology, and Location of brain damage.
Variable
Participants Gender Current age Age at onset Etiology Location
1 Male 31 31 Brain contusion Right frontal
2* Male 49 48 Brain tumor Left
parietal-occipital
3 Female 50 49 Brain tumor Right
frontal-parietal
4 Male 23 22 Stroke Basal ganglia
5 Female 49 48 Stroke Left middle
cerebral artery
6* Female 62 57 Stroke Cerebellum
7 Female 28 27 Brain contusion Frontal
8 Female 41 40 Stroke Cerebellum
9 Male 55 54 Brain contusion Left frontal
10 Male 46 45 OHCA** -
11 Female 63 63 Stroke Right frontal
12 Female 37 34 Stroke Left posterior
communicating artery
13* Male 64 63 Stroke Left pericallosal
artery
14 Male 64 63 Brain contusion Right frontal
*Participants who are excluded in the analyses related to the hypotheses in which performance is a predictor
** Out of Hospital Cardiac Arrest; brain damage as a result of cardio-pulmonary resuscitation
questionnaire about computer experience increases with 1.0 point that the mean score of the questionnaire about the appreciation of the serious game increases with .668 (b = .668, t (12) = 3.487, p = .004). Table 2 provides a summary about the relevant statistical values.
Figure 1 presents a plot which demonstrates the positive relation between computer experience and appreciation.
Figure 1. Positive relation between computer experience and appreciation
Hypothesis 2: performance as predictor of appreciation of a serious game
A simple regression analysis is used to investigate whether the level of performance predicts the level of appreciation of a serious game. Again, all assumptions from the simple regression analysis were checked. The assumptions of linearity and homoscedasticity are met, because the points in the plot are randomly distributed. Furthermore, the assumption about normality is also met, because the residuals are normally distributed. At last, it can be assumed that the residuals are completely independent from each other.
A simple regression analysis was calculated to predict appreciation of a serious game based on performance in keyboard-controlled movement task. No significant regression equation was found (F (1, 9) = .722, p = .418, f² = .080), with an R² of .074 (Table 2).
Hypothesis 3: the effect of computer experience and performance on appreciation of a serious game
In this study is investigated whether the combination of users’ computer experience and performance in a keyboard-controlled movement task has an effect on their appreciation of a serious game, using a simultaneous MRA. All assumptions from the MRA were checked, which were the same assumptions as from the simple regression analysis. The points in the scatterplot are randomly distributed, indicating that the assumptions about linearity and homoscedasticity are met. By making a histogram is shown that the residuals are normally
distributed. This means that the assumption of normality is met. Again, it can be assumed that the residuals are completely independent from each other, because data of participants are collected independently from each other.
A marginally significant regression equation was found between the combination of the two predictors and the dependent variable (F (2, 8) = 3.903, p = .066, f² = .976), with an R² of .494. However, the main effect of computer experience on the appreciation of the game is statistically significant (t (12) = 2.575, p = .033), with an R² of .420 (Part = .648).
Furthermore, the main effect of performance on appreciation is not significant (t (12) = -.671, p = .521), with an R² of .029 (Part = -.169).
Table 2 provides a summary about the relevant statistical values from the three hypotheses.
The minimum required sample size for the main question is calculated using the anticipated effect size (f² = .976), the desired statistical power level (= .80), the number of predictors (=2) and the probability level (=.05) (Cohen, Cohen, West & Aiken, 2013; Soper, 2017), and should be 14.
Table 2. Summary of statistical values from the predictors Computer Experience,
Performance, the combination of Computer Experience and Performance, the main effect of Computer Experience on Appreciation, and the main effect of Performance on Appreciation.
Predictor b Β R² T F P f² CEᵃ .668 .709 .503 3.487 12.158 .004* 1.012 Pᵇ -.017 -.272 .074 -.849 .722 .418 .080 CE + Pᶜ .494 3.903 .066** .976 mCEᵈ .931 .656 .420 2.575 .033* mPᵉ -.011 -.171 .029 -.671 .521 ᵃ CE = computer experience. N = 14 ᵇ P = performance. N = 11
ᶜ CE + P = the combination of computer experience and performance. N = 14 ᵈ mCE = main effect of computer experience
ᵉ mP = main effect of performance
*p<.05 **p<.10
Exploratory analyses
In addition to answering the research questions, exploratory analyses were performed, which may provide more insight.
Difference in time. It would be interesting to know whether the participants actually
improved in time between pretest and posttest during playing of ‘Input Controller
Assessment’. A paired-samples t-test was conducted to compare time in seconds from the starting point to end point of ‘Input Controller Assessment’ at the beginning of the study and at the end of the study. There was a significant decrease in time in controlling the arrow keys between pretest (M = 82.75, SD = 39.88) and posttest (M = 72.45, SD = 33.45); t (10) = 2.578, p = .028.
Controls. At the beginning of the study participants completed questions about two
opportunities of controlling: controlling with the arrow keys, and controlling with the
computer mouse. The questions were based on the satisfaction about the controlling of ‘Input Controller Assessment’, and were answered on a five-point scale from 1 (totally disagree) to 5 (totally agree). The questionnaire about controlling with the arrow keys is highly reliable according to the data (α = .855), and the questionnaire about controlling with the computer mouse is acceptable (α = .747). A paired-samples t-test was conducted to compare the satisfaction in controlling with the arrow keys and with the computer mouse. A significant difference between the satisfaction in controlling with the arrow keys (M = 3.214,
SD = 1.209) and satisfaction in controlling with the computer mouse (M = 3.871, SD = .934) was found (t (13) = 3.243, p = .006).
Discussion
The aim of the present study was to investigate whether users’ computer experience or performance in a keyboard-controlled 3D movement task (or a combination of both) had effect on the appreciation of a serious game. If computer experience and performance prove good predictors for the appreciation of the serious game, the usability of the game can be adapted to these findings. When the usability of the serious game is easy, then the game is more effective (Virvou & Katsionis, 2008). The usability of the serious game, and therefore the appreciation, is measured using a questionnaire including questions from the Flow State Scale (Jackson & March, 1996) and questions about motivating elements in the game (Hou & Li, 2014; Sailer et al, 2017; Wouters et al., 2013). In investigating the usability, the interaction
between user and the serious game should be taken into account. A component which affected this interaction is the personal factor computer experience. Next, the needs of the user are also important in investigating the usability of the game. Fredrickson and colleagues (2010)
suggest that individuals are more satisfied, when they perform better. Therefore, in this study is investigated whether the level of computer experience and the level of performance have effect on the appreciation of the serious game.
In the first research question is investigated whether the level of computer experience predicts the level of appreciation of a serious game. A positive relation between users’
computer experience and their appreciation of the serious game is found. The higher the level of computer experience, the more positive the appreciation of the serious game was. This means that computer experience might be a good predictor for the appreciation of the serious game. Individuals with a higher level of computer experience seem to judge Wayfinder as more motivating, than individuals with a lower level of experience. Research has shown that motivation is a key component that makes the serious game more effective (Wouters et al., 2013). Therefore, the results of the current study suggest that the serious game has effect in individuals with a sufficient level of computer experience. Possibly, participants with more computer experience learn more from the serious game, in comparison with participants with less computer experience (Orvis, Horn & Belanich, 2008). Perhaps, participants with a low level of computer experience should be omitted in the follow-up study, in which the
effectiveness of the game is tested. In addition, it seems important that patients who are going to use the serious game have a sufficient level of computer experience so that brain-injured patients can train their navigation skills independently at home without assistance of a physician. Possibly, in patients with a lower level of computer experience the computer use and computer skills should be increased prior to the use of the serious game.
Furthermore, in this study is investigated whether the level of performance in a
keyboard-controlled 3D movement task predicts the appreciation of the serious game. Results show that performance level on a 3D movement task does not predict the appreciation of the serious game. However, Fredrickson and his colleagues (2010) demonstrated in their study that performance predicts the appreciation of a computerized test battery. In that study cognitive performance is measured using reaction time and correct responses on cognitive tasks. Another way to measure performance could influence the outcomes.
Next, the combination of computer experience and performance in a keyboard-controlled 3D movement task supplies a marginally significant effect on appreciation of the serious game. Perhaps the sample size of this study, consisting of 11 participants in the main
question, is too small. Possibly, using a bigger sample a significant regression equation was found. The minimum required sample size is calculated (Cohen, Cohen, West & Aiken, 2013; Soper, 2017) and should be 14. In short, a sample size of 11 participants is too small for a complete reliable outcome. Although, the main effect of computer experience on appreciation was significant. These findings indicate that in the prediction of appreciation computer
experience is the best predictor. The model in which only computer experience predicts appreciation is better than the model with the combination of predictors. Besides,
performance does not contribute in the combination of predictors to predict the appreciation of the serious game.
Not only the small sample size, but also other limitations may play a role in explaining why no significant regression equation was found when performance was used as a possible predictor. Firstly, this result could be hypothetically explained by late awareness of
performance. Participants were too late aware of their performance in the 3D movement task using the keyboard. 36,4% of the participants (data obtained on N = 11) who played ‘Input Controller Assessment’ for the second time made a comment that he or she seems to be better in controlling the arrow keys and/or that he or she was faster the second time, which is
registered by the examiner. The fact that the posttest of ‘Input Controller Assessment’ was after the completion of the questionnaire about the appreciation could cause late awareness of performance. Possibly, the questionnaire about appreciation of the serious game could be better after the posttest of ‘Input Controller Assessment’. This could have effect on the appreciation. Furthermore, brain-injured patients have generally deficits of self-awareness, and the individuals struggle in experiencing their own performance (Goverover, Johnston, Toglia & Deluca, 2007). These mentioned problems could also have effect on patients’ awareness of their performance, and therefore to the relation with their appreciation. In
addition, the decrease in time between the pretest and posttest of the 3D movement task in this study was significant, which indicates that the second time of controlling the arrow keys was actually faster. This means that participants have practiced long enough, and that they
improved significantly. Secondly, at the beginning of the study participants played ‘Input Controller Assessment’ once with the arrow keys and once with the computer mouse.
Participants were significant more positive about the controlling with the computer mouse in comparison with controlling with the arrow keys, what was analyzed using completed questionnaires about the satisfaction in controlling of the game. In addition, in answer to the open questions, 100% of the participants (data obtained on N = 14) stated that they found controlling with the computer mouse easier, and that they would like to use the mouse in the
serious game. Hypothetically, this opinion of the participants could have effect on the
appreciation of the serious game, because the games of Wayfinder were played with the arrow keys to date. Therefore, in the follow-up study is it recommended that participants play the serious game Wayfinder with the computer mouse, instead of the arrow keys. This could also have effect on the appreciation of the serious game and therefore with the effectiveness of the therapy (Orvis, Horn & Belanich, 2008; Virvou & Katsionis, 2008).
Although the current study did indeed have some limitations, there were also
strengths. A good quality is the innovatory character of this study. Previously, no standardized treatment for navigation impairment has been developed. Considering the high percentage (29%) of patients with a mild stroke and impaired navigation skills which correlates highly with their quality of life (van der Ham, 2013), a therapy to improve the navigations skills is necessary. Thus, the aim to develop an effective serious game which improves navigation skills is innovative and required. Furthermore, the current study utilizes the input of the participants to improve the serious game. A serious game is more effective when the usability of the game is easy (Virvou & Katsionis, 2008), and when the user of the game is motivated (Wouters et al., 2013). The serious game Wayfinder can be adapted to the needs of the user, which consequently makes the game more effective. In the current study is extensively examined what needs to be adapted in the design of the serious game.
Computer experience might be a good predictor for the appreciation of the serious game. The higher the level of computer experience, the more positive the appreciation of the serious game was. In follow-up studies and usage of Wayfinder in brain-injured patients independently at home it would be advisable to ensure a sufficient level of computer experience. Furthermore, performance in a 3D movement task using the keyboard did not significantly affect the appreciation of the serious game. Although improvement was found in patients’ ability to control the arrow keys, in follow-up studies is it recommended that patients play the serious game Wayfinder using the computer mouse. The expectation is that the effectiveness of Wayfinder increases after adaptations according to recommendations. This effectiveness should be further investigated.
References
Claessen, M. H. G., van der Ham, I. J. M., Jagersma, E., & Visser-Meily, J. M. A. (2016). Navigation strategy training using virtual reality in six chronic stroke patients: A novel and explorative approach to the rehabilitation of navigation impairment.
Neuropsychological Rehabilitation, 26(5-6), 822 – 846. doi: 10.1080/09602011.2015.1045910
Coffin, R. J., & Maclntyre, P. D. (1999). Motivational influences on computer-related affective states. Computers in Human Behavior, 15(5), 549 – 569.
doi: 10.1016/S0747-5632(99)00036-9
Cohen, J., Cohen, P., West., S. G., & Aiken, L. S. (2003). Applied multiple
regression/correlation analysis for the behavioral sciences (3rd edition). Mahwah, NJ: Lawrence Earlbaum Associates.
Compeau, D. R., & Higgers, C. A. (1995). Computer self-efficacy – Development of a measure and initial test. Mis Quarterly, 19(2), 189 – 211. doi: 10.2307/249688
DeSmet, A., van Ryckeghem, D., Compernolle, S., Baranowski, T., Thompson, D., Crombez, G., . . . de Bourdeaudhuij, I. (2014). A meta-analysis of serious digital games for healthy lifestyle promotion. Preventive Medicine, 69, 95 – 107.
doi: 10.1016/j.yomed.2014.08.026
Ebner, M., & Holzinger, A. (2007). Successful implementation of user-centered game based learning in higher education: An example from civil engineering. Computers & Education, 49(3), 873 – 890. doi: 10.1016/j.compedu.2005.11.026
Fredrickson, J., Maruff, P., Woodward, M., Moore, L., Fredrickson, A., Sach, J., & Darby, D. (2010). Evaluation of the usability of a brief computerized cognitive screening test in older people for epidemiological studies. Neuroepidemiology, 34(2), 65 – 75.
doi: 10.1159/000264823
Goverover, Y., Johnston, M. V., Toglia, J., & Deluca, J. (2007). Treatment to improve self-awareness in persons with acquired brain injury. Brain injury, 21(9), 913 – 923. doi: 10.1080/02699050701553205
Hess, T., & Gunter, G. (2013). Serious game-based and nongame-based online courses: Learning experiences and outcomes. British Journal of Educational Technology, 44(3), 372 – 385. doi: 10.1111/bjet.12024
Hornbeak, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79 – 102. doi: 10.1016/j.ijhcs.2005.06.002
Van der Ham, I. J. M., Kant, N., Postma, A., & Visser-Meily, J. M. A. (2013). Is navigation ability a problem in mild stroke patients? Insights from self-reported navigation measures. Journal of Rehabilitation Medice, 45(5), 429 – 433.
doi: 10.2340/16501977-1139
Hou, H. T., & Li, M. C. (2014). Evaluating multiple aspects of a digital educational problem-solving-based adventure game. Computers in Human Behavior, 30, 29 – 38.
doi: 10.1016/j.chb.2013.07.052
Jackson, S. A., & Marsh, H. W. (1996). Development and validation of a scale to measure optimal experience: The Flow State Scale. Journal of Sport & Exercise Psychology, 18(1), 17 – 35. doi: 10.1123/jsep.18.1.17
Jaspers, M. W. M. (2009). A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics, 78(5), 340 – 353. doi: 10.1016/j.ijmedinf.2008.10.002
Jordan, K., Schadow, J., Wuestenberg, T., Heinze, H. J., & Jancke, L. (2004). Different cortical activations for subjects using allocentric or egocentric strategies in a virtual navigation task. Neuroreport, 15(1), 135 – 140.
doi: 10.1097/01.wnr.0000097043.56589.b6
Kober, S. E., Wood, G., Hofer, D., Kreuzig, W., Kiefer, M., & Neuper, C. (2013). Virtual reality in neurologic rehabilitation of spatial disorientation. Journal of
Neuroengineering and Rehabilitation, 10. doi: 10.1186/1743-0003-10-17
Lemoncello, R., Sohlberg, M. M., & Fickas, S. (2010). When directions fail: Investigation of getting lost behaviour in adults with acquired brain injury. Brain Injury, 24(3), 550 – 559. doi: 10.3109/02699050903446807
Livingstone, S. A., & Skelton, R. W. (2007). Virtual environment navigation tasks and the assessment of cognitive deficits in individuals with brain injury. Behavioural Brain Research, 185(1), 21 – 31. doi: 10.1016/j.bbr.2007.07.015
Loyd, B. H., & Gressard, C. (1984). Reliability and factorial validity of computer attitude scales. Educational and Psychological Measurement, 44(2), 501 – 505.
doi: 10.1177/0013164484442033
Maynard, D. W., Houtkoop-Steenstra, H., Schaeffer, N. C., & van der Zouwen, J. (2002). Standardization and tacit knowledge: Interaction and practice in the survey interview. Wiley, New York.
Moneta, G. B., & Csikszentmihalyi, M. (1996). The effect of perceived challenges and skills on the quality of subjective experience. Journal of Personality, 64(2), 275 – 310. doi: 10.1111/j,1467-6494.1996.tb00512.x
Mou, W. M., McNamara, T. P., Valiquette, C. M., & Rump, B. (2004). Allocentric and egocentric updating of spatial memories. Journal of Experimental
Psychology-Learning Memory and Cognition, 30(1), 142 – 157. doi: 10.1037/0278-7393.30.1.142 Orvis, K. A., Horn, D. B., & Belanich, J. (2008). The roles of task difficulty and prior
videogame experience on performance and motivation in instructional videogames. Computers in Human Behavior, 24(5), 2415 – 2433. doi: 10.1016/j,chb.2008.02.016 Park, D. C., & Bischof, G. N. (2013). The aging mind: neuroplasticity in response to cognitive
training. Dialogues in Clinical Neuroscience, 15(1), 109 – 119.
Popedavis, D. B., & Twing, J. S. (1991). The effects of age, gender, and experience on measures of attitude regarding computers. Computers in Human Behavior, 7(4), 333 – 339. doi: 10.1016/0747-5632(91)90020-2
Reichlin, L., Mani, N., McArthur, K., Harris, A. M., Rajan, N., & Dacso, C. C. (2011). Assessing the acceptability and usability of an interactive serious game in aiding treatment decisions for patients with localized prostate cancer. Journal of Medical Internet Research, 13(1), 188 – 201.
Sailer, M., Hense, J. U., Mayr, S. K., & Mandl, H. (2017). How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior, 69, 371-380.
doi: 10.1016/j.chb.2016.12.033
Sivan, M., Gallagher, J., Holt, R., Weightman, A., Levesley, M., & Bhakta, B. (2014). Investigating the International Classification of Functioning, Disability, and Health (ICF) framework to capture user needs in the concept stage of rehabilitation
technology development. Assistive Technology, 26(3), 164 – 173. doi: 10.1080/10400435.2014.903315
Soper, D. S. (2017). A-priori sample size calculator for multiple regression [Software]. Available from http://www.danieloper.com/statcalc
Steiner, W. A., Ryser, L., Huber, E., Uebelhart, D., Aeschlimann, A., & Stucki, G. (2002). Use of the ICF model as a clinical problem-solving tool in physical therapy and rehabilitation medicine. Physical Therapy, 82(11), 1098 – 1107.
Ullen, F., de Manzano, O., Almeida, R., Magnusson, P. K. E., Pedersen, N. L., Nakamura, J., Csikszentmihalyi, M., & Madison, G. (2012). Progeness for psychological flow in everyday life: Associations with personality and intelligence. Personality and Individual Differences, 52(2), 167 – 172. doi: 10.1016/j.paid.2011.10.003
Verganti, R. (2008). Design, meanings, and radical innovation: A metamodel and a research agenda. Journal of Product Innovation Management, 25(5), 436 – 456.
doi: 10.1111/j.1540-5885.2008.00313.x
Virvou, M., & Katsionis, G. (2008). In the usability and likeability of virtual reality games for education: The case of VR-ENGAGE. Computers & Education, 50(1), 154 – 178. doi: 10.1016/j.compedu.2006.04.004
Wouters. P., van Nimwegen, C., van Oostendorp, H., & van der Spek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of
