T
HE EFFECTIVENESS AND USABILITY OF SERIOUS
HEALTH GAMES
TO SUPPORT SMOKING CESSATION
Medical Informatics Master thesis
2018
Student:
Sander van Strijp 6103170
Supervision: Prof. M.W. Jaspers, PhD M.E. Derksen, MSc Dr. M.P. Fransen, PhD
2
The effectiveness and usability of serious health games to
support smoking cessation
Student
Sander van Strijp College ID: 6103170
Email: s.vanstrijp@amc.uva.nl
Tutor
Prof. M.W. Jaspers, PhD
Department of Clinical Informatics - J1B-116 Amsterdam UMC, location AMC
University of Amsterdam Meibergdreef 15 1105 AZ Amsterdam Tel: 65269 Email: m.w.jaspers@amc.uva.nl Daily mentor M.E. Derksen, MSc
Department of Public Health - J2-210 Amsterdam UMC, location AMC Meibergdreef 15 1105 AZ Amsterdam Tel: 64494 Email:
m.e.derksen@amc.uva.nl
Mentor Dr. M.P. Fransen, PhDDepartment of Public Health - K2-207 Amsterdam UMC, location AMC Meibergdreef 15 1105 AZ Amsterdam Tel: 63165 Email: m.p.fransen@amc.uva.nl SRP period November 2017 - December 2018
3
Contents
Abstract ... 5
1. General introduction ... 7
2. The effects of game mechanics in serious health games for smoking cessation: A systematic review ... 9
2.1 Introduction ... 9
2.2 Methods ... 9
2.2.1 Search strategy ... 9
2.2.2 Inclusion and exclusion criteria ... 10
2.2.3 Primary and secondary outcomes ... 11
2.2.4 Risk of bias assessment ... 11
2.2.5 Inter-rater reliability ... 11
2.2.6 Data extraction ... 11
2.2.7 Strategy for data synthesis ... 11
2.3 Results ... 11
2.3.1 Study and sample characteristics ... 12
2.3.2 Effectiveness of games in changing smoking behaviour ... 13
2.3.3 Game mechanics ... 13
2.4 Discussion ... 19
2.4.1 Relation with other work ... 19
2.4.3 Voluntary play sessions ... 19
2.4.2 Strengths and limitations ...20
2.4.5 Conclusions and implications ...20
3. A usability study of a serious health game for supporting smoking cessation in disadvantaged (expectant) mothers ... 21
3.1 Introduction ... 21
3.2 Methods ... 21
3.2.1 Heuristic evaluation ... 22
3.2.2 End-user think-aloud protocol ... 23
3.3 Results ... 25
3.4 Discussion... 39
3.4.1 Main findings ... 39
3.4.2 End-users vs experts ... 39
3.4.3 Recommendations ... 40
4
3.4.5 Implications and further research ... 42
4. References ... 42
Appendix ... 46
5
Abstract
Background: Disadvantaged pregnant women have a multitude of problems regarding
smoking cessation, and designing effective smoking cessation strategies for this target group is challenging. Serious health games (SHG) are a promising method for smoking cessation in disadvantaged pregnant women. However, there is little research on what game mechanics (GM) have shown to be effective.
Objective: This thesis had two main goals which were obtained by performing two separate but related studies. The first goal was to assess how the effectiveness of SHG’s on smoking cessation are associated with the presence of certain (combinations of) GM’s. These findings were used by Derksen et al to develop “Kindle”, a SHG to support smoking cessation in disadvantaged (upcoming) mothers. The second goal of this thesis was to assess the usability of Kindle, and give recommendations to its developers on improving the usability of the SHG. Methods: A systematic review was conducted to assess the association between the
effectiveness of SHG’s on smoking cessation and the presence of certain (combinations of) GM’s. We included evaluation studies of SHG’s which intended to positively change smoking behaviour.
For assessing the usability of Kindle, a combination of the Think Aloud (TA) protocol and Heuristic Evaluation (HE) was used.
Results: The systematic review resulted in 17 included SHG evaluation studies. All 17 studies reported one or more positive outcomes related to change in smoking behaviour. The most frequently used GM’s were based on "Theme and genre", "General rewards", "Punishment" and "Graphics and sound". The usability study yielded 77 usability problems using the HE, and 24 usability problems using the TA protocol.
Conclusion: Overall, SHG’s show significant positive effects on smoking cessation and behavioural determinants. The most prevalent GM’s were based on “Reward features” and “Narrative and identity features”. We were unable to answer how effectiveness of SHG’s on changing smoking behaviour is associated with the presence of certain (combinations of) GM’s. Recommendations for further research are 1) to conduct research on a dose-response relationship between playtime and effects on (behavioural determinants of) smoking cessation, 2) comparing different combinations of GM’s using studies with voluntary play sessions, and 3) to include a sufficient follow-up period in evaluation studies.
The usability of Kindle was assessed using a combination of TA and HE. The results confirm the findings of previous usability studies: A combination of usability inspection and usability testing methods reveals more unique usability problems than a single method. TA will disclose more critical usability problems, concerning the usefulness of an application, while HE will mainly result in the disclosure of non-critical problems, concerning the ease of use of an application.
Keywords: Serious Health Games, Game Mechanics, Usability, Smoking Cessation, Pregnant Disadvantaged Women.
6
Samenvatting
Achtergrond: Kwetsbare zwangere vrouwen hebben een scala aan problemen omtrent stoppen met roken. Het ontwikkelen van effectieve interventies om deze groep van het roken af te helpen blijkt een uitdaging. “Serious health games” (SHG) lijken een veelbelovende methode te zijn om stoppen met roken bij kwetsbare zwangere vrouwen te bewerkstelligen. Er is echter weinig onderzoek omtrent welke “game mechanics” (GM) effectief blijken.
Doelstelling: Deze scriptie had twee doelstellingen welke bereikt zijn door het uitvoeren van twee aparte verwante onderzoeken. Het eerste doel betrof het vaststellen van hoe de
effectiviteit van SHG’s zijn geassocieerd met de aanwezigheid van bepaalde (combinaties van) GM’s. Deze bevindingen zijn door Derksen et al gebruikt om “Kindle” te ontwikkelen, een SHG welke kwetsbare zwangere vrouwen ondersteunt in hun proces om te stoppen met roken. Het tweede doel was het evalueren van de “usability” van Kindle. Met deze resultaten hebben wij de ontwikkelaars van Kindle aanbevelingen kunnen geven omtrent het verbeteren van de “usability” van deze SHG.
Methoden: Een systematic review is uitgevoerd om de associatie vast te stellen tussen de effectiviteit van SHG’s voor stoppen met roken, en de aanwezigheid van bepaalde (combinaties van) GM’s. We hebben evaluatiestudies geïncludeerd van SHG’s welke beoogden het
rookgedrag positief te veranderen.
Voor de “usability” evaluatie van Kindle hebben wij een combinatie van Think Aloud (TA) en Heuristische Evaluatie (HE) gebruikt.
Resultaten: De systematic review resulteerde in 17 geïncludeerde SHG evaluatie studies. Alle 17 studies rapporteerden één of meer positieve uitkomsten omtrent verandering van
rookgedrag. De meest frequent gebruikte GM’s waren gebaseerd op "Theme and genre", "General rewards", "Punishment" en "Graphics and sound". Bij het “usability” onderzoek leverde de HE 77 “usability” problemen op, en leverde de TA 24 “usability” problemen op. Conclusie: SHG’s hebben voornamelijk significante positieve effecten op het rookgedrag. De meest voorkomende GM’s waren gebaseerd op “Reward features” en “Narrative and identity features”. Het is ons niet gelukt om te beantwoorden hoe de effectiviteit van SHG’s op het rookgedrag geassocieerd is met bepaalde (combinaties van) GM’s. Aanbevelingen voor verder onderzoek zijn: 1) het uitvoeren van onderzoek omtrent een “dose-response” relatie tussen speeltijd en effect op rookgedrag, 2) het vergelijken van verschillende combinaties van GM’s door middel van studies met vrijwillige speelsessies en 3) het includeren van een toereikende follow-upperiode.
De “usability” van Kindle was geëvalueerd middels een combinatie van TA en HE. Beide methodieken hebben verschillende typen problemen aan het licht gesteld. Waar HE meer niet-kritieke problemen vond op het gebied van gebruiksgemak, vond TA meer niet-kritieke problemen op het gebied van bruikbaarheid.
7
1. General introduction
Tobacco has caused the greatest epidemic of the twentieth century, killing up to half of its users. Every year more than seven million people are killed by tobacco use worldwide. According to the 2017 WHO factsheet on tobacco, “More than six million of those deaths are the result of direct tobacco use, while around 890.000 are the result of non-smokers being exposed to second-hand smoke” [1].
Exposure to tobacco smoke affects all phases of human reproduction – including both female and male fertility [2]. Cigarette smoking during and after pregnancy has adverse health effects for both mother and child, for short and long term [2]–[8].
According to the 2011 Pregnancy Risk Assessment and Monitoring System (PRAMS) data from 24 states in the US, approximately 10% of women reported smoking tobacco during the last three months of pregnancy. Of women who smoked three months before pregnancy, 55% quit during pregnancy. Among women who quit smoking during pregnancy, 40% started smoking again within six months after delivery [9].
In theory, the effects of smoking on the fetus are preventable if women would not smoke during the pregnancy. In practice however, quitting smoking is easier said than done for the majority of people. More than four out of ten people in the US who tried to quit smoking in 2015, were not able to stop for more than one day [10].
The interaction among multiple factors that cause and maintain the addiction makes quitting very complex [11]. The basis of nicotine addiction is a combination of positive reinforcements, including enhancement of mood and avoidance of withdrawal symptoms. In addition,
conditioning has an important role in the development of tobacco addiction [12], [13]. Weakening the conditioning of tobacco usage by influencing behavioural and social
determinants may contribute in treating the tobacco addiction. Studies show that increasing self-efficacy [14], [15] introducing smoke free environments [16] and motivational interviewing [17] can help to reduce tobacco consumption.
A review on socioeconomic status (SES) and smoking by Hiscock et al showed that
disadvantaged individuals have a multitude of problems regarding smoking cessation. Smoking prevalence in these individuals is higher, and they may be more exposed to tobacco’s harms. Disadvantaged individuals also seem to have a higher smoking uptake, and their quit attempts are less likely to be successful. The review states this may be the result of reduced social support for quitting, low motivation to quit, stronger addiction to tobacco, increased likelihood of not completing courses of pharmacotherapy or behavioural support sessions, psychological differences such as lack of self-efficacy, and tobacco industry marketing [18]. Therefore, disadvantaged pregnant women form a window of opportunity for smoking cessation, since most health gain can be achieved in this group and motivation for smoking cessation is relatively high during pregnancy. There is evidence found that interventions tailored to the needs of disadvantaged pregnant women can be effective [19], [20], but smoking cessation remains challenging.
An example of a context with disadvantaged pregnant women is a Dutch nurse-family partnership, ‘VoorZorg’. VoorZorg is a two-year family intervention in which nurses support disadvantaged, young (expectant) mothers with regard to their lifestyle, personal development and the upbringing of their child [21]. At the onset of the VoorZorg program, 43% of the
8
participants smoke. During the program, still 33% of the participants smoke during their pregnancy. Two months after birth, 48% of the women are current smokers [21]. Although VoorZorg succeeded in reducing the number of smokers during the pregnancy, it did not succeed in creating lasting behavioural change after birth. Innovative solutions may be needed for this target group to quit smoking entirely. This intervention will form the context of this project, which will be elaborated later on in the usability study found on page 21.
Serious health games (SHG’s) are a promising method of health promotion for this target group, because it is attractive, enables tailoring, and improves engagement [22], [23]. The most common definition for serious games (SG’s) is games where the primary purpose is not for entertainment. Michael and Chen define SG’s as “games that do not have entertainment, enjoyment, or fun as their primary purpose” [24].
SG’s related to health and healthcare likewise have become more common, and today there exists a large number of them [25]. These games can have direct or indirect positive
physiological and psychological effects on individuals or patients [26], [27], [28] which is exactly the aim of SG’s in health and healthcare. Some examples showing the variety of types and areas for applications related to physical or mental health include: physical fitness, education in health/self-directed care, distraction therapy, recovery and rehabilitation, training and simulation, diagnosis and treatment of mental illness, cognitive functioning, mental and emotional control [29].
SG’s potentially have a multitude of positive outcomes on knowledge acquisition, perceptual and cognitive skills, motor skills, behaviour change, soft skills and social skills, affective and motivational outcomes and physiological and psychological outcomes. However, little is known about how these games should be designed in order to acquire the desired effects [30]. Due to their playful design, it is expected that SG’s for health may be more persuasive at realizing a healthier lifestyle than current care for pregnant disadvantaged women with low socioeconomic status. A recent meta-analysis has shown positive results using SHG’s for behaviour change [26]. Furthermore, designing with, rather than for the target group shows promising results [31]. Usability is a prerequisite for its adoption by this target group and proving its effectiveness [32].
In her PhD study Derksen is systematically developing and evaluating a SHG in co-creation with VoorZorg nurses, clients and their social environment, and game developers aiming at sustainable smoking cessation during and after pregnancy of young, vulnerable pregnant women [33]. This thesis will contribute to Derksen’s development and evaluation of the SHG intervention.
In this thesis I describe two steps in this process of the intervention development: 1) Systematic review
In a systematic review I retrieved evidence on SHG’s whether the effectiveness of SG’s is associated with the presence of certain (combinations of) game mechanics (GM). The results were used as input to develop the SHG for smoking prevention within the VoorZorg program.
2) Usability evaluation
9
recommendations for the developers in order to improve its adoption, and compared Think Aloud testing with Heuristic Evaluation.
2. The effects of game mechanics in serious health games for smoking
cessation: A systematic review
2.1 Introduction
Behavioural support interventions are accepted as best practice in treating tobacco addiction, however current interventions remain to be less effective in disadvantaged individuals [34]. SHG’s are a promising means to change smoking behaviour in these individuals, as they are more engaging than typical behavioural support interventions. SHG’s have positive effects on a multitude of healthy lifestyles and their determinants in various populations[26]. However, despite their overall effectiveness, SHG’s are typically complex in that they consist of multiple potentially interacting GM’s.
GM’s are methods invoked to define the context, goals, rules, setting, types of interaction, and the boundaries within a game [35]. As a result of the complexity of these SHG’s, it is not clear which specific GM’s have been included in existing SG’s, and which of these GM’s have shown to be effective for changing smoking behaviour. To the best of our knowledge, the effects of GM’s in SHG’s for changing smoking behaviour have not yet been specified.
Current research on (effective) mechanisms of SG’s for changing smoking behaviour is still non-existent. The findings of this systematic review will represent a first attempt toward gaining insight into this topic. This may enhance understanding of how SG’s for changing smoking behaviour work and what GM’s are likely to contribute to effectiveness, in turn
informing game developers, researchers and policy makers on effective design choices of future SG’s for health promotion.
This systematic review aims 1) to examine what GM’s are used in SG’s for health promotion on tobacco use; and 2) to assess whether effectiveness of these SHG’s is associated with the presence of certain (combinations of) GM’s.
Research questions:
1. What is the effectiveness of SHG’s on (key behavioural determinants of) smoking behaviour? 2. What GM’s are used in SHG’s for changing smoking behaviour?
3. How are the effectiveness of SHG’s on changing smoking behaviour and key behavioural determinants associated with the presence of certain (combinations of) GM’s?
2.2 Methods
We studied publications on SG’s which intended to positively change smoking behaviour (and key behavioural determinants (such as behavioural intention, knowledge, perceived barriers, skills, attitude, subjective norm, and self-efficacy)) [36].
2.2.1 Search strategy
Two reviewers (MD, SS) and a librarian (JD) systematically searched PubMed (MEDLINE), EMBASE, PsychINFO, and Web of Science for relevant published articles. Intervention database WHO ICTRP and Cochrane were searched as well for (un)published
10
articles/interventions. The search was complemented with hand-searching for studies in relevant reviews as well as snowballing by means of searching for relevant studies in the reference list of included articles.
The search query mainly consisted of combining the concept of SG’s with the concepts of smoking behaviour and health promotion. The search also focussed on evaluation studies by, amongst others, including a multitude effect measures to the query. The full search query can be found in appendix 1.
The selection process consisted of two rounds: a screening based on title and abstract, followed by a selection of full-text articles for eligibility. Both rounds in the selection process were independently conducted by two reviewers (MD, SS). Rayyan QCRI and EndNote were used for screening and data management. Any disagreements were discussed, if no consensus was reached, a third reviewer (MF) was consulted.
2.2.2 Inclusion and exclusion criteria
We included studies on tobacco smoking (e.g. cigarettes, cigars and pipes). Studies on smokeless tobacco (e.g. sucking, chewing, and snuffing tobacco) or on substances other than tobacco (e.g. cannabis), were excluded from this review. No specific populations were excluded in this review. Table 1 shows a complete overview of the inclusion and exclusion criteria that were used.
Table 1. Inclusion and exclusion criteria
Inclusion Definition Exclusion
Serious games “Games that do not have
entertainment, enjoyment, or fun as their primary purpose” [24].
Games which are primarily created for entertainment purposes, applications which do not contain any game mechanics
Smoking behaviour change The intervention should aim to positively change (key behavioural determinants of) smoking behaviour.
Games withoutintent to create a lifestyle change (e.g. treatment support)
Evaluation studies Studies which evaluate the effects of a serious game on (key behavioural
determinants of) smoking behaviour
Studies that only report usability evaluations, player experiences, or case studies
Research designs Randomised controlled trials, non-randomised trials, before-after studies with no control group, case-control studies, and cohort studies
Case studies
11
2.2.3 Primary and secondary outcomes
The primary outcomes were: 1) effectiveness of SHG’s on smoking behaviour and, if available, key behavioural determinants; 2) game elements used in SHG’s for smoking behaviour. Secondary outcomes were conditions in which game elements are effective in changing smoking behaviour and, if available, key behavioural determinants.
2.2.4 Risk of bias assessment
The quality of individual studies was evaluated according to the Study Quality Assessment Tools by the NIH [37]. Two reviewers (MD, MF) independently assessed the quality of each study using these assessment tools. The methodological strength of each study was determined and reported based on a discussion among the authors. When no consensus was reached, a third reviewer (SS) was consulted. Studies were not excluded based on the quality assessment. 2.2.5 Inter-rater reliability
Two raters (SS, MD) independently rated 100% of the search results. Inter-rater reliability of the screening based on the title and abstract was excellent (ICC=0.97) and the inter-rater reliability of the screening based on the full-text was good (ICC=0.8). The quality of the studies using the NIH Quality Assessment Tools was also independently rated by two assessors (MD, MF), yielding an excellent inter-rater reliability (ICC=0.95).
2.2.6 Data extraction
A data extraction form was developed based on the Cochrane data collection form for RCTs and non-RCTs [38]. The form was pilot-tested on the first 5 studies and adjusted where needed. Two authors (MD, SS) extracted data in parallel and then checked the results against each other. Any disagreements were discussed and consensus was reached without the need of a third reviewer. The following items were extracted: full reference; setting; study design; study population characteristics; target smoking behaviour / behavioural determinants; intervention description; the included GM’s, and outcomes on (key determinants of) smoking behaviour. Corresponding authors of interventions meeting the inclusion criteria were contacted twice with a request for additional materials detailing the intervention content when no published materials were available. When not available or no response received, the intervention description in the published included article was used.
2.2.7 Strategy for data synthesis
The effectiveness of SG’s on smoking behaviour and, when available, key behavioural
determinants were extracted from the included interventions and were aggregated. GM’s were extracted using a taxonomy of GM’s [39]. The extracted GM’s were analysed quantitatively (e.g. which mechanics; how many times present; number of GM’s per intervention; combinations of GM’s). The theoretical basis and conditions of GM’s were extracted from the included
interventions and were aggregated. The results of aggregated data were used to analyse associations between effectiveness of SG’s changing smoking behaviour (and, if available, key behavioural determinants) and GM’s.
2.3 Results
The database search yielded 4003 hits, from which 1018 duplicates were removed. Next 2865 articles were excluded after screening the abstract and title. After reading the full texts, 17 studies were retained. No studies were added from other sources. This resulted in a total of 17
12
included papers (Fig. 1).
Fig. 1 Flow chart of study selection process 2.3.1 Study and sample characteristics
Of the 17 included studies, 12 originated from North America, while 5 studies were from Europe. All studies were published in 2007 or later, except one study from 1999. 35% of the studies were of “good” quality, 35% were of “fair” quality, and 30% were of “poor” quality. The most common study design was an RCT (35%), and the second most common study design was a pre-post design (29%).
Most sample sizes were between 25 and 91 participants (65%); however An et al (2013)
conducted an RCT with 1698 participants, having the biggest sample size among the included studies. The ages of the participants across all studies varied from 10 to at least 70 years old. Twelve studies included smokers, while five studies − which studied children − did not contain any smokers. Table 2 shows the study and sample characteristics in greater detail.
The average study duration was five weeks, and in 71% of the studies the participants played the game over multiple sessions. The average play duration per session was 41 minutes, while the average total play duration was 2 hours and 20 minutes. Three studies used physical games (board games), while the remaining 15 were digital-based (table 3).
13
2.3.2 Effectiveness of games in changing smoking behaviour
All 17 studies reported one or more positive outcomes related to behavioural change concerning smoking. Significant positive outcomes were reported in 14 studies (table 4). 2.3.3 Game mechanics
The GM’s can be categorized as follows: “Narrative and identity features” (e.g., the role of character creation and interactive storytelling), “Social features” (e.g., social aspects of video game playing), “Manipulation and control features” (e.g., the role of user input in influencing in-game outcomes), “Reward and punishment features” (e.g., the ways in which players win and lose in video games) and “Presentation features” (e.g., the visual and auditory presentation of video games). Every game mechanic category consists of one or more sub-features. “Reward and punishment features” were most used across the games (82%) closely followed by
“Narrative and identity features” (76%). “Social features” were seen the least (47%) across all games.
The theoretical basis and conditions of GM’s were not reported in any of the studies, and were therefore not included in the results.
14
Table 1. Study sample characteristics.
Study Population (country:N)
Age
(mean:range) Smoking severity Study design
Duratio
n Setting
Quality assessment score
An 2013 United States:1698 24.07:18-30 19.82 cigarettes / day RCT 12 weeks Independently via internet Good
Bordnick 2012 United States:86 47:18-70 25 cigarettes / day RCT 10
weeks Clinic Fair
Brinker 2016 Germany:125 12.75:NR None Cross sectional 1 day School Poor
DeLaughter
2013 United States:30 NR: 19-65+ 13.8 cigarettes / day Pre-post 1 day Test lab Fair
Duncan 2018 United States:25 11.56:11-14 None Pre-post 2 weeks School Good
Girard 2007 Canada:16 50:40-56 20.5 / day No control 2 weeks Test lab Fair
Girard 2009 Canada:91 44:18-65 10+ cigarettes / day RCT 12 weeks Clinic Poor
Gordon 2017 United States:73 39,1:NR NR Pre-post 90 days Online at home Fair
Khazaal 2008 Switzerland:51 43.7:18-65 23 cigarettes / day Pre-post 1 day Hospital, group setting Good
Khazaal 2010 Switzerland:61 30.7:18-65 2-60 cigarettes / day Pre-post 4 weeks University, group setting Good
Khazaal 2013 Switzerland:240 31.5:NR 15 cigarettes / day RCT 4 weeks School/university Poor
Metcalf 2018 United States:61 NR:>18 Any kind of cigarette, tobacco or
alcohol use
quasi-experimental, stratified, wait-list control trial
4 weeks Small office Poor
Nemire 1999 United States:72 NR:12-13 NR RCT 8 weeks Local middle school Poor
Parisod 2018 Finland:151 11:10-13 No current smokers RCT 2 weeks Public schools (grades 4-6) Good
Rath 2015 United States:689 NR:13-24 42% has tried cigarettes before Longitudinal 3
months Online panel Good
Song 2013 United States:62 22.05:NR Once a week 2x2 1 day University Fair
Xu 2014 United States:40 24.10:19-42 13.25 cigarettes / day 2x2 2 days Long Island (NY) community,
15
Table 2. Intervention details
Study Intervention name Co-intervention Comparison Play time /
session Play time (total) Frequency Type
An 2013 Tailored Web Peer coaching C1: Tailored web only C2: Not
tailored NR NR
Once per week (6
weeks) Digital
Bordnick 2012 VR Skills Training Nicotine
Replacement Therapy
C1: Nicotine Replacement
Therapy Only NR NR
Once per week (10
weeks) Digital
Brinker 2016 Photo Mirror
(Smokerface) None None 2 minutes 2 minutes Once Digital
DeLaughter 2013 Craveout None Pre 10 minutes 10 minutes Once Digital
Duncan 2018 SmokeSCREEN None Pre 1 hour 4 hours Twice per week (two
weeks) Digital
Girard 2007 Virtual arm (VR
therapy) None Pre 30 minutes 2 hours
Twice per week (two
weeks) Digital
Girard 2009 Virtual arm (VR
therapy) Counselling C1: Balls intervention 30 minutes 2 hours
Once per week (4
weeks) Digital
Gordon 2017 See Me Smoke-Free None Pre NR NR One month Digital
Khazaal 2008 Pick-Klop None Pre 1h15m 1h15m Once Physical
Khazaal 2010 Pick-Klop None Pre 1h30m 6h Once per week (4
consecutive weeks) Physical
Khazaal 2013 Pick-Klop None C1: Psychoeducation, C2:
Waiting list 1h30m 3h Two sessions Physical
Metcalf 2018 Take control None C1: Waitlist 23 minutes 1h30m 4 sessions (once per
week on avg) Digital
Nemire 1999 Virtual Environment
Discussing personal goals + workbook
material
C1: Life Skills Training 50 minutes 6h40m 8 sessions Digital
Parisod 2018 Fume None
C1: Website with same content, but without game
mechanics; C2: No intervention
19 minutes 76 minutes 4 sessions (in 2
weeks) Digital
Rath 2015 Flavor Monsters None None NR NR NR Digital
Song 2013 Super Smoky None
Self-avatar (yes/no) x Future condition (yes/no) - 4 groups
in total
NR NR Once Digital
Xu 2014 Nameless cooperative
games None
Self-expanding vs. not) x 2 (cigarette cue present vs.
absent)
16
Table 3. All the different sub-features of the 5 different game mechanic categories in each study.
Studies Avatar cre ati o n 1 S tor yte ll in g devi ce 1 Them e and ge n re 1 S oci al U ti li ty 2 S oci al f or ma ti on 2 Lea der boar d 2 Us er i n p ut 3 S ave 3 P laye r ma n ageme n t 3 Non -c o n tro ll abl e 3 Genera l r ew ar d t yp e 4 P un is hme n t 4 Meta -game r ew ar d 4 In ter mi tt en t r ew ar d 4 Near m is s 4 Ev ent f re q uen cy 4 Ev ent du ra ti o n 4 Grap hi cs a n d s oun d 5 In -game a d ver ti si n g 5 Total An 2013 x x x x 4 Bordnick 2012 x x x x x x x x 8 Brinker 2016 x x x x 4 DeLaughter 2013 x x x x x x 6 Duncan 2018 x x x x x x x 7 Girard 2007 x x x 3 Girard 2009 x x x 3 Gordon 2017 x x x x 4 Khazaal 2008 x x x x x 5 Khazaal 2010 x x x x x 5 Khazaal 2013 x x x x x x 6 Metcalf 2018 x x x x x 5 Nemire 1999 x x x x x 5 Parisod 2018 x x x x x x x x 8 Rath 2015 x x x x x x x x x x 10 Song 2013 x x x x x 5 Xu 2014 x x x 3 Total 7 5 11 5 3 1 6 1 2 4 12 9 1 8 2 1 1 9 3 91
17
Table 4. Outcomes.
Study Outcomes studied
Time between last play session and
Follow-up measurement
Outcome effect Outcome (significance Y/N)
An 2013 30-day abstinence from smoking (high=good) 6 weeks 31% vs 23% (C1) vs 11% (C2) Positive (Y)
Bordnick 2012
Craving (low=good), Cigarettes smoked
(low=good), Self-efficacy (low=good) 6 months 1.3 vs 2.1 (C1), 0.41 vs 7.4 (C1) , 1.7 vs 2.5 (C1) Positive (Y), Positive (Y), Positive (Y)
Brinker 2016
"The animation of my face by the Smokerface-App motivates me not to smoke", "I learned new
benefits of non-smoking"
None More than 60% agreement on the two items Positive (NR), Positive (NR)
DeLaught
er 2013 Cravings (low=good) None 3.24(pre) vs 2.99(post) Positive (N)
Duncan 2018
Knowledge (high=good), Self-efficacy (high=good),
Perceived norms (low=good) 10 weeks
4.16(pre) vs 4.88(post), 2.98(pre) vs 3.44(post), 1.81(pre) vs
1.67(post) Positive (Y), Positive (N), Positive (N)
Girard 2007
Quit smoking (percentage, high = good), Number
of cigarettes smoked / day (low = good) 4 weeks 70%, 22.8(pre) vs 12.8(post) Positive (NR), Positive (Y)
Girard 2009
Nicotine dependence (low=good), Tobacco
abstinence (percentage, high=good) 6 months 5.38(C1) vs 3.87, 2%(C1) vs 15% Positive (Y), Positive (Y)
Gordon 2017
Smoking abstinence last 7 days, Smoking abstinence last 30 days, Number of cigarettes
smoked, Confidence in quitting
unknown 4%(pre) vs 46%(post), 32%(post), 17(pre) vs 11(post),
3.62(pre) vs 3.13(post),
Positive (Y), Positive (N), Positive (Y), Negative (Y)
Khazaal 2008
Attitude towards smoking prohibition (high=good),
Intention to stop smoking (high=good) None 71.5(pre) vs 75(post), 61.4(pre) vs 74.2(post) Positive (N), Positive (Y)
Khazaal 2010
Cigarettes smoked / day (low=good), intention to
quit smoking (high=good) None 13.2(pre) vs 11.4(post), 5.6(pre) vs 6.5(post) Positive (Y), Positive (Y)
Khazaal 2013
Attitudes Towards Nicotine Replacement Therapy, Attitudes Towards Smoking (low=good), Smoking Self Efficacy (high=good), Smoking status (non-smoker=good), Number of cigarettes smoked per
day (low=good), Number of quit attempts (high=good)
3 months
(significant increase between T0 and T1, 11.9 decrease between T0 and T2 (int), 10.3 increase between T0 and T2 (int), Non-smoking was sign. higher in intervention than in
control, 10.1(int) vs 12.7(C1) vs 11.9(C2), 9.4 difference between T1 and T2 (int)
(Positive (Y), Positive (Y), Positive (Y), Positive (Y), Positive (Y), Positive (Y)
Metcalf 2018
Self-efficacy (high=good), Intended behaviour
(high=good), Attitude (high=good), 1 week
4,07 (pre) vs 4,01 (post), 4,00(pre) vs 3,89(post), 3,88(pre) vs 3,67(post)
Negative (NR), Negative (NR), Negative (NR) Nemire
1999
Attitudes towards smokers and smoking, Smoking
behaviour 6 weeks
Improvement in intervention group, No differences between groups
Positive in some areas (Y), No difference (N) Parisod
2018
Various outcomes, Acceptability and demand
(high=good) 2 weeks
No differences. Health literacy was already high in all groups, More frequent uses and more than double play time
than C1
No difference (N), Positive (Y)
Rath 2015 Anti-Tobacco Industry (ATI) Index -attitude
measurements (high=good) None Increase from 7.5 (baseline) to 8.0 (3 months) Positive (Y)
Song 2013
Perceived risk (high=good), Attitude towards smoking (high=good), Intention to quit
(high=good)
None 8.14 (future) vs 6.84 future), 7.94 (future) vs 6.81
18
Study Outcomes studied
Time between last play session and
Follow-up measurement
Outcome effect Outcome (significance Y/N)
Xu 2014
effect of: self-expansion on activation of the caudate, cigarette cue conditions in the TPJ,
cigarette cue condition in the amygdala
19
2.4 Discussion
This was the first systematic review studying associations between GM’s and effectiveness in SHG’s to support smoking cessation. Overall, SHG’s show significant positive effects on smoking cessation and behavioural determinants. The most prevalent GM’s were based on “Reward features” and “Narrative and identity features”.
2.4.1 Relation with other work
Desmet et al also found that SHG’s had positive effects on health behaviours (including smoking) [26]. However, at follow-up the positive effects on self-efficacy and intention to change health behaviour were substantially reduced. In our systematic review, 7 out of the 17 included studies did not have a up period. One study was not clear on having a follow-up period. And nine studies did include a follow-follow-up period, which varied between one week and six months. Out of these nine studies, five studies evaluated a SHG in isolation (without any co-interventions). These were the studies by Duncan, Girard (2007), Khazaal (2013), Metcalf and Parisod. Interestingly only Girard (2007) and Khazaal (2013) reported significant positive outcomes on (key behavioural determinants of) smoking cessation. The games in these two studies did not have any GM’s in common.
Taking the follow-up period in consideration is important if one wants to determine whether SHG’s are able to establish lasting behavioural change. If positive effects on smoking cessation are substantially reduced at follow-up, game developers might have to focus on
game-mechanics that keep the player engaged long enough to achieve lasting results. The four studies with a follow-up period that evaluated a SHG combined with a
co-intervention (counselling or pharmacotherapies) all reported positive significant results on (key behavioural determinants of) smoking cessation. This may suggest that combining SHG’s with a different type of intervention might have synergistic effects. These results are consistent with the systematic review by Ranney et al on self-help interventions for smoking cessations. This study reported that self-help strategies for smoking cessation marginally affect quit rates, but also indicated that when self-help is combined with pharmacotherapies and/or counselling a significant increase in cessation can be seen [40].
This systematic review has shown which GM’s are most prevalent in SHG’s for smoking cessation, and how these GM’s relate to the effectiveness on smoking cessation and behavioural determinants. However, since nearly every study reported positive effects on smoking cessation that are difficult to compare with each other, no strong conclusions on effective (combinations of) GM’s can be drawn.
2.4.3 Voluntary play sessions
In 16 out of 17 studies a fixed intervention period was used with a fixed number of play sessions each lasting a fixed time, to evaluate their games. However, when prescribing a serious SHG in an actual care program, the effectiveness will most likely depend on how frequent and how long patients voluntarily will play this game. Parisod et al was the only included study that did use a voluntary intervention period, voluntary number of play sessions and voluntary playtime per session. They compared their serious SHG with a non-gamified version consisting of comparable content, and found that the serious SHG was played twice as much, and was more effective than the non-gamified version.
20
2.4.2 Strengths and limitations
The used search query was built in cooperation with an experienced librarian. Therefore the search strategy was arguably a strong component of this systematic review.
This review had the following three limitations which may have influenced the results: 1) we extracted the included GM’s via the included papers, and did not play most of the games to verify these mechanics. Some mechanics might have been missed due to incomplete or
incorrect game information in the included papers. 2) The results of this review are influenced by the framework used for identifying GM’s. Although we selected the framework that seemed most fitting, some GM’s could have been missed due to the specific framework used. 3) Almost all included studies reported significant positive effects on smoking behaviour and key
determinants. Although this may be legitimate, we cannot rule out that publication bias has influenced our results.
2.4.5 Conclusions and implications
This study has shown that SHG’s supporting smoking cessation are effective overall, and provides insight into the mechanisms within those games. We found that every SHG included at least three of the GM’s in the taxonomy of GM’s that we used. The most frequently used GM’s were based on "Theme and genre", "General rewards", "Punishment" and "Graphics and sound".
We were unable to answer how effectiveness of SHG’s on changing smoking behaviour is associated with the presence of certain (combinations of) GM’s. In order to find causal evidence between (certain combinations of) GM’s and effects, it is necessary to do further research. With the results obtained from this systematic review we will now share our recommendations for future research.
Half of the studies in our review did not include a sufficient follow-up period in their methods, which is not only essential to measure lasting behavioural change, but is also important when assessing the effects between different GM’s as a function of time. For example, it is possible that some GM’s are more successful in changing behaviour on the long term, while other GM’s might only produce positive effects on the short term. In order to measure these differences, we recommend further SHG evaluation studies to include a proper follow-up period.
GM’s which are perceived as more fun, challenging and/or rewarding over the long term will probably result in games being played more often, as opposed to GM’s which are perceived as boring, too simple, and/or unrewarding. If there is a dose-response relationship present between play time and effect, which (up to a certain point) may seem plausible, it can be beneficial to study how we can voluntarily increase the dose (read: play time) of SHG’s.
Therefore we also recommend researchers to study whether such a dose-response relationship exists, and if so, implement voluntary play sessions in their evaluations, to measure how different (combinations of) GM’s influence the voluntary playtime.
21
3. A usability study of a serious health game for supporting smoking cessation
in disadvantaged (expectant) mothers
3.1 Introduction
In the previous chapter we have reported the findings of the systematic review on (effective) GM’s in SHG’s for smoking cessation. These findings were used as input for the study by Derksen et al (earlier referred to in chapter 1), in which “Kindle”, a SHG for smoking cessation support within “VoorZorg” was developed. (Details about “VoorZorg” were reported earlier in chapter 1 of this thesis).
Earlier in 2018, Derksen et al showed that the current interventions supporting smoking cessation in disadvantaged mothers, are not attuned to their motivational stage of change to quit smoking [41]. Based on these findings, it was decided to create Kindle with its intended users: so called co-creation sessions.
Kindle is designed to help (expectant) mothers with low socioeconomic status move through the early stages of change in smoking cessation by increasing, among others, self-efficacy. As stated earlier in chapter 1, self-efficacy has been identified as an important determinant of smoking behaviour change and relapse [14], [15], [42].
This is achieved in Kindle by using GM’s like goal setting, rewards and social functions. The application has separate user-interfaces for clients and nurses. Consequently it allows
VoorZorg Nurses to 1) communicate with their clients by private and group chatting, 2) reward their clients for their real-life achievements by giving one or more hearts to a specific goal, and 3) share daily tips on issues that their clients might have. The clients are able to 1) set goals and receive hearts from their nurse for achieving these goals, 2) communicate with their nurse and peer clients, 3) read/create tips by/for other clients, and 4) write private posts in their digital diary.
As in every game or computer-based application, the design choices influence the ease of use, satisfaction and effectiveness of the application, also known as usability. When designing new technologies and tools, usability remains an integral part in ensuring that heterogeneous target populations are able to interact with the technology as the developers intended [32].
The main objective of this study was to assess the usability of Kindle and to give
recommendations to the developers on improving its usability. The research questions were: 1) what is the usability of the co-created health app with GM’s for sustainable smoking cessation in vulnerable, young pregnant women? 2) What values do two usability evaluation methods bring in and to what extend do their results differ? And 3) what recommendations can be provided to the game developers to optimize the design of the health app?
3.2 Methods
Although Kindle is developed by using input from our systematic review on GM’s in SHG’s for smoking cessation (chapter 2), it is not “played” like a regular game (see figure 2 and 3 for in-app screenshots of the prototype).Therefore we could use conventional usability methods that are proven for testing non-gamified medical applications. A prototype of the health app Kindle was assessed using two usability evaluation methods in parallel. We applied both the Heuristic
22
Evaluation (HE) and the Think Aloud (TA) method [43] for the evaluation of Kindle, to get insight into potential usability problems. The HIMSS usability principles, which are specifically designed for the evaluation of mHealth apps, were used for the classification of the potential usability problems. The results of both evaluations were compared in order to understand the different perspectives offered by the techniques.
Peute et al conducted a usability study of a computerized physician order entry system, in which they compared the results of TA and Cognitive Walkthrough (CW). They found that problems identified by CW and TA were not mutually exclusive, as both CW and TA had overlapping results. However both CW and TA also identified unique problems, showing that a combination of TA and CW can provide greater detail of the potential usability problems [44]. In this current study we hoped to gain insight in whether a similar surplus value could be found by combining TA and HE.
3.2.1 Heuristic evaluation
The usability experts (n=3) who were recruited to perform the evaluation were Medical Informatics graduates of the University of Amsterdam, and had experience with usability evaluations.
23
Fig 3. Screenshot of the admin panel in the nurse-interface of Kindle.
For performing the HE the experts received a guideline containing: 1) A basic explanation about VoorZorg, why Kindle was developed, who Kindle’s end-users are, and what it should accomplish. 2) How to install and open the application on their smartphone device. 3) Information on how to perform the evaluation using the HIMSS usability principles [45] and Nielsen’s 5-point Likert rating scale (from zero: no usability problem to four: usability catastrophe) [46].
No training was given to the experts on how to use Kindle prior to the HE. The experts were asked to systematically evaluate each usability principle by freely exploring the application, and were encouraged to write comments to further explicate the rationale for their ratings. 3.2.2 End-user think-aloud protocol
Participants (n=10) were recruited via personal contact from the VoorZorg program. We included VoorZorg nurses (n=5) and VoorZorg clients (n=5) to test the respective user-specific interfaces. All the participants had prior experience with smartphone applications and none of the participants had prior experience with Kindle.
Following informed consent, participants were instructed to verbalize their thoughts as they received verbal instructions from an evaluator (SS or MD) to complete a series of tasks. The tasks were constructed based on real life scenarios in which all main functions of the prototype could be tested. If during the evaluation a participant stopped verbalizing her thoughts, the evaluator reminded her to do so. If a participant could not complete a given task after
24
exhausting multiple possible solutions, the evaluator would verbally support her to complete that specific task. The series of tasks were always conducted in a fixed order across
participants.
Before starting the series of tasks, participants were given a “warm-up” task to practice thinking aloud. They were asked to add a specific contact to the contacts list while thinking aloud. The series of tasks could be categorized as follows: 1) profile and group (nurse only) creation; 2) group management (nurse only); 3) creating (client only) and managing (nurse only) personal goals (this will be referred to as a “hartewens” (singular) or “hartewensen” (plural) from now on), 3) utilizing the chat functionalities, 4) reading and adding daily tips (this will be referred to “wist-je-dat” from now on), and 5) using the personal diary (client only).
After a participant had completed all tasks, she was asked two questions: 1) what did you like the most about the app and why? 2) What did you like the least about the app and why? Data collection time ranged from 20 to 50 minutes. Sessions took place at the clients’ homes, and the Voorzorg practices. Sessions were recorded via a third-party smartphone app, which recorded the screen, audio, and user inputs. One of the TA evaluators (SS) was responsible for analysing the videos, which consisted of summarizing participants’ utterancesand user inputs. A usability issue was reported when a participant was not able to complete the instructed task in her first attempt. Usability issues were categorized according to the HIMSS usability
principles, which were also utilized for the analysis of the results of the HE. The results of the HE and TA were compared in terms of number of (unique) issues, number of similar issues and the frequency of occurred issues.
25
3.3 Results
The heuristic evaluation yielded 46 and 31 potential usability problems for the client interface, and nurse interface respectively. The TA evaluation yielded 10 and 14 potential usability problems for the client interface, and nurse interface respectively.
Table 1 and Table 2 show the usability problems found by experts and end-users in the client interface categorized by the heuristic principles, for the client- and nurse interface
respectively.
Table 5. Client interface usability test results
1. Simplicity Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendation
General – It is confusing to use multiple icons in order to show in which menu the user finds itself in.
HE expert(s): “There is a
highlighted icon in the navigation bar of the corresponding page, and this same icon (which is non-clickable, but looks clickable) is also put in the right upper corner of the screen.”
Y N 2 out of 3 1.5
Only use the highlighted icon in the navigation bar, and (if necessary) use a title bar with the name of the menu on top.
Chat section – It is unclear
when messages were sent. Y N 1 out of 3 1
Include“date and time stamp” functionality. Chat section (group) - The
functionality to give a heart (or “like”) to a received message is not clear by just looking at it. It is not clear that the heart is a clickable button. Y Y 1 out of 3 experts 2 out of 5 end-users 2
The button should have a different placement on the screen, so the button does not
“blend in” with the message itself.
Diary section – Usage of the lock buttons next to the diary entries is unclear.
End-users had issues deleting a diary post, not knowing that this lock button had to be clicked.
HE expert(s):”The ‘lock’ buttons
next to the diary posts serve no clear purpose.” Y Y 2 out of 3 experts 4 out of 5 end-users
1 Remove the lock buttons.
Hartewensen section - The numbers above the golden
Y Y 3 out of 3
experts 3
Use tiny info buttons to explain the context of these
26 1. Simplicity Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendation
heart-icons are unclear. Experts and end-users don’t understand the context of these numbers, and how these are assigned.
1 out of 5 end-users
watch a one-time tutorial when first launching the app.
Hartewensen section - It’s not clear what the exact function of the “Hartewensen” page is, once a category is selected.
Y N 2 out of 3 3
Use tiny info buttons to explain the context of these
numbers, or let the user watch a one-time tutorial when first launching the app. Profile section - The checkbox
“VoorZorg Cliënt” has no clear goal when making a profile.
Y N 2 out of 3 2
Entirely remove the checkbox, since it has no
utility. Profile section – The user is
forced to upload a picture when creating a profile. This should be optional, since it is not necessary for the app to function properly.
Y N 1 out of 3 2 Give the user the option to
skip this process.
2. Naturalness Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendation
General – HE expert(s): “Most icons
are very unintuitive”. Y N 2 out of 3 3.5
Replace the icons by icons people know from their
own mobile devices and apps. Use material.io for
guidelines.
General - The red cross-icon and green check-icon have no clear meaning in their respective context.
Y N 2 out of 3 3.5 These should be replaced by explicit commands like: “Proceed“, “Go back”, “Save”, “Delete”, etcetera.
Chat section (group) - The chat-icon in the menu is not intuitive and should be more explicit.
Y Y
2 out of 3 experts 1 out of 5 end-users
2 Use material.io for guidelines.
Chat section - It is unknown what the heart-icon does next to the input
Y N 2 out of 3 1.5 The intended
27 2. Naturalness Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendation
field. unknown since the
button does not function. Therefore
a possible recommendation
cannot be given. Chat section (group) - The group chat
and private chat being separate functionalities is confusing. HE expert(s):” I would expect the
group chat to be part of the normal chat functionality. Not a separate menu button.”
Y N 2 out of 3 3
Create a chat functionality with private and group chatting as sub-functionalities. Just like how other chat apps operate (e.g.
WhatsApp, Telegram, Signal). Chat section (group): It is not clear
which messages are sent by the user,
and which are received. Y Y
2 out of 3 experts 1 out of 5 end-users
1.5
Display the user’s messages outlined right at the screen,
and others’ messages outlined
left.
Diary section - It is not clear that the lock icon in the navigation bar is supposed to present a diary. HE expert(s):”The lock icon seems
more like a functionality to lock the main app behind a password, instead of a diary functionality.” Y Y 2 out of 3 experts 1 out of 5 end-users 3
Use an icon that resembles a diary,
and add the title name in the navigation bar as well. (For the sake of consistency, all the navigation icons should have
added titles). Check material.io for more detailed
guidelines. Wist-je-dat section - It is not clear
what the plus-sign is supposed to mean.
N Y 1 out of 5 N.A.
Use text (e.g. “Voeg toe”) instead of a
plus-sign.
Hartewensen section - The pencil-icon to edit the “Hartewensen” seems redundant. Editing is also possible by clicking on the input field.
Y N 1 out of 3 3
Since a hartewens should not be edited frequently
(if not, at all), remove the option
to edit by clicking on the input field. Hartewensen section - The trash-icon
to delete a “Hartewens” seems
Y N 2 out of 3 2 Since a hartewens
28 2. Naturalness Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendation
redundant. Deleting is also possible by clicking on the input field, and erasing the text.
deleted frequently (if not, at all), remove the option
to delete by clicking on the
input field. Hartewensen section - It is not clear
that the “Ga naar de behaalde Hartewensen” text is actually a clickable button.
N Y 4 out of 5 N.A.
Let the button stand out more. Use the guidelines
by material.io.
Profile section – It is unclear when the profile creation is (almost) finished. The green checkmark and red cross icon have no clear meaning.
Y N 2 out of 3 3
Use buttons like “next”, “back” and
“create”, to explicitly let the user know what the result will be.
3. Consistency Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
General - The placement of the main icons of each page are not consistent. The single- and group chat icons are in the top left of the page; while the diary, “Hartewensen” and “wist-je-dat” icons are on the top right.
Y N 2 out of 3 1
Be consistent in icon placement on the screen. Either choose all icons to be
placed right, or left.
Chat section (group) –The group chat does not have the heart-icon next to the input field, unlike the private chat.
Y N 2 out of 3 2
Be consistent in the user-interface across both chat functionalities. (Unless the heart-input is a specific feature
which only should exist in the private chat). Chat section (private) – It is
unclear if the private chat is actually is a private
conversation to one person. HE expert(s): “The private chat
functionality gives me the illusion that it is private, but in the example I see “Anneke van Dam” as well as “Jolanda Seegers” sending me a message in one chat, making it a group
Y N 1 out of 3 2 Make the private chat indeed
29 3. Consistency Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations chat.”
Diary section - The lock-icon which represents the diary in the navigation bar is the same icon as the lock/unlock icon in the top right corner of the diary page. The icons have different meanings, and are therefore confusing.
Y N 1 out of 3 1
Use different icons for the navigation button, and the icon which shows whether the
diary is locked/unlocked. 4. Forgiveness and Feedback Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
Diary section - It is not possible to change your pincode after you’ve created one.
Y N 2 out of 3 3 Create a “my profile” menu where
you can change your pincode.
Profile section - Once a picture is uploaded there is no way to change it.
Y N 3 out of 3 3
Create a “my profile” menu where you can change your profile
picture. Profile section – It is not
possible to change your name after its creation.
Y N 2 out of 3 3 Create a “my profile” menu where
you can change your name. Profile section - The app
gives no feedback how many steps there are for completing a Kindle profile.
HE expert(s): “I have no
idea when I am (almost) done creating my profile”.
Y N 1 out of 3 2
Give feedback about the profile creation process. How many steps
are needed? When is the user almost done creating the profile?
5. Effective use of language Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
Diary section - In the diary app the word “sleutelcode” is used instead of “pincode”.
Y N 1 out of 3 2
Use common terminology that users will definitely understand. “pincode” is used
across multiple other services that individuals use, and is therefore
30 5. Effective use of language Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations This could be confusing.
preferred over “sleutelcode”.
Hartewensen section - It is not clear that “behaalde
hartewensen” list the hartewensen which are already completed.
N Y 2 out of 5 N.A.
It could be that the word “behaalde” is hard to understand for end-users. If that is the case, a replacement like “voltooide”
could be used.
Another cause for the problem could be that, the way that the “huidige hartewensen” is presented, is too similar
of that of “behaalde hartewensen”, and therefore end-users don’t notice the difference between the pages. If this is
the case, a different lay-out could be used to emphasize the difference
between the pages. Profile section - There
is no explanation for creating a profile. HE expert(s):” Why
should I create a profile for this app. Why it is beneficial? Who can find me? How is my privacy secured?”
Y N 2 out of 3 2 Explain why the end-user needs to create
a profile.
Profile section - “Maak hier de app persoonlijk” is confusing when needing to upload a picture.
Y N 1 out of 3 2
It should explicitly say what you want the user to do. In this case: “upload a
picture”. Hartewensen section - The term “Hartewensen” is unknown to experts. Y N 2 out of 3 3.5
If the term is indeed unknown for the end-users, an information button could
be used for explanation. Else, this issue can be ignored. 6. Efficient interactions Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
General - dismissing the keyboard is only possible by clicking “return”. Clicking on the screen does not work,
N Y 2 out of 5 N.A. Let the user dismiss
the keyboard by pressing on a spot
31 6. Efficient interactions Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
although this works in all other apps. on the screen which
is not the keyboard. Chat section – There is no search
functionality in chats.
HE expert(s):”Users should be able to
search for keywords in the chat for efficient information retrieval”.
Y N 1 out of 3 2
Include a search functionality in the
chat.
Diary section - The pincode functionality is broken.
HE expert(s): “The pincode has to be
created again and again every time the diary is opened.”
Y N 1 out of 3 4
After creating the pincode, ask the user for the pincode instead of creating it
again.
Wist-je-dat section – There is no option to browse through existing “wist-je-dat” entries.
Y N 1 out of 3 2
Create an overview of wist-je-dat entries where the
user can scroll through. 7. Effective Information Presentation Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
Diary section – Edit and delete icons are too small and too close to each other.
Y N 1 out of 3 1
Put them further away from each other to avoid
mistakes. Hartewensen section – The
description input field might be too small for some hartewensen.
Y N 1 out of 3 1
Make the description field bigger to allow for more
characters. Hartewensen section – The
green checkmark (apply) button and the red cross (cancel) button are very close to each other, and off-centred.
Y N 1 out of 3 1
Centre the buttons and create more space
between them.
8. Preservation of
context Found by expert
Found by end-user Detected by # of evaluators
Average severity rating
32
9. Minimize cognitive overload
Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
Diary section – Users are unable to give a “subject” (or label) to their diary entries in order to find specific entries more easily.
Y N 1 out of 3 2 Create a subject field which is presented in the overview page of diary entries Legend: Y = yes, N = no
Table 6. Nurse interface usability test results
1. Simplicity Found by expert Found by end-user Detected by # of evaluators Average severity rating Recommendations
General - Experts found it unclear if you can have multiple groups as a nurse.
Y N 1 out of 3 3 Inform the end-user how many
client-groups can be created.
General – It is confusing to use multiple icons in order to show in which menu the user finds itself in.
HE expert(s): “There is a
highlighted icon in the navigation bar of the corresponding page, and this same icon (which is non-clickable, but looks clickable) is also put in the right upper corner of the screen.”
Y N 2 out of 3 1.5
Only use the highlighted icon in the navigation bar, and (if necessary) use a title bar with the name of the menu
on top.
Admin section - It is not intuitive to add a client to your group.
Expert(s): “You can add a
name or phone number. Does the app look in your contact list? Does it look into a list of registered Kindle users? Why can I choose between these two options?”
Y N 1 out of 3 3 Inform the user how clients are found
