Designing DietCare: Gamification elements for monitoring beverage
intake in a health promoting mobile coaching application
SUBMITTED IN PARTIAL FULLFILLMENT FOR THE DEGREE OF MASTER OF SCIENCE
Chris Verbeek
11264128
M
ASTER
I
NFORMATION
S
TUDIES
H
UMAN-
C
ENTERED
M
ULTIMEDIA
F
ACULTY OF
S
CIENCE
U
NIVERSITY OF
A
MSTERDAM
July 21, 2017
1st Supervisor 2nd Supervisor
Dr. Frank Nack Dr. Maartje Poelman
Designing DietCare: Gamification elements for monitoring
beverage intake in a health promoting mobile coaching
application
Chris Verbeek
ABSTRACT
In this paper, three sets of gamification elements were investigated with the purpose of identifying fitting sets to stimulate the measurement of beverage intake in a health promoting mobile coaching application. One control prototype and three gamified prototypes, where each gamified prototype used a different set of gamification elements, were developed and tested in a 7-day experiment with four conditions, namely one control condition and three experimental (gamified) conditions. The studied population consisted of 35 adolescent students residing in the Netherlands. Both data generated by the prototypes and post study questionnaires were collected to evaluate the effectiveness of the gamified prototypes. Although we lack statistical evidence, results indicate that the use of gamification elements had positive effects, regarding the measurement of beverage intake, where the gamified prototypes showed a larger number of beverage submissions compared to the control prototype. Also, results of the post study questionnaires show some indications that the gamified prototypes outperformed the control prototype with respect to the prototypes’ ability to stimulate the submission of beverage intakes. Furthermore, our results also highlight the importance of individual preferences with respect to gamification elements and overall prototype appearance.
General Terms
Gamification, Self-monitoring, Experimentation.
Keywords
Application Design, User Experience, Dietary behavior, Sugar Sweetened Beverage Consumption.
1. INTRODUCTION
The global increase of the consumption of sugar sweetened beverages (SSB) has accelerated rapidly, making SSB the beverage of choice for millions of people spread across the world. In the Netherlands, approximately 1549 million liters of soda, a popular example of a SSB, was consumed (91.5 liters per capita) in 2015, making soda one of the most consumed beverage [1].
Although widely consumed, SSB consumption is a known public health issue and is often associated with obesity, type 2 diabetes, heart disease, dental erosion and numerous other chronic conditions [2]. The consumption of SSB being a major risk factor for type 2 diabetes is further emphasized by the Health Council of the Netherlands, who recommend that the consumption of SSB should be kept to a bare minimum [3].
Several mobile health (m-health) applications try to address these health challenges by delivering health promoting interventions to individuals, stimulate healthy eating behaviors, improve weight management, and facilitate nutritional monitoring [4,5,6]. The increasingly widespread use of gamification elements, in health promoting applications, have led to a proliferation of so called ‘gamified’ applications. These applications typically implement game-like elements such as rewards, achievements, challenges and
other incentives in order to increase motivation and application engagement.
Throughout this thesis the following definition of gamification is used: “the use of game design elements in non-game contexts” [7, p.2].
Reviews of the use of gamification elements in health promoting applications show that gamification has become increasingly popular [8], although few studies make attempts to evaluate the effectiveness of gamification and its usage in the context of SSB consumption.
In order to develop an application that stimulates healthy beverage consumption (e.g. consume less SSB) it is imperative that the application first gains an accurate insight into the user’s dietary behavior, with respect to overall beverage consumption, and the extent to which SSB contributes to the user’s daily drinking pattern. This paper describes the rationale and the development of multiple prototypes of ‘DietCare’, designed to be used in a one-week randomized controlled experiment to measure the effectiveness of sets of gamification elements to monitor beverage intake in a health promoting mobile coaching application.
First, we discuss related work followed by presenting the problem statement and the constructed research question. Then, the identified sets of gamification elements are described, the developed prototypes are explained in detail and the study design is presented. Finally, we discuss the results, identify limitations of the conducted research and suggest areas to be explored for future research.
2. RELATED WORK
Although using a gamification based approach for the purpose of measuring beverage intake is a rather unexplored area, several similar ideas of using gamification as a stimulation tool have been implemented to promote physical activity [9] and motivate students to learn [10,11].
The use of gamification is also notable when looking at mobile applications that focus on measuring and quantifying health related aspects such as physical activities and dietary consumptions. For example, numerous gamified running and nutritional tracking applications exist [12] where gamification is used to expose personal information such as daily calories consumed or time spent running.
2.1 Similar Applications
An example of a gamified application is DietCoach1 which
distinguishes itself from conventional nutrition tracking applications by not only providing a nutritional diary, but also implementing gamification elements. For example, rewarding users for making healthy dietary choices in the form of awarding points, a leveling system based on accumulated points, daily challenges for a healthier lifestyle (e.g. drink 8 glasses of water, park one’s car far away to encourage walking) and the usage of an avatar. One of the key features of DietCoach are the self-control challenges which aim
to stimulate users to avoid certain types of nutrition, including SSB. It tries to accomplish this by rewarding points when the user succeeds in avoiding specific products, such as SSB, for a full day. Although DietCoach does not allow the user to monitor soda intake, it does provide a way to keep track of water consumption and awards the user with points when a glass of water is consumed. Another gamified nutrition tracking application is Foodzy2 which
has a large amount of badges that can be earned. For example, badges are awarded when large amounts of vegetables are consumed, the user has eaten varied and healthy for a specific amount of time, and when the application has been consistently used for a number of days. A notable observation is that Foodzy also rewards its users when less healthy foods or beverages are consumed. For instance, the ‘Pizza Master’ badge is awarded when multiple pizzas are consumed and the ‘Hangover’ badge is awarded when multiple alcoholic consumptions are submitted.
An application that incorporates elements of gamification to monitor dietary intake is 'Snack Track School' [13]. The development of the application is part of a study which aims to improve the snacking pattern of Flemish adolescents between the age of 14 and 16. The application is used in a four week intervention where participants get awarded with points for every snack they enter using the ‘Snack Track’ tool. The amount of awarded points is based on the nutritional value of the consumed snack; the healthier the snack, the more points they receive. To prevent participants to accumulate points by merely keeping track of as many snacks as possible, which can lead to unwanted unhealthy snacking patterns, limitations are built in the application with respect to the point awarding system. Participants are able to enter unlimited snacks, however, will only get awarded for the first ten snacks. To stimulate a more well-balanced snacking pattern and to try to keep ‘non snackers’ engaged, the application also includes the following gratuities in the form of additional points for: (1) a total snack intake of less than seven per day, (2) a snack intake of two/three healthy snack per day and (3) participants who do not consume any snacks but still login for more than three times per day.
2.2 Effectiveness and Criticism of
Gamification
In an extensive literature review, several empirical studies on gamification have been reviewed making attempts to evaluate the effectiveness of gamification and its usage in different contexts such as health, education and commerce [14]. Although the results of the review suggest that gamification provides positive effects, such as making a digital application more fun and engaging, the authors state that these effects are highly influenced by the users using it and the context in which the gamification is being used. A more recent systematic review of the literature on the effectiveness of gamification, in the health domain, also showed evidence that supports gamification and its potential to have a positive impact on both health and wellbeing, and health behaviors [15]. However, the authors state that the results should be interpreted with caution as more than half of the reviewed studies lack the comparison between gamified and non-gamified interventions; these studies mainly focused on making comparisons between gamified interventions or evaluating just a single gamified application.
Two of the main criticisms of gamification are that it is not user-centric, emphasizing on the application owner’s needs instead of the user’s goals, together with gamification often using external rewards (e.g. points, badges) as a motivating factor rather than
trying to intrinsically motivate [16]. The use of these external rewards could potentially even reduce intrinsic motivation [17]. No studies on gamification in the context of measuring beverage intake have been reviewed and as of today, we are not aware of the existence of such a study. For this reason, it remains unclear if gamification is effective in the context that we aim to investigate; stimulate the measurement of beverage intake.
3. CONTEXT & PROBLEM STATEMENT
This thesis is conducted within the context of a running research project at the University of Utrecht.
The aim of the research project is to design a gamified mobile e-coaching application ‘DietCare’ (working title) to improve healthy beverage intake of adolescents from low socioeconomic status. In the context of this research project, mobile e-coaching refers to automated personal coaching by using smartphone technologies. The system will foster awareness about healthy beverage consumption and includes personalized self-regulation behavioral change strategies, such as self-monitoring, planning and setting goals. To support the learning process and behavior change, the system will integrate so-called persuasive strategies, such as a fully automated e-coach and elements of gamification. DietCare’s initial focus is on soda consumption only, where the aim is to decrease soda intake among members of the target audience.
In DietCare, 3 distinctive activities can be identified, namely: (1) monitoring beverage intake, (2) goal setting and (3) providing feedback.
First, the application will allow the users to keep track of their daily beverage intake. Based on this insight, DietCare and the user will set goals, for example reducing soda intake by a specific amount, and make plans on how to achieve these goals. Then, based on set goals and continued monitoring, DietCare can provide appropriate feedback.
Monitoring beverage intake is essential in the mobile coaching process where its results are highly dependent on users that accurately keep track of their beverage consumptions; missing or incomplete data will lead to faulty goal suggestions and feedback. This potential lack of complete and accurate data is a problem and stimulating users to actively use the application and record their beverage intake is a challenge that needs to be addressed. To examine potential solutions, to the aforementioned challenge, we identify sets of gamification elements. This research aims to investigate whether or not these specific selected sets of gamification elements can stimulate the measurement of beverage intake. Therefore, the following research question is asked:
What are fitting sets of gamification elements to stimulate the measurement of beverage intake in a health promoting mobile coaching application?
A sub-question is constructed using the following format:
Does [identified set of gamification elements] stimulate the measurement of beverage intake in a health promoting mobile coaching application?
In order to provide an answer to the research question, we undertook the following initiatives.
We first identified suitable sets of gamification elements by conducting semi-structured interviews with gamification experts and by exploring the current body of literature on gamification. Then, we developed four prototypes to be tested with the target audience; Dutch adolescents from low socioeconomic status. These
prototypes consisted of a control prototype, a prototype which used no set of gamification elements, and three gamified prototypes; each of which incorporated one set of gamification elements. Finally, we tested the prototypes in a one-week randomized controlled experiment followed by the collection of data and analysis.
4. IDENTIFICATION SETS OF
GAMIFICATION ELEMENTS
Two main activities were carried out to identify suitable sets of gamification elements, namely: (1) expert interviews and (2) exploration of existing gamification frameworks and player type models, suggested by these experts. Based on the results of these activities, three suitable sets of gamification elements were constructed.
4.1 Expert Interviews
4.1.1 Setup and Recruitment
In total, four expert interviews were conducted in Dutch, three of which were performed face-to-face and where one was performed via e-mail.
The average length per face-to-face interview was 40 minutes and was recorded using a digital recorder. The group of experts consisted of a gamification researcher (“E1”), two gamified application developers (“E2”, “E3”) and one gamification solution developer (“E4”). In addition to the aforementioned four experts, a fifth expert, the CEO of a company that specializes in developing gamified applications, was contacted for the purpose of conducting an interview (“E5”). Although this appointment was later cancelled due to timeslot issues, he did provide useful information by suggesting a gamification framework/model which later got mentioned by a second expert as well.
All experts were contacted via e-mail and were informed about the purpose of the research and the main focus of the interview. Informed consent was obtained from the experts prior to each of the interviews. Each interview followed a predefined protocol (Appendix A) which included introductory gamification-related questions (e.g. definition, main ingredients, practical applications) and core questions that covered gamification elements that would be appropriate to implement and their potential pitfalls. Finally, successful similar applications were discussed. These interviews were transcribed (Appendix B) and used to create a coding scheme (Appendix C).
4.1.2 Results
Note that all interviews were conducted entirely in Dutch and exemplary quotations were translated into English.
Interviews started with introductory questions where the interviewees were asked about their definition of gamification, its main ingredients and how it is being used in the industry at the moment. All interviewees had similar explanations about what gamification implied, showing that there was a general consensus among interviewees on the definition of gamification, which is “the
use of game elements in non-game contexts” (E2).
Two interviewees mentioned that gamification is often used in healthcare (E2) and health related applications such as running applications (E3).
4.1.2.1 Gamification Elements
All interviewees mentioned the following elements which are often used in gamified applications: (1) badges, (2) progression and performance bars and (3) social interactions, peer pressure and
competitions. Three out of four interviewees also mentioned leaderboards and digital points as potential elements which could be implemented. When looking at the current body of literature on the use of gamification, we see that the elements that were mentioned are commonly used in a wide variety of applications [14, 15]. Two interviewees mentioned the use of avatars where one interviewee mentioned using an avatar as a virtual conversational partner which would ask the users about their beverage intakes (E4), whereas the other interviewee suggested using a young virtual buddy; an avatar that the users need to take care of and might feel responsible for (E1).
Two similar gamification elements mentioned by two interviewees were “instant feedback: if I do well I want the system to tell me that
I am doing well” (E2) and “visual feedback: progression bars that need to be filled, use of colors, etcetera” (E3). It became apparent
that both the visualization of progress and instantly notifying the user of a significant happening, for example ‘leveling up’ or obtaining a new badge, were two must-have elements.
Three interviewees (E1, E2, E4) mentioned the importance of randomness with respect to rewarding users with points and badges. For example, although letting the users know upfront when they will receive a certain badge can be motivating, awarding points and badges at random might make using the application “more exciting” (E4) and can create the feeling of “the system recognizes my
behavior” (E2).
4.1.2.2 Concerns and Potential Pitfalls
Interviewees also expressed their concerns about using some of the aforementioned elements as motivators, stating “a leaderboard only
motivates the top 10% and might even demotivate the others” (E2)
and “one of the potential pitfalls of competitions is that people might
quit because they are not winning and have the feeling that they will never catch up” (E4). These concerns could be addressed by making
a relative leaderboard; a leaderboard where one only sees users with slightly higher or lower ranks. Still, it became evident that implementing any kind of competition and working with leaderboards should be approached with caution.
Also, it was mentioned that when rewarding the users with points or badges, for keeping track of beverage intake, it is of great importance that they do not get rewarded for the individual submission as this might lead to cheating behavior; submitting multiple times to get more points (E2). Therefore, some limitations need to be built in order to prevent these behaviors from occurring. One interviewee felt very strongly about using multiple gamification elements in a single application, stating “I would never
recommend using only one gamification element, unless you have a very specific target audience” (E1).
Finally, one interviewee highlighted the importance of “keeping a
link to the real world” (E2), meaning that one should always keep
in mind that the gamification is only implemented to support the goal the application means to achieve. A second interviewee made a similar remark and mentioned that “[…] the biggest flaw I see in
many gamified applications is that the game has started to live its own life” (E4).
4.1.2.3 Similar Applications
No gamified applications in the context of stimulating the measurement of beverage intake were mentioned. However, interviewees did mention popular applications that use gamification elements such as MyFitnessPal3, Runkeeper4 and Balance-it5,
together with successful games such as Pokémon Go6 and Candy
A notable observation was that two experts mentioned the so called ‘Bartle taxonomy of player types’ [18], one of the oldest gamer psychology models, and the more recent ‘Marczewski’s player type framework for gamification design’, which has several commonalities with Bartle’s player types (E1, E3) [19]. Two other experts (E2, E5) strongly suggested to look at the ‘Octalysis’ gamification framework [20].
Both Bartle’s and Marczewski’s player type models will be discussed in the following section as both of them, together with the suggestions mentioned by the experts, laid the foundation of the developed prototypes. Although the Octalysis framework was examined, it was not implemented in the design of the intended prototypes. This was mainly due to the fact that the Octalysis framework is very detailed, consisting of five separate levels (layers), and partly focuses on improving the longevity of gamified systems; designs that can last over decades. Because this paper was written in the context of a master’s thesis, with only limited time available, implementing such a detailed framework would not be feasible, therefore this gamification framework will not be discussed.
4.2 Player Type Models
The so called ‘Bartle’s player types’ is one of the most referenced theories on game psychology where the following four player types were identified based on gaming preferences: achievers, explorers, socializers and killers. Each of these player types were characterized by having specific preferences with respect to gaming elements. For example, achievers focus mainly on obtaining points, prizes and badges whereas explorers just want to learn or discover new things. Socializers primarily focus on the social aspects of a game, such as interacting with other players or working in teams, while killers just want to beat other players in a competitive setting. These classifications were based on observing player interactions in so called multi-user dungeon (MUD) games.
However, more often than not, a gamified application does not resemble an actual game but only uses game-like elements. Because of this reason, Bartle’s player types may not be entirely accurate in the context of a non-game application.
Marczewski took another approach to creating user types, in the context of a gamified system, which builds on Bartle’s player types. A six user type system was created where the following types were identified: (1) socializers: motivated by relatedness and want to interact with others, (2) free spirits: motivated by autonomy and want to create and explore, (3) achievers: motivated by mastery and want challenges to overcome, (4) philanthropists: motivated by purpose/meaning and want to enrich the lives of others, (5) players: motivated by rewards and will do whatever is needed to collect and (6) disruptors: motivated by change and want to disrupt the system. Additionally, an extensive list of gamification elements that support these various user types were identified, such as quests, guilds, unlockable content, lotteries and avatar customization. These aforementioned user types are divided into three categories, namely users that are willing to play (players), users that are less willing to play (socializers, free spirits, achievers and philanthropists) and users that are not willing to play (disruptors).
Both Marczewski and two interviewees (E1, E3) highlight the importance of designing for multiple user types as “some players
find competitive elements motivating while others find social interaction more important” (E1). This design approach also takes
into account that most users cannot be assigned to one single user type as it is likely that they display traits of other user types to some degree.
4.3 Gamification Sets
Based on aforementioned results, three sets of gamification elements were created where each set consisted of multiple elements which primarily focused on one user type, as described by both Bartle and Marczewski.
4.3.1 ‘Challenges and Rewards’ set
This set focused on the Marczewski’s ‘Players’ and Bartle’s ‘Achievers’ user type. These user types are motivated by rewards, badges, achievements and collecting points, therefore the ‘Challenges and Rewards’ set consisted of the following gamification elements: badges, achievements, levels, points and a clear visual presentation of progression.
4.3.2 ‘Avatar’ set
The ‘Avatar’ set focused on Marczewski’s ‘Philanthropists’. This user type needs to understand the meaning or purpose of what they are doing. One of the suggested gamification elements that supports these types of users, mentioned by Marczewski, is care-taking; allow users to take a parental role. This is in line with one of the expert’s recommendations; implementing a virtual buddy that the users need to take care of and might feel responsible for (E1). It was decided to implement an avatar which would gradually move from a sad-looking state to a happy state, and vice versa, depending on the user’s behavior.
4.3.3 ‘Social’ set
This set primarily focused on Marczewski’s and Bartle’s ‘Socializers’ user type. However, because we wanted to include some type of competitive element in one of the gamification sets, for the purpose of specifically targeting competitive users which were less targeted in the previously described sets, it was decided to include competitive team elements where team points could be collected. Because these competitive elements and team points were implemented, this set could also be potentially appealing to Marczewski’s ‘Players’, and Bartle’s ‘Killers’; players that like competitive elements and like to beat other players. The elements used in this gamification set are (1) a competition element, (2) a team element (3) a social pressure element and (4) team points. Additionally, the following mandatory features were set by the researcher. All prototypes should have features that:
Allow participants to submit their beverage intake and its amount (M1)
Allow these beverage submissions to be viewable by the participants at any moment (M2)
A total of four preliminary prototypes were developed where one prototype used no elements of gamification whatsoever and where three prototypes used one of the previously described sets.
5. FOCUS GROUP
In order to pre-test and collect opinions about the three preliminary prototypes, where each prototype implemented one of the three aforementioned gamification sets, a focus group that consisted of members of the target audience, namely Dutch ‘middelbaar beroepsonderwijs’ (mbo) adolescent students, was constructed. This focus group was used to elicit perceptions about the preliminary prototypes to see if any additional requirements needed to be set and to uncover any potential usability issues that may influence prototype adaptation. The focus group was used to analyze three different aspects of the overall research project, namely, (1) the needs and requirements of DietCare as a whole, (2) gamification elements to stimulate the measurement of beverage intake and (3) the needs and requirements of the ‘eBuddy’;
DietCare’s virtual coach which will guide the user and provide feedback throughout the whole coaching trajectory.
5.1 Setup
Members of the focus group were recruited at the ‘Wellantcollege Rijswijk’, a mbo school located in Rijswijk, the Netherlands. It included 13 students (7 males, 6 females). The focus group session lasted one hour and was divided into three rounds lasting 20 minutes each. Students were split up into three groups and assigned to one of three tables. Each table was overseen by one researcher who was responsible for guiding a planned discussion, covering one of the three project-related aspects, and allowing students to interact and respond to each other (Figure 1).
Figure 1. Focus Group Setup
At the 18th minute mark of each round an alarm would go off
indicating that the round was about to end, leaving room for the researchers to wrap up the discussions. When a round was finished, the groups switched tables. All discussions were recorded, using a smartphone, were transcribed (Appendix D) and used to create a coding scheme (Appendix E).
5.2 Materials
To demonstrate the prototypes, both a touchscreen laptop and two smartphones were placed in the middle of the table as well as printed versions of all prototypes (Figure 2).
Figure 2. Focus Group Materials
In addition, a smartphone was placed on the table for audio recording purposes.
5.3 Procedure
First, students were placed on the opposite side of the table so that the researcher could clearly see, hear and observe the students’ interactions with the prototypes. The smartphone was set to record at the beginning of each round.
After the students sat down, a volunteer from each group was asked to perform a few basic tasks such as switching between pages and making beverage submissions. The goal here was to identify any usability issues that might influence prototype adaptation. The researcher acted as an observer, took notes and questioned the student after the tasks were completed to see whether or not any difficulties were encountered. Fellow members of the group were also asked if they had any suggestions to potentially improve the usability of the prototypes.
Secondly, all gamified prototypes were explained both verbally, with the help of the printed versions, and by demonstration using a laptop which had all four prototypes installed. The students observed and were asked for their opinions, suggestions for improvement and general remarks.
5.4 Results
Before presenting the results of the group session, it has to be noted that not all students actively participated in the discussions; three students were very shy and did not express any opinions even when asked directly. This means that the results presented below do not reflect the views and opinions of the whole group, but only of those that did actively participate. The discussions were conducted in Dutch and exemplary quotations were translated into English. None of the volunteers, that tested the prototypes, experienced any difficulties while performing basic tasks such as submitting beverage intakes and navigating through the various pages of the prototypes. Both the students that volunteered and the students that observed mutually agreed that everything was clear and made no comments or suggestions to potentially improve the prototypes with respect to usability.
When students were asked to express their attitudes towards the three gamified prototypes, it became immediately clear that the avatar prototype was perceived as being the least favorable prototype by both male and female students. One of the main reasons for this was that the avatar appeared to be too ‘girly’, making it very unattractive for male students. Remarks were made such as “this is more for girls I think” (Group1, Girl1) and “A boy
would probably see this as a joke” (Group1, Girl2). Two male
students specifically mentioned the avatar and its ability to cry, stating “The crying deters me […]” (Group3, Boy1) and “I could
not care less if it started to cry” (Group2, Boy1). The students made
several suggestions to potentially improve the avatar prototype, such as making five avatars and letting the user choose which one they would like to use (Group3, Girl1) and making the avatar ‘grow up’ instead of transitioning the avatar from a sad (crying) to happy state (Group3, Girl2). None of the students expressed that they found the avatar prototype to be the most preferable out of the three gamified prototypes, except for two females (Group1, Girl2 and Girl3) but only if the avatar would be customizable (e.g. possibility to change hair, adjust clothes).
Two out of nine students, that expressed their personal preference with respect to the prototypes, stated that they preferred the social prototype and one students stated that it was a toss-up between the social and the challenges prototype. No noteworthy remarks regarding the social prototype were made other than one female student, who mentioned that inactive teammates could potentially have a negative effect on one’s own submission behavior; feeling less inclined to make submissions because teammates show similar behaviors (Group3, Girl1).
The same student suggested combining the social and the challenges prototype because “Well you also just do this for
this to see how well you are doing and not how well your team is doing” (Group3, Girl1). Four students stated that they preferred the
challenges prototype. Again, no noteworthy comments were made regarding the challenges prototype other than the aforementioned suggestion of combining the social and the challenges prototype. When asked whether or not the students felt that the gamification elements would stimulate them to keep better track of their beverage intake, compared to an application that would use no gamification elements whatsoever, all of the students mutually agreed that it would.
When asked about the visual appearance of the prototypes, four out of six responses were positive stating that the prototypes looked good. One student found the appearance to be too whole-colored and suggested to use brighter colors (Group2, Boy4), and one student found the layout to be too green (Group3, Girl1).
Finally, some individual remarks were made such as incorporating social media to share progress (Group2, Boy4), adding online rankings (Group3, Boy1) and adding messages to remind users to submit their beverage intakes (Group2, Boy1).
Based on these results, one major addition and one small alteration were made. Because it was very clear that the avatar, used in preliminary version of the avatar prototype, was perceived to be too girly and unsuitable for male users, it was decided to add an additional avatar. This avatar tried to address the issue of being too girly by having a man-like appearance and by not displaying any sort of sad emotions such as crying (see section 6.2.2). Finally, one small adjustment was made to the color scheme of all prototypes which resulted in slightly brighter colors.
6. PROTOTYPES
6.1 Overview
The control prototype covered all the mandatory features that were set (M1, M2) but did not utilize any gamification elements. The three gamified prototypes extended these basic features with one of the previously defined sets of gamification elements (see section 4.3).
6.1.1 System Specifications
Although the installable Android package files (APK) were developed using the Intel XDK development environment8, the
resulting smartphone applications merely functioned as a web browser which connected the applications to a website specifically tailored to be viewed and used by a smartphone.
The prototypes were specifically developed to be run on Android9
4.4+ devices. The prototypes also required a stable connection, preferably Wi-Fi or 4G, to the internet to allow for smooth transitions between pages, and 25 MB free internal storage space.
6.1.2 Basic Features
All prototypes incorporated three basic features, namely the possibility to submit beverage intake (Figure 3a, Figure 3b) and its amount (Figure 3c), together with a logbook functionality (Figure 3d); a dedicated page where the participants could review all of their submissions. These two features could be accessed by clicking the ‘home’ and the ‘logboek’ (“logbook”) buttons, located on the bottom menu of each prototype. Additionally, each prototype had a “day assessment” feature which was implemented to allow participants to state, for each day of the study, how accurate their beverage submissions were.
6.1.2.1 Beverage Intake Submission
During a beverage intake submission, six types of beverages could be selected which were water, soda, energy drink, yoghurt drink, alcohol and others. After the beverage type was selected, four different units were presented, namely a sip, a glass, a can and a bottle which corresponded to 30ml, 250ml, 330ml and 500ml respectively. In addition to unit selection, the amount of a unit could also be specified by dragging the slide bar to the right, increasing the units, or left, decreasing the units.
6.1.2.2 Logbook
All beverage submissions could be reviewed by accessing the logbook page. Here, all submissions were presented both textually, ordered by date, and visually in the form of two pie graphs. One graph represented the total amount of submissions per beverage type, whereas the second graph represented the total amount of volume (ml) per beverage type.
6.1.2.3 Day Assessment
A day assessment could be submitted by tapping the ‘Nu invoeren’ (submit now) button displayed next to the day to be assessed (Figure 4). Participants could then self-assess their intake submission accuracy using a 5-point response format, ranging from 1 to 5 where 1 equaled ‘did not keep track of anything’ and 5 equaled ‘kept track of everything’, for each day of the study.
a) Beverage Submission b) Beverage Type Selection c) Beverage Amount Selection d) Beverage Log
Figure 4. “Dagbeoordeling”
Assessments could only be submitted for days in the past.
6.2 Gamified Prototypes
Each gamified prototype extended the aforementioned basic features with one of the previously described sets of gamification elements.
6.2.1 ‘Challenges and Rewards’ Prototype
The status of the gamification set used in this prototype (e.g. current level, amount of accumulated points, completed achievements) could be viewed by accessing the ‘challenges’ page (Figure 5).
Figure 5. ‘Challenges’ page overview
Accumulating a sufficient number of points resulted in the participant ‘levelling up’. As the current level increased, the amount
of points needed to advance to the next level increased accordingly; the higher the level, the more points were needed to ‘level up’. Points could be earned by collecting badges and completing a daily achievement. A total of three badges could be earned and were awarded when participants: (1) made a first beverage submission, (2) reached level 3 and (3) consistently kept track of their beverage intake for 5 days in a row.
The daily achievement of 30 points was awarded when three individual beverage submissions, on the same day, were made. Submitting over three separate beverage submissions did not result in the acquirement of additional points.
6.2.2 ‘Avatar’ Prototype
The status of the avatar could be viewed by accessing the ‘avatar’ page (Figure 6).
Figure 6. ‘Avatar’ page overview
Upon opening the prototype for the first time the participant was given the choice to select one of two avatars. The participants could switch avatars at any moment.
The progression bar displayed the avatar’s current emotional state ranging from its initial state, either sad (avatar 1) or mad (avatar 2), up to its ending state which is happy and joyful. In addition, two intermediary states were added so both avatars could gradually shift from their initial state to their ending state and vice versa (Figure 7).
Figure 7. Avatar’s emotional states
The avatar’s emotional state depended on how consistent the participants kept track of their beverage intakes. For example, when four consecutive days of beverage intakes were submitted, the avatar would appear to be very happy. Alternatively, the avatar’s emotional state would gradually return back to its initial state if days were skipped submission-wise, or would stay at its initial state if no submissions were made whatsoever.
6.2.3 ‘Social’ Prototype
The status of the gamification elements used in this prototype (e.g. amount of team points, team members, team progression) could be viewed by accessing the ‘community’ page (Figure 8).
Figure 8. ‘Community’ page overview
In this prototype, two equal-sized teams, team red and team blue, competed against each other where each team consisted of 5 members. A set amount of daily team points could be collected by individual team members which accumulated over time. Each of the team members could contribute a single team point each day, by submitting their beverage intake, which would be added to the team point total. At the end of each day, the accumulated daily team points were reset. Both of the team point totals were displayed at the top of the community page and were visualized by a two-colored graph.
Additionally, the status of each team member was presented at the bottom of the community page.
7. STUDY DESIGN
7.1 Participants
7.1.1 Study Population
The population of interest for this study was all ‘middelbaar beroepsonderwijs’ (mbo) adolescent students residing in the Netherlands. The study population that was chosen for this specific experiment consisted of all persons who were willing to participate and who met the following criteria: (1) currently following an educational program at a Dutch mbo school, (2) aged 16 or above and (3) in the possession of an Android smartphone.
Several strategies were used to recruit study participants. First, participant were recruited by visiting four Dutch mbo schools located in Utrecht, the Netherlands and seeking out potential study participants. These schools included: the ‘Wellantcollege’10, the
‘Grafisch Lyceum’11, the ‘Sport Academie’12 and the ‘ROC
Midden-Nederland’13.
Second, a snowball sampling approach was used where future participants, collected from aforementioned schools, were asked to recruit subjects from among their acquaintances such as classmates, friends and students from other schools.
Using both strategies resulted in a sample size of 35 (Appendix F). The study population consisted of 17- to 26-year-old students which included 25 females (71%) and 10 males (29%). The participants attended 15 different schools in total, located in 12 cities throughout the Netherlands.
7.2 Design and Procedure
A one week, four-group, randomized controlled experiment was conducted that started Monday June 5th 2017 up until and including
Sunday June 12th 2017.
All participants were requested to provide the researcher with basic information such as name, age, sex, educational background and attending school, via email. Additionally, informed consent forms were sent out to all participants and consisted of two parts, namely an informational section and the request of consent (Appendix G). A total of four groups were formed where one group was assigned to the control prototype, the control group, and each of the three remaining groups were assigned to a different gamified prototype, the experimental groups.
The participants were assigned randomly and individually to either the control group or one of the three experimental groups, so that groups of near equal size could be formed. The participants who kept track of their beverage intakes, for the full duration of the study, and submitted the post study questionnaires were compensated with a gift card (€ 10) from various shops.
All participants were given their assigned prototypes, a few days prior to study initiation, which they were instructed to use for a week. At the end of the week, each participant was given a post study questionnaire.
7.3 Materials and Measures
In addition to the four prototypes that were utilized throughout the course of the study, two measures were used, namely the day assessment feature, implemented in all prototypes, and the post study questionnaires.
7.3.1 Prototype Day Assessment
Participants self-assessed their intake submission accuracy using a 5-point response format, ranging from 1 to 5 where 1 equaled ‘did not keep track of anything’ and 5 equaled ‘kept track of everything’, for each day of the study.
7.3.2 Post Study Questionnaires
A 21-item questionnaire was designed to both measure and obtain insights into the participants’ attitudes and opinions towards the prototypes with respect to usability, visual appearance, the degree to which the prototypes stimulated keeping track of beverage intake and overall prototype usage (Appendix H).
The questionnaire consisted of both open-ended questions, and closed-ended questions in the form of dichotomous and scaled answering-options. Items were divided among three sections, namely general questions which focused on the usability and the visual appearance of the prototypes, specific questions which focused on the gamification elements and the closing questions where participants were asked to rate the application and express
whether or not they felt that the prototypes were useful for them. Finally, participants were asked if they had any additional feedback they would like to share.
7.4 Analyses
Data were analyzed using only descriptive statistics due to a small sample size. Using any kind of statistical tests would yield low statistical power resulting in a high margin of error, rendering any result meaningless.
8. RESULTS AND ANALYSIS
8.1 Study
Out of the initial 35 participants, four discontinued their participation immediately. When asked about the specific reasoning behind this decision, all four participants stated that they had trouble installing the prototypes due to Android’s security warning, which prevents applications that did not originate from the Google play store from installing, and did not understand how to bypass this alert.
Out of the 31 remaining participants, one participant dropped out after day 1, two participants dropped out after day 2 and one participant dropped out after day 3, resulting in four dropouts spread across the week of the study. Two dropouts were caused by severe illness, one dropout was caused by sudden application failure and one participant did not specify as to why she dropped out. Note that all data generated by the dropouts are omitted in upcoming sections. The characteristics of the 27 remaining participants that used the control prototype (“Control”), the ‘Challenges and Rewards’ prototype (“Game-1”), the ‘Avatar’ prototype (“Game-2”) and the ‘Social’ prototype (“Game-3”) are displayed in Table 1.
Table 1. Characteristics of the participants in all four groups
Gender Age
Groups Female Male Total
x̄ SD n f (%) n f (%) n Control 6 85.7 1 14.3 7 20.9 2.1 Game-1 5 71.4 2 28.6 7 21.4 1.4 Game-2 4 80.0 1 20.0 5 20.8 3.9 Game-3 4 50.0 4 50.0 8 19.5 1.8
The majority of participants were female (70%) resulting in groups that consisted of relatively few men with the exception of Game-3. A relatively large difference in age between participants was observed in Game-2 where ages ranged between 17 and 26.
8.1.1 Beverage submissions and Dropouts
A total of 637 (23.6 on average, per participant) beverage submissions were recorded spread across Control, 1, Game-2 and Game-3 where each group was responsible for 131, Game-211, 130 and 165 submissions respectively. The group average submissions are displayed in Table 2.
Table 2. Average group beverage submissions stratified by day
Day Week Averages Groups 1 2 3 4 5 6 7 Control 3.6 3.7 2.7 1.9 2.6 2.3 2.0 18.7 Game-1 5.0 4.6 4.3 5.0 3.1 3.9 4.3 30.1 Game-2 4.0 3.8 4.8 4 2.2 4.0 3.2 26.0 Game-3 3.1 3.9 2.8 2.6 2.5 3.3 2.5 20.6
Table 2 shows that Game-1 had the highest weekly average followed by Game-2, Game-3 and finally the Control group. No clear trend can be identified other than a slight decrease in overall beverage submissions towards the end of the week. For Game-1 and Game-2, day 5 (Friday) seems to be an outlier displaying a relatively low amount of submissions compared to the rest of the week.
The 637 beverage submissions contained a total volume of 380.8 liters (14.1 liter on average, per participant).
In total, 17 participants (63%) consistently made at least one submission each day, for the duration of the study. The remaining 10 participants (27%) were responsible for a combined total of 15 days which were absent from any beverage submission (“skipped days”). Table 3 displays the skipped days across all groups for each day of the study, including a weekly total and ratio.
Table 3. Total skipped days per group stratified by day
Day Total days/% Groups 1 2 3 4 5 6 7 Control 0 0 1 2 1 1 2 7 / 14% Game-1 0 0 1 0 0 0 0 1 / 2% Game-2 0 0 0 0 0 0 0 0 / 0% Game-3 1 0 3 0 1 0 2 7 / 13% Total 1 0 5 2 2 1 4 15 / 8%
Table 3 shows that the majority of skipped days occurred in the Control group and Game-3; seven days were skipped in both groups. Only one participant from Game-1 skipped a day, submission-wise, and participants from Game-2 consistently made at least one submission each day.
A more detailed overview of the submissions, including skipped days, can be found in Appendix I and Appendix J.
8.1.2 Day-assessments
An overview of the average day-assessments per group are displayed in Table 4.
Table 4. Average day-assessments per group stratified by day and week Day Week Groups 1 2 3 4 5 6 7 x̄ SD Control 4 4.1 3.4 3.3 3.8 3.4 3.1 3.6 0.4 Game-1 4.6 4.4 4 4.4 4.4 3.7 4.7 4.3 0.3 Game-2 4.4 4 3.8 3.8 2.8 3.8 4 3.8 0.5 Game-3 3.9 4.8 3 4.1 3.6 4.1 3.2 3.8 0.6
As shown in Table 4, all gamified prototypes received higher day-assessment scores, on average for the duration of the study, compared to the control prototype, where Game-1 received the highest score, on average.
8.2 Post Study Questionnaires
The post study questionnaire first determined ‘Usability’ which contained five scaled statements and ‘Visual Appearance’ which contained three scaled statements. Then, three scaled questions, which used a 5-point response scale, together with one question that used a scale from one to ten, were asked which aimed to elicit the participants’ attitudes towards the degree to which the prototypes stimulated them to submit their beverage intake. Additionally, in
order to determine how often the participants used the prototypes and how often they forgot to submit any beverage intakes, including the reasons why this occurred, two scaled and one open questions were asked. Four open questions were asked to determine the participants’ overall attitudes towards the prototypes (both positive and negative aspects), if the participants felt that the prototype was useful for them and if they had any suggestions or comments. Finally, participants were asked to rate the prototype as a whole, using a scale from one to ten.
Out of the 27 participants that actively participated in the one-week experiment, 25 (93%) submitted the post study questionnaires. Both the participants that did not respond, P6 from Game-1 (female) and P5 from Game-2 (male), did not specify as to why they did not submit the questionnaires.
Unless stated otherwise, all scaled statements used a 5-point response scale which were coded as 1= Strongly disagree, 2= Disagree, 3= Neutral, 4= Agree and 5= Strongly agree.
8.2.1 Usability
The usability questions inquired about the usability of the prototype in terms of easiness of: usage (Q1), navigation (Q2), submitting beverage intakes (Q3, Q4) and whether the prototype responded in a timely manner (Q5). The attitudes towards prototype usability are displayed in Table 5.
Table 5. Attitudes towards prototype usability
Groups x̄ median mode SD min max
Q1 Control 4.7 5 5 0.5 4 5 Game-1 4.5 4.5 4, 5 0.6 4 5 Game-2 4.5 4.5 4, 5 0.6 4 5 Game-3 4.2 5 4.5 5 0.9 3 5 Q2 Control 4.3 4 4 0.5 3 5 Game-1 3.7 3.5 3, 5 1.2 2 5 Game-2 4.3 4.5 5 1.0 3 5 Game-3 4.1 4 4, 5 0.8 3 5 Q3 Control 4.4 5 5 1.0 3 5 Game-1 4.3 4.5 5 0.8 3 5 Game-2 4.5 4.5 4, 5 0.6 4 5 Game-3 4 4 4 0.5 3 5 Q4 Control 4 4 5 1.2 2 5 Game-1 4.5 5 5 0.8 3 5 Game-2 4.5 4.5 4, 5 0.6 4 5 Game-3 4.1 4.5 5 1.1 2 5 Q5 Control 4.3 5 5 1.0 3 5 Game-1 3.2 3.5 4 1.5 1 5 Game-2 4.8 5 5 0.5 4 5 Game-3 3.6 4 4 0.9 2 5 Qa Control 4.4 5 5 0.8 2 5 Game-1 4.0 4 5 1.1 1 5 Game-2 4.5 5 5 0.6 3 5 Game-3 4.0 4 4 0.9 2 5
a = all questions combined
Table 5 shows that the majority of participants largely agreed with all five questions and expressed fairly positive attitudes towards the prototypes with respect to usability.
8.2.2 Visual Appearance
The visual appearance questions asked about whether participants liked the appearance of the prototypes (Q1) and if the pages of the prototypes (Q2) and menu buttons (Q3) were neatly organized. The attitudes towards the visual appearance of the prototypes are displayed in Table 6.
Table 6. Attitudes towards prototype visual appearance
Groups x̄ median mode SD min max
Q1 Control 2.9 3 3 0.9 1 4 Game-1 3.8 3.5 3 1 3 5 Game-2 4.3 4.5 5 1 3 5 Game-3 3.4 3.5 4 0.7 2 4 Q2 Control 3.4 3 3 0.5 3 4 Game-1 4.2 4 4 0.8 3 5 Game-2 4.5 5 5 1 3 5 Game-3 4.1 4 4 0.6 3 5 Q3 Control 3.6 4 4 0.5 3 4 Game-1 4.2 4 4 0.8 3 5 Game-2 5.0 5 5 0 5 5 Game-3 4.4 4 4 0.5 4 5 Qa Control 3.3 3 3 0.7 1 4 Game-1 4.1 4 4 0.8 3 5 Game-2 4.6 5 5 0.8 3 5 Game-3 4.0 4 4 0.8 2 5
a = all questions combined
Table 6 shows that the majority of participants from the experimental groups expressed either neutral or positive opinions towards the statements regarding visual appearance. The Control group seemed to have the most neutral attitudes where one participant strongly disagreed with Q1.
8.2.3 Prototype Stimulation
Participants were asked to, using a scale from one to ten, rate the prototypes with respect to the degree to which they felt that the prototypes stimulated them to submit their fluid intake (Table 7).
Table 7. Beverage submission stimulation: group mean ratings
Groups x̄ median mode SD min max
Control 5.9 6 5, 6 1.3 4 8
Game-1 7.3 7 7 1 6 9
Game-2 7 7 7 0.8 6 8
Game-3 7 7 6, 8 0.9 6 8
Table 7 shows that the Control group scored the lowest on average whereas the experimental groups had similar higher scores. Three additional scaled questions were asked to further determine attitudes towards prototype stimulation. These questions inquired about the prototypes’ ability to stimulate participants to: keep track of beverage intakes (Q1), submit beverage intakes more often (Q2) and make more accurate submissions (Q3). The attitudes towards the prototypes being stimulative or not are displayed in Table 8.
Table 8. Attitudes towards prototype stimulation
Groups x̄ median mode SD min max
Q1 Control 3.1 3 3 0.7 2 4 Game-1 3.0 3 3 0.6 2 4 Game-2 3.5 4 4 1 2 4 Game-3 3.8 3.5 3 0.9 3 5 Q2 Control 2.9 3 3 0.7 2 4 Game-1 3.3 3 3 1 2 5 Game-2 3.8 3.5 3 1 3 5 Game-3 3.0 3 3 0.5 2 4 Q3 Control 3.0 3 3 0.8 2 4 Game-1 3.3 3 3 1 2 5 Game-2 3.5 3.5 3, 4 0.6 3 4 Game-3 3.0 3 3 0.9 1 4 Qa Control 3.0 3 3 0.7 2 4 Game-1 3.2 3 3 0.9 2 5 Game-2 3.6 4 4 0.8 2 5 Game-3 3.3 3 3 0.8 1 5
Table 8 displays very scattered responses across all groups. Because the differences among group means are very small, we can only state that, according to these responses, all groups had fairly neutral attitudes towards the prototypes with respect to stimulation of submitting beverage intakes, with the exception of Game-2 which expressed slightly more pronounced (positive) attitudes compared to the Control group.
8.2.4 Tendency to Forget
To determine how often the participants tended to forget to submit beverage intakes, the following scaled statement “I often forgot to submit my beverage intake” was asked. The attitudes towards this tendency to forget are displayed in Table 9.
Table 9. Attitudes towards the tendency to forget to submit beverage intake
Groups x̄ median mode SD min max
Q1
Control 3 3 3 0.8 2 4
Game-1 2 2 2 1.1 1 4
Game-2 2 2 2 0.8 1 3
Game-3 2.4 2 2 1.2 1 4
Table 9 provides hints that the Control group had a slightly stronger tendency to forget to submit beverage intakes compared to the gamified groups.
8.2.5 Overall Usage
The overall usage question inquired about how often the participants used the prototypes each day, on average. The following five answering-options were given (translated): 1 time, 2 times, 3 times, 4 times and 5 or more times. The responses are displayed in Figure 9.
Figure 9. Overall prototype usage responses
Figure 9 displays that all groups showed scattered responses with the exception of Game-1 where participants seem to have used the prototype at least 3 times a day, on average.
8.2.6 Open-ended Questions
In total, three open-ended questions were constructed to determine what the participants liked about the application and gamification elements (Q1), what they disliked (Q2) and, if applicable, how many days of submitting beverage intakes were skipped and the reason behind this (Q3). The coding scheme of all responses to the open-ended questions can be found in Appendix K. Note that exemplary quotations were translated into English.
One common response, which was mentioned by multiple participants from the Control group (P2, P3), Game-1 (P5) and Game-3 (P2), was that the ‘Others’ beverage category covered too many beverage types and that “it could be divided into more
categories” (P3, Control). Another common response was that
participants disliked the fact that they could not edit their submissions after they have been made.
Out of all groups, the Control group showed the largest amount of negative responses regarding the visual appearance of the prototype. Two out of seven respondents stated that they found the layout to be too simple and one respondent, notably the only male in the group, just thought it was “really ugly” (P1, Control). In addition, one participant stated that she thought the sound, when
making a beverage submission, was rather loud. With respect to prototype usability, only positive responses were submitted. No comments were made about the prototype’s ability to stimulate making beverage submissions by participants of the Control group. Game-1 showed similar responses with respect to the overall appearance of the prototype and sounds used, where one participant stated that the layout could be improved and two participants mentioned that they did not like the sounds. When looking at the prototype’s ability to stimulate making beverage submissions, the opinions seem to be divided. Two participants stated that they found the gamification elements to be stimulating whereas one participant stated that “For me personally, the game elements did not really
stimulate to keep track” (P4, Game-1). One participant explicitly
stated that she only made beverage submissions for the sake of research and the fact that she got compensated for it, not because of the game-elements. Two participants stated that they disliked the fact that the daily achievement, making at least three beverage submissions a day, was the same for each day.
The responses from Game-2 were very limited. The majority of responses were positive where only one participant expressed a negative statement about the visual appearance of the avatar, namely that it looked very “childish” (P2, Game-2). Two other participants actually liked the “cute” (P4, Game-2) appearance and the way the avatar’s appearance would change from day to day. With respect to visual appearance, no responses from Game-3 were expressed other than one participant who stated that she thought it looked very good. The majority of responses were positive and addressed the specific, team-based, gamification elements. Participants stated “I liked the fact that you worked with points” (P2, Game-3), “I thought the teams were a good idea” (P3, Game-3), “I
liked the fact that you were assigned to a team and that you could see how many points your team had” (P4, Game-3) and that the
game-elements were “challenging” (P8, Game-3). Only one participant stated that he disliked the fact that he did not know who his teammates were.
The most common reasons for skipping a day, submission-wise, was being too busy or that the participants simply forgot to submit.
8.2.7 Overall Rating and Sense of Usefulness
Participants were asked to rate the overall prototype, using a scale from 1 to 10. Also, an open question was asked to determine if the participants felt that the prototype was useful for them. The prototypes’ overall ratings are displayed in Table 10.
Table 10. Prototype overall group mean ratings
Groups x̄ median mode SD min max
Control 6.4 7 7 0.8 5 7
Game-1 7.3 7 7 1 6 9
Game-2 8 8 8 0.8 7 9
Game-3 7 7 7 0.8 6 8
Table 10 shows that the Control prototype received the lowest overall score on average, followed by Game-3, Game-1 and finally Game-2.
Notably, all responses with respect to the prototype’s perceived usefulness, from 17 participants spread across groups, were positive and mentioned that the prototypes created insights into the participants’ current drinking behaviors.
1 1 1 2 1 1 3 3 2 3 3 1 2 1 0% 20% 40% 60% 80% 100% Control Game-1 Game-2 Game-3
