K. Miesenberger et al. (Eds.): ICCHP 2014, Part I, LNCS 8547, pp. 494–501, 2014. © Springer International Publishing Switzerland 2014
Gamification for Low-Literates: Findings on Motivation,
User Experience, and Study Design
Dylan Schouten1, Isabel Pfab2, Anita Cremers3, Betsy van Dijk2, and Mark Neerincx1 1
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, The Netherlands {D.G.M.Schouten,M.A.Neerincx}@tudelft.nl
2 University of Twente, Enschede, The Netherlands Isabelpfab@gmail.com, E.M.A.G.vanDijk@utwente.nl
3
TNO, Soesterberg, The Netherlands Anita.Cremers@tno.nl
Abstract. This study investigated the effects of the gamification elements of
scaffolding, score and hints on the user enjoyment and motivation of people of low literacy. In a four-condition within-subjects experiment, participants per-formed mental spatial ability tests with the aforementioned elements. Quantita-tive results were inconclusive, but post-test interviews provided insights on the limited effectiveness of the gamification elements. Complex questionnaire wording, high task difficulty, and an improperly situated task environment all contributed to ceiling effects in the influence of scaffolding. Score was found to be ineffective without proper contextualization connecting the numerical score to clearer performance measures. Finally, the underused hints functionality has indicated the need for adequate ‘mixed initiative’ support.
Keywords: Literacy, Gamification, Motivation, User Enjoyment.
1
Introduction
In modern Dutch society, people of low literacy engage in low levels of societal par-ticipation, due to an insufficient grasp of information and communication skills. Computer tools to support societal participation learning for this demographic already exist, but the design of these tools is not optimal. First, people of low literacy suffer from low self-efficacy with regard to learning and the use of information and commu-nication technology (ICT), which translates into reduced motivation to act. And second, the current tools are experienced as complicated and unengaging. This study aims to experimentally investigate whether or not gamification elements can improve the user enjoyment and raise the motivation of people of low literacy using software.
The paper is set up as follows. This chapter provides background information on low literacy and low societal participation in the Netherlands, the issues with current computer support tools related to societal participation learning, and the concept of gamification. Chapter 2 presents the experimental methodology, including information
on experimental design, measurement operationalization, participant demo-graphic information, and materials used. Chapter 3 gives an overview of the results. Chapter 4 offers discussion on study outcomes and recommendations for future work.
1.1 Low Literacy, Societal Participation, Information and Communication Skills
At around 1.1 million people, close to 10% of the Dutch work force is currently clas-sified as being 'low-literate' [6]. The concept of low literacy encompasses not only a deficient grasp of basic reading and writing ability, but also indicates a lack of the information and communication skills needed to effectively participate in modern information societies [14]. Murray et al [10] report on a standard of functional literacy that is associated with higher societal participation, independence and learning, which is seen as the minimum level required for effective participation in society [14].
Societal participation is the common term for goal-directed behaviors enacted in the context of a social structure. Schouten [13] provides a model of societal participa-tion consisting of two axes: a formal-to-informal social context axis and an informa-tion-to-communication skill axis. Examples of problematic societal participation for low-literates in this model are engaging with formal institutions through writing and through interactive voice response systems, reading and understanding traffic signs while driving, and using email and Skype to stay in touch with far-away friends.
1.2 Support Tools, User Enjoyment, Motivation
One common method for assisting low-literate people in language learning and so-cietal participation is the use of software support tools. However, Schouten [13] as-serts that currently available tools in these areas are not optimal. Particularly in the areas of user enjoyment and motivation, low-literate users often run into issues. Schouten [13] posits that people of low literacy often suffer from low levels of self-efficacy with regard to both learning and ICT use, which negatively impacts motiva-tion. Negative affect towards learning and towards ICT play a role as well. While many low-literate people express a desire to learn, the (perceived) complexity of sup-port tools acts as an additional impediment.
User enjoyment, a subset of user experience, indicates users’ general pleasure or displeasure with a system. Based on Gajadhar et al [3], user enjoyment encompasses several concepts. Positive and negative affect refer to the user’s general affective state towards the software. Annoyance measures judgment about the software’s usability. Finally, competence and challenge indicate respectively the user’s sense of being good at using the software, and the difficulties encountered during use. All of these concepts are self-reported. Preliminary reports from low-literate user workshops [13] show issues with regard to support software and user enjoyment. High levels of com-plexity exacerbate negative attitudes towards ICT, causing frustration and annoyance. Motivation is a powerful predictor for software use and learning outcome. This study specifies four kinds of motivation, based on Guay et al [4]. Intrinsic motivation refers to engaging in an activity ‘for its own sake’. Extrinsic motivation, engaging in an activity to obtain a certain goal, is divided into identified regulation (where the
activity itself is seen as valuable and self-chosen) and external regulation (where the activity is strictly seen as a means to an end). Finally, amotivation refers to a lack of desire to engage in an activity.
1.3 Gamification
Gamification is becoming increasingly ubiquitous. Deterding et al [2] define it as "the use of game design elements in non-game contexts" (p.1). Within the context of this study, gamification could prove useful in increasing user enjoyment and motivation with regard to societal participation support. Gamification has already been applied in astronaut training software to improve individual motivation [1] and to improve user engagement in e-learning [9].
Three specific gamification elements are seen as being particularly interesting in this context. Scaffolding refers to gradually increasing difficulty level and reducing guided support in track with user progress [2]. This matching of user skill and task difficulty could make the user feel more competent at tasks, and implementations of similar ideas have been shown to influence intrinsic motivation [12].
Hints, implemented as helpful guides and comments designed to facilitate progress and avoid frustration [16], can be seen as a form of learning support, aimed at reduc-ing experienced challenge. Good use of hints supports players in solvreduc-ing problems on their own, rather than just ‘giving’ them the solution [12]: hints used in this learning-support way could increase intrinsic motivation.
Finally, Zichermann and Cunningham [17] consider the numerical performance measure of score to be "an absolute requirement for all gamified systems" (p.36). Score and reward systems have been linked to increased extrinsic motivation in users [16]. Furthermore, score’s provision of feedback and social-competitive context to the user [17] links it to possible higher positive affect with regards to a task.
1.4 Research Question
To our knowledge, studies into gamification specifically with low-literate users are nonexistent. There are reasons to suspect that general findings about gamification do not directly apply to a low-literate demographic, however. Information and communi-cation skill deficiencies cause problems with software use at large, likely impeding the effectiveness of gamification [14]. The addition of extra elements to software can create perceptions of high complexity and necessitate learning new functionality and new information structures, which would exacerbate problems derived from this. Finally, van Linden and Cremers [8] report that low-literates, among other things, often show reduced abstract thinking skills: this could reduce the effectiveness of certain gamification elements, such as score.
This study aims to investigate the effectiveness of the gamification elements of scaffolding, score and hints in improving the user enjoyment and motivation of people of low literacy. The principal research question is: Can the use of gamification in ICT software applications enhance motivation and user enjoyment for people of low literacy using these applications? The following hypotheses are tested:
1. Scaffolding
(a) Scaffolding increases intrinsic motivation in low-literates. (b) Scaffolding increases self-reported competence in low-literates. 2. Score
(a) Score increases external regulation in low-literates. (b) Score increases positive affect in low-literates. 3. Hints
(a) Hints increase intrinsic motivation in low-literates. (b) Hints decrease experienced challenge in low-literates.
2
Method
2.1 Experimental Design
This study employed a counterbalanced mixed-method within-subjects design. To measure the effect of gamification on user enjoyment and motivation, four experi-mental conditions were used: three conditions corresponding to the scaffolding, score and hints gamification elements, and one control condition.
Each condition consisted of the participants completing a mental spatial ability test (SPAT). Developed by Neerincx et al [11], the SPAT involves matching a provided three-dimensional figure with a correctly rotated version of that same figure, chosen from a set of alternatives. While this test is non-verbal, it has a high base level of difficulty and complexity that supposedly provides room for support initiatives like gamification to have clear effects. Research shows that low-literate participants per-form relatively poorly at this test, exacerbating this effect [8]. Figure 1 shows a screenshot of the SPAT-application used during the experiment.
Fig. 1. Screenshot of Spatial Ability Test (SPAT) software used in the study. The participant
should indicate which of the three right-most figures is identical to the left-most figure. In the control condition, participants were asked to solve a set of 15 SPAT puzzles. Puzzles ranged in difficulty from 2 to 5 answer possibilities, and a predetermined random difficulty order was used. Each puzzle had a time limit of 15 seconds. Other
conditions were differentiated as follows: in the scaffolding condition, the predeter-mined random difficulty order was changed to a sequential difficulty order, gradually increasing the puzzles in difficulty level: 2 answer categories, then 3, then 4, then 5. In the hints condition, a ‘hint’ button was added to the user interface: participants could click this button once per puzzle to remove one incorrect answer, a maximum of ten times total. In the score condition, participant performance was displayed at the end of the condition, in a '[number correct]/15' format.
2.2 Participants
With the help of the Dutch Stichting Lezen & Schrijven, 17 low-literate participants were recruited for this study. 12 female and 5 male subjects participated, ranging in age from 33 to 71 years (mean=51.94 years, SD=11.255).
2.3 Measures
In order to measure user enjoyment and motivation in low-literate participants, a short questionnaire was created. This questionnaire was adapted from the Game Experience Questionnaire (GEQ) [7] and the Situational Motivation Scale (SIMS) [4], containing five questions to measure self-reported competence, annoyance, challenge, and posi-tive and negaposi-tive affect, and four questions to measure intrinsic motivation, identified regulation, external regulation, and amotivation. All questions were rated on a 5-point Likert scale. Question wording was simplified to better fit the low-literate partici-pants. Participant performance on the test, defined as the number of puzzles solved correctly, was also measured in all conditions.
2.4 Procedure
Two participants were present in each experimental session, participating in parallel. A brief introduction of the task was given before starting. The test was demonstrated, paying special attention to operation, and any questions were answered.
Each participant completed all four experimental conditions in sequence: condition sequences were counterbalanced. Participants filled out the enjoyment/motivation questionnaire after each condition; experimenter help in reading questionnaires was offered, but not enforced. After completing the experiment, participants also filled out a questionnaire measuring demographic information, computer experience and gam-ing preferences. Finally, short interviews were held. The intent of these interviews was to gain more in-depth insights into participant motivation and enjoyment. Inter-views were semi-structured: questions like 'what did you think of this game?' and 'did you notice any differences between different play sessions?' were used as predeter-mined starting points, with follow-up questions based on insight.
3
Results
Questionnaire results were analyzed using linear and quadratic regression analysis. Scores for positive affect, intrinsic motivation and identified regulation were high
across participants, while other scores (including SPAT scores) were more varied, with no immediate trends. None of the six hypotheses were confirmed (see Table 1).
Table 1. Overview of regression analysis results as related to the six hypotheses
1a: Scaffolding, Intrinsic Motivation F(3.64) = .082 p = .969 Unconfirmed. 1b: Scaffolding, Competence F(3.64) = .777 p = .511 Unconfirmed. 2a: Score, External Regulation F(3.64) = .104 p = .957 Unconfirmed. 2b: Score, Positive Affect F(3.64) = .022 p = .995 Unconfirmed. 3a: Hints, Intrinsic Motivation F(3.64) = .118 p = .960 Unconfirmed. 3b: Hints, (Experienced) Challenge F(3.64) = .476 p = .700 Unconfirmed. The follow-up analysis found two interesting significant results. First, a linear rela-tion between competence (R^2=.066, F(2,64)=4.57, p=.36) and performance indicates that self-reported competence was a good predictor of actual competence. And second, a quadratic relation between positive affect (R^2=.209, F(2,64)=4.671, p=.000), intrinsic motivation (R^2=.109, F(2,64)=3.931, p=.023), identified regula-tion (R^2=.216, F(2,64) = 8.800, p=.000) and performance seems to indicate a rela-tion between skill and enjoyment. Analysis of computer self-efficacy and gaming preferences showed no significant outcomes: values for both constructs measured across the range, with no trends to speak of.
4
Discussion and Conclusions
Based on quantitative analysis, none of the hypotheses could be confirmed. As this represents a deviation from expectations, possible explanations are offered next.
Generally speaking, this study seems to have underestimated the issues low-literates experience with regard to information and attention complexity. Simply deal-ing with the ‘base’ SPAT proved complex enough for participants to require their full attention. Furthermore, questionnaire wording was found to be overly complex, even after editing. The motivation questions were particularly misunderstood, with partici-pants often verbalizing one answer and noting another. The concept of ‘motivation’ may be too abstract for this demographic to be measured in questionnaires.
This study reaffirms the vital importance of strongly taking the constraints asso-ciated with low literacy into account (c.f. [8]). While attempts were made to adapt measuring instruments to the target demographic, issues of complexity, abstraction, and information overload were still encountered.
The lack of results for scaffolding can be attributed to two ceiling effects. First, almost all participants reported high motivation and enjoyment, both in questionnaires and in post-test interviews, simply from participating in the study. The experimental task was too 'fun', and seen as a game, which led to a high baseline for enjoyment and motivation even in the control condition. These findings confirm the importance of properly situated test design [14]. An experimental design that was less enjoyable, framing the situation as a problem or a test, could have avoided this issue.
Task difficulty presented a second ceiling effect. Perceived difficulty did not change as a function of the number of answer possibilities, but instead seemed de-pendent on participant skill levels: participants were either consistently good at the task, or consistently bad, with little change between conditions. Particularly for the latter group, and counter to expectations, the difficulty of the experimental task influ-enced outcomes in an undesired way: the base task difficulty was so high that the effect of the experimental manipulations was too small to make any real difference.
It could also be argued that since the SPAT measures the stable trait of mental rota-tion skill, little room would be left for difficulty manipularota-tion through scaffolding to begin with. While this notion does not invalidate this study’s scaffolding manipula-tion, given that measurements focused on more affective factors, it does provide a compelling alternate reason for why this manipulation had so little effect.
The lack of results for score can be attributed to insufficient context: displaying score in only one condition lacked any progress and skill information. Participants also did not always report ‘seeing’ the post-test score screen; as no equivalent screen existed in other conditions, this could be due to information overload or attentional division. Finally, the numerical nature of score was difficult to some participants.
Earlier assumptions about the application of this gamification element should be rethought. The combination of information and attention overload and interpreting numeric information could mean that a visual representation of score as a motivator is simply not suited for low-literates. Future studies should attempt to clarify this.
The lack of results for hints can be attributed to two factors. First, about 50% of participants reported not 'seeing' the hints button in the condition it was present: as with score, this could represent information or attention issues. Second, the partici-pants that did see the hints button reported a desire to not use it: they saw the test as a game, and wanted to test their skills. A similar attitude was visible with the question-naires: participants would never ask for help with reading the questions, but any offer of help was always gladly accepted. These findings seem to indicate the need for bet-ter design of support options: mixed initiative design principles could be used to achieve a better balance in offering and enforcing help [5].
Finally, the quadratic relation between positive affect measures and performance seems similar to flow theory and the Zone of Proximal Development [12], which present the optimal user experience as a between frustration and boredom. The notion that societal participation learning for low-literates is subject to flow considerations could be used as a starting point for designing better software to this effect.
References
1. Cornelissen, F., Neerincx, M.A., Smets, N.J.J.M., Breebaart, L., Dujardin, P., Wolff, M.: Gamification for Astronaut Training. In: The 12th International Conference on Space Op-erations, Stockholm, Sweden (2012)
2. Deterding, S., O’Hara, K., Sicart, M., Dixon, D., Nacke, L.: Gamification: Using Game Design Elements in Non-Gaming Contexts. In: Proceedings of CHI 2011, pp. 1–4. ACM (2011)
3. Gajadhar, B.J., de Kort, Y.A.W., IJsselsteijn, W.A.: Influence of social setting on player experience of digital games. In: CHI 2008. ACM, Florence (2008)
4. Guay, F., Vallerand, R.J., Blanchard, C.: On the Assessment of Situational Intrinsic and Extrinsic Motivation: The Situational Motivation Scale (SIMS). Motivation and Emo-tion 24(3), 175–213 (2000)
5. Horvitz, E.: Principles of mixed-initiative user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: The CHI Is the Limit (CHI 1999), pp. 159–166. ACM (1999)
6. Houtkoop, W., Allen, J., Buisman, M., Fouarge, D., van der Velden, R.: Kernvaardigheden in Nederland: Resultaten van de Adult Literacy and Life Skills Survey (ALL). Uitgave Expertisecentrum Beroepsonderwijs (2012)
7. IJsselsteijn, W.A., de Kort, Y.A.W., Poels, K.: Game Experience Questionnaire (2007), http://www.gamexplab.nl
8. van Linden, S., Cremers, A.H.M.: Cognitive Abilities of Functionally Illiterate Persons Relevant to ICT Use. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 705–712. Springer, Heidelberg (2008)
9. Muntean, C.I.: Raising engagement in e-learning through gamification. In: Proc. 6th Inter-national Conference on Virtual Learning ICVL, pp. 323–329 (2011)
10. Murray, T.S., Kirsch, I.S., Jenkins, L.B.: Adult Literacy in OECD Countries: Technical Report on the First International Adult Literacy Survey, National Center for Education Statistics, USA (1997)
11. Neerincx, M.A., Pemberton, S., Lindenberg, J.: U-WISH Web usability: methods, guide-lines and support interfaces (Report TM-99-D005). TNO Human Factors Research Institute, Soesterberg, The Netherlands (1999)
12. Obikwelu, C., Read, J., Sim, G.: The Scaffolding Mechanism in Serious Games. In: Fun and Game 2012, Toulouse, France (2012)
13. Schouten, D.G.M.: Improving Societal Participation for Low-Literates and Non-Natives Through Computer Support Tool Design (2013) (Unpublished)
14. Schouten, D., Smets, N., Driessen, M., Hanekamp, M., Cremers, A.H.M., Neerincx, M.A.: User Requirement Analysis of Social Conventions Learning Applications for Non-Natives and Low-Literates. In: Harris, D. (ed.) EPCE 2013, Part I. LNCS, vol. 8019, pp. 354–363. Springer, Heidelberg (2013)
15. van Staalduinen, J.P., de Freitas, S.: A Game-Based Learning Framework: Linking Game Design and Learning. In: Kyhine, M.S. (ed.) Learning to play: exploring the future of education with video games, pp. 29–54. Peter Lang, New York (2011)
16. Warren, S.J., Dondlinger, M.J., Barab, S.A.: A MUVE Towards PBL Writing: Effects of a Digital Learning Environment Designed To Improve Elementary Student Writing. Journal of Research on Technology in Education 41(1), 113–140 (2008)
