Guest Quest: A Review of Game Based Hospitality
Teaching
Master Thesis - Information Studies: Game Studies
Author: Mandy Sepers Student number: 10003710 Supervisor: Frank Nack
Final version: 23rd of August 2014
University of Amsterdam - Faculty of Science
First examiner: Frank Nack Second examiner: Sander Bakkes
Guest Quest: A Review of Game Based Hospitality
Teaching
Mandy Sepers (10003710)
ABSTRACT
This paper describes an evaluation of the learning game Guest Quest. Through a pre-test, play session and post-test, this project examines if players achieve their learning goals. User tests and interviews explore which game mechanics and aesthetics influenced the learning experience. The paper then offers a review on how these aspects of a game worked as effective teaching tools and how they could be used more effectively in parts that did not work out as planned.
General Terms
Design, Human Factors, Theory
Keywords
Guest quest, e-learning, game mechanics, user test, hospitality
1. INTRODUCTION
In 2012 Taskforce Innovation Utrecht started a project called Smart Experience Actuator (SEA). It aims are applying games and game concepts in tourism, retail and events. Because these sectors address new, young target groups it allows developers to test innovative applications. For their third pilot they created the assignment to make game-based hospitality training aimed at employees in the Dutch ‘horeca’1
. A stakeholder in this project is Tourism Utrecht. Together they aim at making Utrecht the most hospitable city of the Netherlands by 2017. After a pitch, the company Shapers got the job and started building the game: Guest Quest, a web based point-and-click-adventure that takes the player on a guest’s journey in the catering industry.
After a short and intense developing process, the game was tested through several game sessions. As interviews in the SEA [21] describe, the tests not only stimulated discussions about the content of Guest Quest itself but also about hospitality in general. Participants reflected on their behavior and that of their colleagues. About this Niels Floor, director of Shapers, says: “It was an indication that the game should be part of a broader training scheme.. Guest Quest on its own was not enough to make people more hospitable (…). That wasn't the intention either. The idea was to open people's eyes. If that is successful (…) the client is already more or less satisfied.” [21]
Because reflection and understanding are important skills for learning and changing behavior, it was concluded that the product
1
The Dutch term ‘horeca’ refers to the industry of hotels, restaurants and cafés. In this paper, it is later referred to as catering industry.
was at least good enough to please the client. However, achieving this resulted in neglecting the greater possibilities of game based learning. Even though the ambition was still there, and any remaining skepticism towards game based learning was overcome, no further research was done on the achievement of the game’s learning goals.
This means the question remained: Do players achieve their learning goals?
This paper describes a research on this question. It starts with a review of existing thoughts on game based learning and a description of the game. After this, the paper describes the evaluation method, namely the definition of the learning goals and based on those the evaluation questionnaire. The results of this evaluation are then presented and discussed. The paper ends with conclusion and future work.
2. RELATED WORK
Over the past decades, many articles have been written about the possibility of using games for educational purposes. This section touches on some of the discussed possible benefits and drawbacks of game based learning. It also examines some research on testing these learning experiences.
2.1 Possible Benefits of Game Based Learning
Learning is inherent to games. To proceed in a game, the player has to engage with a system with a certain set of rules that may not be clear from the beginning. It is a natural way of learning. [8, 19, 21] With the growing use of new media in everyday life, playing digital games becomes an inherent part of our present-day society. The enormous amount of time spent on playing games facilitates players with complex skills such as problem solving and collaboration, but also with plain knowledge. Furthermore, it shows the engagement that comes along with it [6, 14].
Engagement is one of the most celebrated characteristics of games. It is even debated if real learning can occur without engagement or motivation [6]. The impressive game design of the entertainment game World of Warcraft immerses millions of player for an average of 23 hours a week to collaboratively complete epic quests. [14, 22] Game design is indeed important to achieve such remarkable results [17].
Apart from being pleasant to spend time on, digital games combine the benefits of traditional lectures and hands-on training. They are cost-effective and have low physical risk [1] and at the same time allow for tailored learning paces and immediate feedback [22].
2.2 Possible Drawbacks of Game Based
Learning
Apart from the possible benefits of educational games, the drawbacks have been discussed equally often. One of the most apparent arguments against educational games is that games are distracting. The game world and everything that comes along with it distracts players from achieving the learning goals that are just a small part of the whole [18]. This is specifically the case for people that do not have many experience in using game consoles. Even though we spend more and more time on games, still not everyone has access to them and some people simply did not grew up with digital games. Especially for those who do not yet have much gaming experience, the controls and game environment may come too unnatural. [14, 17]
Another problem is the design of games. It requires well designed games to keep players engaged and to transmit the learning goals. Because game design is a young and growing field of research, mistakes are easily made and many decisions are not based on scientifically proved knowledge but rather on experience or instinct [3, 17]. The goals of a game have to be aligned with the learning goals, which due to the complex nature of today’s video games may be hard to accomplish.
2.3 Testing Learning Experiences
Game design for educational purposes is at its infancy, which results in research on existing learning games. This requires a theoretical approach on existing knowledge about learning and teaching as well as extensive reviews of the game-based learning experience’s design. Models are created to integrate learning theory, flow theory and game design. These models offer insight in important design elements such as feedback mechanisms, clear goals and personalized levels of difficulty. [10, 15, 23] But once these and other important game design decisions are taken into account and a game is realized, how does one evaluate the outcome?
An important part of game design – or any design – is user testing. The iterative design process often contains several test sessions in order to improve the user experience and to fix bugs. However, a user experience says little about a learning experience. When the first is bad, the latter will also decline, but when the first is good, it does not guarantee that the learning will be successful. Implementing a particular learning test offers new possibilities to improve teaching capacities of a game. [7, 13, 24]
What does a learning test look like? As the design process of Guest Quest needed a review on the teaching capacities, the research described in this paper tries an approach of doing so. First of all, there needs to be a test to measure the state of knowledge before the game is played. The same knowledge has to be tested after a play session. [24] Interviews or questionnaires similar as those used in user experience test can be done, only now focused on the learning experience. If certain information does not seem to have come across, recordings of the play tests offer a notion of the functionality of implemented game mechanics. The precise way in which this approach is used for Guest Quest is described in chapter 4 and 5.
3. GUEST QUEST
Before diving into our evaluation, it is helpful to offer a notion of the look and feel of Guest Quest. This section gives an overview
of the game Guest Quest created by Shapers for Tourism Utrecht. It describes its mini games in terms of game play and mechanics. Guest Quest is a web based point and click game, made to teach employees in the catering industry about hospitality. It focusses specifically on employees in grand cafés, which in Utrecht are often student at their side job [21]. Players follow a guest’s journey while he visits a restaurant. The game is composed around six mini games that each represent a step in the journey (Figure 1). The player acts as an employee in the restaurant and the more hospitable he is, the more tips he will receive.
Figure 1: The home screen shows six buttons that lead to the mini games and a leader board.
Every mini game has three levels. During a level, a maximum of 3 stars can be earned, based on how well the player did. With every wrong decision, a star is lost. At least one star is needed to finish a level. When all levels are completed, the player can proceed to the next game. The player will also receive points in the form of tips depending on his performance. The tips are added up if a game is played multiple times. This scoring element is merely there to stimulate the player. Is gives the game some replay value and encourages the player to make the best choice available. However, it is not necessary to achieve a set amount of money to go further in levels or games. Theoretically, a level can be won if the player constantly makes choices that are neither bad (i.e. makes them lose a star) or good (i.e. earns them tips).
In the following sections the 6 different mini games will be described in more detail as it is important that the reader understand their individual setups and aimed for goals.
3.1 Make a reservation
In the first mini game, a reservation call is created in the form of a divided screen (Figure 2). On the left there is the guest asking for availability. On the right de drawing of a woman represents an employee.
Figure 2: The player can choose one of three responses in a conversation.
The player gets to choose between three options every time it is his turn in the conversation. For the best fitted response, the player receives a tip. The next best answer allows the player to go to the next step, but does not reward him with a tip. The worst response results in a feedback animation, depending on the part of the game. For example, if the player chooses to let the phone ring for over seven times, the feedback animation show the guest looking mad and a screen that says “conversation ended”. After this a pop-up screen appears that gives the player a tip about dealing with this type of situations. If the conversation is completed in a satisfying way, an animation shows the arrival of happy guests and a pop-up screen tells how they will have a pleasant evening.
3.2 Welcome Under the Dom
This game is about welcoming the guests and offering them a table. It is set in a restaurant with tables, a bar and a kitchen. The waitress automatically walks around from bar/kitchen to the tables to serve guests. Incidentally new guests will enter. After clicking on the guest, the waitress can interact with the guest in different ways: making eye contact, shaking hands, taking the coats, offering a place at the bar and finally, offering a table (figure 3).
Figure 3: Clicking on a guest opens up a menu for interaction.
Depending on what the character is doing at the time of interaction, the player can receive a tip or lose stars. While serving dinner or drinks, for example, the player is not allowed to shake hand or take a coat because the waitresses’ hands are full. Making eye contact on the other hand, is always possible. There is
a time limit on welcoming the guest. If one of the interactions is performed, extra time and a tip are added to the game. A feedback system is implemented in the form of a pop-up screen that offers a hint when the player made a mistake.
3.3 Empathize With the Guest
The thirds game is about understanding the type of guest to be served. Players must offer their guest what meets their preferences. The game shows a (group of) guest(s) and a notebook that offers four options from the menu (Figure 4). Two of these are wrong and will cost the player a star. The third is alright and allows the player to proceed. The last one is perfect and bears a tip. During a level, the player serves three guests. Only one star is needed to proceed to the next level. In other words, the player only needs to give one good (but not necessarily perfect) answer.
Figure 4: Before time runs out, the player should offer the guest one of the four options. In this example: coffee and cake, coke and snacks, milk and cheese, Belgium beer and nuts.
The rating of an option is based on the weather, time and type of guest. It differentiates between nationalities, gender, and age. For example, the women should be offered white wine or cappuccino, whereas men should be offered beer and coffee. If a wrong answer is given, a pop-up screen tells the player why this was not the best choice according to the game designers.
3.4 What Belongs to Whom?
In this game, the player has to take up order and serve the drinks to the right guests. First, guests order their drinks. The order is written down automatically on an onscreen notebook. Secondly, a countdown screen and the drinks are shown. After this the guests reappear and the player sees a tray with the drinks from a first person perspective (Figure 5).
Figure 5: Drinks have to be dragged to the right guest.
It becomes a memory game where difficulty increases in different manners. While proceeding through the levels, lists of drink
become longer, the time of taking an order becomes less and people change places. When a drink is dragged to the wrong guest, a pop-up screen tells the player to relate order to a guest’s appearances. This also costs a star.
3.5 Watch and See
The fifth game is about handling complaints. The player has to be aware of unhappy people and deal with their problems accordingly. The setting resembles that of the second game (3.2 Welcome under the Dome). Guests spawn automatically on different tables. Clicking on a guest makes the waitress move towards them. The waitress will ask them how they are doing. When they look happy, they will tell her they are doing fine. If they look sad or angry, they have a complaint. The player has four ways of responding to the complaint (Figure 6).
Figure 6: If guests look unhappy, they want to complaint. The player gets four options to respond with.
Only one of the four options is right. A wrong choice results in the loss of a star. There are different kinds of wrong options: the player does not apologize, the player blames someone else (the guest, colleagues or the restaurant), or the player doesn’t take the complaint seriously. A pop-up will appear that tells why the choice was wrong. It also gives a hint on what to do next time.
3.6 See You Next Time
In this game, guests receive their bill and pay. The player has to return them the correct change. It starts with the same scene as game two and five. Guests will raise their hand and a patience counter starts to decrease. The player clicks on a player to let the waitress present the bill. The next screen shows a receipt, money and the guests telling how much money they gave (Figure 7).
Figure 7: The player can click on the money to add it to the change.
To give back change, the player can click on the right coins and bills. The middle of the screen shows the total amount of change
that the player is about to hand over when he clicks on the OK button. There is no possibility to reverse and action. If the correct amount of change has been given, the player receives a tip. If not, he loses a star and a pop-up screen tells the player that guests will get angry. Due to restrained recourses, especially the last mini game was not fully implemented with all learning subjects as planned, but rather as a closure of the guest’s/player’s journey. Even though returning the correct change is important for a guest’s experience, at this point it is more about counting than about hospitality. When the three levels are completed, the guest’s journey has come to an end.
4. ANALYSIS OF GUEST QUEST
LEARNING GOALS
To answer the research question ‘Do players of Guest Quest achieve their learning goals?’ an evaluation procedure was set up that consisted of two parts. A game can be investigated in terms of mechanics, dynamics and aesthetics [9]. The recommendations that eventually will be made would be mainly based on mechanics and dynamics. Therefore the first part investigated the learning goals and game mechanics that were implemented to achieve them. Several methods were used to create a complete overview. There were interviews with three persons that worked on the game. Adding to this, the game design documents offered a more structured insight on what the learning goals of the game are. In these documents, not only the learning goals were discussed, it also offered insight in the way they were implemented. To get an even better understanding of these mechanics, I extensively played the game myself. This was in order to discover as many paths as possible and experience most of the game mechanics. The latter was important to get a grasp of which mechanics might be of importance for the learning goals. This knowledge is necessary to discuss the outcome of the second part of the research. Also, in the design process there tends to be a gap between plans and outcome. Examining both the documents and the game thus offered the opportunity to see which plans did and did not come to development.
Secondly, based on the learning goals analysis, a test was created to see if certain learning goals can be achieved through the game. It included two questionnaires and a play session for 13 participants. The play tests were recorded in order to have access to the game dynamics mentioned above. The results are evaluated with regards to the game design (See chapter 5. Evaluating Guest Quest on Learning Goals).
4.1 Methods
To be able to test the learning goals, one must have a clear view of what they are. Thus before creating the questionnaires, it was necessary to gain insight in Shapers’ design process. Through interviews with three employees of the company it became clearer what the game was supposed to teach. Interviews were done with Niels Floor, Maarten van Broekhoven and Henna de Koning. They had worked with hospitality expert Hans van Spronsen who is specialized in teaching and coaching hospitality. The interviews were semi-structured, following 3 main questions: ‘What is the goal of the game’, ‘How do you try to achieve this goal’ and ‘What is the user group?’. The conversations took 30 minutes and during this time the interviewees did not access the game. They did use what they called the ‘Applied Game Design Canvas’, their design model, to explain their starting-points. [21]
Answers were unanimous about the goal being ‘Teaching players about hospitality’ where the players are ‘People that work in grand café’s’. However the way to achieve this goal was unclear, at least in terms of game mechanics. The narrative design was mentioned several times as a way to make player aware of their role in the experience of a customer. As the title suggest, the game should offer the player a journey through the experiences of a guest. To get a more structured view on the design and implementation of the goals, I gained access to the game design documents.
4.2 Findings
With the use of the interviews, the design document and several hours of playing the game, learning goals were extracted. Some of them where about acquiring certain skills but most included facts about or tricks in hospitality in the catering industry. The skills involved remembering orders and combining tasks.
4.2.1 Learning Goals
Presented here are the learning goals as they will be evaluated in chapter 5. Evaluating Guest Quest on Learning Goals. The ones in Italic are not implemented. Highlighted are the keywords that will be used in qualifying the answer to test if learning has occurred.
Game one:
- Obtain necessary information (date, time, amount of persons) - Obtain extra information (limitations, preferences, occasions) - Repeat the appointment
- Answer the telephone before it rings the fourth time - Speak with a friendly attitude.
- Smile while talking.
- Show interest in the guest and pay attention to their needs. Game two:
- Guests need to feel that they have been seen and are welcome - This can be achieved by: making eye contact, greeting,
shaking hands or taking their coat
- Combining this with other tasks, employees must at least pay
attention to the entrance and make eye contact when a guest
comes in. Game three:
- Determine the right product to offer your guest - Pay attention to time, weather, gender and age - Also look for other contexts such as ethnics or groups - Explain the procedure for ordering
- Adjust your timing to certain guests Game four:
- Remember which order belongs to whom
- Relate a plate to a guest (instead of for example a place)
Game five:
- Be proactive and ask guests how they are doing (even if they seem happy)
- Take complaints seriously - If possible, do something about it
- Try to avoid complaints Game six:
- Give back the right change - Take time for a warm goodbye
- Be professional and hospitable even without tips
Due to limited resources only the first goal was implemented. The last mini game is basically about counting and functions rather as a closure of the game than as a part of the learning experience. This mini game was therefore not included in the research furthermore.
4.2.2 Implementation of the Goals
In the design documents, four types of learning goals were identified, based on the work of Kolb et al. on experiential learning [12]. These were: cognition, experience, perception and behaviour. However, “that behaviour is not part of Guest Quest very much yet” [21]. The game focusses on understanding concepts from hospitality and applying them in-game. To decide on the right way to evaluate the learning objectives, it was necessary to investigate to what extend players are expected to use their gained knowledge.
The set learning objectives can be described in terms of Bloom’s classification on learning goals [19]. This taxonomy offers insight in the extent to which students are expected to deal with the acquired knowledge. In Bloom’s pyramid, the levels start from knowledge develop further via comprehension, application, analysis and synthesis to evaluation. The levels can be tested in different ways, as described in Table 1.
Table 1: Explanation of levels by Sosniak [19] Level Way of testing
Remembering: can the student
recall or remember the information?
define, duplicate, list, memorize, recall, repeat, reproduce state
Understanding: can the
student explain ideas or concepts?
classify, describe, discuss, explain, identify, locate, recognize, report, select, translate, paraphrase
Applying: can the student use
the information in a new way?
choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write
Analysing: can the student
distinguish between the different parts?
appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test
Evaluating: can the student
justify a stand or decision?
appraise, argue, defend, judge, select, support, value, evaluate
Creating: can the student
create new product or point of view?
assemble, construct, create, design, develop, formulate, write
In Guest Quest, teaching occurs mainly through feedback. The player is expected to operate in a world with a certain set of rules and knowledge about hospitality procedures. These procedures are not taught, but when applied in a wrong way or denied at all,
feedback is given (as described in section 3. Guest Quest). Looking back at the mentioned learning goals and the description of the game, we can see to what extent the goals are tested according to the revised Bloom’s taxonomy.
The goal of the first mini game is to teach people about a reservation call. During the game, players are shown three options for a next step in a conversation on the phone. In the design documents the following learning goals have been discussed:
- Obtain necessary information (date, time, amount of persons)
- Obtain extra information (limitations, special preferences, special occasions)
- Check if you and the guest understand the appointment - Answer the telephone before it rings the fourth time - Speak with a friendly attitude.
For each step in the conversation there is an option that reveals a bad attitude or uninterested behaviour. There are also options that sound nicer and ask for more information about the guest (i.e. pay attention to needs and obtain (extra) information). Looking at these learning goals as pieces of information, users have to remember, understand and apply them to play the game at its best. For all mini games, the information has to be at least understood and applied. In most areas, players can also analyse the information in a way. For example when one mistake does not mean the end of a level, players can test and experiment with the information that they possess at that point. However, there is never the possibility to evaluate ones decisions by ways of defending, argues or judging the choices made in the game. In the game there is also no room for creating, as players follow a set path and design. Thus, in the following test we focus on the development of knowledge on hospitality concepts and its application.
5. EVALUATING GUEST QUEST ON
LEARNING GOALS
Learning that Guest Quest is a mainly knowledge based educational game, using a pre and post-test for the participant’s expertise on hospitality concepts seems accurate. [24] An increase in score on the test after playing the game compared to the score before playing the game would then suggest that learning has occurred. If the score decreases the game might teach the wrong things. If it stays the same no learning seems to occur. This chapter describes the tests and evaluates the results.
5.1 Study Design
The test exists of three parts: (1) filling in a questionnaire, (2) playing the game and (3) filling in a comparative questionnaire. The purpose of the first questionnaire was to acquire information demographics and previous experience with hospitality courses. It also asks specific questions regarding the learning goals of the mini games to see what the answer to these subjects are before the participants played the game. For example, two questions related to the first mini game are “What is the goal of a reservation call?” and “Which information is necessary to obtain from such a call?”. Answers were given in free text and later coded as described in chapter 5.4 Quantifying Qualitative Data.
After filling in the questionnaire, participants are asked to play the game at www.gastvrijheidsgame-utrecht.nl. They all use the same guest account on which all the mini games are unlocked.
When finished playing the game (either by completing it or feeling like quitting), the participants fill in a second questionnaire. It contains the same questions regarding the learning goals as mentioned in the first questionnaire. After a set of questions about one mini game and its learning goals a print screen of the game is included. The image comes along with the following question: “Do you feel you learned something in the mini game? If no: what do you think is the reason for this? If yes: what did you learn and how do you think you learned it?” At the end of the list some extra questions were added:
- What do you think of the visuals of the game?
- Where there things you did not understand? If yes: what did you not understand and how do you think that can be?
These will help to reveal problems outside the game mechanics, for example with visuals or in understanding text.
5.2 Procedure
Participants were asked to take place behind the laptop. The browser is opened with three tabs. From left to right, the tabs contain the first questionnaire, the game and the second questionnaire. To capture the screen during the test, Dell Webcam Central2 is running too. Before starting the participants were briefed with the following information (translated from Dutch): “You are going to play the game Guest Quest. It is a game about hospitality in catering industry. You start with filling in the first questionnaire (show tab 1). When you finish, you can play the game. There are six mini games, you play them from left to right and can only proceed if you finish the first level of a game. However, you are allowed and encouraged to play more of a mini game if you like to (show tab 2). Once you finished the game – or don’t want to play anymore – you can fill in the second form (show tab 3). During the test the screen and sound will be captured. If you have any questions, ask me. You can stop at any moment during the test.”
5.3 Participants
Thirteen students from the University of Amsterdam and varying institutes for applied sciences were recruited by the researcher in the Netherlands. There was no compensation in any form for participating in the study. Because of technical issues during their test, three participants were excluded from the sample. This resulted in a sample of 10 participants (M = 23.30 years old, SD = 1.77), of which four were male and six female. Furthermore, the participants had varying amounts of experience in catering industries (zero to seven years) and gaming (zero to 20 hours a week).
5.4 Quantifying Qualitative Data
The data retrieved from the questionnaire had to be grouped and quantified. The amount of experience in catering industry was divided into three groups: no experience, some experience (experience in places where hospitality is relevant, but not in the catering industry, or less than a year in catering industry) and considerable experience (more than a year in catering industry). Also the game time was divided in three groups: little game time per week ( < 1 hour), medium game time per week (1 to 5 hours) and much game time per week ( > 5 hours).
2
Software used on Dell computers to – amongst other things – capture the screen. <http://www.dell.com>
To be able to analyse knowledge improvement, the answers to questions regarding learning goals (in other word: the test questions) were marked 1 to 4. With 1 being wrong (no keywords included), 2 being not totally wrong (some keywords included), 3 being not totally right (some keywords missing), and 4 being perfect (all keywords included). The used keywords are highlighted in 4.2.1. Learning Goals.
5.5 Statistical Analysis
A single factor repeated-measures (within-subjects) design was used. Eta-squared (η2) was used to determine effect sizes ( .02 = small effect size, .13 = medium effect size, and .26 = large effect size; [4] and were reported for all analyses.
5.6 Results and Evaluation
The test described in 5.1 Study Design offers insight in the practical outcome of playing Guest Quest.
Firstly, an analysis has been made of the knowledge score before playing the game (M = 2.52, SD = .56), and after playing the game (M = 3.08, SD = .42). A repeated measures one-way analysis of variance (ANOVA), with condition (pre-game vs. post-game) as the within-subjects factor was conducted. The results show that the game had a significant positive effect on the knowledge score, with a strong effect size, F(1, 9) = 16.96, p = .003, η2 = .65.
In the post-test participants answered more questions more correctly. In some cases, the answers in the pre-test were not completely wrong, but they did not hit the correct terms and keywords. In other words, in these cases it does not necessarily mean that the players learned a new concept, but at least they learned to understand the questions and the learning goals of the game. They showed improvement towards a clearer expression of the concepts, which might indicate a better understanding in terms of Bloom’s taxonomy.
If answers are changed between questionnaires, they are changed in a way that fit the learning goals. Answers from the post-test were never scored lower than those from the pre-test. This means participants at least did not learn the wrong thing.
Secondly, a repeated measures ANOVA has been conducted to determine the effect of the game and varying levels of hospitality business experience (no experience, some experience, considerable experience) on knowledge scores. The average knowledge scores for each level of experience can be found in Table 2. The results show that the game had a strong significant positive effect on the knowledge score for the group with considerable experience, F(1, 5) = 7.86, p = .038, η2 = .61. Some results of the other two groups had to be withdrawn due to technical issues, therefore the same statement cannot be made for ‘no experience’ and ‘some experience’. However, we do see a remarkable result in the pre and post score between different groups. As expected, the ‘no experience’ group started off with lower scores than the ‘some experience’ and ‘considerable experience’ group. Compared to the post test, participants with no experience did not only increase more, they even ended with higher scores than the group with considerable experience. This research does not offer insight in the ground of this result, but it may be speculated that the latter group tends to answer questions more out of working experience than game experience.
Table 2: Effect of the game on varying levels of experience
Experience Mean
Statistic Std. Error No experience Pre 2,2000 0,00000
Post 3,4000 ,20000 Some experience Pre 2,3000 ,50000 Post 2,8000 ,40000 Considerable
experience
Pre 2,7000 ,24631
Post 3,0667 ,16865
Thirdly, a repeated measures ANOVA has been conducted to determine the effect of the game and hospitality training (training vs. no training) on knowledge scores. The average knowledge scores for each group can be found in Table 3. The results show that the game had a strong significant positive effect on the knowledge score for the no-training group, F(1, 6) = 14.43, p = .009, η2 = .71, but not for the training group, F(1, 2) = 3.57, p = .119, η2 = .64.
In other words, it seems that participants with prior experience in hospitality courses tend not to change their answers in the direction of Guest Quest’s learning goals. A participant noted “I did not learn anything new, but it was a nice reminder of thing I learned in previous courses”.
Tabel 3: Effect of the game and hospitality training
Training Mean Statistic Std. Error No training Pre 2,3143 ,16246 Post 2,9714 ,16578 Training Pre 3,0000 ,34641 Post 3,3333 ,17638
Also, a repeated measures ANOVA has been conducted to determine the effect of the game and game time per week (little, some, much) on knowledge scores. No significant effect has been found, F(2, 7) = 3.56. p = .084, η2 = .51. The effect size suggests that a bigger sample, it might show significance.
Lastly, a final repeated measures ANOVA has been conducted to determine if there were any differences between the games (game 1 to game 5). The average knowledge scores for each knowledge construct addressed by the individual games, can be found in Table 4. The results show that there was no significant difference between the games, F(4, 36) = .31, p = .873, η2 = .033. In plain English, according to this analysis each game was made equally well.
Tabel 4: Difference of effect between the games
Mean Statistic Std. Error Game1_Pre 2,8000 ,32660 Game2_Pre 2,7000 ,15275
Game3_Pre 1,7000 ,30000 Game4_Pre 2,5000 ,42817 Game5_Pre 2,9000 ,27689 Game1_Post 3,5000 ,34157 Game2_Post 3,2000 ,13333 Game3_Post 2,2000 ,32660 Game4_Post 3,2000 ,32660 Game5_Post 3,3000 ,21344 Valid N (listwise)
However, some questions never or rarely reached an answer scored at 4 points. This was the case for game three (none of the participants gave an answer that scored 4 points) and game two (only two of the participants scored perfectly). Also, for the fifth game, 4 out of 10 participants scored perfectly, but two of them did that already before playing the game. Thus only 2 out of 8 participants showed improvement to an answer that is considered perfect.
5.7 Unachieved Goals
The fact that mini-games two, three and five (almost) never reach the perfect answer can be explained in different ways. It might be the case that the questions were not clear. However, participants did not ask for explanation during the test. In the questionnaire they were also asked to highlight any difficulties, but they never mentioned these questions. They did mention difficulties with game three, which is about upselling and making suggestions for guests (see section 3.3 Empathise with the Guest). Participants mentioned that they recognised the need to stereotype to play the game at its best, but they did not agree that this is a good thing in real life. Therefore they did not use this in their answers to the questions around upselling. Some examples of remarks are: “Apparently I have to generalize the guest, I don’t know if a agree”,[P3] “Apparently I have to make guessing based on stereotypes”,[P4] and “I don’t think I should make assumptions based on group compositions or guests’ appearance”.[P6] For mini games two and five, there were no such remarks. This suggests that for these, an explanation might lay in the game mechanics that hold players back from observing or learning certain goals. Diving into the test questions, it appears that there were certain keywords from the learning goals that never reappeared in the post test answer. For the second game this was ‘taking a guest’s coat’ and ‘shaking a guest’s hand’, for the third game it was ‘ask a guest how he is doing’.
Analysing the recordings of the second game revealed insight in a possible explanation for the absent keywords ‘shaking hand’ and ‘taking coat’. As explained in section 3.2 Welcome Under the Dome, when a guest enters the café, the player has certain ways of interacting with them: making eye contact, shaking hands, taking
the coats, offering a place at the bar and offering a table. However, only making eye contact was always enough to complete the level. Shaking hands and taking coats offered extra tips, but sometimes took too much time. This way, players did not seem encouraged to perform these actions.
The fifth game seems to deal with a similar timing problem. In this game the player must reply correctly to complaining guests (3.5 Watch and See). One of the learning goals was to ask guests how they are doing to be able to moderate problems before they escalate. However, two mechanics seem to prevent players from learning this. First of all, there is no incentive for asking guests how they are doing until they already appear unhappy. Secondly as the game progresses, the café is full of unhappy guests which obstructs the player from asking how guests are doing in time anyway.
This study revealed some problems with teaching certain knowledge, namely that of ways to welcome a guest (taking a coat and shaking hands) and dealing with complaints (to prevent this by pro-active behaviour). However, offering solutions to these requires implementation of new mechanics and another series of tests. This will be discussed in the next chapter.
6. DISCUSSION
The results of a test on the subject of hospitality showed that players were able to use their knowledge outside the gaming environment. By answering the questions they showed understanding of hospitality concepts that where covered in the learning goals. Other tests, for example in real life role playing or at work could show if the game causes an even deeper learning. Fresh views on the difference between in-game and out-of-game use of the knowledge would need a research where the in-game results are compared to the answers of a test outside of the game. Even though the findings reveal that only part of participants’ learning capacity is addressed, the test did show that for these parts, the game did well. In general, the player showed significant improvement in test score after playing the game compared to the score they had beforehand. To make more significant statements, the test should be done by a larger group of participants. This is especially necessary when it comes to findings between different groups within the population. It would be even more interesting to compare the results with a research that uses the same questionnaires on employees who only followed a traditional training.
Lastly, the research revealed that certain key elements of the learning goals did not seem to come across. An analysis of the play tests’ recording suggested problems with the implementation of certain game mechanics. To see if these could be optimised, one needs to implement an adaption of the mechanics and redo the test to see if it makes a difference.
Of course the test set-up has some problematic points that could have influenced the results and hence the findings. For example participants were allowed to ask about the questions if they did not understand them, however not everyone did this. It may be the case that some of them interpreted the questions wrongly. This was only partially accounted for during the analyses of the answers. Also all the participants played on the same account, which might have led to a lesser motivation, since tips are stacked on one big pile and the personal achievement was small. Finally, as stated before, in-game data could have been used if the set-up was more structured.
7. CONSLUSION
Guest Quest was created to teach employees in the catering industry about hospitality. This paper started with the research question “Do players achieve their learning goals?”. It reviewed the use of this game as a learning tool. It positioned the game in the course of hospitality teaching.
As play tests during the game design process revealed, players liked the game and the user experience was as expected. The client would at least be satisfied with the discussions that emerged after discussing the game with catering employees. This showed that Guest Quest functions well as a part of a bigger hospitality course.
Examining the learning goals it became apparent that the two most complex steps in Bloom’s revised taxonomy of learning, evaluating and creating, where no inherent part of the game. At this point, these can be achieved by having debriefings and discussions as mentioned above.
Finally, testing players for their knowledge on hospitality concepts showed that it improved after playing the game. However at least two learning goals were never achieved, which might have been caused by troubled game mechanics. Players were hardly able to learn that a pro-active attitude might intercept complaints and shaking hands or taking a coat can be added to greeting in order to be more hospitable. Implementing more time and incentives for these actions is suggested, as it might result in a better learning experience.
8. ACKNOWLEDGMENTS
This paper would not be here if it was not for professor dr. Frank Nack, who has been able to guide me through the roaring months of the end of my studies. Of course I would not be able to study at all without my loving and caretaking family, which I am blessed to be part of. I would also like to thank professor dr. ing Sander Bakkes, for taking the time to read my paper and attend my defence. Of course I am very grateful that Niels Floor from Shapers gave me the opportunity to take a look at their work so closely. Last but not least, my special thanks to Marvin Soetanto MSc, who turned my plain English into significance.
9. REFERENCES
[1] Bainbridge, W. S. (2007). The scientific research potential of virtual worlds. science, 317(5837), 472-476.
[2] Becker, K. (2005). Games and learning styles. In Special Session on Computer Games for Learning and Teaching, at the The IASTED International Conference on Education and Technology~ ICET 2005.
[3] Bogost, I. (2007). Persuasive games: The expressive power of videogames. Mit Press.
[4] Cohen, Jacob. (1988). Statistical power analysis for the behavioral sciences / Jacob Cohen (2nd ed ed.). Hillsdale, NJ: Hillsdale, NJ : Lawrence Erlbaum Associates.
[5] De Freitas, S., & Jarvis, S. (2007). Serious games-engaging training solutions: A research and development project for supporting training needs. British Journal of Educational Technology, 38(3), 523.
[6] Green, C., & Bavelier, D.(2012). Learning, attentional control, and action video games. Current Biology, 22(6).R197-R206.
[7] Horton, W. (2001). Evaluating e-learning. American Society for Training and Development.
[8] Huizinga, Johan. Homo Ludens Ils 86. Routledge, 2014. [9] Hunicke, R., LeBlanc, M., & Zubek, R. (2004, July). MDA:
A formal approach to game design and game research. In Proceedings of the AAAI Workshop on Challenges in Game AI (pp. 04-04).
[10] Kiili, K. (2005). Digital game-based learning: Towards an experiential gaming model. The Internet and higher education, 8(1), 13-24.
[11] Kim, B., Park, H., & Baek, Y. (2009). Not just fun, but serious strategies: Using meta-cognitive strategies in game-based learning. Computers & Education, 52(4), 800-810. [12] Kolb, D. A. (1984). Experiential learning: Experience as the
source of learning and development (Vol. 1). Englewood Cliffs, NJ: Prentice-Hall.
[13] Mashable(2012).
http://mashable.com/2012/05/18/educational-video-games-how-to. Laste accessed: 22-08-2014.
[14] McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. Penguin. [15] Prensky, M. (2005). Computer games and learning: Digital
game-based learning. Handbook of computer game studies, 18, 97-122.
[16] Reuss, R. L., & Gardulski, A. F. (2001). An Interactive Game Approach To Learning in Historical Geology and Paleontology. Journal of Geoscience Education, 49(2), 120-29.
[17] Schell, J. (2008). The Art of Game Design: A book of lenses. CRC Press.
[18] Slideshare (2009).
http://www.slideshare.net/DeirdreB/promoting-active-learning-using-games. Last accessed: 22-08-2014.
[19] Sosniak, L. A. (1994). Bloom's Taxonomy. L. W. Anderson (Ed.). Univ. Chicago Press.
[20] Tannahill, N., Tissington, P., & Senior, C. (2012). Video games and higher education: what can “Call of Duty” teach our students?. Frontiers in psychology, 3.
[21] Taskforceinnovatie(2013). http://taskforceinnovatie.nl/media/files/seaenglish.pdf. Last accessed: 22-08-2014. [22] Newmedia. http://www.newmedia.org/game-based-learning--what-it-is-why-it-works-and-where-its-going.html. Last accessed: 22-08-2014.
[23] Van Eck, R. (2006). Digital game-based learning: It's not just the digital natives who are restless. EDUCAUSE review, 41(2), 16.
[24] Whitton, N. (2010). Game engagement theory and adult learning. Simulation & Gaming.
