Facial Expression Recognition for
Distinctive Game Events and Personality Profiles
Master Thesis - Information Studies: Game Studies
Author: Sara Larsson Student number: 10662642 Supervisor: Sander Bakkes Final version: 1st of August 2014
University of Amsterdam - Faculty of Science
First examiner: Sander Bakkes Second examiner: Frank Nack
Facial Expression Recognition for
Distinctive Game Events and Personality Profiles
Sara Larsson
University of Amsterdam
sara.larsson.91@gmail.com
ABSTRACT
Personalisation in video games is becoming increasingly important for the player’s enjoyment. Current methods are insufficient to pro-vide a fully personalised experience since they only map player in-tent to what is shown on screen and cannot adapt to the player’s feelings. This study has evaluated facial expression recognition as an affective method to increase personalisation in video games. By looking at the emotional response from players during game events in three different games the study aims to create deeper understand-ing in how game events and player personality affect the emotional output. Findings show that more extrovert players show stronger emotions, also that unexpected and lethal game events elicit stronger emotions from players. It was further observed that players rarely keep a neutral expression when playing video games. There-fore it is important to take both the personality of the player and the game event into account for facial expression recognition. With more research it would be possible to find ways to determine a player’s personality by looking at the response from a game event.
Keywords
Affective feedback, video games, facial expression recognition, game event, personality profiles, affective gaming
1. INTRODUCTION
Video games (henceforth referred to as games) have in recent years become the most popular and economically successful entertain-ment industry (ESA 2014). It is a rapidly growing and evolving medium and during the last decade games have become increas-ingly realistic in their visual and auditory presentation. However Artificial Intelligence (AI) has yet to reach a high realism. AI is important in games, and is used to personalise games to indi-vidual players. Today most games strive to create a more personal-ised experience and is therefore expected by players (Zad, An-gelides and Agius 2012, cited in Bakkes, Tan and Pisan 2013). Lui, Agrawal, Sarkar and Chen (2009)) notes that there has been a gen-eral dissatisfaction in players with the current games due to the in-capability to provide optimal challenge levels towards the individ-ual player. Today game developers are trying to increase personal-isation with dynamic difficulty adjustment based on the player’s performance during the game. Though Lui and his colleagues (2009) argue that a problem with dynamically personalising the game based on the player’s performance is that it is not always ap-propriate to change the difficulty level. For example if a player en-joys completing difficult tasks it is not fitting to decrease the diffi-culty, even if the player has failed on the task multiple times. To improve personalisation in games one possibility is to use affec-tive feedback. According to Ambinder (2011) it could be the next step for games. He argues that the current feedback methods are one-dimensional, that they can only map player intent to what is shown on screen and cannot yet adapt the game to the player’s feel-ings and therefore lack the information to fully personalise a game. However affective feedback in games is a relatively new field and the research that has been done so far is limited in a focus on simple games. A positive note is that most of these studies have found pos-itive results of increased player enjoyment when testing affective feedback (e.g. Chanel, Rebetez & Bétrancourt 2011, Mandryk & Atkins 2007, Lui et al 2009, Xiang, Yang & Zhang 2013). The lack of research poses a problem for game developers. According to Gil-leade and Allanson (2005) it is important that a game can contain its affective feedback loop; if the player becomes aware of how the feedback loop works the game loses its affective nature. Therefore it is important that the game developers are aware of what the ef-fects of affective feedback are.
There are several different methods to measure affective feedback, such as heart rate measures, galvanic skin response and, electroen-cephalography (EEG). Facial expression recognition is a method that easily could be implemented in consumer products since it is possible to measure with a simple webcam. Facial expression recognition is also less intrusive than other methods which often requires something to be attached to the body (Ambinder 2011, Christy & Kuncheva 2014, SightCorp 2013b). Therefore this study will focus on facial expression recognition.
To make facial expression recognition applicable in modern games this study will aim to explore how both the game and the player affect facial expression recognition. To do this the research will look at how strong emotions players show during game events. However since there is no clear definition of what a game event is this will first be defined. Further the following research questions will be explored: (1) Do game events elicit a stronger emotional response than the rest of the game? (2) What variables in a game event help trigger emotions? (3) Does the player’s personality af-fect the elicited emotions?
By finding answers to these questions it would be possible to get a better understanding of how facial expression recognition can be used in a game, and how different types of players react to different types of game events. This could be used both for implementing dynamic changes in a game, and during the development when game events are designed.
2. THEORETICAL BACKGROUND AND
RELATED WORK
This section will explore two different subject: affective gaming (2.1) and facial expression analysis (2.2). In the part concerning af-fective gaming studies that has been done so far will be discussed; what methods have been used and what conclusions have been drawn. Facial expression analysis will focus more on discussing if facial expressions is versatile to apply to facial expression recogni-tion in games.
2.1 Affective Gaming
Affective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affective behaviour. It is an interdisciplinary field spanning com-puter science, psychology, and cognitive science (Tao & Tan 2005). Affective gaming is almost identical to this explanation, with the difference that it is implemented in games. The field of affective gaming is relatively new, the research that has been done so far is therefore limited. So far eye movement desensitization (EMD), electrodermal activity (EDA) and heart rate measurement (HR) are the most common methods tested, mostly due to their low computing costs, but there are various other methods such as facial expression recognition, (EEG), eye movement, and body tempera-ture (Ambinder 2011, Christy & Kuncheva 2014).
Chanel (2011), Mandryk (2007), Lui (2009) with colleagues have used psychophysiological signals (EDA, HR, electrocardiography (ECG) and EEG) to measure the player’s involvement in a game (Tetris and Pong). By finding levels representing boredom, engage-ment, and anxiety they try to adjust the game’s difficulty to keep the player in the flow channel. They could all report a significant improvement of player experience and challenge compared to if the difficulty was changed based on the player’s performance in the game. In addition Chanel and his colleagues (2011) found that adapting the game difficulty according to the player’s emotions is important to maintain the player’s engagement in the game. Xiang, Yang and Zhang (2013) did a study with facial expression recognition where they detected four different moods of the player: frustrated, excited, relaxed, and bored. They used these moods to determine the difficulty level in Tetris. They found that the players preferred the game when the difficulty was decided with the affec-tive method instead of one based on performance.
However even though these studies show that affective feedback have a positive effect on the player’s experience they are limited by the use of simple classical games (Tetris, Pong). On top of the list
of most popular games of 2013 there are games like Diablo III, Grand Theft Auto V, and Call of Duty: Ghosts (ESA 2014). These games are more complex than Tetris and Pong and have more areas to personalise. Bakkes with colleagues (2013) present eight differ-ent areas that can be personalised: space, missions, characters, game mechanics, narrative, music/sound, player matching (multi-player), and difficulty. The main challenge is to combine personal-isation techniques, and to incorporate this into a game’s design. Drachen, Nacke and Yannakakis (2010) did a study to determine if affective feedback such as EDA and HR would be feasible in com-mercial game development. To test this they let the players play three different first person shooter (FPS) games while measuring their heart rate and EDA, every five minutes the games were paused and the players were asked to report their experience during the session. This was later compared to the psychophysiological data to see if there is a correlation between the measured data and self-reports from the players. They found a significant correlation (p < 0.01), concluding that these methods would be reliable to use in commercial games. Although there is no report if there was any difference in emotion output between players, which is an im-portant aspect for the feasibility of affective methods.
Gilleade and Allanson (2003) made a pilot study where they worked to develop a SDK that could be used to develop affective games, with a focus on content such as story, music and difficulty for games that have a heavy involvement for the player, such as action and adventure games. They used heart rate to decide which of the following states the player was in: bored, tired, content, ex-cited, and ecstatic, these states were then mapped to different changes in the game. With the SDK they then made an action game where they tried to optimise the changes to occur in such a way to optimise the player’s state, but to prevent too strong emotions since that can lead to exhaustment. In the initial testing group of eight participants six thought that the affective version of the game was more fun and that it would have a longer lifespan than the control game (without affective feedback). They conclude that affective feedback has a big impact on games, and in combination with learn-ing algorithms it could become powerful by rememberlearn-ing how the specific player reacts to different situations and optimize the game experience for each individual player.
2.2 Facial Expression Analysis
Facial expression is a well explored field that gives a sturdy base to apply in other fields, such as affective gaming. Most studies agree on six basic emotions: anger, disgust, fear, happiness, sadness, and surprise (Keltner, Ekman, Gonzaga, & Beer 2003). Though this list is quite short if the number of words that explain or express an emo-tion (e.g. interest, pride, relief) is taken into account. What is spe-cial with the six basic emotions is that they all have a distinct faspe-cial expression. Ekman together with Friesen (1971), and Izard (1971) found that facial emotions are universal through different cultures. People perceive expressions in the same way in the isolated islands of New Guinea as in the western world. Though there are differ-ences in display rules; who can show which emotions, to whom, and when.
Emotions are triggered by events, reactions to something that hap-pens, and can be of different intensity depending on what the event was (Ekman 1993). Though not every event triggers an emotion, and an event does not necessarily trigger the same emotion with different individuals. Ekman found that there is large individual dif-ferences in a number of aspects concerning the timing of facial ex-pressions. For example the time between the triggered event and
the expression differs, as well as how long it takes for an expression to decay.
There has been research that tried to determine if there can be emo-tion without facial expression. Though according to Ekman (1993) the evidence for either case is fragmentary. Tassinary and Cacioppo (1992) found evidence that people may show no facial change even though they report feeling emotions and manifest changes in auto-nomic nervous system activity. While Tomkins (1963) proposes that facial activity always is part of an emotion. It is possible for people to ‘fake’ facial expressions when they do not feel any emo-tion (Ekman, 1985). Though there is some evidence that false ex-pressions can be distinguished from real exex-pressions by the absence of certain muscular movements in the face that most people cannot trigger voluntarily.
3. METHOD
This section will describe crucial elements of this study, such as defining what a game event is, and what games will be used in the experiment. First game events will be defined (3.1), since no prior definition has been made, this is important to not misunderstand what a game event is, or to cause confusion about why and what events in the games will be looked at. Followed by what games and events are employed in the experiment (3.2), and a description of InSight (3.3), the software used for facial expression recognition. Finishing with explaining what data will be gathered from the ex-periment for further analysis (3.4).
3.1 Defining Game Events
Games and their building blocks are and have for a long time been troublesome to define. Different people have tried, and different definitions have been presented (Schell 2008, p. 27-36), but they have yet to decide on one coherent definition. For this study a def-inition for game events is necessary, something that has yet to be defined. To find an accurate definition for this definitions for build-ing blocks of a game will be analysed.
When defining the building blocks of a game it has been most com-mon to start from the absolute core, for example mechanics (e.g. rules and goals) and aesthetics (e.g. look and sound) (Schell, 2008, Zimmerman & Salen, 2004). However a game event cannot be de-fined as a singular entity, but is a combination of the core parts. Looking at how others have defined other parts of a game; Schell (2008) states that ‘puzzles’ are part of the game, in this case not just jigsaw puzzles, but anything that makes the player stop and think. It does not necessarily have to be conscious thinking, for example ‘in what order should I shoot the enemies to take the least damage’ but could be more conscious as ‘what strategy would work best against this opponent’. Koster (2013) refers to games as puzzles to solve, small concentrated chunks for our brains to chew on. He con-tinues to state that for a player to ‘beat’ a game they need to find the existing patterns of the game. While the general definition for the word event is: “an event is a thing that happens or takes place, especially one of importance” (Oxforddictionaries.com) or “an event is something that occurs in a certain place during a particular interval of time” (Dictionary.com).
With these definitions it would be possible to say that a game event is something that makes the player stop and think, it could be a pat-tern, and it is something that happens or takes place during a par-ticular interval of time. A definition could be that a game event is a happening that breaks the player’s current pattern.
This definition makes it easy to exclude items like basic player in-put such as walking or shooting. A game event could possibly be
enemies spawning, a cutscene starting, or a boulder blocking the player’s path. It is also possible for a game event to apply on ferent parts of a game, for example narrative or gameplay, and dif-fer between if they are triggered by the game or by the player. To narrow down events further it is important to find the boundaries between these different parts.
Figure 1: A diagram showing the relation between the type of game event and the trigger.
For this study a simple division between gameplay and narrative events will be made, and a division between events triggered by the game, or triggered by the player (fig 1). Narrative events are events that aim to move the story forward, such as cutscenes and dia-logues, while gameplay events are events the player actively has to react to, such as enemies spawning or a puzzle that needs to be solved. Player triggered events are events that are triggered by an action by the player, such as pressing a button, while game trig-gered events are trigtrig-gered by the game, for example enemies spawning. There might be combinations of narrative and gameplay events, for example a boss fight, which can both help to drive the story forward, while at the same time interactive. These cases will for the thesis be treated as gameplay events due to their interactive nature.
The thesis will focus on game triggered gameplay events since these are events that have an important and clear role in all games. Since these events are triggered by data from the game they would benefit from affective feedback from the player and a framework proposing how this could work together to optimally (for the player’s experience) trigger an event.
3.2 Games Employed in the User Studies
In this study three different games will be used, the games all come from different genres to be able to analyse if responses are genre specific. The games used are Tomb Raider from 2013 (Square Enix Ltd 2013), Limbo from 2010 (Playdead 2010), and Call of Duty: Black Ops from 2010 (Activision Publishing, Inc 2012). In every game a section from the beginning is used, this covers about ten minutes playtime, depending on the skill of the player. These sec-tions was chosen due to the minimum amount of cues the player
has to learn (e.g. that a rough wall means that it is possible to climb it). These sections all have a variety of game events, only the game triggered gameplay events will be analysed.
Tomb Raider is a third person action adventure game where the player controls Lara Croft on an island full of archaeologically in-teresting locations, trying to solve environmental puzzles and get-ting past enemies. The section from Tomb Raider that is used for testing starts just after the intro movie and ends when the player exits the cave and is presented with the title screen, it has six game triggered gameplay events: (1) player sees a dead body on the wall, (2) grabbed from behind, (3) boulder falls and blocks the path, (4) the floor collapses, (5) grabbed from behind, (6) player has to crawl up a slope.
Limbo is a 2D platformer and puzzle game, guiding the player with non-verbal narrative through a noir world, the player controls an unnamed boy trying to solve puzzles to move forward and figure out the fate of his sister. The section used for testing starts at chapter three and ends when the player gets loose from the spiderweb co-coon in the ceiling. It has three game triggered gameplay events: (1) boulder rolls towards the player, (2) spider encounter, (3) caught in spider web.
Call of Duty: Black Ops is a first person shooter, the player com-pletes missions as a member of an elite Special Forces unit in a modern day setting. Engaging in covert warfare, classified opera-tions, and explosive conflicts. The section used for testing starts when the player lands after the zipline in the Operation 40 mission and ends when the player kills Fidel Castro. There is six game trig-gered gameplay events: (1) melee kill a soldier, (2) enemies at the stair, (3) entering the first house, (4) entering the second house, (5) opening the first door, (6) opening the second door.
In each game event the variables unexpectedness, lethality, dura-tion (time), and violence have been identified. These variables have three classifications: (1) small/short presence of the variable, (2) medium, and (3) big/long, which all events were classified with. This to later test what variables have an impact on the elicited emo-tion. Further if the player had previously played the game, or if a game event was failed and had to be replayed a weight was put on the variables to give a more correct representation of the game event; unexpectedness was weighted to small, and lethality taken down one step.
3.3 InSight SDK
For this study the InSight SDK is employed, it is an individual face analysis software toolkit developed by Sightcorp in collaboration with the University of Amsterdam. With the use of a regular webcam InSight can integrate face analysis and facial expression recognition in third party applications (Sightcorp 2013a). InSight can be used to detect and analyse facial emotion, head pose, gaze, eye movements, age, and gender of the user (Sightcorp 2013b). The accuracy of InSight is high, the reported average accu-racy of the emotion recognition is 93.2% (Sightcorp 2014), this is the part of the SDK that this study will make use of. InSight can track tiny movements of facial muscle in an individuals’ face and translate this into the six basic facial expressions, and in addition a neutral state.
The main_loop script of the SDK has been slightly altered; a timer function has been added which helps the SDK sample the emotional state ten times per second instead of every frame. This change was made to get a more clear time span of when the emotions were shown and to make the data easier to work with. To make it easier
to pair data with participants a minor change to add a timestamp as title to the output file was made.
3.4 Data Acquisition
To be able to explore the hypotheses three different types of data will be gathered during the experiment: (1) facial expression clas-sifications, (2) demographics, (3) personality profiles.
From InSight facial expression classifications will be gathered. By monitoring the player’s face InSight will write out the level of ex-pressions shown (ranging from 0 – 1) ten times per second for each of the six basic expressions as well for a neutral expression. This is also recorded together with the game to be able to pair correct time span with a game event.
The participants will fill out a demographics form, which includes questions such as age, gender, how often and what game genres they play, and what they would think of a camera monitoring them to enhance the game play when playing a game.
To assess if the personality of the player is correlated with emo-tional output the participants fill out a Big-Five factor self-report test. The test consists of 50 questions, evaluating the participant’s personality level in extraversion, agreeableness, conscientiousness, emotional stability, and intellect, the test was developed by Gold-berg in 1992 (IPIP 2014).
4. EXPERIMENTAL SETUP
This section will go through the setup of the experiment in such a way to make it possible to recreate. Starting with the hypotheses of the research (4.1), continuing with participants (4.2), test environ-ment, equipment (4.3), procedure (4.4), and lastly how the data will be evaluated (4.5).
4.1 Hypotheses
The hypotheses for the experiment is mostly based on Ekman’s studies of facial expression recognition, since the emotional re-sponses highly rely on psychology and is not something that can specifically be altered by affective feedback. Though by under-standing how the emotional responses work it would be possible to purposely trigger an emotional response with the help of affective gaming. Therefore the hypothesis are:
The player will keep a neutral state during most part of the game, but give high emotional responses at events. This since Ekman
(1993) reports that events are the trigger to emotions.
Different events elicit different emotion (Ekman 1993). Therefore
certain variables will contribute towards a stronger emotional re-sponse from the player.
Ekman (1993) reports that emotion output can change depending on personality. Therefore stronger and longer emotions would be
observed together with a high extraversion personality trait.
4.2 Participants
In total 11 people participated, they were recruited through personal and social media networks. The majority (72%) of the participants were male (N = 8), 28% were female (N = 3). The age of the par-ticipants varied from 20 to 25 years old (M = 22,36, sd = 1,69). Level of highest education completed was in majority bachelor de-gree (N = 11) and a few high school (N = 2). Most participants reported that they play games daily (N = 8), a couple weekly (N=2) and one less than once a month (N = 1). When asked how they would feel if a camera was monitoring them while playing a game
in order to enhance the game experience the majority (N = 7) swered ‘good’, some (N = 3) ‘I do not know’, and one person an-swered ‘bad’.
Due to problem in the facial expression recognition data only tests from eight participants could be used. The InSight software could not always classify a correct neutral state which lead to some con-fusion in the data and occasionally created too much noise for the data to be evaluated correctly.
4.3 Test Environment and Equipment
The testing took place in a well lit room at the University of Am-sterdam Science Park. The participant was seated about 60 cm in front of a 23 inch PC screen, the PC was equipped with a Microsoft Wired Keyboard 400, Logitech Corded Mouse M318e, and Logitech Webcam HD C270. The participant was wearing Creative HS-850 headphones to increase immersion to the game and to can-cel outside noises. Further the PC had an I7 2600K CPU, NVIDIA Quadro 6200 graphics card and 12 gigabyte RAM, due to the insuf-ficient power of the graphics card the graphical settings on all games were set to lowest.
On the PC the games (Tomb Raider, Limbo, Call of Duty: Black Ops) were installed through Steam, the InSight software was streamed from a laptop with Chrome Remote Desktop (Google 2014) due to a limited amount of licenses. Both the games and In-Sight were recorded with Open Broadcaster Software (Open Broad-caster Software 2014), a free open source screen recording pro-gram. InSight was writing comma separated values of the emo-tional state of the participant to a .txt file ten times per second.
4.4 Experimental Procedure
The participant was first welcomed to the testing and got an expla-nation about the procedure, he/she was then asked to sign a consent form to show that he/she understood the testing and accepting the session to be recorded. After this the participant was asked to fill out a demographics form and a personality test.
When this was done the three games were played in a randomised within subjects order. Before each game started the participant got a brief explanation of the controls and was asked to look neutral at the screen, this to be able to set up the InSight software properly. Between the games the participant was asked if he/she had played the game before, and in that case how long ago and how thoroughly, and if he/she usually play similar types of games. To finish the par-ticipant was thanked with a candy bar. The procedure in total took around 45-60 minutes depending on how quick the participant fin-ished the games. The acquired data is subsequently analyzed in or-der to test the hypotheses.
4.5 Evaluation Criteria
The data will be evaluated in five steps, both to remove noise and to explore the hypotheses. Due to confusion within the InSight soft-ware it is not possible to examine different emotions, only the strength of the neutral emotion will be evaluated.
To remove noise an exponential smoothing with alpha of .3 is ap-plied to the facial expression classifications. Then a neutral state in every game for every player will be identified, the neutral state will be used for comparison with the data from the game events. The neutral state is compared with every game event in the correspond-ing game with a one tailed paired t-test that is applied to every time span of .5 seconds, the p-values are then averaged to get the correct p-value for the whole event. In a similar manner the effect size for every game event will be calculated; the neutral state is compared
to the game events from corresponding game. This will generate results to test the first hypothesis: that players give high emotional
responses at events.
To examine the second hypothesis: that certain variables
contrib-ute more towards higher emotions, a correlation between the
vari-ables unexpectedness, lethality, duration, and violence towards ef-fect size will be made. Several correlations will be done, both for all games together, and for each game separately. This would give results showing how strong correlation the different variables have with the effect size of the events.
A similar correlation test will be conducted to test the third hypoth-esis: that stronger emotion would be found in more extravert
per-sonalities. The effect size and variance of the events will be tested
towards the five personality traits: agreeableness, conscientious-ness, emotional stability, and intellect. This will be done for all games, and for each game separately. Making it possible to see if an extravert personality leads to higher and more varying emotions.
5. RESULT
Here the results from the analysis of the data from the experiment will be presented. It will handle each research area (hypothesis) in order, starting with if game events elicit significant emotional re-sponse from the players (5.1). Continuing with how the variables of the game events are correlated to the effect size of the events (5.2), and if personality traits are correlated with the effect size or variance of the game events (5.3).
5.1 Strength of Emotions in Game Events
To confirm that players show high emotional response at events a one tailed paired t-test was conducted, testing the neutral state of the player towards every event. The null hypothesis was that there is no difference between events, p > .05. The majority (N = 60) of the events (N = 88) were rejected by the null hypothesis, meaning that 60 events elicit significant emotion with the player, leaving 28 events that did not elicit significant emotion. This is also supported by Cohen’s effect size, with 28 events showing insignificant differ-ence towards the neutral state (d < 1). Further 14 events showed small difference towards events (d < 1.5), 12 medium difference (d < 2) and 34 big difference (d > 2), giving a clear indication that events indeed elicit emotions.
From the experimental results it is possible to conclude that the hy-pothesis that players will keep a neutral state during most part of
the game, but give high emotional responses at events. The t-test
showed that almost 70% of the game events elicited significant emotional response, something that was further supported by Co-hen’s effect size. Additionally CoCo-hen’s effect size showed that more than 50% of the game events that showed significant emo-tional response had a big difference from the neutral state.
5.2 Game Event Variables
Correlation tests were made to look for correlation both within each game, and for all games together. The effect size was tested towards duration, unexpectedness, lethality, and violence of the game events (table 1). The experimental results for the correlation test for all games together showed a significant correlation between effect size and unexpectedness, p(86) = .001, p < .05, and also between effect size and lethality, p(86) = .001. The other variables, duration and violence, showed no significant correlation with the effect size.
Table 1: Correlation and significance between effect size and the variables of game events.
When the game events were tested within each game similar results could be found. In Limbo there was a significant correlation with unexpectedness, p(22) = .001, p < .05, and lethality, p(22) = 0.001, there was no significant correlation with the other variables. The same applied to Tomb Raider; unexpectedness: p(32) = .001, p < .05, lethality: p(32) = .001. Call of Duty however only showed sig-nificant correlation between effect size and unexpectedness, p(28) = .001, p < .05, not with lethality, p(28) = .17, like Limbo and Tomb Raider, nor with violence or duration.
The hypothesis that certain variables would contribute towards a
stronger emotional response from the player can be confirmed with
these results. In all games unexpectedness and lethality was signif-icant correlated with the effect size, this also applied to Limbo and
Tomb Raider. While in Call of Duty only unexpectedness was sig-nificant correlated.
5.3 Personality Profiles
A similar correlation test was made to test if the personality of the player relates to the effect size or variance in the neutral emotion during events. With all games grouped together a significant corre-lation between extraversion and variance, p(86) = .02, p < .05, and also between extraversion and effect size, p(86) = .017. None of the other personality traits showed any significant correlation with ei-ther variance or effect size. However when testing the games sepa-rately no significant correlation was found between extraversion and variance or effect size. In Limbo and Tomb Raider no signifi-cant correlation was found between any of the personality traits. In Call of Duty there was a significant correlation between variance and agreeableness, p(28) = .001, p < .05, and also between variance and intellect, p(28) = .01.
In the results it is possible to find a significant correlation between extraversion and effect size, confirming the hypothesis that
stronger and longer emotions would be observed together with a high extraversion personality trait. However when looking at the
games separately this is not present, which might be due to a too small sample size.
6. DISCUSSION
In this section the results of the report will be discussed, assessing the importance of the data, what it could be used for and observe problems with facial expression recognition. This will be done in three parts, starting with analysing the results from the experiment and how this could be implemented in games or game development (6.1). Further the problem with InSight will be discussed, to find
how this can be prevented and what problems it would cause if im-plemented in games (6.2). Finally how important emotional states (such as boredom and frustration) could be assessed and the risks with this will be reflected upon (6.3).
6.1 Facial Expression Recognition in Games
From the results it is possible to see a trend through all three of the games where unexpectedness and lethality are the main contribu-tors toward a high emotional response from the player. Therefore it is not necessary to make big distinctions between game genres. However these are natural responses, if something unexpected hap-pens people often react strongly to it, especially if it at the same time is something dangerous, therefore it is not that surprising that the same reaction applies when playing games. Call of Duty how-ever was an exception, lethality did not significantly contribute to-wards high emotional response. This might be since it is a more intense game than Limbo and Tomb Raider, with a threat of ene-mies spawning often present.
When implementing facial expression recognition in a game these results could be useful to game developers, to be able to trigger or avoid high emotions, and also to use for a guide on how the player behaves. Since there was a significant correlation between extra-version and effect size it is however important to not generalise all players. An emotional response that is small for an extravert player might be high for an introvert player. If the game would handle all players the same way it would give incorrect responses between players. To assess the personality of the players it might be possible to use a generic game event based on a more extensive testing of how players react to game events, to be able to connect the emo-tional response of the event with the correct personality profile.
6.2 Perceived Neutral Expression
Unfortunately due to too much noise and confusion in the data it was not possible to make a deeper analysis of what specific emo-tions player show at events. To set up InSight the player was asked to look neutral, and this might have caused the problem, since when the player started to play the game InSight sometimes perceived another emotional as the neutral. This might be caused by two things, either the player has a wrong perception of what their neu-tral state, or that the player keep a different neuneu-tral state when ing a game. That the player keep a different neutral state when play-ing a game is highly possible since a game always requires input and attention from the player, therefore it would be possible to as-sume that this would not cause a fully neutral state with the player. This is an important problem to keep in mind, and would be an in-teresting topic for future work. It might be possible to solve by ei-ther using a facial expression recognition that learn the player’s ex-pressions, but this is not optimal since it takes time and can cause problem if learned the wrong way. Another possibility would be to have the player play a neutral part of a game, for example just walk-ing around without any game events, while the software is initial-ised, it would then sample the game neutral state as the neutral state.
6.3 Assessing Emotions
Ekman (1996) states that not everyone reacts the same way to an event. This causes problem in how to assess what a good or bad reaction from a player at a game event would be. Since none of the expressions that would be expected to use in a game, for example frustration, and boredom, is present in the software it is important to know how to find these states. Would all players show the same expression for these states, or can they be expressed in different ways? This is an important question that needs to be answered
be-fore facial expression recognition could be implemented in a com-mercial game. Without knowing this there is a high risk that the system would interpret the affective feedback in the wrong way, causing a bad personalisation experience for the player.
With this it can be concluded that facial expression recognition would be an interesting method for affective gaming in more com-mercial products. It is less intrusive and more accessible than other methods. However more research and testing is needed before it can be implemented successfully. Most of the participants were positive to such a system, which is a good sign. Though by dialogue with the participants it became clear that the condition for this would be that the data would only be used locally in the game, and not be available to the game developers or any third party.
7. CONCLUSION
A t-test was conducted on the game events compared to a neutral state for each game, the effect size of each game event was also calculated. From the experimental results it is possible to conclude that the hypothesis that players will keep a neutral state during
most part of the game, but give high emotional responses at events.
The t-test showed that almost 70% of the game events elicited sig-nificant emotional response, something that was further supported by Cohen’s effect size. Additionally Cohen’s effect size showed that more than 50% of the game events that showed significant emotional response had a big difference from the neutral state. To test the hypothesis that certain variables would contribute
to-wards a stronger emotional response from the player a correlation
test was made between lethality, unexpectedness, duration, and vi-olence in relation to the effect size. From this it can be seen that in all games unexpectedness and lethality was significant correlated with the effect size, this also applied to Limbo and Tomb Raider. While in Call of Duty unexpectedness was the only variable signif-icant correlated with effect size.
A correlation test was also made to find if the personality traits (ex-traversion, agreeableness, conscientiousness, emotional stability, and intellect) was correlated with effect size or variance of the game events. In the results it is possible to find a significant correlation between extraversion and effect size, confirming the hypothesis that stronger and longer emotions would be observed together with
a high extraversion personality trait. However when looking at the
games separately this is not present, which might be due to a too small sample size.
With these conclusions it is possible to see that there are several things affecting the strength of the emotional response from the player. Both personality and specific variables from the game events are contributing to the emotional response. Therefore it is important to take both the personality of the player and the game event into account for facial expression recognition. With more re-search it would be possible to find ways to determine a player’s personality by looking at the response from a game event.
8. FUTURE WORK
To make facial expression recognition viable in commercial games more research is needed. This study had a limited number of par-ticipants and would need to be done on a bigger scale for more re-liable results. It would then also be possible to find a way to connect the emotional response from a player with a personality profile. Something to think of is to use the game neutral state of the player instead of a perceived neutral state, it should then be possible to deeper analyse specific emotions of the player.
It would also be necessary to test how important states of emotion, such as boredom and frustration, is expressed by players, and if it is possible to find a general mix of base emotions, or if it differs between players. Further it is important to also do a more extensive research of how players would receive facial emotion recognition implemented in a game.
9. ACKNOWLEDGMENTS
I would like to express my gratitude to Professor Sander Bakkes, my research supervisor, for his useful critiques and encouragement throughout the whole research. I also wish to acknowledge the in-valuable support from Robert van den Born. Diederik Roijers and Marvin Soetanto for help with the statistical analysis. I would fi-nally want to thank Professor Frank Nack who agreed to serve as my second reader.
10. REFERENCES
Activision Publishing, Inc (2012) Call of Duty: Black Ops [online] Re-trieved from http://www.callofduty.com/blackops/home [Ac-cessed 14 July 2014]
Ambinder, M. (2011, 02). Biofeedback in Gameplay: How Valve Measures Physiology to Enhance Gaming Experience [Video file] Retrieved from http://www.gdcvault.com/ play/1014734/ Bio-feedback-in-Gameplay-How-Valve
Bakkes, S., Tan, C. T., & Pisan, Y. (2013, July). Personalised gaming: a motivation and overview of literature. In Proceedings of the 8th Australasian Conference on Interactive Entertainment: Playing the System (p. 4). ACM.
Chanel, G., Rebetez, C., Bétrancourt, M., & Pun, T. (2011). Emotion assessment from physiological signals for adaptation of game dif-ficulty. Systems, Man and Cybernetics, Part A: Systems and Hu-mans, IEEE Transactions on, 41(6), 1052-1063.
Christy, T., & Kuncheva, L. I. (2014). Technological Advancements in Affective Gaming: A Historical Survey. GSTF Journal on Com-puting, 3(4).
Dictionary.com, LLC (2014). Event | Define Event at Dictionary.com
[online] Retrieved from
http://dictionary.refer-ence.com/browse/event [Accessed 8 May 2014]
Drachen, A., Nacke, L. E., Yannakakis, G., & Pedersen, A. L. (2010, July). Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games (pp. 49-54). ACM.
Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of personality and social psychology, 17(2), 124.
Ekman, P. (1985). Telling lies: Clues to deceit in the marketplace, mar-riage, and politics.
Ekman, P. (1993). Facial expression and emotion. American Psycholo-gist, 48(4), 384.
(ESA) Entertainment Software Association (2014). Essential Facts About the Computer and Video Game Industry [online] Retrieved from: http://www.theesa.com/facts/pdfs/esa_ef_2014.pdf [Ac-cessed 29 June 2014]
Gilleade, K., & Allanson, J. (2003). A toolkit for exploring affective interface adaptation in videogames.
Gilleade, K., Dix, A., & Allanson, J. (2005). Affective videogames and modes of affective gaming: assist me, challenge me, emote me. Google (2014) Chrome Web Store – Chrome Remote Desktop [online]
Retrieved from
https://chrome.google.com/webstore/de- tail/chrome-remote-desktop/gbchcmhmhahfdphkhkmpfmi-henigjmpp/details?hl=en [Accessed 14 July 2014]
(IPIP) International Personality Item Pool, (2014). New IPIP 50 Item Scale [online] Retrieved from http://ipip.ori.org/New_ IPIP-50-item-scale.htm [Accessed 19 June 2014]
Izard, C. E. (1971). The face of emotion.
Keltner, D., Ekman, P., Gonzaga, G. C., & Beer, J. (2003). Facial ex-pression of emotion.
Koster, R. (2013). Theory of fun for game design. “O’Reilly Media, Inc.".
Liu, C., Agrawal, P., Sarkar, N., & Chen, S. (2009). Dynamic difficulty adjustment in computer games through real-time anxiety-based af-fective feedback. International Journal of Human-Computer Inter-action, 25(6), 506-529.
Mandryk, R. L., & Atkins, M. S. (2007). A fuzzy physiological ap-proach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4), 329-347.
Open Broadcaster Software (2014) Open Broadcaster Software [online] Retrieved from https://obsproject.com/ [Accessed 14 July 2014] Oxford University Press (2014). Event: Definition of event in Oxford
dictionary. [online] Retrieved from http://www.oxforddictionar-ies.com/definition/english/event [Accessed 8 May 2014] Playdead (2010) LIMBO [online] Retrieved from http://playdead
.com/limbo/ [Accessed 14 July 2014]
Salen, K., & Zimmerman, E. (2004). Rules of play: Game design fun-damentals. MIT press.
Schell, J. (2008). The Art of Game Design: A book of lenses. CRC Press.
Sightcorp, (2013a). About us - Face Analysis Technologies Company [online] Retrieved from http://sightcorp.com/about/ [Accessed 21 April 2014]
Sightcorp, (2013b). InSight SDK - Individual Analysis Software [online] Retrieved from http://sightcorp.com/insight/ [Accessed 21 April 2014]
Sightcorp, (2014). InSight SDK 3.2 [online] Retrieved from http://sightcorp.com/docs/insight/latest/index.html [Accessed 21 April 2014]
Square Enix Ltd (2013) Tomb Raider [online] Retrieved from http://www.tombraider.com/ [Accessed 14 July 2014]
Tao, J., & Tan, T. (2005). Affective computing: A review. In Affective computing and intelligent interaction (pp. 981-995). Springer Ber-lin Heidelberg.
Tassinary, L. G., & Cacioppo, J. T. (1992). Unobservable facial actions and emotion. Psychological Science, 3(1), 28-33.
Tomkins, S. (1963). Affect/imagery/consciousness. Vol. 2: The nega-tive affects.
Xiang, N., Yang, L., & Zhang, M. (2013). Dynamic Difficulty Adjust-ment by Facial Expression. In Informatics and ManageAdjust-ment Sci-ence V (pp. 761-768). Springer Lond
