MA Thesis
The Silent Accent:
Can We Make Language Judgements of Nativeness based solely
on Facial Expressions?
Kimberley Christina Francis
Student Number: S2703424 MA in Multilingualism
Department of Applied Linguistics and Frisian Language and Culture Faculty of Arts
Rijksuniversiteit Groningen Supervisor: Dr. Nanna Hilton
Table of Contents
Acknowledgements………..……….3
Abstract ………..4
Chapter 1: Introduction………... 5
Chapter 2: SLA, Facial Expressions and Nativeness ………....7
2.1 Introduction ………7
2.2 Facial Expressions in L2 attainment and Nativeness………..……….…….9
2.3 Perception of Nativeness ……….………..….14
2.4 Cross-Linguistic Facial Expression and L1 Transfer ………..………..…..16
2.5 Facial Expressions and the Issue of Universality ……….….…...18
2.6 Research Questions ……….……….……….21
Chapter 3: Methodology ………...22
3.1 Experiment ……….22
3.2 Pilot ……….………...22
3.2.1 Methodology ………..….………22
3.3 Issues with the Pilot Study and Redeeming Them………..23
3.3.1 Manual Coding Systems and Their Limitations………25
3.3.2 Coding the Footage………..25
3.4 Main Study ……….……….27
3.4.1 Materials ………...27
3.4.2 Participants ………...27
3.4.3 Procedure ………...……….29
3.5 Visual experiment ……….………..30
3.5.1 Procedure and Materials ………..30
Chapter 4: Results ……….33
4.1 Facial Expression Analysis ……….33
4.2 Survey Responses ………...40
4.2.1 Accuracy of the Judges ………..………..40
4.2.1.1 Photos ……….………40
4.2.1.2 Videos ……….………43
Chapter 5: Discussion ……….46
5.1 Analysing Facial Expressions ……….………..46
5.2 Accuracy of the Judges ……….………47
Chapter 6: Conclusion ………...49
Appendices ………51
Appendix A: Facial Action Coding Sheet ……….…….………..…….51
Appendix B: Control Group Photographs used in the Visual Experiment …….……..…57
Appendix C: A Count of Action Unit (AU) performed by each participant ……….……58
Acknowledgements
This thesis would not have been possible without the help, insight and support of many individuals. I would like to thank my supervisor, Nanna Hilton, who
encouraged and supported me throughout the process of combining facial
expressions and linguistics. I would also like to thank Hanneke Loerts for giving me feedback and guidance.
I would like to express my gratitude to all of the people that came to the recording studio and so kindly allowed me to film them for the experiments, and to all of those that participated in the online survey.
I’m ever grateful to my dear friend, Angus Smith, for unknowingly being the partial inspiration for my thesis topic, for proof-reading parts of my thesis and for helping me with parts of the experiment.
I would like to thank my family who have supported me tremendously throughout my academic career. Their knowledge, ideas and helpful suggestions always help shape my work, my thesis being no exception.
Finally, I would also like to show my appreciation to my friend Keeley Gregory for her advice, motivational pep talks, and for spending countless hours on excel looking at statistics and graphs with me. I could not have completed this without you.
Abstract
There has been much research into the cultural component of facial expressions. However, despite these findings, second language acquisition (SLA) research is yet to explore facial expressions as a paralinguistic element. The existence of a critical period for achieving a native-like level is still ongoing and heavily debated,
nonetheless, facial expression is yet to be seen as an element of a target language, and thus, have not been included in the debate regarding the critical period.
Knowledge about the connection between language background and the use of facial expressions may add to the debate and help us gain insights into the cognitive
processes of an L2 learner.
This thesis, compares the facial expressions of Dutch near-native, advanced and intermediate L2 English learners with the facial expressions of British English native speakers. Dutch and British participants are asked to watch a short clip and give a narrative description of the story. The narrative description is recorded and analysed for variety in expression. These recordings are shown to English native and Dutch speakers who judge the language culture of the speaker, in an attempt to see if facial expression contributes to the notion of appearing native-like.
Chapter 1: Introduction “Vultus est index animi”
The face is the mirror of the mind
-Saint Jerome (Letter 54; 1975: 251)
Over the years, there has been a vast body of work produced which includes facial expression as the main focus. This research has continued to grow over the years. The scientific study of the facial expression of emotion began with Darwin (1872), who gathered evidence that some emotions have a universal facial expression. Darwin proposed principles explaining why particular expressions occur for
particular emotions. Nonetheless, Darwin has been criticised for not “considering at any length when, how, and why emotional expressions are reliable or misleading” (Ekman, 2003). More recent advances in facial expression research have involved computer animation and gaming (e.g. Albrecht et al., 2002; King et al., 2003;
Hazlett, 2006), advertising (Hazlett & Hazlett, 1999; Ohme et al., 2009) and camera technology (De Marsico et al., 2012). In contrast to the vast amount of work focusing on facial expression, research involving the linguistic aspect of facial expressions or paralinguistics, is “unfortunately, a neglected step-child at most” (Rauch, 1999 in Schuller & Batliner, 2013).
Marsh et al., (2003), ponder the possibility of distinguishing language cultures solely from facial expressions. Findings have indicated that facial expressions of emotion can contain nonverbal accents that identify the expresser’s nationality or culture. Marsh et al., (2003) asked Japanese nationals and Japanese Americans to guess the nationality of photographs displaying Japanese nationals and Japanese American posers bearing various facial expressions. It was found that there was a greater accuracy when the judges judged the nationality of posers from the same culture. They consider whether facial expressions of emotion in this instance may act as a universal language; one with regional accents. They concluded that “cultural
Despite these findings, second language acquisition (SLA) research is yet to explore facial expressions as a paralinguistic element of SLA. The existence of a critical period for achieving a native-like level is still ongoing and heavily debated,
nonetheless, facial expression is yet to be seen as an element of a target language and thus, have not been included in the debate regarding the critical period. Knowledge about the connection between language and facial expression may add to the debate and help us gain insights into the cognitive processes of an L2 learner.
Looking at the facial expressions of L2 learners at different stages of proficiency, in comparison to native speakers, will give an insight as to whether or not L2 speakers acquire facial expressions as a specific stage on their way to fluency. Asking native and non-native speakers to give language judgements on these differences will highlight whether or not the acquisition of ‘native-like’ facial expressions assists the L2 learner in achieving a native-like competency, and thus, being recognised as belonging to the target language community. During this thesis, the perception of nativeness and facial expressions will be addressed. Additionally, it will be
Chapter 2: SLA, Facial Expressions and the Perception of Nativeness 2.1 Introduction
It is predicted that roughly one out of four English users in the world are considered native speakers (Crystal, 2003). But what constitutes a native speaker? Is there a systematic way of defining or characterising this term?
According to Davies, (1991) the term native speaker was first referenced by Bloomfield (1933 in Davies, 1991) as “the first language a human being learns to speak is his native language; he is a native speaker of this language.” Davies (1996) states that this is the ‘bio-developmental definition’ of a native speaker. In Cook’s (1999) view, by this definition, to be a native speaker is ‘an unalterable historic fact; individuals cannot change their native language any more than they can change who brought them up.” However, this definition has been noted as somewhat too
restricting. A first learned language can be replaced by a language that is acquired later (although perhaps not completely forgotten). With more frequent and fluent use of a later-acquired language, the first may become “no longer useful, no longer generative or creative and therefore no longer ‘first’” (Davies, 1991; Lee, 2005). Thus, the ‘bio-developmental definition’ (Davies, 1996) excludes the idea of language maintenance. This argument does not only question the core meaning of the term but it brings us to explore the shared characteristics of the native-speaker.
Lee (2005) isolates six defining features of a native speaker that are in agreement with numerous scholars in the field of SLA and language teaching.
“1. The individual acquired the language in early childhood (Davies, 1991; McArthur & McArthur, 1992; Phillipson, 1992; Cook, 1999) and maintains the use of the language (Kubota, 2004; McArthur & McArthur, 1992),
2. The individual has intuitive knowledge of the language (Stern, 1983; Davies, 1991; Cook, 1999),
4. The individual is communicatively competent (Davies, 1991; Medgyes, 1992; Liu, 1999), able to communicate within different social settings (Stern, 1983),
5. The individual identifies with or is identified by a language community (Davies, 1991; Johnson & Johnson, 1998)
6. The individual does not have a foreign accent (Coulmas, 1981; Medgyes, 1992).”
Cook (1999) states that the characteristics of a native speaker are “variable and not a necessary part of the definition of native speaker; the lack of any of them would not disqualify a person from being a native speaker.” Cook’s (1999) argument highlights many examples of native speakers that do not necessarily fit into the criteria
presented by Lee (2005). For example Lee’s (2005) third characteristic defines a native speaker as able to produce fluent, spontaneous discourse. However, Cook (1999) argues that many native speakers are far from fluent in speech and have to communicate via alternative means, for example Stephen Hawking and Helen Keller. Lee’s (2005) fourth characteristic, defines a native speaker as able to communicate within different social settings. On the other hand Cook (1999) argues that some native speakers function poorly in social settings. Lee’s (2005) fifth characteristic refers to the idea that native speaker’s identify with a language community. However, as Cook (1999) argues, native speakers are free to disassociate themselves completely from their L1 community, politically or socially, without giving up their native
speaker status.
Among Lee’s (2005) six essential features of the native speaker there is one that cannot be disputed, that is that the native speaker is an individual that acquired the language in childhood. According to Cook (1999), an individual who did not learn a language in early childhood cannot be a native speaker. Cook (1999) asserts that L2 students cannot be turned into native speakers. To do so would alter the core
Lee (2005) suggests that L2 learners may be able to reach a native-like proficiency by embodying some of the six characteristics of the native speaker presented. However, many of these characteristics are highly debatable, with the exception of the bio-developmental definition (which without being re-born is an impossibility for the L2 learner).
For the L2 learner to reach native-like levels we must first be able to define the ability of the native speaker. However, to clearly define the abilities or expertise (Rampton, 1990) of the native speaker is problematic. De Bot et al. (2005) refer to the
impossible task of defining full command of a language (which is often attributed to native speakers). De Bot et al. (2005) declare that it is impossible to define the amount of knowledge bilinguals should have in their L2 in order to be classified as native-like.
This chapter will explore different views regarding the aim of the L2 learner, whether it is possible to reach native-like levels, the ultimate goal of the L2 learner and the role that facial expression plays in L2 attainment. This will lead to a discussion on the perception of nativeness and past research exploring cross-linguistic influences on facial expression. The different views regarding facial expressions and the issue of universality will then be discussed, leading to the research questions of this thesis.
2.2 Facial Expressions in L2 attainment and Nativeness
There are few studies in regard to the combining of facial expressions and SLA. Research into classroom teaching of facial expressions, specifically, mostly focuses on the development of nonverbal communication skills in students with social
perception deficits. An example of this is Minskoff (1980), who presents a four-stage teaching approach for teaching nonverbal communication skills. Minskoff (1980) suggests that the perception of facial expressions can be taught by using tasks in which the student’s attention is directed to the critical facial features. The student should be asked to imitate the expression with a mirror and then they are assessed by making judgements of facial expression photographs.
cues in conjunction with SLA. As Gullberg (2006) points out, the SLA of gesture is a desperately under researched area, despite there being many advantages of learning “paralinguistic cues such as facial expressions and gestures” (Amores, 2015). As Amores (2015) discusses, the learning of these cues will facilitate the learners overall comprehension of a language. This is an important element which aids
communication, since some cues may not be cross-culturally applicable (Galloway & Rose, 2015). Consequently, without this knowledge, cross-cultural communication difficulties may arise (King et al., 2003).
Undoubtedly, the education of paralinguistics is not without its problems. As
Gullberg (2006) declares, “the acquisition of gestural repertoires represents a major challenge to language learners and educators alike who, by and large, have to
establish on their own what to learn and teach.” As Gullberg (2006) highlights, the issue with formally teaching cross-cultural gesticulations is finding a suitable method for teaching gesture forms. This involves selecting relevant gestures, interpreting the meanings of the relevant gestures and explaining to students the appropriate rules of usage. Gullberg (2006) states that “in many cases it is not known how emblems are actually used by native speakers, nor how much variation there is in form.” In other words, the native speaker may use forms of gesture which are far from the
conventionalised gestural domain. It is this variation that makes it harder to determine which gestures are part of the native speaker’s repertoire.
Using facial expressions whilst teaching may also be seen as a useful tool for understanding vocabulary in a target language. As suggested by Sime (2006)
this way” (Macaro et al., 2015). Although this was initially difficult for students, Macaro et al. (2015) found that the students benefited far more in the end as results showed that teacher’s L2 explanations, with restricted facial expressions and gestures led to students having a better understanding and also better recall of new words later (Macaro et al., 2015). The restricting of facial expressions may improve word retention in the L2, however, excluding facial expression from the teaching process may simultaneously lead to students being deprived of the experience of witnessing how facial expressions and gestures are performed in the target language and thus, miss out on a learning experience that may aid their communication with native speakers.
Research in SLA has traditionally tended to concentrate “on beginning or
intermediate learners rather than advanced learners” (Spolsky, 2000). Nonetheless, interest in advanced learners has begun to increase in recent years and with this interest has brought the discussion as to whether the ultimate aim of a learner should be to appear native-like.
Cook (1992) suggests that the SLA research paradigm should be integrated with one of multilingualism, namely that the advanced L2 user should be considered
independently and characterised in their own right rather than being categorised as a language learner at a specific stage on the way to becoming fully native-like. Cook (1999) states that the norms of successful L2 users should be used as the standards for L2 teaching and learning rather than native speaker norms. Regardless, the discussion as to the factors which contribute to an L2 learner achieving a native-like level continues to be explored.
Much evidence has shown that the age of acquisition plays an important role in achieving nativeness. For example evidence that immigrant children can achieve a native-like level in their second language, but their parents cannot reach the same level (Johnson & Newport, 1989).
This leads us to the Critical Period Hypothesis (CPH), which goes back to Lenneberg (1967), who speculates the maturational constraints on language acquisition.
their L2. This was as a result of the decrease in brain plasticity connected to maturation.
It has been much debated whether or not it is possible to achieve nativeness as an adult learner. There has been much discussion as to what percentage of adults reach nativelike levels (e.g. Birdsong, 1999, 2005; Krashen & Seliger, 1975). Nevertheless, some hold the view that no adult L2 learners ever achieve entirely nativelike levels (Bley-Vroman, 1989 in Abrahamsson & Hyltenstam, 2009).
Furthermore, it seems that it is not only the age of acquisition that contributes to achieving nativeness but there are numerous additional factors, for example the motivation and aptitude of the learner, and the amount of time and effort a speaker is able to put into the acquisition of the language (Bialystok, 2001).
Additionally, setting has shown to contribute a particular influence over SLA. For example, those that learn a second language in a more formal setting have appeared to have different results to those that learn in a naturalistic way. Formal settings are those settings in which a language is taught and learned intentionally, this may be in a classroom or other instructional techniques such as websites, books or an
audiolingual method which provide language training (Spada, 2015). This is
compared to naturalistic SLA which occurs in an informal setting (i.e. the playground or the workplace) where learners are expected to “pick up” the L2 through
observation and interaction with others in much the same way that children acquire their L1 (Spada, 2015).
been consistent across different ages, genders and cultures (Matsumoto & Hwang, 2011).
It seems that there are few tools for testing the acquisition of facial expression and gesture. One of the few existing tools was created by Jungheim (1995) who created the Gesture Test (Gestest) and the Nonverbal Ability Scales (NOVA) which assesses English second language learner’s comprehension of gestures and nonverbal
behaviour. The system uses multiple-choice questions with video prompts, testing the comprehension of culturally specific gestures. There is also a series of rating scales to evaluate learners’ use of gestures, gazes and head nods during an interview task. This system was criticised for needing broader units of analysis beyond the verbal message which are likely needed to fully understand learners’ pragmatic ability (Sayoko, 2008). Gullberg (2006) comments on the lack of assessment tools for nonverbal behaviour, stating that “the absence of assessment instruments for gesture is perhaps surprising given that learners’ gestures do influence native addresses.” Gullberg (2006) concludes, that without a tool for assessment, the
acquisition of gesture may not be open to formal instruction or explicit learning. The absence of an appropriate tool to assess the student’s learning of paralinguistic cues means there is no measurement for the progress of the students and thus, no
measurement for the effectiveness of the chosen teaching method. Therefore, despite the obvious advantages, it may not be possible to implement formal facial expression instruction into a formal or classroom setting.
Overall, there is one thing that most SLA researchers agree upon when it comes to L2 learning and that is that early starters are likely to attain high levels of linguistic competence. These high levels may often be comparable to native speakers, whereas later learners are less likely to reach comparable levels. Of course there are
cross-cultural paralinguistic cues must be learnt informally. The role that informally
learning these cues contributes to nativeness is certainly unclear and it is evident that further research is needed into the role of facial expressions in conjunction with SLA.
2.3 Perception of Nativeness
The notion of ‘nativeness’ perception is explored by Kerswill (2001), who declares that the perception of nativeness “is an issue which lies at the centre of Labovian sociolinguistics: that of the ‘speech community’” (Kerswill, 2001). Labov (1972) defines the speech community as a community that is “not defined by any marked agreement in the use of language elements, so much as by participation in a set of shared norms; these norms may be observed in overt types of evaluative behavior, and by the uniformity of abstract patterns of variation which are invariant in respect to particular levels of usage” (Labov, 1972). Kerswill (2001) states that part of
Labov’s definition of the speech community involves evaluation of the linguistic features or varieties in use. Kerswill (2001) states that Labov’s definition highlights that varieties which do not ‘belong’ to the speech community will be identified by members as not belonging in their evaluation. The identification of a native member within a speech community is an important issue as “so much sociolinguistic analysis depends on the linguist’s, and presumably also the native listener’s, ability to identify what is and what is not a part of a given speech community” (Kerswill, 2001). As Kerswill (2001) points out, in order to test for judgements of nativeness, one must present “[…] ‘native listeners’ with samples of speech which differ, in an externally motivated and quantifiable way, in linguistic features which the linguist believes to be criterial for the speech community being examined.” In the case of gesture,
Gullberg (2006) claims that people from communities instinctively know that people in other cultures differ in their gesture use. This in turn forms a system in which those from the ‘native’ community, group individuals “on the basis of a certain observed uniformity in gestures, which also allows them to distinguish one group from another” (Gullberg, 2006). Gullberg (2006), suggests that there is a perceivable systematicity in gesture use and that “the forms gestures take are also governed by cultural norms.” As Labov (1972) suggests, there are particular patterns and
be included in the perception of nativeness. The connection between paralinguistics and the perception of nativeness will be identifiable if, as Kerswill (2001) suggests, those belonging to the group are presented with samples that differ from the community’s linguistic norms and features.
However, as Kerswill states, it cannot be assured that there will be a “perfect
agreement among the judges as to which samples are ‘native’ and which are not.” As Kerswill and Williams (1999) highlight, native judgements may “be systematic, though they relate to a complex sociolinguistic reality.” Kerswill (2001), reinforces this idea stating that “speech communities presumably differ in the degree to which they are set off linguistically from other communities with which they are in contact.” This brings Kerswill to question whether or not this degree of difference is echoed in the judgements of natives. Kerswill and Williams (1999), in exploration of this notion, found that listeners from, Hull, England, were more able to identify speakers from their own city than were listeners from the southern towns Reading and Milton Keynes when exposed to voices from their home towns. Kerswill and Williams (1999) argued that the linguistic distinctiveness of Hull led to the higher recognition rates, while the variation of dialects in the Southern towns Reading and Milton Keynes led to poorer recognition rates. Therefore, it seems that distinctiveness is the key to judgements of nativeness. If a community has a distinctive specific linguistic feature than the recognition rates will be higher when natives are confronted with samples from their own speech community.
Overall, when it comes to judgements of nativeness, it seems that distinctiveness is key. As Gullberg (2006) suggests, differences in gesture allows language
increased if the linguistic group in question have, not only noticeably unique facial expressions, but are also presented with samples from their own language
communities that are set in contrast with samples from a linguistic group that differs significantly in facial gestures from their own.
2.4 Cross-Linguistic Facial Expression and L1 Transfer
There are no specific studies highlighting the facial expressions of British and Dutch natives and L2 attainment. Hooijschuur (2014), set out to investigate whether it was possible for native speakers of British English to identify the nativeness of Dutch and British English speakers, based solely on gesture. Hooijschuur (2014), asked male near-native Dutch L2 speakers of English, native speakers of English, and a control group of low proficient Dutch L2 speakers of English to recall a cartoon narrative. Hooijschuur (2014) recorded the gestures used during the production of these narratives. This was then played to English native speakers who were asked to rate participants on their nativeness. The results revealed no clear differences between the gestures of the three groups regarding amount, types, and placement, and no significant differences between the ratings of the three groups when it came to nativeness. The results indicated that it is not possible to judge nativeness solely based on gestures. However, male judges were more accurate in their ratings than the females.
These results are surprising as it has been typically assumed that cultural differences exist in the types of gesture that accompany speech (Kendon, 1981). Turkish,
Japanese and English speakers have been shown to use different kinds of gestures whilst describing motion events (Kita, 2000; McNeill, 2000; Kita & Özyürek, 2003). Furthermore, bilinguals who speak English in addition to a language with high gesture rates, such as Spanish or French, are more likely to gesture in English than monolingual English speakers. It was also noted that high rates of gesture occurred regardless of English being the L1 or L2 (Pika et al., 2006).
There is much evidence to suggest that the L1 may have an influence on gesture patterns. Previous research has suggested that L1 transfer involves mother-tongue patterns which are carried over and used as a strategy in the early stages of L2
competing and one of these languages must be suppressed in order for the bilingual to function in one language. Transfer has typically been seen as a more frequent phenomenon in the early stages of SLA, however, advanced L2 users have also shown L1 transfer in their L2 (Montrul & Slabakova, 2003; Navarro & Nicoladis, 2004). Language proficiency has also been shown to affect cross-linguistic gesture use. Nicoladis et al. (1999), when observing five boys, in Quebec, Canada, from age 2 to 3 and a half months that were raised bilingually with French and English (one
language primarily from each parent), found that the children used more conventional and deictic gestures more frequently in their weaker language. Conventional gestures are movements that are part of a learned, shared, symbolic systems. In terms of facial gestures this would include such movements as nodding or shaking the head (Pelachaud & Poggi, 2002). Deictic gestures are gestures which direct the attention to a specific point. In terms of deictic facial gestures this would include moving the direction of eyes or head toward a particular point (Pelachaud & Poggi, 2002). Additionally, it has been noted that Swedish L1 intermediate learners of French and French L1 learners of Swedish used more deictic gestures in their L2 than in their L1 (Gullberg, 1999). Similar findings of deictic gestures in the L2 have also been found in advanced L1 English learners of Spanish and Spanish L1 learners of English (Sherman & Nicoladis, 2004). These results all suggest the possibility that conventional facial expressions such as nodding or shaking the head and deictic facial expressions such as moving the direction of eyes or head toward a particular point (Pelachaud & Poggi, 2002) may be used more frequently by L2 users (Gullberg, 1999; Sherman & Nicoladis, 2004).
The connection between different language verb expression and eye movement has also been observed (Bassetti & Cook, 2011). Flecken (2011), when investigating direction of attention and the use of progressive aspect by early Dutch–German bilinguals, found that despite the bilinguals having acquired the means to express progressive aspect in Dutch, their use showed a pattern that differed from
patterns when participants verbalised events. Bassetti and Cook (2011) conclude that these findings lead us to believe that “describing motion may be linked to an aspect of cognition.” Thus, there is a possibility of a connection between spoken language and facial gesture. Hoetjes (2008), found that English speakers only gesture the path of movement, while Dutch speakers will also incorporate the object in their gestures. For example, English belongs to the first verb inventory and only has put, one single placement verb, whereas Dutch belongs to the second and has two, one referring to the horizontal placement leggen ‘lay’ and the other to vertical placement zetten ‘set’ (Gullberg, 2009). Hoetjes’ (2008) findings lead us to believe that differences
between the facial expressions of Dutch and English speakers may be observed. In conclusion, past research indicates that the use of the L2 and having weaker language skills may increase the use of conventional and deictic facial expressions. Therefore, language proficiency may act as an influencing factor in the production of facial expressions and gesture use. Flecken (2011) and Hoetjes’ (2008) findings also highlight the possibility that there may be a noticeable difference in the eye
movements of Dutch participants in comparison to the British, but as to what extent this will affect their other facial movements it is uncertain. This leaves us to ponder as to whether distinguishable differences in eye movement and facial gesture may be associated with the paralinguistic repertoire of the British English speech community and the Dutch speech community. Furthermore, if these attributes do exist, then it leaves us to question if these results will correlate with the judges perception of nativeness.
2.5 Facial Expressions and the Issue of Universality
The issue of whether facial expressions are universal or culturally specific has been long debated. Charles Darwin’s (1872) seminal works, The Expression of the
Emotions in Man and Animals, briefly states that all humans communicate six basic
internal emotional states (happy, surprise, fear, disgust, anger, and sadness) “using the same facial movements by virtue of their biological and evolutionary origins” (Susskind et al., 2008 in Jack et al., 2012). Darwin’s universality hypothesis has been long debated in the biological and social science field (Susskind et al., 2008 in Jack et al., 2012). There has been much evidence to support Darwin in his hypothesis.
facial expressions all over the world. To explore this notion further Ekman and Friesen (1978) created the Facial Action Coding System (FACS) when looking for a means to objectively measure facial activity for behavioural science investigations of the face. Ekman and Friesen (1978), using palpation, knowledge of anatomy, and videotapes (Donato et al., 1999), assigned an action unit (AU) for each facial muscle that contracts and changes the appearance of the face (Ekman & Friesen, 1978). Studying these changes, Ekman and Friesen selected 46 Action Units to correspond to each independent motion of the face (Donato et al., 1999).
This system is then learned and used manually by a coder. The trained human FACS coder decomposes an observed expression into the specific AUs that produced the expression. These 46 AU’s can also be combined. For example, smile expression is considered to be a combination of “pulling lip corners (AU12 + 13)” (Essa &
Pentland, 1997). This system is meant to provide “precise specification of the dynamics (duration, onset, and offset time) of facial movement in addition to the morphology (the specific facial actions which occur)” (Donato et al., 1999). Altogether there are more than 7,000 of these AU combinations that have been observed (Ekman & Friesen, 1982). Using FACS to measure and photograph
expressions, Ekman (2003) discovered that when showing these pictures to even in remote and isolated communities, such as those in Papa New Guinea, people were still able to recognise the basic emotions from pictures of Western faces. Ekman believed that this was enough evidence to confirm that facial expressions of emotion have, as Susskind et al. (2008, in Jack et al., 2012) call it, “biological and
evolutionary origins.”
Many recent cross-cultural comparisons of facial expressions have shown that
individuals have more difficulty understanding the expressions of those from foreign cultures. Jack et al. (2012) claimed to have found evidence that facial expressions of emotion are not universal. Jack et al. (2012) claimed that when analysing the facial movements of Western and Eastern faces their data demonstrates “cultural
divergence on both counts.”
various facial expressions. It was found that there was a greater accuracy when the judges rated the nationality of posers from the same culture. They concluded that “cultural differences are intensified during the act of expressing emotion, rather than residing only in facial features or other static elements of appearance. This evidence suggests that extreme positions regarding the universality of emotional expressions are incomplete” (Marsh et al., 2003). Marsh et al. (2003) claim that their findings “likely consist of very slight variants in the muscle movements, or in the relative intensity of these movements.” They claim that these slight cultural differences “seem not to be explicitly detectable” using a manual facial coding system but are somehow detectable in experiments using nationality judgements.
Matsumoto and Assar (1992), attempt to explore this phenomenon further and question whether or not bilinguals differ in their facial expression judgements due to the differences in the degree of assimilation of different cultures in the same individual. Matsumoto and Assar (1992) explored this notion by asking bilingual subjects in India to judge facial expressions of emotion in two separate sessions, one conducted entirely in English, the other in Hindi. They found that bilingual subjects at a University in India recognised anger, fear, and sadness more accurately in English than in Hindi. These findings were found to show that behaviour or in this case judgements, differ in bilinguals from language to language. This provides “some support for the notion that judgments of emotion by bilinguals differed as a function of the language in which the judgments were made” (Matsumoto & Assar, 1992). Much evidence suggests that there are universal facial expressions which display the six basic emotions, but there are also variations of these expressions which differ from culture to culture. If Marsh et al.’s (2003) claims are true, that cultural variety of facial expressions lies in micro expressions, this would mean that these cultural indicators are very small. As Ekman (2003) states, these expressions are “very fast facial movements lasting less than one-fifth of a second.” These expressions not being easily read or taught may imply that viewing facial expressions at a normal speed will not reveal any obvious cultural differences and therefore an accurate cultural judgement. However, past research involving nationality judgements and facial expression has shown a higher accuracy when judges are faced with an
expression from their own language community. In addition, Matsumoto and Assar’s (1992) results show that the language in use may affect the judgements of the judges.
2.6 Research Questions
There is evidently a great possibility that a difference in facial expressions exists between language communities.
This thesis will investigate the ability to identify nativeness of speakers based solely on facial expressions. It will also seek to explore whether the participant’s English proficiency plays a role in these judgements.
The following questions will be answered.
1. Are there clear differences between the type and frequency of facial expressions produced by Dutch and English subjects? Are these results in correlation with their English proficiency?
Chapter 3: Methodology 3.1 Experiment
The experiment for this study entails still images and video recordings of narrative by both British English natives and Dutch participants. The Dutch participants are divided into 3 groups according to their English level. The facial expressions found in these recordings will be analysed to see if there are differences between the groups. One minute of footage will be shown to British and Dutch natives, who will be asked to judge the English nativeness of a participant based solely on their facial
expressions.
3.2 Pilot
3.2.1 Methodology
Ten males of Dutch and British origin in their early 20’s studying full-time in English at the University of Groningen were asked to come to the recording studio to watch a 5 minute video clip and then state their opinion. The participants were aware they were being filmed and were asked to sit at the table and place their hands on a red mat and not to move them from the mat for the duration of the experiment. This was to prevent face touching and shifting body positions which would create problems when coding the footage. The experimenter sat opposite the participant at a table with a laptop and a camera and once the film had begun, the experimenter did not look at or engage with the participant but sat in front of them so that the participant was aware of their presence. A 5 minute clip was selected from the British TV show
Russell Brand’s Ponderland (2008). The experimenter sat in the room whilst the
participant watched the video as the social context effect has been proven to be particularly strong, especially in cases of laughing and smiling. “These activities are performed infrequently by solitary subjects” (Russell & Fernández-Dols, 1997). Therefore, in order to produce more facial activity and expressions the experimenter remained in the room, sitting opposite the participant whilst viewing the clip. After viewing the footage, participants were asked what happened in the clip and what they thought about what they saw. The participants were then asked to give a
The coding system FACES (Kring & Sloan 1991) was used to measure the frequency, duration and size of the facial expression. Additionally, it was noted whether or not the clip produced a more positive or negative reaction.
Due to multiple errors within the pilot study, no conclusive results were produced. The errors that occurred during the pilot study indicate that amendments must be made.
3.3 Issues with the Pilot Study and Redeeming Them
Two participants’ results were disregarded from the pilot study. One for chewing gum throughout the experiment and another for having smoked marijuana before entering the recording studio. These occurrences “may actually inhibit natural expressive displays” (Kring & Sloan, 1991). Therefore, more stringent checks should be performed by the experimenter.
The visual stimuli used in the pilot study (Russell Brand’s Ponderland, 2008) will be altered as the participants in the pilot study produced a spoken narrative for an average duration of 1.03 minutes. Making participants produce a longer narrative would ensure that there is enough footage to be able to select a one minute clip that contains minimal face touching. It has been speculated as to whether a short oral narrative was produced by participants in the pilot study because the stimuli, being completely in English, may be too difficult for some of the L2 English speakers, thus, the stimuli in question may require an extensive vocabulary to comprehend and talk about in great length. To eliminate this concern and to potentially give a more
accurate measure of facial expressions, the stimuli must encourage the participant to be more loquacious.
Past research has revealed that cartoons encourage participants to talk and gesture for a reasonable period of time (McNeill, 1992; Alferink, 2008). This idea will be replicated and the participants will be shown a cartoon and then asked to recall what they have seen. The participants will be shown the cartoon Tom and Jerry, episode 34, Kitty Foiled (June 1, 1948). This is shorter than the stimuli in the pilot study and runs for a duration of 3.37 minutes. Tom and Jerry has been used in other
simplicity of this content, as it is intended for children, meant that it will be
understood by all participants. Additionally there is no speech featured in the chosen cartoon, thus, it will be ensured that there would be no language comprehension issues when viewing the stimuli.
The red mat from the pilot study, in which participants were asked to place their hands during the duration of the experiment, will be taken away. As participants reported, this practice feels too unnatural and makes you feel conscious of your body language. Thus, to encourage natural speech patterns the mat will be taken away. As a participant in the pilot had guessed the purpose of the study, extra lengths will also be taken to ensure that the experiment objective was hidden from participants. Feedback from the participants suggested that they were extremely intimidated having the experimenter sit in front of them. Two of the Dutch participants also tried to engage in conversation with the experimenter during the film. There were also issues with external noise as the participants had let themselves into the room and had not closed the sound-proof door. The experimenter will leave the room when the participants view the stimuli, this will ensure that the sound proof door will be closed and that a light can be activated notifying passers-by to be silent as filming is in progress. Upon re-entering the room, the experimenter will sit next to the participant rather than in front of the camera as in the pilot study. This will lower the anxiety and distraction noted by participants in the pilot study but also ensure that some social context remains which potentially will make participants more expressive. Additionally, three participants’ results had to be disregarded due to lighting and technical difficulties. Therefore, extra precautions will also be taken to ensure that lighting and camera equipment are working efficiently by running a one minute filming test before the participant enters the room.
3.3.1 Manual Coding Systems and Their Limitations
To train in FACS, a coder must spend over 100 hours learning the 46 action units (AUs) that Ekman and Friesen (1978) assigned to each facial muscle that contracts and changes the appearance of the face (Ekman & Friesen, 1978). Once trained, each minute of the video takes approximately one hour to score. (Donato et al., 1999). In contrast, Kring and Sloan (2007) claim that FACES is “a less time consuming
alternative to measuring facial expression that is aligned with dimensional models of emotion.” The FACES system asks coders to watch 5 minutes of silent footage of a subject observing emotional stimuli. The coder must note the frequency, intensity, valence, and the duration of the participant’s facial expressions (Kring & Sloan, 2007). FACES requires coders to make judgements on whether or not an expression is positive or negative. Since the interest of the study does not concentrate on the valence of facial expressions, this element of FACES is not suitable to code the video footage. FACES also asks participants to give a self-evaluation of their emotions. However, as the study is not concerned with emotion this element of FACES will not be used as it has the potential to influence the coder’s evaluation.
Kring and Sloan (2007) note that FACES is not without its problems. It is not suitable for clips “longer than 5 minutes” (Kring & Sloan, 1991) and there are a “number of things that may be problematic for coders” (Kring & Sloan, 1991) such as moments when the participant is not paying attention to the stimuli, shifting body positions, covering parts of the face with a hand, wearing glasses or chewing gum (Kring & Sloan, 1991). With such numerous requirements it is a very specific situation in which FACES is appropriate as a manual coding system.
In practice, neither the FACS nor the FACES manual coding system is a wholly appropriate choice for this experiment. In conclusion, elements of both coding systems have been selected to create a tool suitable for coding this experiment.
3.3.2 Coding the Footage
One minute of footage will be selected from each recording in which the participant recalled narrative regarding the stimuli. The only precursors to selecting the one minute of footage is that there is minimal touching of the face and that the participant stays in shot of the camera, as these interferences may block facial
coder at a speed of x0.10 as the slowing down of the footage may reveal more expressions (Ekman & Rosenberg, 1997). Similarly to the coding system FACES (Kring & Sloan, 1991), the footage will be watched by the coder without any sound. Head movements and eye movements will be recorded as facial expressions.
Expressions will be categorised according to Kring and Sloan’s definition. “[…] an expression is any change in the face from a neutral display (i.e., no expression) to a non-neutral display and back to a neutral display” (Kring & Sloan, 1991) and
secondly it is “a change from one non-neutral display to another non-neutral display […] For example, someone may begin smiling, but instead of returning to a neutral display begin frowning” (Kring & Sloan, 2007).
A “neutral display” (Notarious & Levenson, 1979; Kring & Sloan, 1991; Mitra &
Acharya, 2007) is defined as a face in which the movements of display are interpreted as a having a physiological attachment rather than a psychological interpretation. Therefore, it will be down to the discretion and interpretation of the coder to make a distinction between these two occurrences. Considering this
interpretation, it should be noted that facial movements connected to speech will not be recorded as expressions. As Kring and Sloan (1991) say in the FACES manual, “In the event that coders are faced with rating a segment in which a participant is
talking, attempts should be made to identify an expression independent of the
talking. For example, if a participant smiles and then begins talking, the smile should be recorded as an expression” (Kring and Sloan, 1991). Using this as a guideline, the coder will be able to identify the individual’s expression independent of talking. The variations that occurred in participant’s facial expression will be described and categorised using the descriptions in the manual coding system FACS (Ekman and Friesen, 1978). Once the variety in facial expressions has been assumed, these facial action units (AUs) will be divided and categorised in accordance to their placement on the face (brows, cheeks, chin and jaw, eyes, head, lids, and mouth). This
categorisation may accentuate differences between the four language groups in regards to the types of AUs used.
Once the AUs have been categorised according to their position on the face, a
will highlight the diversity in each participant’s facial performance during the one minute narrative sample.
3.4 Main Study
3.4.1 Materials
The experiment was conducted in a sound proof recording studio at the University of Groningen. The room had lights built-in which were all used to ensure that the footage was well lit and clear. The room also had a light which showed recording was in progress. This was activated during the experiment. All footage was recorded on a video camera with a built-in microphone. The camera was placed on top of a tripod. A classic Tom and Jerry clip was shown on a Lenovo Yoga 13 Laptop. The
participants also filled in a short survey on the laptop which asked them details about their age, name, studies, level of English and other languages they spoke.
3.4.2 Participants
The participants consisted of 12 heterosexual males age 19-24 that were in full-time education at university in Groningen, Netherlands, and studying for either a
Bachelor’s or Master’s Degree.
The sexuality of participants was taken into consideration as Hooijschuur (2014) found the sexuality of the speaker to have an impact on the frequency of gestures used. The homosexual participant’s in Hooijschuur’s (2014) research gestured more than heterosexual participant’s thus affecting the overall result.
Male participants were also chosen as it has been shown that gender can influence gesture rates (Hostetter & Hopkins, 2002).
Recruitment occurred via word of mouth, therefore, all participants, although are not all immediate friends, were all interconnected through the same social circles. All participants were living in Groningen at the time of the experiment. All 12
Table 1: Level of English of English Proficiency and Age Range for each group. Group Level of English Proficiency Number of
Participants Age Range
Group 1 Native 3 22
Group 2 Near-Native Proficiency 3 20-22
Group 3 Advanced Proficiency 3 21-24
Group 4 Upper Intermediate 3 19-21
[Group 1] The British English native speaking participants were comprised of three British males. Two participants were from Yorkshire, England and spoke with Northern English Yorkshire dialects. The third British English participant was from South London and spoke with a subtle Multicultural London English (MLE) accent which had been softened after he undertook elocution lessons.
The nine Dutch native speakers with English as an L2 were divided into three levels of English ability based on the language of their current study, the duration of their past studies which were full-time and conducted in English, their time spent living with an English native speaker, their time spent living in an English speaking country and their self-professed English levels and their levels obtained via studying English formally. Considering each of these factors, the information was then calculated to decide on the English proficiency level for each individual. All of the Dutch
participants had some English abilities and fit more than one of these criteria.
[Group 2] The three near-native L2 English speakers were Dutch native speakers that attended The International School Groningen (ISG), a high school in Groningen, Netherlands, which requires a native to high fluency in British English (Bonnema, 2015). Two of the participants studied full-time in English at ISG and one
participated in bilingual education at ISG that was evenly split between Dutch and English. The participant that undertook bilingual high school education was the only Dutch participant to have lived in an English speaking country, having lived in
Australia for 6 months. All three participants in this group are currently studying BA International Relations full-time in English at the University of Groningen and had lived with an English native speaker for over a year. Thus, the near-native proficiency participants have each been studying and interacting at a high level in English for almost 10 years.
their only participation in a programme that has been conducted in English on a full-time basis. They have all lived with a native English speaker for at least a year and therefore they write, listen, read and speak English on a daily basis.
[Group 4] The upper intermediate English L2 group were all Dutch native speakers. Group 4 had all lived with a native English speaker for some period of time. All the participants in this group are currently studying in Dutch and have never
participated in any full-time education in which English was the language of
instruction. Considering all these factors and their self-proclaimed results of formal testing, this placed Group 4 at English level B2: Upper Intermediate on the Common
European Framework of Reference for Languages (Byram & Parmenter, 2012).
Three of the Dutch participants were raised bilingually; one participant from Group 3 and one participant from Group 4 stated that they were Dutch and Frisian native speakers and one participant from Group 2 identified himself as a Dutch and
Indonesian native speaker. Therefore, one participant from each Dutch native group had two native languages and differed in their English abilities.
3.4.3 Procedure
The Dutch and British participants were asked to come to the recording studio at the University of Groningen to participate in a short memory test in which they would watch a 3 minute video and recall what they had seen. Upon entering the studio there was a sign on the door which read “Experiment in Progress: Kimberley Francis- 3 Minute Video Memory Test.” The participants were then asked to sit down in the studio at a table which had a laptop placed in front of them out of reach. In front of the table stood a video camera on a tripod. The participants were then told that recording had begun. They were then given instructions to watch the short 3 minute video and pay attention to the details as they would be asked questions about the video once it had ended. The experimenter then pressed play and the words ‘Tom and Jerry Thesis Memory Test.mp4’ appeared on the screen, the clip then began to play and the experimenter left the room. A light was then activated outside of the studio which notified passers-by to be silent as recording was in progress. After 3 minutes the experimenter returned to the studio and stood behind the camera
participant (a metre away) but did not look at or engage with the participant. The presence of the experimenter gave this situation some form of social context which would potentially increase the participants’ facial expressions.
Once the participant said they could not remember anymore, the camera was turned off and they were asked to complete a short online survey. The participants were then asked if they knew the purpose of the experiment. All participants believed it was a memory test. When the true purpose of the experiment was revealed some
participants continued to believe it was a memory test and asked for their results.
3.5 Visual experiment
3.5.1 Procedure and Materials
For the visual experiment, the one minute of footage that was previously coded was cut and re-sized using Adobe Premier CS6.
A control experiment was used in addition to the video footage in which the judges were asked to make cultural language judgements from still images of participants displaying neutral faces. This was used to ensure that judgements would be made on the participants’ facial expressions from the video footage, and not external factors such as their looks or clothes. Still images were taken from the recordings of the participants viewing the stimuli. These screenshots were then pasted together in groups using Adobe Photoshop CC (See Appendix B). In these images the
participants were not speaking and had their eyes fixed on the stimuli with their facial muscles relaxed. A “neutral display” (Notarious & Levenson, 1979; Kring & Sloan,1991; Mitra & Acharya, 2007) is defined as a face in which the movements of display are interpreted as a having a physiological attachment rather than a
psychological interpretation. Therefore, it will be down to the discretion and interpretation of the coder to make a distinction between these two occurrences. The control images and the one minute video footage of each participant was divided into three groups (See Table 2).
Table 2: A Table Showing the English level and the Participant Number of the Control Group and Video Group used in Each Survey
Video Group Control Group British Native English Dutch Native Near Native English Dutch Native High English Dutch Native Intermediate
Video Group 1 Control Group 3 BN1-EUA DH1- JEL DI1- DIC DL1-SJO Video Group 2 Control Group 1 BN2-ALI DH2-JOR DI2-WYT DL2-MAR Video Group 3 Control Group 2 BN3-ANG DH3-ROD DI3-RIN DL3-ATE Table 3: A Table Showing Which Control and Video Groups Were Used in Each Survey
Video Group Control Group
Survey A Video Group 1 Control Group 1
Survey B Video Group 2 Control Group 2
Survey C Video Group 3 Control Group 3
The surveys were created using SurveyGizmo. The first page asked participants to answer questions regarding their age, gender, nationality, current place of residence, places lived, languages and education level. Age and gender are particularly
important as these have been proven to have significant influence upon how a face is evaluated (Bäckman, 1991; Brigham & Barkowitz, 1978; Ebner, 2008; Golby, et al., 2001; Isaacowitz et al., 2006; Mather & Carstensen, 2003; Ruffman et al., 2008). Judges were first shown one of the control groups and told that some were Dutch and some were British and they had to label them accordingly. Then the first video from one of the video groups was shown. The participant was told. ‘This person is speaking English but there is no sound. Are they a British English Native speaker? [Yes/No]’. Each page had the option to leave a comment and the final page asked participants if they experienced any technical difficulties or would like to leave any comments regarding the survey. The judges were also asked if they recognised any of the participants featured as this may be a possibility due to the survey being shared on social media. Bunyan (2009 in Abidin & Kolberg, 2013), calculated that the average degree of separation for all users on Facebook was 5.73 degrees (Bunyan, 2009 in Abidin & Kolberg, 2013), thus, there is a risk attached to sharing the survey on social media as someone may recognise a participant featured in the survey and consequently know the nationality and first language of the participant in question. Therefore, this question was asked as an extra precaution.
All participants were found via an advertisement on the social media website
nationals to participate. 159 judges responded to the three surveys. 61 males and 98 females. 112 people were Dutch and 47 were British.
Chapter 4: Results 4.1 Facial Expression Analysis
A table was created in Microsoft Excel, for each participant, noting a number count of each action unit (AU) made by participants (see Appendix C).
The AUs were adapted from Ekman and Friesen’s (1978) FACS. It should be noted that the AU numbers here do not necessarily correlate with Ekman and Friesen’s (1978) work. Unlike FACS (Ekman & Friesen 1978), AUs are recorded separately rather than combined.
There were 46 variations of facial expression that were found when viewing the footage. These were put into a table and given an AU label number, a description and photograph that demonstrated the expression (see Appendix A). These facial action units (AUs) were then divided and categorised in accordance to their placement on the face (brows, cheeks, chin and jaw, eyes, head, lids, and mouth).
Table 4: Facial Action Units (AUs) Divided and Categorised in Accordance to Their Placement On the Face.
Section of the Face AUs
Brow AU-1 AU-2 AU-3 Cheeks AU-5 AU-14 AU-15
Chin and Jaw AU-18
AU-11 AU-12 AU-13 AU-16 AU-17 AU-19 AU-20 AU-21 AU-22 AU-23 AU-24 AU-25 AU-27 AU-29 Nose AU-7
A total number count of facial expressions was calculated from each participant’s one minute narrative sample (see Table 5) and a mean total of expressions per second were calculated based on these results.
Table 5: Total Number Count of Expressions and Mean Total of Expressions of per second of Participants’ Facial Expressions in a one minute sample
Group Participant Number of
Expressions Mean Total of Expressions of per second
BN1-EUA 114 1.89 British Native English BN2-ALI 30 0.50
BN3-ANG 67 1.08
DH1-JEL 51 0.85 Dutch Native Near Native English
DH2-JOR 114 1.09 DH3-ROD 124 2.06 DI1-DIC 75 1.25 Dutch Native High English DI2-WYT 113 1.83
DI3-RIN 118 1.96
DL1-SJO 128 2.13 Dutch Native Intermediate
English DL2-MAR 117 1.95
Figure 1: Individual Number Count of Participants’ Facial Expressions in a One Minute Sample.
The British native group have the biggest amount of variation (British Native
English: SD= 42.099) with subject BN2-ALI from this group (see Figure 1) producing the least amount of facial expressions of all participants in any group and only
generating a mean of 0.50 facial expressions per second. This is in contrast to BN1-EUA (see Figure 1) from the same group. The contrast in results led to the British Native group having the highest range in results of all the language level groups (British Native English; Range= 84) in comparison to the Intermediate L2 English group whose group only produced a range of 25.
Looking at the variation of AUs produced by participants (see Appendix C) reveals that there were AUs that were not performed by the British Native group but were performed by the Dutch L2 English participants.
The AUs that were not performed by the British natives were collected (see Figure 2) and compared, using the number count of each AU performed.
113 51 75 103 30 114 113 117 65 124 118 128
BRITISH NATIVE DUTCH NATIVE NEAR NATIVE ENGLISH
Figure 2: A Graph Representing the Number Count of Each AU That Was Not Performed by the British Native Group But Were Performed by Dutch L2 English Participants.
Glancing at Figure 2, there does not seem to be a significant correlation between English language levels and the AUs performed by subjects. Perhaps for the
exception of the frequent occurrence of AU-11 the Nasolabial Deepener and perhaps AU-21 Lip Funneler which have occurred more frequently in the lower English level groups and remain unseen in the native and near-native groups.
Once the AUs had been categorised according to their position on the face, a
percentage of each individual’s facial movements were then calculated. The stacked bar chart, Figure 3, highlights the diversity in each participant’s facial performance during the one minute narrative sample.
1 2 2 2 1 5 9 10 1 1 1 5 1 3 1 5 1 1 1 2 1 1 B N 1 -E U A B N 2 -A L I B N 3 -A N G DH1 -J E L DH2 -J O R DH3 -R O D D I1 -D IC D I2 -W Y T D I3 -R IN D L 1 -S J O D L 2 -M A R D L 3 -A T E N UM BE R O F E XP R ES SIO N S
Figure 3: A Stacked Bar Chart Showing the Diversity in the Participant’s Facial Expressions During the One Minute Sample. The Chart Includes the Percentage of Each Facial Expression Category Performed in the X-axis
Figure 3 reveals that all participants performed more eye, head and mouth
movements than any other facial expression category. Every group; Native, Advanced and Intermediate performed 0.24% of head movements with the exception of the Near Native group which used 0.27% of head movements.
Figure 4: A Pie Chart Depicting the Percentage of Head Movements in Each Language Group.
Figure 5: A Chart Showing the Average Number of Eye
Movements in Each English level Group with the Percentage of Eye Movements Being Represented on the X-axis. 0.24 0.27 0.24 0.24
Head Movements
Native Near Native Advanced Intermediate 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% British Native 1 British Native 2 British Native 3 Near Native 1 Near Native 2 Near Native 3 High 1 High 2 High 3 Intermediate 1 ntermediate 2 Intermediate 3 14% 3% 7% 23% 7% 2% 3% 9% 1% 6% 12% 9% 9% 10% 2% 6% 2% 1% 3% 4% 2% 2% 23% 3% 2% 2% 3% 27% 20% 33% 18% 42% 41% 37% 29% 2% 21% 24% 14% 18% 37% 18% 26% 22% 30% 14% 20% 39% 24% 19% 29% 9% 3% 10% 3% 2% 3% 1% 1% 2% 3% 28% 40% 30% 13% 21% 21% 20% 39% 49% 41% 36% 46% 2% 2% 2%Brow % Cheeks % Chin and Jaw% Eyes % Head % Lid % Mouth % Nose %
The Near-Native group used the most facial expressions in the eye movement
category. The Near-Native group performed 0.34% of eye movements in comparison to the lowest percentage, 0.19% performed by the Intermediate English group.
Figure 6: A Pie Chart Depicting Each Language Group’s Percentage of Mouth Movements.
In both eye and mouth movements the Intermediate group were amongst the lowest percentages of all the language level groups (Intermediate: Eye movements=0.19%, Mouth movements= 0.04%).
Conventional facial expression Head Nod (AU-46) was only performed by one participant DI3-RIN. The conventional facial expression Head Shake (AU-45) was performed the most frequently by the British Native Group with eight Head Shakes in total (BN1-EUA=3; BN2-ALI =5), then the Near Native group who produced six Head Shakes in total (DH1-JEL=2; DH2-JOR =4) and then the Intermediate English group who produced three Head Shakes in total (DL2-MAR=2, DL3-ATE=1).
The Advanced group did not perform any Head Shakes (AU-45) nor did BN3-ANG from the British Native group, DH3-ROD from the Near Native group or DL1-SJO from the Intermediate group.
Table 6: Count of the Conventional Facial Expressions AUs Performed by Participants
ANOVA shows that there was no significant effect of the English level of groups and the amount of conventional expressions performed (AU-45: f (3, 8) = 1.44, p= .301; AU-46: f (3, 8) =1 p=.441.
Facial expressions which are potential deictic gestures such as moving direction of eyes or head during speech, were analysed in concordance with the English language level groups.
ANOVA shows that there was no significant effect of the English level of groups and the potential deictic eye movement action units performed (AU-37: f (3, 8) =2.223,
p= .163; AU-38: f (3, 8) =2.241, p= .161; AU-39: f (3, 8) =.806, p= .525; AU-40: f (3,
8) =.589, p= .639; 41 f (3, 8) = .235, p=.869, 42 f (3, 8) =3.077, p= .091, AU-43 f (3, 8) =.296, p=.827 and AU-44 f (3, 8) =.444, p= .728).
Table 7: Count of the Potential Deictic Eye Movement AUs Performed by Participants
ANOVA shows that there was no significant effect of the English level of groups and the potential deictic head movement action units performed (AU-31: f (3, 8) =.546,
p= .665; AU-32: f (3, 8) =.477, p= .707; AU-33: f (3, 8) = 1.521, p= . 282; AU-34: f (3,
8) = 1.194, p= .372; AU-35: f (3, 8) =.269, p= .846; AU-36: f (3, 8) = .414, p= .747). Table 8: Count of the Potential Deictic Head Movement AUs Performed by Participants
4.2 Survey Responses
4.2.1 Accuracy of the Judges
4.2.1.1 Photos
Dutch and British judges were asked to make cultural language judgements from still images of participants displaying neutral faces.
Table 9: Frequency of the Judges’ Nationality that Rated the Control Photograph
Judges were shown four photographs in which they must make native language judgements. From the 159 judges, 112 judges were Dutch and 47 were British. 182 ratings were made by British judges and 446 ratings were made from Dutch judges. This led to 628 judgements being made in total. The majority of participants
identified the nationality of 3 out of the 4 photographs they were shown correctly. Only two participants were unable to correctly identify any of the control
photographs.
Table 10: The Frequency of Correct Answers to Control Photograph in Correlation with the Gender of the Judges
In total 61, (Percentage=38.4%) survey respondents were male and 98
(Percentage=61.6%) were female. Pearson Chi-Square revealed that there is no association between the gender of the judges and accuracy of their control photograph judgement (χ² (4) = 3.278, p=.512).
Nationality of Rater Frequency
Dutch 112
British 47
Total 159
Correct Ratings Gender Total
