The Effects of Emotional Expressions and Gaze Direction –
Social Referencing in an Eye Tracking Paradigm
Miriam Hollarek 10608753 September 2014 Research Internship Department of Psychology University of Amsterdam
First Supervisor: Maartje Raijmaakers Second Supervisor: Evin Aktar
Abstract
The ability to use others’ interpretation of situations to guide one's own behavior, referred to as social referencing is a crucial mechanism for infants to learn about novel situations. Along these lines, an adult’s reaction to an object is known to influence infants' behavior towards the object. This study aimed to investigate how adults’ different emotional expressions influence infants' processing of novel objects and how the adult's gaze direction is related to this. Eye tracking data was recorded from 15-month-old infants during picture presentations of a female stranger who exhibited fearful, happy, sad or neutral facial expressions, either looking away or towards a novel object. Results show, firstly, that pupil dilation for objects that were previously presented with a neutral expression was significantly larger than pupil dilation for objects that were
presented in fearful, happy or sad conditions. Secondly, during paired presentation of objects and faces, infants shifted their gaze significantly faster from happy faces to the object as compared to neutral and fearful expressions. Gaze direction neither affected pupil dilation nor time of gaze shifts. Differences between early processing of faces and objects as well as implications for the negativity bias are discussed.
The effects of emotional expressions and gaze direction – Social referencing in an eye tracking paradigm
Social referencing (SR) is defined as the ability to coordinate one’s gaze between objects and people and to use others' interpretation of ambiguous situations to guide one’s own
understanding and behavior in the situation (Feinman, 1982). SR is a key mechanism for early social learning in infancy (Senju, Csibra, & Johnson, 2008). It provides the base for developing theories of mind (Striano & Rochat, 2000) and is crucial to learn how to regulate emotions and behavior (Schmitow & Stenberg, 2013). Infants as young as 10 months are capable of SR
(Striano & Rochat, 2000) and it has been a robust finding in observational research since decades (Klinnert, Emde, Butterfield, & Campos, 1986; Hirshberg & Svejda, 1990; Aktar et al., 2013). Infants use familiar adults as well as strangers as a referent in novel situations (Gredebäck, Fikke, & Melinder, 2010), showing a preference for the most knowledgeable person in that situation (Schmitow & Steinberg, 2013).
An evolutionary advantage of SR has been proposed as it serves as a mechanism to learn about the environment, without directly facing the threats in potentially dangerous situations (Vaish, Grossmann, & Woodward, 2008). To be able to learn via SR infants need to understand the meaning of their referent’s emotional expression and recognize at what object this emotion is directed. Infants’ processing of emotional expressions and gaze direction is therefore essential for SR processes.
Around 7 months of age (reviewed in Grossmann, 2010), infants start to differentiate between different emotions. From 12 months on, research has repeatedly found a preference for threat relevant visual stimuli (Leppänen & Nelson, 2012). For example fearful expressions are
more readily detected and attended to longer as compared to happy expressions (Yang, Zald, Blake, 2007; Nelson & Dolgin, 1985). This asymmetry for negative stimuli (negativity bias) has been established for adults, but its precise ontogenesis is still under debate (Vaish et al., 2008). A preference for fearful stimuli could have evolved to promote survival by facilitating processing of threats in the environment (Leppänen & Nelson, 2012).
After encoding their referent’s emotion, infants need to direct their attention to the relevant object by following their referent’s gaze.By the age of 8 months infants have fully acquired the ability to follow others' gazes (Gredebäck et al., 2010). Previous studies showed that infants’ processing of the adult’s gaze differs for object directed and non-object directed gazes (Hoehl, Reid, Mooney, & Striano, 2008). Using an EEG paradigm, it was found that infants showed enhanced processing for objects when a face was previously directed towards the object as opposed to objects from which a face was turned away (Reid, Striano, Kaufman, & Johnson, 2004). The authors proposed that the direction of gaze of the referent facilitates object encoding by biasing the infant's attention towards the jointly attended object (Reid et al. 2004). Indeed studies showed that infants’ looking time and negative central component increased for objects from which a face had previously looked away (Hoehl, Wahl, Pauen, 2014). The authors suggest that uncued objects remain more novel and thus require more attention. From an evolutionary perspective, gaze following is adaptive because it facilitates faster detection of threat (Haxby, Hoffman, & Gobbini, 2002).
To fully appreciate a novel situation, infants need to combine the information of the emotional expression and the direction of gaze of their referent. This combined effect of emotion and gaze has been investigated in various age ranges using different paradigms. Observational research found that 12 months old infants hearing an emotional outburst actively
seek out information from a referent and relate the expressed emotion to a jointly attended toy. During this task infants looked more often at their referent in response to expressions of disgust as compared to joy (Moses, Baldwin, Rosicky, & Tidball). Using EEG paradigms in three and seven month old infants comparison of event related components showed a larger negative central component for objects which had previously been paired with a fearful expression as compared to a neutral expression, however this effect was only found if the face was directed at the novel object (Hoehl, Wiese, & Striano, 2008; Hoehl, Palumbo, Heinisch, & Striano, 2008). The authors concluded that a fearful expression of an adult directed at a novel object enhances infants' attention paid to that object (Hoehl, Wiese, & Striano, 2008). Similarly three month old infants also showed a larger negative central component during presentations of objects paired with happy expressions as compared to neutral expressions, but again only when faces were looking towards the object (Hoehl & Striano, 2010).
This study firstly aimed to investigate the underlying mechanisms of the previous findings. ERP studies cannot directly examine the nature of this general enhanced processing. In order to examine the visual components of the observed enhanced object processing for emotionally cued objects, this study used an eye tracking paradigm. Eye tracking data not only tracks allocation of overt visual attention, but also allows for inferences about autonomic arousal by tracking pupil dilation (Gredebäck, Johnson, & von Hofsten, 2010). It hereby offers objective measurements in high special and temporal resolution free from all observational bias.
We secondly aimed to differentiate between effects of general negative or positive emotional salience and effects of specific emotions (e.g. fear). In addition to fearful and neutral expressions we therefore also included happy expressions as an additional positive and sad expressions as an additional negative reference. Pupil dilation was recorded during the first
presentation of a novel object and again during the second presentation, after it had been paired with a sad, happy, neutral, or fearful expression. Faces either looked away from the object or towards the object. During paired presentations of the face and the object we recorded time of gaze shifts from the emotional faces to the novel object.
Based on previous ERP studies we hypothesized firstly, that infants would exhibit enhanced pupil dilation for objects that had previously been paired with sad, happy, or fearful expressions as compared to neutral expressions. Secondly, we expected enhanced pupil dilation if the gaze of the face was directed towards the novel object as opposed to away from it. Thirdly, we expected that infants would shift their gaze faster from fearful faces to the novel object compared to happy and neutral conditions. And lastly, we hypothesized that faster gaze shifts would be enhanced for conditions where the face was directed towards the novel object as opposed to away from it.
Method Participants
We initially recruited 64 infants (mean age = 15.3 months, SD = 0.5 months, 29 female) for the study. Due to failed tracking 13 infants were excluded from the analysis, leaving 51 participants. All infants were healthy and without any visual or neurological deficits. They were recruited via the municipality who sent letters to a random sample of women who recently had a baby.
Stimuli and Apparatus
The novel objects consisted of 16 unfamiliar objects from the database Novel Object and Unusual Name (Horst, 2009). Pictures of facial expression were used from the Radboud Faces
Database (Langner et al., 2010). We used neutral, fearful, happy, and sad faces by a Caucasian female model, with a gaze turned 45 degrees to the left or to the right. Eight colorful animated pictures moving in synchrony with different sounds were used to grab infants' attention between each trial (Gredebäck et al. 2010).
The study used the Tobii T120 with Tobii studio eye-tracking system with a sampling frequency of 120Hz. Stimuli were displayed on an 17 inch TFT monitor integrated in the Tobii T120 system.
Procedure
The current study investigated infants' pupil size changes to unknown objects that had previously been paired with emotional facial expressions directed towards or away from the object. (1) Pupil dilation during observation of the object was the dependent variable to quantify early attention allocated to the objects. (2) Latency effects of gaze shifts were investigated using time of the first gaze shift from the face to the object during paired presentation. The following section describes the study design and data collection procedure:
Infants came to the baby lab with one of their parents. Parents were informed about the procedure and signed an informed consent sheet. The infants were placed in an elevated car seat using the seat belt for fixation, while their parent sat behind them. Luminance in the room was kept constant across measurements. A screen was located 65cm in front of the car seat. Each session started with a 5 point calibration (Gredebäck et al. 2010). After successful calibration, the task and eye tracking recording started.
Two stimuli blocks, each consisting of eight trials, were presented. Each block always started with the neutral face to avoid potential carryover effects of the emotional salient
appeared alone in the center of the screen (2 sec) followed by a blank screen (0.5 sec). Next, a face displaying an emotional expression first appeared alone (1 sec) and then next to the object (2 sec), again followed by a blank screen (0.5 sec). The trial finished with the object reappearing alone for two seconds (Figure 1). Each emotion condition was presented twice in consecutive trials with different gaze directions. The order of emotions and gaze direction were randomized. Gaze directions were reversed in the second block to account for side bias.
Figure 1: Trial sequence.
The experimenter observed the infant and repeated the presentation of ‘attention getters’ when necessary to direct children’s attention back to the screen. Once all tasks were completed
the parent and infant were thanked for their participation and received 10 euro and a children’s book or a stuffed animal as compensation.
Data reduction and analysis
Pupil dilation. Pupil dilation was computed as the difference between the mean pupil
dilation during the second object presentation and the mean pupil dilation during the first object presentation. Pupil dilation scores that were above or below three standard deviations from the mean of an individual infant's distribution were replaced by their individual mean value. When pupil dilation data was missing for only one eye, the data of the other eye was used for
interpolation. The mean pupil dilation data of both eyes was then computed and used for further analysis. The pupil dilation data was interpolated linearly when data was missing for less than 500 msec. Trials were removed if the infant was looking at the stimuli for less than 500 msec - that is 25 percent of the time during the paired presentation. After interpolation, mean difference scores were still missing for 35 percent of all observations.
Gaze shift. Sad condition trials were removed for the analysis of gaze shift, since sadness
was not expected to influence time to gaze shift, and in fact there was only one valid observation for gaze shift in the sad condition. To analyze the time until gaze shift areas of interest were defined for the face and the object. Time to gaze shift was computed for each paired presentation as the time from the first fixation on the face to the first fixation on the object. When data of visual fixation was missing for less than 500 msec the visual fixations were interpolated with the observations of the previous location. For cases where a new event started during this time, visual fixations were not interpolated. Scores that deviated with more than three z scores of time to gaze shift of all children were replaced with the mean value of each emotion respectively. After interpolation time until gaze shift was missing for 36 percent of all observations.
A repeated multilevel analysis was used to analyze the data to account for nested data within participants and for the many missing observations. Maximum Likelihood estimation was used to estimate the parameters. Repeated factors were type of emotion, block repetition and gaze direction. No significant random effects were found when comparing models with random effects of emotion and gaze direction for the dependent variables pupil dilation, CHI² (2) = 0.0, p > 0.5 and gaze shift, Chi² (2) = 5.4, p > 0.5.
Results
Pupil dilation. A Multilevel model fixed analysis with pupil dilation as the dependent
variable and emotion and gaze direction as the independent variables was performed using sad and looking towards the object gaze as a references. The analysis showed a significant positive intercept (EstIntercept) = 0.1030, SE = 0.026, t (174.65) = 3.937, p < 0.001 and a significant effect of neutral emotion (EstNeutral) = 0.09073, SE = 0.025, t (494.10) = 3,592, p < 0.001. This suggests that the difference score for pupil dilation increases for objects that were paired with a neutral expression as compared to all other emotions. No effects of fearful or happy expressions was significant on the difference score of pupil dilation (EstFear) = -0.00735, SE = 0.026, t (492.34) = -0.285, p = 0.776, (EstHappy) = -0.00262, SE = 0.026, t (494.00) = -0.100, p = 0.920. Moreover, looking away or towards the object did not significantly affect the difference score of pupil dilation (EstAway) = -0.01427, SE = 0.018, t (493.23) = -0.786, p = 0.432. The observed and predicted difference score of pupil dilation correlated significantly r = 0.181, p < 0.01. By squaring the correlation we obtain a measure of the global effect size R² and conclude that the model explains 3.3 percent of the variability of pupil dilation.
Gaze shifts. A Multilevel model fixed analysis with time of gaze shift as the dependent
variable and emotion and gaze direction as the independent variables was performed using neutral and looking towards the object gaze as a references. The analysis showed a significant effect for the happy condition (EstHappy) = -54.30, SE = 24.26, t (367.12) = -2.238, p = 0.026, showing that infants looked significantly faster to objects that were paired with a happy face than with a neutral face. The fearful condition did not significantly differ from the neutral condition (EstFear) = -20.34, SE = 25.19, t (373.12) = -0.808, p = 0.42. Moreover there was no significant effect of looking towards or away from the object (EstAway) = 16.98, SE = 24.09, t (367.11) = -0.705, p = 0.48, nor any significant interaction effect
(EstAway * Fear) = 44.75, SE = 25.71, t (373.35) = 1.25, p = 0.21, (EstAway*Happy) = 25.26, SE = 35.05, t (369.04) = 0.721, p = 0.47. The observed and predicted difference score of pupil dilation correlated significantly r = 0.147, p < 0.01. The global effect size R² is thus 0.021 and the model explains 2.1 percent of the variability of gaze shifts.
Discussion
The present study investigated the influence of emotional expressions and gaze direction on infants' object processing. Infants were presented with pictures of a female stranger either looking away or towards a novel object. Firstly, pupil dilation analysis showed significantly smaller pupil dilation for objects that were previously paired with happy sad or fearful faces as compared to objects that were previously paired with neutral faces. Secondly, analysis of gaze shifts showed that the time until the first gaze shift from the face to the object during paired presentations was significantly faster in happy conditions as compared to neutral conditions.
Gaze shifts during fearful conditions were not significantly faster than during neutral condition. Direction of gaze neither significantly influenced pupil dilation nor gaze shifts.
The data showed a significant difference between the pupil dilation during object presentations that were previously paired with neutral expressions as compared to objects that were previously paired with fearful, happy, or sad expression. However the direction of this effect was opposite to our hypothesis. We hypothesized larger pupil dilations in the emotional condition, but our data reviled larger pupil dilations in the neutral condition. We did not anticipate these findings because previous studies found larger pupil dilation for emotional conditions. Infants showed enhanced pupil dilation at 6 and 12 month while watching videos of peers in distress or experiencing happiness as compared to neutral states (Geangu et al., 2011). However, this study investigated infants’ reactions to the emotional facial expression itself, not to an object associated with an emotional expression. Research that investigated infants’ object processing on the other hand, report results similar to our study. In an ERP paradigm infants’ negative central component increased for objects in the neutral condition (Hoehl et al., 2010), indicating enhanced processing for objects in neutral conditions. Thus, it might be the case that infants respond with enhanced pupil dilation for emotional faces but not for objects associated with them.
A possible explanation for the observed enhanced pupil dilation during neutral trials is the lack of information that infants receive from their referent when he or she displays an ambiguous neutral facial expression. Therefore, these objects might elicit more arousal in the infant because they cannot rely on their referent for cues on how to behave but have to evaluate the object on their own. In order to make these evaluations, infants might require additional processing for object presented during neutral conditions.
A negativity bias in early development which states that infants' attention is biased towards negative stimuli (Vaish et al. 2008) is not directly supported by the current study. However, this study focuses on object processing and our results suggest that the effect of emotional versus neutral expressions might differ for processing of faces and processing of objects associated with emotional expressions. It seems plausible that the negativity bias is only evident for processing of faces and as opposed to objects. Therefore we cannot rule out the existence of a negativity bias in infants. The negativity bias might simply be restricted to facial expressions and does not generalize to object processing.
Our second hypothesis concerned the time until infants’ first gaze shift from the face to the novel object. Here we hypothesized (1) that infants would shift their gaze faster from the face to the object during fearful conditions (2) that infants would shift their gaze faster to objects when the face was directed towards the object, and (3) that looking towards the object during fearful conditions would result in even faster gaze shifts. Contrary to our hypotheses however analysis showed (1) significantly faster gaze shifts during happy conditions as compared to fearful and neutral conditions, (2) no effect of looking towards or away from the object, (3) and no interaction between emotion condition and looking away or towards the object.
Longer times until the first gaze shift for fearful and neutral conditions could be explained by disengagement inhibition (Leppänen et al., 2009). The ambiguity of the neutral condition and the novelty and arousal of the fearful condition might prolong the time until infants shift their gaze to the object because these faces attract infants' attention for longer time periods. Happy faces, on the other hand, are relatively familiar to infants and, thus, might be easier to disengage from. Moreover, following the gaze of a happy face might be followed by a visual reward which would motivate infants to shift their gaze faster.
Again, contrary to our hypothesis and previous studies (Hoehl et al., 2008) infants' reactions to the presented objects did not differ when faces were directed towards or away from the object. When comparing age groups, one study found that three and six month old infants showed different object processing of cued objects as compared to uncued objects, while no different processing was found for nine month old infants (Hoehl & Striano, 2010). The effect of gaze direction on object processing might be more pronounced for younger infants and might thus not be evident in our sample of 15 month old infants. An alternative explanation would be that gaze cuing an object facilitates processing via other mechanisms than autonomic arousal or visual attention, so the eye tracking data is not sensitive to the differences that gaze direction has on object processing.
Interestingly, there was only one infant in the sad condition who switched its gaze from the sad face to the object at all. All other infants either only looked at the face, only at the object or looked away during sad trials. Different explanations could account for this fining: Firstly, infants might avoid looking at any sad stimuli in general (Gredebäck et al. 2010). Secondly, infants might not be motivated to follow an adult's sad gaze because they neither expect threat, as in the fearful condition, nor a reward as in the happy condition. Thirdly, infants are usually not exposed to sad faces and may thus fail to disengage from the face before the object disappears (Leppänen et al. 2009).
One limitation of the current study is the fixed position of the neutral condition at the beginning of each block. Novel stimuli enhance pupil dilation (Naber, Frassle, Rutishauser, & Einhauser, 2013). Hence, it is plausible that first presentations - always neutral conditions - enhance pupil dilation. We would optimally want to randomize all conditions, however for our purpose of identifying arousal it is crucial to prevent neutral conditions from being contaminated
with carryover effects of the emotional conditions because once arousal is triggered it is unlikely to go back to zero for neutral conditions. Since carry over effects of the emotional conditions might make it completely impossible to compare neutral and emotional conditions, the fixed position of the natural conditions is preferred to complete randomization.
Another issue of the study worth mentioning is power. The sample size was computed beforehand and should be sufficient to detect the hypothesized effects but 35% missing
observations were not anticipated. Due to this power loss more subtle effects in the data such as differences between the types of emotions or interaction effect with gaze direction might simply not be detected.
The current study was one of the first to investigate infants’ object processing via eye tracking. Our results showed that infants respond with enhanced pupil dilation to objects in neutral as compared to happy, fearful, or sad conditions. Moreover infants shifted their gaze from the face to the object faster for objects in happy conditions as compared to fearful or neutral conditions. The study demonstrates that infants’ enhanced processing of faces with emotional as compared to neutral expressions - as found in previous studies - does not necessary generalize to processing of objects. In order to disentangle the two processes future studies should directly compare infants’ reactions to faces displaying emotional expressions to infants’ reactions to objects paired with these emotional faces.
Concerning the negativity bias this study suggests that there is no such bias in the
processing of objects paired with negative emotions. Nevertheless, this bias could still be present in face processing and the question remains whether all negative emotions or only threatening stimuli can elicit the proposed bias. Differentiating not only between the effect of specific types of emotions and general emotional salience but also between different mechanisms of object
processing can be difficult. The current study demonstrates that to overcome these obstacles future studies should include additional positive and negative emotional salient stimuli for baseline comparison to be able to draw conclusions about specific types of emotions. In addition to that future studies should continue to employ various different techniques to investigate infants’ processing of emotion related stimuli. Adding eye tracking paradigms to observational research and ERP studies contributes to the methodological pluralism that is needed to resolve the remaining questions.
References
Aktar, E., Majdandžić, M., de Vente, W., & Bögels, S. M. (2013). The interplay between expressed parental anxiety and infant behavioural inhibition predicts infant avoidance in a social referencing paradigm. Journal of Child Psychology and Psychiatry, and Allied
Disciplines, 54(2), 144–56.
Feinman, S. (1982). MERRILL-PALMER QUARTERLY. Merrill-Palmer Quarterly (1982- ),
28(4), 445–470.
Gredebäck, G., Fikke, L., & Melinder, A. (2010). The development of joint visual attention: a longitudinal study of gaze following during interactions with mothers and strangers.
Developmental Science, 13(6), 839–48.
Gredebäck, G., Johnson, S., & von Hofsten, C. (2010). Eye tracking in infancy research.
Developmental Neuropsychology, 35(1), 1–19.
Grossmann, T. (2010). The development of emotion perception in face and voice during infancy.
Restorative Neurology and Neuroscience, 28, 219–236.
Haxby, J. V, Hoffman, E. a, & Gobbini, M. I. (2002). Human neural systems for face recognition and social communication. Biological Psychiatry, 51(1), 59–67.
Hoehl, S., Palumbo, L., Heinisch, C., & Striano, T. (2008). Infants’ attention is biased by emotional expressions and eye gaze direction. Neuroreport, 19, 579-582.
Hoehl, S., Reid, V., Mooney, J., & Striano, T. (2008). What are you looking at? Infants’ neural processing of an adult's object directed eye gaze. Developmental Science, 11(1), 10-16.
Hoehl, S., & Striano, T. (2010). The development of emotional face and eye gaze processing. Developmental science, 13(6), 813-825
Hoehl, S., & Striano, T. (2010). Infants' neural processing of positive emotion and eye gaze. Social neuroscience, 5(1), 30-39.
Hoehl, S., Wahl, S., & Pauen, S. (2014). Disentangling the effects of an adult model's eye gaze and head orientation on young infants' processing of a previously attended object. Infancy,
19(1), 53-64.
Hoehl, S., Wiese, L., & Striano, T. (2008). Young infants’ neural processing of objects is affected by eye gaze direction and emotional expression. PloS One, 3(6), e2389. Horst, J. S. (2009). Novel Object and Unusual Name (NOUN) Database. Retrieved, from:
http://www.sussex.ac.uk/wordlab/noun
Klinnert, M. D., Emde, R. N., Butterfield, P., & Campos, J. J. (1986). Social referencing: The infant’s use of emotional signals from a friendly adult with mother present. Developmental
Psychology, 22(4), 427–432.
Leppänen, J. M., & Nelson, C. a. (2012). Early Development of Fear Processing. Current
Directions in Psychological Science, 21(3), 200–204.
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H.J., Hawk, S.T., & van Knippenberg, A. (2010). Presentation and validation of the Radboud Faces Database. Cognition & Emotion,
24(8), 1377—1388.
Naber, M., Frassle, S., Rutishauser, U., & Einhauser, W. (2013). Pupil size signals novelty and predicts later retrieval success for declarative memories of natural scenes. Journal of Vision,
13(2):11, 1–20.
Nelson, C. A, & Dolgin, K. G. (1985). The generalized discrimination of facial expressions by seven-month-old infants. Child Development, 56(1), 58–61.
Quinn, P. C., Uttley, L., Lee, K., Gibson, A., Smith, M., Slater, A. M., & Pascalis, O. (2008). Infant preference for female faces occurs for same- but not other-race faces. Journal of
Neuropsychology, 2(1), 15–26.
Reid, V. M., Striano, T., Kaufman, J., & Johnson, M. H. (2004). Eye gaze cueing facilitates neural processing of objects in 4-month-old infants. Neuroreport, 15(16), 2553–5. Schmitow, C., & Stenberg, G. (2013). Social referencing in 10-month-old infants. European
Journal of Developmental Psychology, 10(5), 533–545.
Senju, A., Csibra, G., & Johnson, M. H. (2008). Understanding the referential nature of looking: infants’ preference for object-directed gaze. Cognition, 108(2), 303–19.
Striano, T., & Rochat, P. (2000). Emergence of selective social referencing in infancy. Infancy,
1(2), 253–264.
Vaish, A., Grossmann, T., & Woodward, A. (2008). Not All Emotions Are Created Equal: The
Negativity Bias in Social–Emotional Development. Psychological Bulletin 134(3), 383–
403.
Yang, E., Zald, D. H., & Blake, R. (2007). Fearful expressions gain preferential access to awareness during continuous flash suppression. Emotion (Washington, D.C.), 7(4), 882–6.
