GENDER MISMATCH DETECTION AND PREDICTION IN VISUAL WORLD PARADIGM: THE EFFECT OF EARLY LOOKS ON L1 AND L2 PROCESSING OF
GENDER IN NORWEGIAN
Anastasia Shaboltas
Under the supervision of Professor Dr. Irina Sekerina at the City University of New York and Dr. Bjørn Lundquist at the University of Tromsø
Abstract
In the present study the effect of early looks on the pattern of predictive gender processing in the Visual World Paradigm (VWP) is investigated. Using previously collected data from Johannessen et al. (in prep.) I compare predictive looks to target of L1 and high- proficient L2 Norwegian speakers in the trials where they looked at the distractor when hearing a gender-marked article in a speech stimulus (i.e., had early looks at the distractor) with the trials where early looks of the participants were at the white space or the target. In the experiment the participants heard a sentence such as Jeg tenker på en avbilda bil (’I am thinking of a depicted car’) and saw two pictures of the same or different gender – one representing a target noun mentioned in the sentence and one representing a distractor. In such experimental setup the predictive effect of gender condition is seen as the increased proportion of looks to target in different-gender condition where a gender-marked article can be used as a predictive cue in contrast to same-gender condition where a gender-marked article cannot be informative for locating the target.
High-proficient L2 speakers of Norwegian showed a temporally different profile of the predictive effect with earlier increase in looks to target in different-gender condition in the trials with early looks at the distractor than in the trials with early looks at the white space. These results support the proposed hypothesis that the effect of gender condition can be driven by two temporally distinct types of processing depending on early looks – earlier gender mismatch detection in case of early looks at the distractor and later prediction in case of early looks at the white space. However, L1 control group did not show a temporal difference similar to high-proficient L2 group, although the effect of gender condition was larger in the trials with early looks at the white space. I suggest that overall results do not provide direct evidence of L1 and L2 speakers having qualitative or just quantitative difference in gender processing due to substantial variation in early looks and difficulties to
assess the factor of early looks statistically. In order to address this issue and target gender processing more efficiently, I propose a new three-panel visual stimuli design based on two- picture display from Johannessen et al. (in prep.) with adding an extra panel to the display.
Key words: prediction; gender mismatch detection; gender processing; eye-tracking;
Visual World Paradigm.
Acknowledgments
I would like to thank my supervisors, without whom this work could not be possible, for transforming a difficult journey of thesis writing into a useful and educational experience.
Björn Lundquist for supporting me through all the stages of the process from the idea of the thesis to the data analysis, especially for creating friendly and calm atmosphere of cooperation even when I was isolated above the Arctic Circle. Dr. Irina Sekerina for helping me stay on track and make my writing process more thoughtful, for valuable comments on how the text can be improved and correcting the mistakes, for teaching me how to convey my thoughts more clearly. I would also like to thank everyone at the University of Tromsø who were kind to participate in pilot testing of the new experiment.
Table of contents Contents
Abstract ... iii
Acknowledgments... v
Table of contents ... vi
Contents ... vi
LIST OF TABLES ... viii
LIST OF FIGURES ... ix
Chapter 1 ... 1
1 Introduction ... 1
1.1 Prediction in comprehension... 1
1.2 Experimental methods of studying prediction ... 4
1.2.1 EEG ... 4
1.2.2 The Visual World Paradigm (VWP) ... 5
1.3 Prediction in L1 and L2 ... 8
1.3.1 Grammatical gender as a cue for prediction in L1 ... 10
1.3.2 Grammatical gender as a cue for prediction in L2 ... 12
Chapter 2 ... 16
2 Data reanalysis and the current study ... 16
2.1 Grammatical gender as a predictive cue in VWP ... 16
2.2 The current study and research questions ... 18
2.3 Gender in Norwegian ... 22
2.4 The dataset from Johannessen et al. (in prep.) ... 23
2.5 Secondary data reanalysis ... 28
2.6 Results ... 29
2.6.1 Goal and predictions ... 29
2.6.2 Early filler condition (Figure 2.9) ... 30
2.6.3 Early distractor condition (Figure 2.10) ... 31
2.6.4 Early target condition (Figure 2.11) ... 32
2.7 Discussion ... 33
Chapter 3 ... 38
3 Future directions and a new stimuli design... 38
3.1 Introduction ... 38
3.2 Method ... 39
3.2.2 Materials ... 39
3.2.2 Apparatus and procedure ... 41
3.3 Preliminary results ... 43
3.4 Discussion ... 44
4 Conclusion ... 46
References ... 48
Appendix ... 55
Auditory stimuli in the experiment with a three-panel design ... 55
LIST OF TABLES
2.1 Examples of Norwegian nouns in indefinite, definite, double definite form and nouns with adjectives... 23 2.2 A time-line of a speech stimulus ... 26 2.3 Regression coefficient table for L1 speakers and high-proficient L2 speakers of Norwegian ... 33 3.1 Experimental design with three panels and six variations of the items in same- and different-gender conditions ... 43
LIST OF FIGURES
2.1. An example of same-gender condition (de schoen ‘the-COM shoe’ vs. de lamp ‘the- COM lamp’) and different-gender condition (de schoen vs. het huis ‘the-NEUT’ house) in Dutch from Brouwer, Sprenger, and Unsworth (2017). ... 17 2.2. An example of Norwegian stimuli in same-gender (en bjørn ‘a-MASC bear’ vs. en gris
‘a-MASC pig’) and different-gender condition (en bjørn vs. et esel ‘a-NEUT donkey’) from Lundquist, Rodina, Sekerina, and Westergaard (2016). ... 17 2.3. An example of the increase in proportion of looks to target in Norwegian L1 speakers, who use gender information at the article predictively, in different-gender condition (blue line) compared to same-gender condition (red line) (Johannessen et al., in prep.). ... 18 2.4. Proportion of looks to target in the trials with early looks at the distractor in L1 control group. ... 19 2.5. Proportion of looks to target in the trials with early looks at the white space (here, early filler) in L1 control group. ... 20 2.6. Proportion of looks to target in L1 control group and high-proficient L2 group with the effect of gender condition shown as a separation of a blue line (different-gender condition) from a red line (same gender condition) (Johannessen et al., in prep.) ... 24 2.7. An example of the visual stimulus in different-gender condition from Johannessen et al.
(in prep.): et egg ‘an-NEUT egg-NEUT’ and en løk ‘an-MASC onion-MASC’. ... 25 Figure 2.8. The eye-tracking results of L1 Norwegian control group. Separation of red and grey lines marks lexical access when the proportion of looks to the target in same-gender condition (red line) increases as compared to the looks to the distractor (here Comp., grey line) (Johannessen et al., in prep.)... 27
2.9. Proportion of looks to target for L1 control group and high-proficient L2 group in the trials with early looks at the white space (here early filler). ... 30 2.10. Proportion of looks to target in L1 and high-proficient L2 groups in the trials with early looks at the distractor. ... 31 2.11. Proportion of looks to target in L1 and high-proficient L2 groups in the trials with early looks at target. ... 32 3.1. An item in same-gender condition with the target on the left (en bil ‘a-MASC car’) and distractor on the right (en sko ‘a-MASC shoe’)... 40 3.2. An item in different-gender condition with a pink panel on the left, a target in the middle (et tre ’a-NEUT tree’) and a distractor on the right (en bil ’a-MASC car’)... 40 3.3. Proportion of looks to target of L1 Norwegian speakers in the trials with a distractor in the middle of the screen and in the trials with a pink panel in the middle of the screen. ... 44 3.4. Proportion of looks to target of L1 Norwegian speakers in the trials with early looks at the distractor and in the trials with early looks at the white space (here early filler) from Johannessen et al. (in prep.). ... 44
Chapter 1
1 Introduction
1.1 Prediction in comprehension
Various studies provide evidence that people use prediction when comprehending language. Predictive view of comprehension suggests that a person can use higher-level information provided by the context to create expectations about what comes next in the input. Evidence has been found that in order to facilitate language processing, a person can use different types of information from the context that allows prediction to happen, at the levels of semantics (Altmann & Kamide, 1999; Federmeier & Kutas, 2001; Dijkgraaf, Hartsuiker, & Duyck, 2019), syntax (Van Berkum, Brown, Zwitserlood, Kooijman, &
Hagoort, 2005) and phonology (DeLong, Urbach, & Kutas, 2005; Ito, Pickering, & Corley, 2018).
However, the predictive approach to comprehension in the literature was often set against the integration view which suggested that a comprehender was just integrating incoming information into preceding context in a bottom-up manner to construct a higher- level meaning (De Long et al., 2005; Gussow, Kapnoula, & Molinaro, 2019). Experimental evidence was also subjected to ambiguous interpretations. For instance, in EEG studies, it has been found that the N400 effect is inversely correlated with cloze probability of a word, that is, the probability of a word being used in a specific context (Kutas & Hillyard, 1984) (see Section 1.2.1). The more predictable from the context a word was the smaller was the amplitude of the N400 effect, elicited at this word. From the integration point of view, such a reduction happens because the information retrieved from the semantic memory with the help of the context is rapidly integrated with perceptual input about the word. From the prediction
point of view, reduction of N400 reflects confirmation of the predictions that a comprehender actively forms about the words (Van Petten & Luka, 2012).
In the present study, I will follow Pickering and Gambi (2018) who define prediction as a pre-activation of “linguistic information before processing the input that carries this information” (p. 1003). They suggest that pre-activation is what distinguishes prediction from integration. Kochari and Flecken (2019) note that in order to demonstrate such pre-activation, the effect should be measured not at the content word as was the case with the above- mentioned N400 component, but at the forms preceding the content word, i.e., at the pre- nominal articles and adjectives. These pre-nominal forms can be used by the comprehender as informative cues to predict the upcoming input. In the present thesis, I will focus on exploring one type of cues, gender information, realized at the gender-marked article in Norwegian (see Sections 1.3.1 and 1.3.2).
There is no unified view on the nature of prediction and how it is organized. The earlier view on prediction considered it to be an all-or-nothing phenomenon while later accounts suggest that prediction is graded and probabilistic in nature (Kuperberg & Jaeger, 2016). In their review article, Kuperberg and Jaeger describe a probabilistic framework of cognition on which they base their hierarchical generative framework of language comprehension. This framework is based on the assumption that at any time point, a comprehender has specific hypotheses or beliefs about the speaker’s message. These beliefs are incrementally updated at each new incoming word of the unfolding sentence. The degree of belief updating at each cycle is known as Bayesian surprise – computational expression of prediction error. It is important to note that Kuperberg and Jaeger’s model suggests an hierarchy of linguistic representations, where a comprehender realizes her ultimate goal of comprehension in understanding a message by using higher-level information to predictively pre-activate lower-level information.
There are other models of how prediction functions and what role it plays in language processing. For instance, Pickering and Garrod (2013) proposed an integrated theory of language processing based on the mechanism of action and action perception. This theory suggests that production and comprehension are interwoven with each other that allows a comprehender to predict an incoming utterance. At the core of this theory are forward models that can be seen as computational devices for generating predictions. According to this model, prediction-by-simulation (also called prediction-by-production in Pickering & Gambi (2018)) plays a central role in predictive processing, when a comprehender covertly imitates, that is, uses her production system, the unfolding utterance of a speaker, and then derives a predicted utterance percept with the help of forward modeling. Prediction-by-simulation route is more likely to be used when a speaker and a comprehender either know each other or share some common background. If it is not the case, a comprehender can rely on prediction- by-association based on her personal experience in communication with other people (Pickering & Garrod, 2013).
Some researchers have pointed out possible weak places of integrated theory proposed by Pickering and Garrod (2013). For instance, Chang and colleagues (Chang, Kidd,
& Rowland, 2013) have criticized it for not taking into consideration a developmental perspective where prediction can be viewed as a part of learning process, connected to language acquisition. In addition, Pickering and Garrod’s model represents a situation of an ideal conversation without accounting for individual differences in native and non-native processing (Festman, 2013). I will elaborate more on predictive processing in L1 compared to L2 in a Section 1.3 below.
1.2 Experimental methods of studying prediction
1.2.1 EEG
A lot of evidence for predictive processing comes from the EEG literature. This evidence is not limited to above-mentioned N400 component, related to predictability of a word, but also touches upon how people make use of different types of linguistic information from the context, i.e., semantics, syntax or phonology, to predict the upcoming input.
Although the EEG method is not the main focus of the present study, I would like to cover one of the most important EEG studies that explored the role of phonology as a cue for prediction, and the debate that followed around reliability of this evidence. In the study by DeLong and colleagues (DeLong et al., 2005), the participants read sentences like The day was breezy so the boy went outside to fly a kite/an airplane. The authors used phonological regularity of English (the indefinite article is used in form of a before the words starting with a consonant and an before the words starting with a vowel) and manipulated cloze probabilities of the article and a noun in relation to preceding context (a kite being more probable continuation for the sentence). Cloze probability of articles and nouns was measured in an offline sentence completion task, also called cloze task, and corresponded to the percentage of people who used a particular article or noun to complete a sentence, truncated before the article or the noun. The authors found that with increase of cloze probability of a noun, amplitude of N400 decreased, thus, replicating the results obtained in previous studies (Kutas & Hillyard, 1984). In addition, the articles also elicited N400 negativity. Although the effect was in general smaller than at the nouns, its amplitude also varied dependent on the expectancy of the article. These results demonstrated that the readers not only integrated incoming words into mental representation of the sentence but also actively formed probabilistic predictions about specific upcoming words.
This evidence has been criticized by Nieuwland et al. (2017) reporting the results from nine laboratories. They successfully replicated the effect found at the noun but failed to find any effect at the article. Yan and colleagues (Yan, Kuperberg, & Jaeger, 2017) analyzed the results from Nieuwland et al. and came to the conclusion that the effect at the indefinite article was actually replicated if analyzed in terms of surprisal, that is, a log-transformed cloze probability. Yan and colleagues underlined the importance of this debate for future research because it emphasizes the following key questions about predictive processing:
when we predict, to what extent and at what levels of representation.
In order to address the previous debatable evidence on phonological prediction, in a recent study, Ito and colleagues (Ito, Gambi, Pickering, Fuellenbach, & Husband, 2020) aimed to compare the effects of phonological and gender-based prediction. The participants read a sentence with an expected continuation (e.g., un incidente ‘an accident’) or with another plausible but unexpected noun that either began with a different phonological class (consonant vs. vowel, e.g., uno scontro ‘collision’) or having a different gender (e.g., un’inondazione ‘flooding’). Ito and colleagues found different ERP effects at gender mismatch and phonologically mismatched articles. This dissociation is claimed by the authors to be the additional evidence for predictive processing whereas in case of integration they would have expected the same ERP-component both for phonological and gender mismatch conditions. Their findings also support the central role of the prediction-by-production mechanism because the participants activated gender information before phonology the way it happens during production.
1.2.2 The Visual World Paradigm (VWP)
The most straightforward way to demonstrate prediction is to use an experimental method that will show pre-activation of linguistic representation of a word before the
comprehender encounters this word (Pickering & Gambi, 2018). One of the methods that meets this criterion is eye-tracking. Eye-tracking provides a naturalistic way to measure underlying processing during language comprehension when the comprehender’s looks can be time-locked to the input (Tanenhaus & Truswell, 2006). In addition, eye movements reflect not only linguistic or visual processing separately but demonstrate the updating of mental representations based on the information received auditorily and visually (Huettig, Rommers, & Meyer, 2011).
For the purposes of this study, I will specifically focus on the studies utilizing eye- tracking in the Visual World Paradigm (Cooper, 1974) in spoken sentence comprehension. In a standard VWP comprehension study, a participant listens to an utterance and looks at an experimental display (a computer screen or tabletop) while her eye movements are being recorded. Visual display can consist of semi-realistic scenes, single images or even words if the researchers are interested in orthographic processing (Huettig et al., 2011). VWP experiments also differ in type of the instructions a participant receives. Action-based tasks require a participant to act out the instruction they heard while looking at the objects without any contextual scene (e.g., move a certain object). In contrast, during passive listening, participants usually look at a contextually rich scene and hear a sentence describing it (Dussias, Kroff, & Gerfen, 2013).
The rise of the VWP in psycholinguistic research on speech processing started with the seminal study by Tanenhaus, Spivey-Knowlton, Eberhard and Sedivy (1995). The authors demonstrated how the processing of syntactically ambiguous sentences depends on referential context represented by the real objects. The participants were presented with a set of objects and heard an ambiguous sentence Put the apple on the towel in the box where on the towel can refer to a destination or have a modification function. In the one-referent condition, the set of objects contained an apple on the towel, a towel, a box, and a pencil. The two-referent
condition had two apples (an apple on the towel and an apple on the napkin), a towel, and a box. The results showed that in the one-referent condition, the PP on the towel was interpreted as a destination, where the apple should be put, while in two-referent condition, the participants interpreted the PP on the towel as a modifier.
The researchers also started using VWP in exploring predictive sentence processing.
An example of a seminal study in this domain is by Altmann and Kamide (1999). The authors were interested in how lexico-semantic information provided by the verb can influence referential processing. The participants saw a semi-realistic visual scene depicting a boy with a toy train, a toy car, a balloon, and a cake around him. On the appearance of visual scene they heard either a sentence with a verb compatible with only one target object The boy will eat the cake or the sentence with a verb that permitted four objects from the scene including a target object (e.g., The boy will move the cake.). The participants had to judge if a sentence matched the picture and respond “yes” or “no” on the button box. The results showed that the probability of fixating a target object between verb onset and target noun onset in “eat”
condition was higher than in “move” condition. This study demonstrated that people can use semantic and syntactic information available at the verb to incrementally process the sentence and find the right referent in the visual context.
In the early VWP studies, the researchers were also interested in how spoken word recognition unfolds and what cues can influence the activation of competing lexical candidates in the mental lexicon. For instance, VWP has been used to study if gender cues can influence spoken word recognition. In the study by Dahan and colleagues (Dahan, Swingley, Tanenhaus, & Magnuson, 2000), French-speaking participants were presented with four pictures: a target noun (e.g., bouton ‘button-MASC’), a phonological competitor of different gender (e.g., bouteille ‘bottle-FEM’) and two unrelated nouns. When the spoken instruction was Cliquez sur les boutons, ‘click on the-AMB buttons’, with the ambiguous
definite plural article before the noun, the participants looked at the phonological competitor more than at the unrelated objects as the target word was unfolding over time. In contrast, when the spoken instruction contained gender information (Cliquez sur le bouton, ‘click on the-MASC button’), the participants fixated a target object faster and the definite article eliminated the effect of phonological competitor.
Weber and Paris (2004) broadened this line of research by using visual world paradigm to investigate how gender marking affects spoken word recognition in non-native language and what the origin of linguistic gender effect is. In their experiment, native French speakers highly proficient in German were presented with a set of four pictures in the corners of the screen and a fixation cross in the middle. The task was to listen to a sentence in German, e.g., Wo befindet sich Die Perle? ’Where is the-FEM pearl?’ and click on the target.
Each set contained a target noun, a competitor that had the overlapping phonemic onset with the target and two unrelated distractors. In all the trials, the target noun had the same gender in both German and French but in half of the trials, the target and the competitor had different genders in French. By manipulating gender of the nouns in the native language of the participants, the authors aimed to test if French listeners use the French gender information to recognize German words. If so, it would support the view that the gender effect is not a form- based but rather a grammar-based phenomenon. Indeed, the results provided evidence for this view as in same-gender trials, French listeners fixated competitors more than unrelated objects but in different-gender trials, early looks at the competitor were eliminated.
1.3 Prediction in L1 and L2
While there is a lot of evidence that L1 speakers are able to process their native language predictively, studies about non-native processing yield mixed results. Some studies point out the important role L1 grammatical system plays in L2 processing (Rossi, Gugler,
Friederici, & Hahne, 2006). Other researchers find the level of proficiency in L2 to be crucial for prediction (Hopp & Lemmerth, 2016). According to Pickering and Gambi (2018), due to poorer proficiency of L2 speakers in producing language, they are less likely to use prediction-by-production. Indeed, the correlation between production and predictive processing was shown experimentally even for high-proficient L2 speakers. For instance, L1 English advanced learners of German had a correlation between correct gender assignment in production and prediction of gender in comprehension (Hopp, 2013).
To provide an explanation to the limited ability of L2 speakers in predictive processing, Grüter, Rohde, and Schafer (2014) formulated RAGE (Reduced Ability to Generate Expectations) hypothesis. According to this hypothesis, non-native speakers can integrate information similar to native speakers but cannot really utilize proactive processing, i.e., make predictions. However, more recent studies suggest that L2 speakers can use various information sources as cues for prediction and differences between native and non- native speakers are more gradual than categorical (Clahsen & Felser, 2018). Kaan (2014) mentions multiple factors that can influence predictive processing in L2: frequency bias (i.e., how frequently a learner encounters a certain linguistic combination); competitor effect (i.e., during comprehension there can be a co-activation of information in L1 and L2 so the comprehender keeps competing predictors in parallel and is therefore less likely to make a specific prediction); quality of lexical represention (i.e., how good a comprehender’s knowledge of a word or form is); task-induced strategies, and other factors, such as cognitive abilities.
Dijkgraaf, Hartsuiker, and Duyck (2019) used the VWP to test if Dutch-English bilinguals demonstrate different semantic activation in their L1 and L2. The participants heard a sentence and looked at a four-picture display with an experimental picture (target word or semantically related competitor) and three unrelated pictures. Although in both L1
and L2 semantically stronger competitors attracted more fixations, the effect of semantic relatedness was stronger and started earlier in L1 than in L2. Thus, predictive processing happened in both languages but the semantic prediction in L2 was less efficient.
L1 and L2 speakers also differ in phonological predictive processing. Ito, Pickering, and Corley (2018) used similar paradigm as Dijkgraaf et al. (2019), but the participants saw an item containing a picture of a target object, an English phonological competitor, a Japanese phonological competitor or an unrelated object. Both L1 and L2 speakers looked at a target object predictively but Japanese L2 speakers of English were slower and, in addition, did not fixate the English phonological competitor before the target word onset like L1 speakers did. Thus, the authors claim that they found no evidence for phonological prediction in L2 speakers. The results from these two studies support the idea that non-native processing also involves prediction as native processing but can differ in strength and in types of linguistic information a speaker relies on.
1.3.1 Grammatical gender as a cue for prediction in L1
In the present study, I focus on one type of linguistic information, that can be used predictively, namely, grammatical gender. In many languages, nouns are classified in different lexical gender classes; gender can also be realized in morphosyntactic agreement between the head noun and its modifiers inside the noun phrase, e.g., determiners and adjectives (Hopp, 2016). This morphosyntactic agreement is also known as gender agreement (or concord) and is based on the generalization that determiners and adjectives must have the same gender as the noun they are associated with (Grüter, Lew-Williams, & Fernald, 2012).
The evidence for predictive use of gender often comes from ERP studies of languages that demonstrate gender agreement. In a study by Otten and Van Berkum (2009), Dutch participants with low and high working memory capacity (WMC) had to read a highly
constraining mini-story, where the first sentence set a context and the second sentence contained a target noun which could be predicted by this context. A target noun was preceded by a definite article of common de ‘the-COM’ or neuter het ‘the-NEUT’ gender. The authors also included several intervening words between the target noun and the article in order to make sure that any effect on the article would not reflect processing of a noun which usually immediately follows the article. The results showed that prediction-inconsistent determiners elicited early (300-600 ms) ERP negativity relative to consistent determiners. This effect is claimed to indicate reader’s pre-activation of anticipated noun and its gender, before the reader encountered a target word. Interestingly, participants with low WMC displayed additional late negativity about a second after a gender-mismatching determiner that could reflect either working memory load or difficulties with inhibiting disconfirmed predictions.
However, Kochari and Flecken (2019) failed to replicate Otten and Van Berkum’s (2009) results. Kochari and Flecken did not find a significant negativity modulation on the definite gender-marked article caused by predictability of the target noun. Thus, no reliable evidence for lexical pre-activation of the noun was found. The authors mentioned important methodological factors that could affect the results, that is, rate of word presentation, number of fillers, and possible small size of the effect on the article in general. In addition, they underline the significance of open questions for the studies using grammatical gender agreement for investigating lexical prediction: When the form of the article mismatches with the predicted noun, what kind of processing does the effect at the articlereflect? It is unclear if this effect demonstrates difficulty in integrating the article with preactivated noun or if the form of the article is pre-activated together with the lexical form of the noun which was not encountered yet.
In summary, the above mentioned studies rely on “gender-dependent paradigm”
(Otten & Van Berkum, 2009, p. 97) where the preceding word (i.e., an article or an adjective)
either matches or mismatches in gender a predicted target word. If, in mismatching or different-gender condition, there is an effect at the preceding word, this can indicate that a participant actively predicted the upcoming input. In addition to EEG-studies, in Section 1.3.2 I review VWP studies based on the similar paradigm with the pre-nominal article having the same or different gender with the visual stimulus representing a target noun. In this paradigm not a linguistic input (grammatical or ungrammatical sentence), but visual stimuli (matching or mismatching in gender with the article) are being manipulated. In the current study, I use the latter paradigm with visual stimuli manipulation to study predictive processing in L1 and L2 that I describe in more detail in Chapter 2.
1.3.2 Grammatical gender as a cue for prediction in L2
Besides studies on how native speakers use gender information to predict incoming input, there is a line of research addressing non-native gender processing and differences between L1 and L2 speakers. One of the hypotheses that aims to explain this difference is the Lexical Gender Learning Hypothesis (Grüter et al., 2012). This hypothesis states that the assymetry can be explained by the way children acquire their first language and adults learn a second language. Young children encounter non-segmented auditory input with determiner- noun sequences. Using probabilities of co-occurring between determiners and nouns, children can isolate nouns and also map determiners to abstract nodes representing gender. Thus, due to the strong association between determiners and nouns, gender-marked forms can be used as predictive cues for processing of associated nouns. In contrast, adult L2 speakers usually learn on the basis of smaller units and they do not compute co-occurence probabilities of determiners and nouns to extract gender information. This leads to gender class representations being weaker in adult learners than in native speakers and gender cues becoming less informative (Hopp, 2016).
In VWP studies, a gender-marked article serves as a referent to an object whose noun has a specific gender and if a participant looks at a gender-matching object when hearing the article, it indicates that gender is used predictively. Lew-Williams and Fernald (2007) showed that as early as at two to three years Spanish children can already make use of gender marking at the article in order to facilitate speech processing. In Spanish, there are two genders, masculine and feminine, and they are marked at the definite article before the noun (la-FEM_SING and el-MASC_SING). In the experiment, the children saw a pair of objects of the same gender (e.g., la pelota ’ball-FEM’ vs. la galleta ’cookie-FEM’) or of different gender (e.g., la pelota vs. el zapato ’shoe-MASC’) and heard a sentence referring to one of the pictures on the screen (e.g., Encuentra la pelota ’Find the ball’). The children fixated the referent object of the sentence in different-gender trials faster than in the same-gender trials.
Although the children showed similar pattern to the adult control group, they were in general slower in shifting their gaze to the target both in the same- and different-gender conditions.
Using the same materials and procedure as in the above-mentioned study, Lew- Williams and Fernald (2010) conducted eye-tracking experiments with L2 adult learners and focused on the comparison of L1 and L2 gender processing. The advanced L2 speakers of Spanish looked at the pairs of the pictures on the screen and heard a sentence with the name of one of the pictures. One experiment was based on familiar nouns. The second experiment consisted of pictures denoting novel nouns in order to control for frequency of exposure to article-noun pairs. The results have shown that in the experiment with familiar nouns, in contrast to L1 native speakers, L2 learners of Spanish failed to use the gender-marked article to identify the target picture quicker. In case of novel nouns, L2 speakers like L1 speakers could use a gender cue predictively but L2 speakers were in general slower than native speakers.
Some studies show that syntactic similarity between L1 and L2 can affect gender processing. Foucart and Frenck-Mestre (2011) manipulated gender agreement violations between determiners and nouns and also between adjectives and nouns. L2 German advanced speakers of L1 French read sentences with different gender agreement violations word-by- word on the screen. The results showed that L2 participants exhibited a similar P600 effect as native speakers of French at determiner-noun violations where German and French share similar syntactic rules. However, when agreement rules varied across languages, L2 speakers did not show such an effect. This result suggests that syntactic similarity between L1 French and L2 German has an effect on gender processing.
Syntactic similarity between the languages which in case of gender is sometimes referred to as gender congruency effect is also used in studies where gender is explored as a predictive cue. Lemmerth and Hopp (2019) compared the use of gender cues in simultaneous and successive Russian-German bilingual children. Importantly, the authors operated with two types of gender congruency in Russian and German. One is lexical gender congruency that stands for words having same or different gender in the two languages. Another one is syntactic gender congruency that stands for similarities or differences in how gender is syntactically realized in languages (e.g., in Russian, there are no articles but in German, gender is realized at the article). In the experiment, the children heard a question and saw four objects on the screen, three of which including the target, mentioned in the question, were of the same color. In same-gender condition, these three objects were of the same gender and in different-gender condition, the target differed in gender from the other objects. The children heard two types of questions, either with gender marked at the article (1-a) or with gender marked at the adjective (1-b). Syntactic congruency has been shown to have no effect on predictive processing in bilingual groups. Both groups, just like a German monolingual control group, demonstrated faster reaction times in different-gender trials compared to same-
gender trials. However, lexical congruency had an effect on predictive processing in successive bilinguals who processed gender predictively only at congruent nouns, while simultaneous bilinguals performed like a control group.
(1) a. Article condition Wo ist Der/Die/Das gelbe [N]
Where is the MASC/FEM/NEUT yellow [N]
‘Where is the yellow [N]?’
b. Adjective condition
Wo ist ein kleiner/s gelber/s [N]
Where is a small MASC/NEUT yellow MASC/NEUT [N]
‘Where is a small yellow [N]?’
Chapter 2
2 Data reanalysis and the current study
2.1 Grammatical gender as a predictive cue in VWP
VWP studies can be divided into two types by the design of stimuli presentation. In the first type of experiments, participants hear different linguistic stimuli but look at the unvarying visual display corresponding to these linguistic stimuli while for the second type of experiments it is crucial to manipulate visual stimuli as well as auditory input. In the research on gender processing one example of the first type of the paradigm is the above-mentioned study by Dahan et. al. (2000), where the participants looked at the screen with four pictures and heard a sentence either with a target noun, preceded by a gender-marked article, which could be used as a predictive cue, e.g., Cliquez sur le bouton, ‘click on the-MASC button’, or preceded by the ambiguous plural article, which could not be used predictively, e.g., Cliquez sur les boutons, ‘click on the-AMB buttons’. By comparing fixation patterns at the target in the two conditions, we directly see how the presence or absence of a distinctive feature in the sound stimulus affects the looks. It is the manipulation of the linguistic stimuli only that affects the possibility of predicting the target noun.
In the present study I focus on another way of testing if gender information is used predictively, based on presentation of visually constraining stimuli with the objects of different gender and visually non-constraining stimuli with the objects of the same gender.
Visual display in this paradigm can contain two (Figure 2.1) or more objects with additional distractors (Figure 2.2).
Figure 2.1. An example of same-gender condition (de schoen ‘the-COM shoe’ vs. de lamp
‘the-COM lamp’) and different-gender condition (de schoen vs. het huis ‘the-NEUT’ house) in Dutch from Brouwer, Sprenger, and Unsworth (2017).
When hearing the sentence with the target noun in visually constraining or different-gender condition, the participants can use a gender-marked determiner before the target noun as an informative cue to move their gaze toward the target. In the non-constraining or same-gender condition, a gender-marked determiner refers both to the target and distractor objects, thus, cannot be used predictively. In contrast to the first type of design, in this paradigm, it is not linguistic stimuli but semantic or syntactic features of the objects depicted on the screen that affect the possibility of predicting the target noun.
Figure 2.2. An example of Norwegian stimuli in same-gender (en bjørn ‘a-MASC bear’ vs.
en gris ‘a-MASC pig’) and different-gender condition (en bjørn vs. et esel ‘a-NEUT donkey’) from Lundquist, Rodina, Sekerina, and Westergaard (2016).
As a result of alternating between constraining and non-constraining visual contexts, the effect of gender condition can manifest itself in facilitation of looks to the target object which matches a gender-marked article; therefore, we see the increased proportion of looks to target in different-gender condition compared to same-gender condition (Figure 2.3).
Figure 2.3. An example of the increase in proportion of looks to target in Norwegian L1 speakers, who use gender information at the article predictively, in different-gender condition (blue line) compared to same-gender condition (red line) (Johannessen et al., in prep.)
2.2 The current study and research questions
The data from Johannessen et al. (in prep.) with methodology explained in more detail in Section 2.4 also revealed that L1 group was not homogeneous in the participants’ looking patterns. In VWP experiments participants can exhibit individual variation in how they explore the visual screen early in the trials and where they look at the onset of disambiguating cue (a gender-marked article in this case). In L1 control group, the effect of gender condition manifests itself in a different pattern in the trials where participants happened to look at the
distractor object when hearing a gender-marked article (Figure 2.4) compared to the trials where participants looked at the white space at the article onset (Figure 2.5) with larger effect of gender condition in the latter type of trials. In later analysis, I use the definition early looks, referring to looking patterns of participants when hearing a gender-marked article in the beginning of the trials, described in more detail in Section 2.5.
Figure 2.4. Proportion of looks to target in the trials with early looks at the distractor in L1 control group.
In the present study I analyzed previously collected data by Johannessen et al. (in prep.) (see Section 2.4 and Section 2.5) with an aim to explore the effect of gender condition in high-proficient L2 speakers of Norwegian compared to L1 control group and formulated the following research questions:
(1) Do high-proficient L2 speakers show similar pattern in the effect of gender condition dependent on their early looks compared to L1 speakers?
(2) If there is a temporal difference, is gender processing of high-proficient L2 speakers just qualitatively different from L1 speakers, being slower in general, or do high- proficient L2 speakers rely on different type of processing depending on where they look early in the trials?
Figure 2.5. Proportion of looks to target in the trials with early looks at the white space (here, early filler) in L1 control group.
I hypothesize that there can be two patterns in processing of gender cues. One is based on detection of gender-mismatch when a participant hears a gender-marked cue and simultaneously looks at the object with a different gender, while another pattern reflects pre- activation and predictive processing later in the trials. Assuming that detection of gender- mismatch happens faster than prediction, I expect to find earlier increase of looks to target in different-gender condition in the trials, where participants’ early looks are at the distractor, which does not match a gender-marked article, than in the trials where participants look at the white space or the target at the moment of hearing the disambiguating article. In the latter case speakers cannot rely on gender mismatch detection but on predictive processing of gender cues.
While EEG studies provide evidence for temporally different ERP components which can be attributed to different types of processing, such as earlier prediction and later integration effects in N400 window (Nieuwland et al., 2020), VWP studies with manipulation of visual context do not provide direct evidence for a claim why one visual stimulus can be harder to integrate than the other, thus, making a distinction between prediction and integration or other types of processing problematic. In addition, if variation in the data from Johannessen et al. (in prep.) can potentially be attributed to individual strategy or chance, it does not give us sufficient understanding on how participants use morphological cues in comprehension. In Section 2.3 I provide a more detailed description of gender in Norwegian, the main language of the experiment. In Section 2.4 I present the method and the data from Johannessen et al., and in Section 2.5 – secondary reanalysis of these data comparing L1 and high-proficient L2 speakers of Norwegian. In Chapter 3 I address the problem of individual variation in VWP by suggesting a modified visual stimuli design, where I aim to have a better control for early looking patterns and target gender processing more efficiently.
2.3 Gender in Norwegian
Trosterud (2001) defines gender as different classes of nouns, manifested in how associated words behave. He has a dynamic view on Norwegian gender system which suggests that gender systems can undergo changes depending on how each generation speaks and uses grammar. For example, Norwegian gender system traditionally included three genders: masculine, feminine and neuter. However, currently, this system is gradually changing; in some dialects, for instance, in Tromsø and Oslo, feminine gender is disappearing and the indefinite article ei-FEM is being replaced by the indefinite article en/ein-MASC (Rodina & Westergaard, 2015). The change in Norwegian gender system is not a uniform process. It has been found that speakers of the dialects, where FEM gender is disappearing, do not use the article ei-FEM predictively regardless of whether they continue to produce it or not, therefore, their comprehension might be affected earlier than production (Lundquist et al., 2016). Due to gradual change of Norwegian gender system and taking into consideration that the data for Johannessen et al. study (in prep.) was collected in Oslo and Tromsø where such a transition takes place, in the current study, I investigate predictive processing using two-gender distinction: MASC (or common) and NEUT.
Gender assignment in Norwegian that is an inherent property of the noun is non- transparent and the gender can be identified only when a noun appears with an associated word (e.g., ei bok ‘a book-FEM’) (Lohndal & Westergaard, 2016). Thus, manifestation of gender in Norwegian comes in form of agreement inside the noun phrase when nouns agree in gender with indefinite articles, attributive adjectives, possessive pronouns, demonstratives and prenominal determiners (Johannessen et al., in prep.). It can be expressed pre-nominally as a free morpheme and post-nominally as a bound morpheme (Rodina & Westergaard, 2015). The example of pre-nominal expression is the indefinite articles en-MASC, ei-FEM and et-NEUT (Table 2.1-a). Definite articles are realized post-nominally in form of suffixes –
en-MASC, -a-FEM and -et-NEUT (Table 2.1-b). In addition, gender can be expressed both pre-nominally and post-nominally at the same time in double definite forms (Table 2.1-c).
When expressed on adjectives, gender in Norwegian is present only in indefinite singular form and has a two-way distinction: one for MASC/FEM or common and one for NEUT (e.g., grøn-MASC/FEM, grønt-NEUT ‘green’) (Table 2.1-d) with one exception – adjective liten-MASC/lita-FEM/lite-NEUT ‘little/small’ (Table 2.1-e) (Rodina & Westergaard).
Table 2.1
Examples of Norwegian nouns in indefinite, definite, double definite form and nouns with adjectives
MASC FEM NEUT
(a) en bil ei seng et egg
a car a bed an egg
(b) bilen senga egget
car.DEF bed.DEF egg.DEF
(c) den bilen den senga det egget that car.DEF that bed.DEF that egg.DEF (d) en grøn bil ei grøn senga et grønt egg
a green car a green bed a green egg (e) en liten bil ei lita senga et lite egg
a small car a small bed a small egg
2.4 The dataset from Johannessen et al. (in prep.)
For the purpose of the present study I used previously collected data of the speakers who exhibited predictive gender processing, namely, a control group of L1 Norwegian speakers (n = 19) and a group of high-proficient L2 speakers of Norwegian with Greek, Russian and Turkish L1s (n = 33). In a gender comprehension task, high-proficient L2 speakers demonstrated smaller effect of gender condition than L1 control group (Figure 2.6).
Figure 2.6. Proportion of looks to target in L1 control group and high-proficient L2 group with the effect of gender condition shown as a separation of a blue line (different-gender condition) from a red line (same gender condition) (Johannessen et al., in prep.)
The data was collected in a VWP experiment which included a presentation of a visual stimulus on the screen and playback of an auditory stimulus. As an example of a visual stimulus, the participants saw two coloured pictures with one image depicting a target noun mentioned in the auditory stimulus and another image depicting a distractor noun (Figure 2.7). All the images represented nouns of either MASC or NEUT gender. All the items were divided into two main conditions: same-gender condition when the target and the distractor shared the same gender and different-gender condition when the objects had different gender.
In addition, lexical congruency was manipulated so that the L1 speakers saw half of the items with a target of the same gender in Norwegian as in their L1 and half of the items with a target that had different gender in Norwegian and their L1. The participants saw 128 experimental items and 128 filler items (256 in total) during the experiment.
Figure 2.7. An example of the visual stimulus in different-gender condition from Johannessen et al. (in prep.): et egg ‘an-NEUT egg-NEUT’ and en løk ‘an-MASC onion-MASC’.
Auditory stimuli consisted of a carrier phrase Jeg tenker på (’I am thinking of’) followed by an indefinite gender-marked article, an adjective without gender marking and a target noun (2).
The sentences were recorded in Oslo dialect and acoustically adapted in Praat to make all the phrases identical without any unwanted prosodic cues. A gender-marked article en-MASC or et-NEUT agreed only with a target noun in different-gender trials and with both target and distractor noun in same-gender trials. In this set-up a participant can predictively use gender information from the article only in different-gender trials that allows the effect of gender condition to be potentially present in the graphs. In the filler trials a carrier phrase did not (2) Jeg tenker på en avbilda bil
I am thinking of a-MASC depicted car-MASC
‘I am thinking of a depicted car.’
include a target noun and the indefinite article. The filler carrier phrase had a reference to two objects (3).
It is important to mention that the words in the phrase meaningful for the eye-tracking analysis had a specific onset as shown in Table 2.2. The participial adjective avbilda
’depicted’ had consistent duration of 788 ms. This provides a substantial time for manifestation of prediction triggered by the gender-marked article as it takes around 200 ms to launch a saccade (Altmann, 2011) and also prevents processing of a noun and an article as one unit (Grüter et al., 2012).
Table 2.2
A time-line of a speech stimulus
Time of onset 0 ms 920 ms 1192 ms 1980 ms
Jeg tenker på en avbilda bil I am thinking of a depicted car
‘I am thinking of a depicted car.’
In the beginning, two time slots (50 ms each) were identified for the analysis based on the eye movement patterns in the L1 Norwegian control group. First, late time slot was identified as the last slot before the lexical access triggered by the onset of the target noun.
Lexical access was visually detected as the increase in proportion of looks to the target as (3) Jeg tenker på to avbilda ting
I am thinking of two depicted things-PL
‘I am thinking of two depicted things.’
compared to the looks at the distractor in same-gender condition (Figure 2.8). The early time slot was set at 500 ms prior to the late time slot. I use these previously identified early and late time slots in the reanalysis of the data to fit mixed linear regression models for each of the slots separately. Early time slot should not be confused with the factor of early looks which I introduce later in this chapter.
Figure 2.8. The eye-tracking results of L1 Norwegian control group. Separation of red and grey lines marks lexical access when the proportion of looks to the target in same-gender condition (red line) increases as compared to the looks to the distractor (here Comp., grey line) (Johannessen et al., in prep.).
2.5 Secondary data reanalysis
One of the goals of the present thesis was to explore if the effect of gender condition is qualitatively different depending on the participants’ early looks, i.e., where they looked at the moment when they heard a gender-marked article; and to investigate if these early looks have an effect on temporal profile of predictive processing. In order to explore possible interaction of early looks with the effect of same- and different-gender condition I extracted the data with the proportion of looks in the time window from the article onset to 250 ms after the article onset and added these new data to the main dataset. As a result, the final data set included an additional factor of early looks with three levels (‘targ’, ‘dist’ and ‘fill’), representing looks at the target, the distractor, and the white space respectively in the time window from the onset of a gender-marked article until 250 ms after.
I divided the original dataset into three subsets according to the early looks of the participants: one subset with early looks at the target, the second one with early looks at the distractor, and the third one with early looks at the white space. I used the lme4 package in R (Bates, Mächler, Bolker, & Walker, 2015) to fit mixed logistic regression models for each of these three subsets separately. Proportions of looks at the target in the early time slot or in the late time slot (annotated Bin in the models) served as a dependent variable and were coded 0 (not looking at target) or 1 (looking at target). Fixed effects included group of speakers (L1 Norwegian control group or high-proficient L2 Norwegian group annotated Group), gender condition (same- or different-gender condition annotated cond), and interaction between group of speakers and gender condition (Group x Cond). As random effects, I included intercepts for participants as minimal configuration to ensure model convergence. Maximal configuration also included by-participant slope for condition and intercept for the items.
In Section 2.6 below, I report the coefficients of mixed logistic regression models (Table 2.3) for above-mentioned subsets of the data separately referring to them as early filler
