Autobiographical Narrative Modulates Visual
Perception by Fostering Expectations
15-8-2014 Werner de Valk 5927145 Supervisor: Machiel Keestra Co-assessor: Julian Kiverstein MSc in Brain and Cognitive Sciences Cognitive Science track University of Amsterdam
2
Index
Introduction: Narrative and Perception ... 3
Chapter 1: Narrative as a Cognitive Tool ... 7
Elements of Autobiographical Narrative ... 7
Self-referential processing. ... 7
Autobiographical memory. ... 8
Basic Elements of Narratives ... 9
Narrative imagery. ... 9
Theory of mind. ... 9
Event selection. ... 9
Expectancy Fostering Elements ... 10
Sequence. ... 10
Causality. ... 12
Conclusion ... 14
Chapter Two: Perception Modulation ... 15
Top-down Modulation of Visual Perception ... 15
Empirical Demonstrations of Visual Modulations ... 16
Sequence. ... 16
Causality. ... 17
Other Perception Modulating Factors... 17
Memory. ... 18 Intention. ... 18 Attention. ... 18 Conclusion ... 19 General Conclusion ... 20 References ... 23 Appendix ... 26
3
Introduction: Narrative and Perception
When two persons are looking at an object, do they see the same thing? And do their live stories have to do anything with this? Imagine Peter and Ronald, sitting next to each other and looking at an object that is hard to recognize: their dog is eating something, but what is it? Peter might have forgotten his glasses, while Ronald could be colour-blind, altering their rudimentary visual perception.1 But differences in perception might not be limited to the quality and content of one’s vision; they also rely on higher order cognitive processes. Here we will focus on those processes that are fundamental elements of narrative, and hypothesize that autobiographical narrative modulates perception.
An important idea we base this hypothesis on is that the brain is not a passive reacting machine, but constantly makes predictions. We will argue that narrative is a mechanism supporting these predictions, after discussing two theories based on implications of new insights in the nature of neural information processing. Traditionally, this information processing is viewed as hierarchically organized (Gilbert & Sigman, 2007). This is most evident in the visual system, where visual information is first analyzed on a simple, feature-based level. Following the pathway up the hierarchy, information becomes more and more complex. This way, the brain can be thought of as a computational machine, passively waiting for new sensory input. However, recently the importance of top-down interactions has become more clear.
The brain benefits from updating information and creating predictions about what it will encounter next. The theoretical frameworks that stress this are Bar’s proactive brain (Bar et al., 2006; Bar, 2003, 2007, 2009) and Friston’s predictive coding (Friston & Kiebel, 2009; Friston, 2005, 2010; Kilner, Friston, & Frith, 2007). According to Bar, the brain continually generates predictions about the future, based on memory.2 Our perception might rely on a proactive link between input and these predictions, and Bar suggests that the latter is equally important as the first.
According to this framework, rudimentary information coming from early sensory areas is sent quickly to higher levels, where representations and associations between them are stored. Analogies are derived quickly from this gist information, which are then linked with the representations in memory. This leads to activation of associated representations. These are sent back as predictions about the perceived stimuli, towards early sensory areas, where they facilitate perception. The co-activation of related items provides a focused prediction about what is likely to be experienced.
These predictions (and the expectations in which they result) play a large role in cognition in general, by reducing cognitive load and preparing the brain for upcoming action. Information processing is minimized, which helps guide people’s thoughts, plans, actions and perception. Examples of effects of minimized information processing are habituation, inhibition of return, change and repetition blindness (Bar, 2007). In concordance, unexpected information processing receives increased amounts of processing resources. Examples of this include perceptual pop-out and automatic attending toward novel and surprising stimuli. The ‘decision’ to reduce or increase the amount of processing resources might rely on an evaluation of the match between perceptual input and these top-down predictions: resources are reduced after a high match, when the match is low an unexpected stimuli must have been encountered, which requires additional processing.
1 Of course, all circumstances being equal, individual differences in subjective experience might still exist.
Qualia – the experience of a headache, the perceived shades of red in the evening sky – are hard (or even impossible) to compare. Here, we will limit ourselves to differences in early visual processing that are not a result of differences in incoming retinal information.
2 The past, future and present are linked here; memory plays a role in both what happened and what will
happen. This is supported by the idea of the prospective brain, which broadly states that a fundamental feature of the brain is to use stored information to predict and simulate events in the future (Schacter et al., 2007). We will go into this in a bit more detail in the first chapter.
4 An important implication of this theoretical framework is that the predictions are formed continuously. Since they are based on associations, association forming should therefore be a continuous process in the brain. This is supported by recent research suggesting a strong connection between the neural mechanisms supporting associative processing and the brain’s ‘default network’ (Bar, Aminoff, Mason, & Fenske, 2007). Activity in brain regions supporting this default mode is found when subjects are not engaged in any specific task (Raichle et al., 2001); it represents processes that happen continuously. Association forming might be so important that it activates this default network during rest, which leads to a continuous updating of subsequent predictions (Bar et al., 2007).
The second framework stressing that prediction forming is a basic neural process is Friston’s model of the Bayesian brain and predictive coding (Friston & Kiebel, 2009; Friston, 2005, 2010; Kilner Friston, & Frith, 2007). According to this theory, the brain is “an inference engine that is trying to optimize probabilistic representations of what caused its sensory input” (Friston, 2010, p. 130). This enhances our capability of making predictions and forming expectations. Friston argues that this inference might broadly be based on minimizing the brain’s free energy (2005). He assumes that the brain models the world as a hierarchy, running from cause of perceptual input to percept (Friston & Kiebel, 2009). Perceptual processing is the reverse of this model; it tries to make sense of sensory data. To mimic this, Friston used a model of sensory data processing and took the reverse of it, relying on the free-energy principle. The mechanisms behind this reversed hierarchical interference are equivalent to empirical Bayes3, with its basic mechanism being to suppress prediction error/free energy in cortical hierarchies.
Prediction error is minimized through recurrent or reciprocal interactions within hierarchies in the cortex (Kilner et al., 2007). Higher levels predict representations in the level lying below. These predictions are projected to this lower level through a backward connection. In the lower level, prediction error units compare this prediction with the current representation. The prediction error that they calculate is sent upwards again to the units in the higher level. This leads to an adjustment of the predictions about the lower level representations, reducing overall prediction error. This up-and-down adjustment of predictions continues until a cause of input is generated that has the highest probability, which happens when prediction error is minimized throughout all levels, and conditional expectations are optimized.4 In summary, the brain uses probabilistic information about future events all the time and these Bayesian processes are general mechanisms in the brain (Clark, 2013).
Predictions rely on different sources of information; they might not always be based on associations between representations like in the proactive brain framework, or on the statistical processes described in the predictive coding theory. Another important additional factor is the main subject in this thesis: narrative structuring. When events are unlikely to happen, mere statistical predictive processing would not make our brain to expect them. However, their connection could be very important to a person because they play a relevant role in his or her narrative. In that case, this person might have great expectations of perceiving this event.
3 Using Bayesian inference, the current believe about an hypothesis (H) is updated based on new evidence (E).
This follows from Bayes’ rule:
. In words: the posterior probability of a hypothesis given
the available evidence, or P(H|E), is derived from the probability to find evidence in case the hypothesis is true, or P(E|H), times the probability that the hypothesis is true without prior evidence, or P(H), divided by the general probability that the evidence is found. So, the likelihood of a hypothesis follows from a combination of the inherent probability of this hypothesis to be true, and its compatibility with the evidence that was found.
4 These predictions are not of the same nature as discussed throughout the rest of this thesis. Rather, sensory
effects are predicted from their cause. These predictions are no forecasts of what sensory states are likely to be perceived in the future (Kilner et al., 2007). However, this framework does indicate the importance of top-down modulation, which – as we will see later – plays an important role in perception modulation.
5 But what are narratives exactly? Looking on the back of pack of corn flakes, what distinguishes the short story for the young reader, with the section about the ingredients? And do people really structure their lives like stories?
Narratives can be thought of as a cognitive tool; they are a way of organizing information (Keestra, 2014). One of the main features of narrative might be to use linguistic resources in order to foster expectations and anticipations. When information is structured in such a narrative form, this reduces the amount of information that has to be processed, and can make ongoing information processing more efficient. This helps focus one’s attention, avoids costly counterproductive actions, and supports the integration of intensions.
A number of features are needed before something can be called a story, according to the review presented by Mar (2004). The first fundamental feature of narratives is that they contain a causal event-structure: each story is a description of a series of actions and events occurring over time, with a causal link between them5. Because of this causal link, the events need to occur in a logical order where effect follows cause. The second feature is coherence; every element in a narrative has to be significant for the full story. This significance is determined by the intentions and goals of the characters in the story.6 To categorize the text on the back of the pack of corn flakes as a story, it needs these two features, together with the basic elements of narratives: a main agent, his goal, elements impeding or facilitating the progress towards that goal, and the setting in which this takes place.
Information relying on such a structure is not limited to books, movies, and the like. Indeed, it is often claimed that narrative construction is akin to the formation of our own biographical self-narrative, where memories are selected and ordered to build a coherent representation of the self (Mar, 2004; Habermas & Bluck, 2000). People often express a narrative representation of their lives, playing the main character in the story that is their autobiography.7
But why would do they do this? A possible reason might be that it functions as a means to create a unified sense of self (Schechtman, 2007).8 According to Schechtman’s (2007) narrative self-constitution view, “we constitute ourselves as persons by forming a narrative self-conception according to which we experience and organize our lives” (Schechtman, 2007, p. 162). This is an interesting idea in the light of our hypothesis, since it suggests that narrative self-conception plays an important role in one’s experiences. These experiences might ‘follow’ perceptual processes – which results from the idea that lower order visual processes result in an experience of visual content higher up the hierarchy. But perhaps, in accordance with a predicting brain, the opposite effect could also exist. Might this effect be so strong that predictions about future content of one’s experience actually influence one’s perception?
In this thesis, we will try to defend that our hypothesis, that biographical narrative modulates perception by fostering expectations, is at least plausible. Unfortunately, not a lot of direct research has been performed regarding this issue. We will therefore only be able to discuss indirect evidence that defends the plausibility of our hypothesis. Will discuss studies indicating that perception can be modulated by expectations, as well as those suggesting that expectations could be fostered by live stories. Discussing what is available will allow us to suggest future research that might test our hypothesis more directly.
5
We will discuss causal connections in more detail in the first chapter, when we discuss expectancy fostering elements of narratives.
6 In well crafted narratives, the significance of certain elements is often not stated explicitly, and it is the task of
the audience to infer why they are relevant to the goals of the main character. For this, the reader assumes that when an event or character is mentioned, this is in a certain way relevant to the main character’s goals.
7Strawson (2004) disputed the idea that people generally do this, taking himself as an example of an Episodic –
someone who does not figure himself within a broad time course. Because this discussion between him and
Schechtman (2007) goes beyond the scope of this essay, we will have to limit ourselves to agreeing with Schechtman, and assuming that the majority of autobiographies are like stories.
8
6 As we have seen, brains can be thought of as continuous prediction creating machines, and narratives might function as additional predictors. In the first chapter we will further discuss this potential role of autobiographical narrative. We will give an overview of the different elements of narrative, and the cognitive processes supporting these, by looking at autobiographical narratives as well as short-term narratives, assuming that the elements that the latter contains are also fundamental features of autobiographical narratives. After this, we will focus more deeply on processes that might foster expectation forming directly: sequence and causality.
In the second chapter, we will discuss the idea that visual perception is not just a passively waiting process, processing incoming information in a bottom-up fashion, but it is flexible and mediated in a top-down fashion, by higher order processes, one of them being predictions. We will first discuss how the theoretical framework of a predicting brain supports this. Then, we will discuss how expectations can function as modulating factors that continuously influence other lower order processes like perception, by focussing on the elements sequence and causality again.
The conclusions that we will draw from these two chapters indicate that it is plausible that Peter and Ronald’s early visual processing differs, even when their incoming retinal information is equal, because they are looking at the same black-and-white object (as not to annoy colour blind Ronald) and because Peter found his glasses. Indeed, there might be something different in their live stories which results in different expectations, modulating their perception.
7
Chapter 1: Narrative as a Cognitive Tool
We base our hypothesis on the idea that autobiographical narrative fosters the creation of expectations. In the current chapter we will see that narratives are based on a selection of relevant events that are placed in meaningful temporal and causal order, indeed fostering expectations.
As said, empirical research that supports the plausibility of our hypothesis is limited, which especially is the case with regard to autobiographical narrative. Therefore, we will focus on some general elements of narrative and try to draw more specific conclusions from this about autobiographical narrative. For this, we will discuss elements that are contained by some types of narrative while they do not play a role in others. Based on the available research, we can distinguish autobiographical narrative (e.g. Ainley, Maister, Brokfeld, Farmer, & Tsakiris, 2013; Young & Saver, 2001) from short-term narrative (e.g. operationalized using scripts; Crozier et al., 1999; Knutson, Wood, & Grafman, 2004). Elements included by the latter are more basic, contained by all types of narratives, while additional features are required to create the former. We will start discussing elements that are only contained by autobiographical narratives: self-referential processing and autobiographical memory. Then we will focus on more basic ones: narrative imagery, theory of mind, event selection, to end with those that foster expectations most directly: sequence and causality.
This choice of these elements is partly supported by a meta-study performed by Mar (2004), who reviewed studies that measured brain activity underlying story comprehension and production. The most used paradigm to investigate story comprehension was to compare normal story processing with the processing of reversed or incomprehensible stories. No widely tested research paradigms exist for story production yet, but some attempts do exist (e.g. Braun, Guillemin, Hosey, & Varga, 2001). Based on overlapping areas between production and comprehension, areas representing mechanisms that support narrative in a general sense could be located. These areas were found to play a role in simulation, ToM, attribution of mental states, information selection, and (temporal) ordering of events.
In addition, we will provide an overview of brain mechanisms that underlie the different elements of narrative (see Appendix for a summarizing table). This is important, because from this we can make predictions about future research. If top-down modulation of visual perception would be found to originate in one of the areas supporting elements of narrative, this might add to the plausibility that narrative plays a role in perception modulation.
Elements of Autobiographical Narrative
Long-term autobiographical narratives contain a description of a number of events/actions that follow each other over a long period of time. Because of this long time span, neurocognitive research is difficult. Autobiographical narratives differ from other narratives in their source: instead external (mostly fictional) events, they are based on events that one has experienced by oneself. I.e. these narratives are self-referential and rely on autobiographical memory.
Self-referential processing. Different concepts of self have been proposed throughout the years, which could roughly be divided into three types, the proto-self that covers sensory and motor domains; the core or mental self; and the narrative self that binds future, past and present (Northoff et al., 2006). Northoff et al. (2006) argue that all kinds of self have in common that they compromise the processing of stimuli that are self-referential; stimuli that people experience as having a strong relation to their own person. For example, whereas a picture of the house where you grew up is self-referential, a picture of a random other house is not.
Of course, self-referential processing can have different forms, and two important ones are related to the proto-self and the narrative self: processing of representations of one’s own body, and of the content of one’s autobiographical narrative. Ainley et al. (2013) compared the effect of these
8 two on the awareness of one’s interoceptive senses (processing afferent information that originates from within the body). Subjects either looked at a photograph of their own face (representing their bodily self), or read self-relevant words (representing their narrative self). Words representing the narrative self included their first name, hometown, school, university course, the name of their best friend and the name of the most important person in their life. As was predicted, interoceptive awareness (as measured by the accuracy of self-counted number of own heartbeats) increased in both conditions. This indicates that focussing on one’s autobiographical narrative is a strong form of self-reference that even influences the perception of one’s own body.
This research also demonstrates the limitations of studying autobiographical narrative. This was
operationalized here by using self-relevant words, but it is evident that one’s biographical self constitutes more than a simple list of words. The content of autobiographical narratives is difficult to manipulate, and individual differences are too complicated to draw clear inferences from. Therefore, it would help to find a feature that many long-term narratives share, while different in another group of narratives. By comparing these groups with each other, mechanisms behind the feature they differ in can be investigated. A way to do this is to look at cultural differences in narrative.
An important cultural distinction concerning narrative exists between individualistic (Euro-American) and collectivist (non-Western) cultures, in the way they perceive the self in relation to important others. In Western cultures, social experience is more concentrated in the form of an individualized life story than in collectivistic cultures (Herman, 2001). In other words, a difference between the autobiographical narratives of the members of these groups is in how much their self is associated with others. Self-referential processing corresponds for both Western and Chinese subjects with activity in medial prefrontal cortex (mPFC) and anterior cingulate cortex (ACC), as compared to processing of information about others (Zhu, Zhang, Fan, & Han, 2007). However, processing information about the mother corresponded to activity in the mPFC in Chinese subjects as well, whereas it did not for Western ones. This indicates that the self-referencing aspect of autobiographical narration might differ between cultures, and that other people play a more fundamental role in one’s life story in collectivistic cultures than in Western ones.
Autobiographical memory. Autobiographical narrative also differs from other narratives in that it relies on the memory of autobiographical events. The core brain network supporting the processing of this kind of information is thought to consist of left medial PFC, left medial and lateral temporal and posterior cingulate cortex, temporoparietal junction, and the cerebellum (Svoboda, McKinnon, & Levine, 2006).
Interestingly, recent research suggest that memory mechanisms are also used to make predictions about the future; they play a crucial role in the ability to make future simulations (Schacter, Addis, & Buckner, 2007). Brain regions that become active when remembering past events stored in autobiographical memory also get activated when people imagine non-existing future events. Since memory configuration and narrative simulation overlap, autobiographical memory plays a role in past narrative as well as simulation of future narrative.
But what happens when these memories are lost? Bilateral brain damage in the amygdalohippocampal area’s results in isolated memory impairment; new ideas can only be held in memory for about 30-90 seconds. Autobiographical memory remains intact either until the injury or some years before. This injury results in two kinds of narrative impairments: ‘arrested narration’ and ‘unbounded narration’. In the first, individuals are able to create an autobiographical narrative, but only until their injury. After that, their self-narrative simply stops.
Individuals that express unbounded narration generate self-narratives that do not rely on memories – without being aware of this (Young & Saver, 2001). These confabulated narratives give an indication of the strength of the tendency to narrate one’s live – even if no memory to build on is present. So, memory mechanisms might in fact be more important for future narrative simulation than a necessary mechanism to create narratives about the past – although they are of course necessary when one favours an autobiographical narrative that is somewhat in line with reality.
9
Basic Elements of Narratives
We will now turn to the more basic features; elements that are part of every kind of narrative. After discussing three basic elements – narrative imagery, theory of mind, and the selection of events – we can focus on the elements that are the most important in answering the main question of this thesis; those that play a role in mediating expectations.
These elements are all supported by research using short-term narratives. The advantage of investigating narrative on a small timescale is that this allows manipulating variables directly and measuring the direct (neuro)cognitive effects. Typical research paradigms of short-term narrative focus on the rendering of a small amount of perceived events. We will have to assume that the elements that were found using these short-term narratives are also essential for autobiographical narratives.
Narrative imagery. When reading or listening to a story, people often visualize the described events. Thinking about the future of one’s biography requires simulation as well. A way to study this element of narrative is by presenting subjects with brief narrative scripts, and asking them to image themselves engaged in the events that were described (Sabatinelli, Lang, Bradley, & Flaisch, 2006). Imaging led to an increase of activity in supplementary motor area, left inferior frontal gyrus, and right lateral cerebellum. These areas are known to play a role in executing actions or planning and them. Therefore, narrative imagery might be cognitive tool that helps prepare for future actions.
Theory of mind. As we saw in the Mar’s (2004) definition of narrative, basic elements include the goal of the main agent. Often, these goals are not stated explicitly; so readers have to infer what goes on in the mind of the characters. Theory of mind (ToM), or ‘mentalizing’ is the ability to infer the mental states of others (Mar, 2011). ToM could be employed in understanding stories, because understanding fictional others might require similar processes as those involved in understanding mental states of non-fictional others (Mar, 2011).
In a meta-study, Mar (2011) looked at studies in which subjects read stories that could only be comprehended by applying mental interference, and stories that did not need those mentalizing abilities. He compared these with studies applying other techniques that did not use stories, like presenting cartoons to test false-beliefs, presenting shapes moving either randomly or implying social behaviour, or contrasting mental state judgements with judgements of gender. Mar looked at overlapping brain activity between the different ToM measuring paradigms. Based on this (and on other studies), the brain network supporting ToM includes mPFC, bilateral temporoparietal junctions, posterior cingulate cortex (pCC) and precuneus.
Mentalizing might not be limited to thinking about fictional others, but could also be employed when thinking about oneself in the future (i.e. future events in one’s autobiographical narrative). A central part of the mentalizing network is the mPFC. This area can be identified as a brain area supporting fundamental aspects of social-cognitive functioning (Mitchell, Banaji, & Macrae, 2005). Importantly, ventromedial PFC is involved both when mentalizing trials of others, as well as during self-referencing tasks. 9 Furthermore, children with autism spectrum disorders – which is known to result in deficits in ToM understanding – show impaired thinking about future self (Jackson & Atance, 2008). Evidently, stories are virtually always about people. To understand and create them, one has to understand the characters – even when that character is yourself in the past or future.
Event selection. Mental inference does not have to be applied everywhere – only when an action is important for the rest of the story. For this, it is important to select relevant events. We stated earlier that an important feature in narrative is coherence; every feature in it is significant for the full
9
However, the fact that brain areas are shared between inferring states about others as well as thinking about oneself can also be used as evidence for a reverse direction. According to the ‘simulation theory’, inference about others’ mental states is derived from one’s own individual experiences (Mitchell et al., 2005).
10 story. One could therefore argue that selection does not have to play a role in narratives, since every event has to be selected. But it does in the production of stories, in understanding the most important events when comprehending stories, and especially when creating one’s autobiographical narrative – building a narrative out of episodic memories requires selecting those events that are important for the overall story, and discarding those that are not.
According to Mar (2004), two fundamental aspects of narrative comprehension and production are causal temporal ordering (which we will discuss in the next section) and selection. Partiot, Grafman, Sadato, Flitman and Wild (1996) showed that selection of elements within a narrative is a special process, that is supported by mechanisms differing from those supporting ordinary chronological ordering. Subjects were performing identification tasks while PET-scans were performed. They had to select an anomalous event within a script (for example ‘put on a swimsuit’ within a script about attending a wedding), or detect errors in the order of scripted events, with font selection within scripts as a control.
Brain imaging results as a result of event ordering will be discussed below. Important to note here is that these activations differed from those supporting anomalous event selection. Event selection led to increased activation in the left medial frontal gyrus, left anterior cingulate gyrus and anterior left superior temporal gyrus.
Selecting relevant events might play an indirect role in making predictions; events become relevant for the rest of the story because they affect other events (e.g. lead to actions). We will now focus onto two fundamental features of narrative that play a direct role in predicting those future events.
Expectancy Fostering Elements
Now that we have discussed the different kinds of narratives and most of the elements they include, we can zoom on what is most important. We need to know whether it is plausible that autobiographical narrative fosters the creation of predictions, after which we can see whether these predictions alter our perception.
Here, we will discuss two fundamental elements of narrative, sequencing and causality. We will see that in essence narratives are based on a selection of relevant events that are placed in meaningful temporal and causal order, which fosters the formation of expectations. Again, these elements are studied using short-term narratives, and we will have to assume that the implications that can be drawn from them – that they foster expectation formation – also apply in case of autobiographical narratives.
Sequence. What is the role of event-sequencing in narrative, and does it foster expectations? Can the underlying mechanisms and their neural correlates of narrative sequencing tell us something about its relation with perception modulation? The element of sequence is included in every type of narrative. Numerous definitions of narrative stress the importance of its sequential nature: “A narrative is an account of events occurring over time” (Bruner, 1999, p. 6)10; or it can be defined as a “sequentially organized representation of a sequence of events” (Herman, 2001, p. 30)11. This sequential nature forms the basis of autobiographical narratives as well: the sequence of experienced events is projected into the bigger story of one’s live. Mechanisms that support sequential processing have become more clear in the last few decades. We will discuss three
10 Spatial information plays an important role as well. A narrative representation might be a sequence of
‘there-and-thens’, where spatial and temporal information are blended together in spacetime (Herman, 2001) – but we will limit ourselves to the temporal domain.
11 We have to note that in narrative representations, this sequence may be altered by the use of flashbacks,
11 methods used to study this, first look at their cognitive/behavioural results and then discuss the neural correlates of these.
A popular method to study this is to use scripts. Scripts are a kind of hierarchical knowledge structure, like schemas are. They are generally used in computational and simulation studies of cognitive processes but also in the domains of language and development (Keestra, 2014). Here we will focus on the kind of scripts that constitute representations linked by associations that follow a story syntax: a temporally organized sequence of actions (Crozier et al., 1999).
Crozier et al. (1999) used fMRI to measure the brain activity of participants who were asked to detect errors of sequences within scripts or within sentences. ‘Get dressed/take a shower,’ is an example of script sequence error they used; ‘the message twice/announced was,’ a sentence sequence error (Crozier et al., 1999, p. 1470).
Not all event sequencing is of the kind that is fundamental to narratives. Creating an (autobiographical) narrative is more than placing information in a chronological order. As we have seen, it constitutes selecting and ordering events or actions belonging to the same story. Knutson, Wood and Grafman (2004) compared the brain activity corresponding to these two kinds of information sequencing: chronological ordering, and sequencing events belonging to one story. In the chronological ordering task subjects placed items belonging to the same semantic category (historical facts) in the right order. To measure mechanisms supporting the sequencing of events belonging to a story, a script ordering task was used.
Lastly, Cohn, Paczynski, Jackendoff, Holcomb and Kuperberg (2012) looked at mechanisms that support sequential image comprehension. They showed comics that varied in narrative structure and semantic relatedness and compared subjects’ ERP responses to the target images: images that were equal in every comic, and also shared their place in the image sequence (e.g. third picture always). This resulted in four different types of sequences: Normal, expressing both structure and meaning; Scrambled, in which both semantic relatedness and narrative structure was lacking; Semantic only, in which narrative structure was missing; Structural only in which narrative structure was present but the images were not semantically related. To illustrate the last example, which might be a bit hard to imagine: in the first image someone prepares an action; then a (different, semantically unrelated) action is performed; the image depicts a continuation (in general, e.g. if a ball was thrown in the previous image, now someone or something is moving in the same direction); and the sequences closes with an ending situation.
These three studies led to partly converging results. No large cognitive or behavioural differences were found between the different conditions. First of all, subjects did not show enhanced detection capabilities when performing script-sequencing as compared to word-sequencing: both error detection rate and reaction time did not differ between the conditions (Crozier et al., 1999). No behavioural differences that are important for the current discussion were found in the study of Knutson et al. (2004) either. Lastly, reaction times were fastest in the Normal sequence condition, slower in both Semantic only and Structural only condition and slowest in the Scrambled condition. This suggests that both a narrative structure and a semantic link between the elements support processing (Cohn et al., 2012). A narrative structure could add processing qualities because images following such a structure might be easier to predict, since they consist of actions that follow each other in an understandable sequence.
Although no behavioural differences were found in the study of Crozier et al. (1999), they reported some interesting neural correlates.12 Comparing script-sequencing with word-sequencing as a control showed that event sequencing corresponds with increased activity in bilateral middle frontal gyrus, left supplementary motor area (SMA) and left angular gyrus. Because left SMA and left angular gyrus are thought to be involved in movement planning, the authors argued that here they played a role in simulation of actions represented in the script (which is in part confirmed by the earlier discussed study of Sabatinelli et al., 2006). The bilateral middle frontal gyrus would thus be
12 This shows an advantage of brain imaging results: although accuracy and reaction times could be equal, it
12 more specifically involved in activating representations stemming from long-term memory and sequencing them. This is confirmed by Partiot et al. (1996; discussed above), and more recently by Knutson et al. (2004). Knutson et al. (2004) showed that sequence processing in both tasks led to bilateral activation of the middle frontal gyrus.13 However, the script order task resulted in increased activation in the right inferior frontal gyrus, whereas the chronological order task led to an increase of activity in the left inferior frontal gyrus. This further narrows down the exact areas that play a role in narrative sequencing, with the right inferior frontal gyrus being specialized in the temporal ordering of information with a narrative nature. So, the causal-event structure of (autobiographical) narratives requires a special kind of processing, supported by differentiated brain areas.
Lastly, the N300/N400 response was largest in response to the target image in both comics with lacking narrative structure and semantic relatedness and those with only narrative structure; intermediate with only semantic relatedness and smallest when both were present (Cohn et al., 2012). The researchers proposed that narrative structure and semantic relatedness together facilitate processing of upcoming images (which was indicated by the N300/N400 response). Furthermore, because sequences missing both semantic relatedness and narrative structure evoked a larger left-lateralized anterior negativity than those lacking only narrative structure, and because this negativity differed from the N300/N400 response, it was argued that sequential image processing might be based on a narrative structure that is independent of semantic relatedness.
In summary, sequencing appears to be a fundamental part of narrative, and narrative sequencing is a special process distinct from chronological ordering and semantic relatedness, with underlying differentiated neural processes. But do they foster expectation formation? As we saw in the introduction, the brain is a proactive machine, continuously generating predictions about the future. We suggested it partly bases these predictions on narratives, and an element of these narratives might indeed be the sequence of previously experienced events. When event B has followed event A repeatedly, it makes sense to expect this short sequence of events to occur the next time, so that perceiving event A results in expecting event B again. This is even the case when the two are not causally related but just tend to follow each other in a sequence regularly. Numerous activities playing a role in either life’s narrative or in stories follow such a distinctive temporal order. For example, taking a bus requires informing the bus driver that you want to ride along, waiting for the bus to stop and the door to open, getting in, paying, finding a place to sit down, waiting for your stop – in that specific order. Events within such a sequence might be stored in memory together with their temporal position, in a script structure. Such a script includes single events that are connected to each other, with a sequence between them: they follow each other in an absolute order. When such a script is activated the temporal order allows us to predict which event is bound to happen next (Landgraf, Raisig, & Van der Meer, 2012).
Events are not only linked because they belong to the same sequence. Another connection is a causal chain, to which we will turn next.
Causality. In our understanding of the world, causal14 structure might be a more important feature than sequence or temporal order. Indeed, causal structure is often said to be ‘the glue of the
13 This is again broadly confirmed using lesion studies, where healthy participants are compared with frontal
lobe patients in event-sequencing tasks. For example, subjects were asked to generate scripts with demanding event-sequencing properties (Godbout, Cloutier, Bouchard, Braun, & Gagnon, 2004); or to sequence 20 cards with certain actions depicted on them (Sirigu et al., 1995). In both studies frontal lobe patients performed worse during the sequencing tasks: as compared to healthy subjects they came up with less events when generating scripts, and in both studies they made more sequence errors. A problem with these lesion studies is, however, that they are not precise: although frontal areas do include the middle frontal gyrus, they encompass a much larger area. No strong conclusion can be drawn based on these studies solely, but they do provide additional support for experimental brain imaging studies.
14
The concept of causality, and how it can actually be inferred, is a difficult philosophical question. Here we will limit ourselves to perceived causal connections only. Whether these rely on true causality, and what true
13 universe’, or – more specific – of our image of the universe. But is it also a feature of narratives? If it is, could it be a mechanism that modulates perception? And here as well, we have to see what role it plays in expectation forming.
Mar (2004) states that the presence of a causal event-structure is one of the fundamental characteristics of narratives. Clearly, this is linked with sequence: the temporal information of events (i.e. the sequence in which they occur) informs causal judgements. Herman (2001) confirms this: “narrative understanding depends fundamentally on pragmatic principles according to which interpreters assume that if Y is mentioned after X in a story, then X not only precedes but also causes Y” (p. 21).15 Such a causal structure might play a role in autobiographical narratives as well.16 Here, motives, intentions and subsequent actions function as the causal connections.
A way to investigate the cognitive effect of causal interference between events in a narrative is to vary the relational strength between elements within scenarios (Kuperberg, Lakshmanan, Caplan, and Holcomb (2006). Kuperberg et al. (2006) asked their subjects what they thought the second sentence in a three-sentence scenario would be. Based on this, the scripts were divided into three groups, differing in level of relatedness. For a scenario to be included in the unrelated group, subjects had to be unable to infer the second sentence from the first and third: “The boys were unsure about the weather. The next morning they had many bruises”; with the second sentence being “At noon they started to hike” (Kuperberg et al., 2006, p. 346). In the intermediately related sentences, subjects were not able to infer the second sentence directly, but rather its explicit result: “The boys were having an argument. The next morning they had many bruises,” with the second sentence being: “they got more and more angry.” Lastly, for a scenario to be part of the highly related group, subjects had to be able to infer the second sentence. The first and third sentences were the same; the second sentence was: “they hit each other.”
The researchers then asked (other) subjects to provide causal relatedness judgements about the three-sentence scenarios. Based on these judgements, causal interference was only generated in the intermediately related sentences. In the highly related sentences, no interference was needed because the cause was stated explicitly within the sentences, and in the unrelated sentences the causal relation could only be guessed. Causal connections in stories are most similar to those depicted in the intermediately related scripts, indicating that causal inference plays an important role in narrative comprehension.
Reading intermediately related sentences correlated with longer reaction times, as compared to reading unrelated and highly related sentences, which demonstrates that causal interference requires extra processing resources. Neural activity followed the same pattern: activity in the bilateral superior medial PFC and the right inferior prefrontal gyrus17 increased the most when reading intermediately related sentences. This indicates that these brain regions play a role in causal interference processing within short-term narratives, and it is plausible that they do too when it comes to autobiographical narratives.
Now that we have seen that causal event-structure is indeed a fundamental part of narrative, we can focus on its role in forming expectations. Unfortunately, to our knowledge, no studies exist in which the difference in expectation forming is compared between sequencing and causal
causality is exactly, might not be that important in narrative, or in its role in perception modulation. So the focus lies more on causal inference.
15
Still, people do not always infer a causal connection from recurrent sequences. Going back to the bus example, (virtually) always after an individual has entered a bus, it starts riding – but (s)he knows that this does not mean that busses start running because (s)he enters them.
16
Interestingly, when individuals that suffered from a traumatic experience were asked to write about the traumatic events, the usage of causal words was linked with stronger health improvement. Incorporating negative events into one’s biographical narrative in a coherent way, based on a causal structure, might reduce negative effects (Pennebaker, 1993).
17 Or Brodmann area (BA) 47, which is a subsection of the larger right inferior frontal cortex; a region that was
14 connections. But we will see that based on conceptual difference, causal interference might create even stronger expectations than mere sequences.
Support for the role of causal inference in creating expectations can be found in language structures. Language is an important medium of expressing narratives (be it autobiographical or fictive ones), and every language contains words indicating a kind of coherence between actions. Importantly, words pointing towards a causal connection between actions are found to lead to faster processing times (Mak & Sanders, 2013). This indicates that words expressing causality help to form expectations about future information.
Causal connections differ from sequential relations. Whereas in recurrent sequences one expects B to happen because it has preceded A in the past, in causal interference one expects B to follow A when A is a factor of B. Based on a mere sequence, one expects a future event to happen because it has been part of the same sequence before, but it does not necessarily have to happen. Other obstructing events can alter the sequence. Going back to the bus example: the bus driver could miss your hand and not stop, or you might not find a place to sit down because the bus is too crowded. Expectations based on causal relations are stronger: if A is the cause of B and you are performing A, you can strongly expect B to occur. Hitting your knuckles on glass has a sound as effect, so when moving your knuckles towards a glass surface (in our example, the door of the bus, to catch the bus driver’s attention and still let you in) you can expect a knocking sound with strong certainty.
Conclusion
Here, we have tried to provide an overview of the different elements that narratives include, with a special focus on those that foster expectations. First, we looked at elements that were especially important in the formation of autobiographical narratives. Long-term autobiographical narrative is different from other narratives in that it is self-referent and it is based on memories. Then, we focussed on more basic elements, assuming they are essential for autobiographical narratives. These elements included imaging, ToM, event selection as basic elements, and sequencing and causal inference as important elements fostering expectations. We saw that in order to create a narrative, one has to select relevant events and place them in an order that is both temporally and causally logical. Such sequential and causal relations between events showed to foster expectations. We based this on research using short-term narratives, but since we argued that these elements are fundamental features of every type of narrative, it is plausible that these expectation fostering qualities apply to autobiographical narratives as well.
It is evident that sequencing and causal inference play a large role in numerous higher order processes, with an important one being expectation forming. If higher order processes in general proactively modulate (visual) representations, the sequential and causal glue must play a role here as well. We will discuss top-down perception modulation in more detail when we turn to the next chapter.
But before that, let us go back to Peter and Ronald, and discuss some differences in their autobiographical narratives. We learn that Peter is leaving for his holiday overseas tomorrow, and that it is ‘the story of his life’ that things always go wrong during the sequence of events that occur in the last two days before he leaves. Because of this, he has missed several planes and he is now on the lookout for bad things to happen.
Ronald was just pausing from work and does not expect anything relevant to happen during his break. He does not expect events to occur that will have a causal effect on events in the feature. But Peter, on the other hand, is slowly forming an expectation, based on events in his autobiographical narrative that have obstructed him from leaving the country before…
15
Chapter Two: Perception Modulation
In our introduction, we referred to insights suggesting that visual perception is not a bottom-up process, but is flexible and mediated by top-down processes, one of them being predictions. These predictions are based on associations between representations in memory (Bar et al., 2006; Bar, 2003, 2007, 2009), or on statistical processing (Friston & Kiebel, 2009; Friston, 2005, 2010; Kilner et al., 2007). In the previous chapter, we have added to this that autobiographical narrative might be used as a cognitive tool that fosters prediction forming as well. For our hypothesis, that autobiographical narrative modulates perception, this implies that if expectations appear to be perception modulating factors, they might be the link through which narrative modulates perception.
So can predictions or expectations that result from elements of narrative (i.e. sequence and causal relations) actually modulate one’s perception? Here, we will first discuss frameworks suggesting that top-down visual perception modulation is possible in theory. Then, we will focus on expectations resulting from narrative, in their role as a top-down modulating factor. We will look at demonstrations of modulation by expectation, by discussing the effects of causal and sequential relations between visual events. Three additional perception modulating elements – memory, intention and attention – are discussed lastly, because they might play an (indirect) role in the perception modulating effects of expectations.
Top-down Modulation of Visual Perception
The advantages of having a brain that constantly makes predictions are evident when it comes to visual processing. A reason why visual processing has to rely on these predictions is that the incoming information is just too complex to be based on a hierarchy from simple feature detection to higher order complex visual processing. In addition, visual information is very ambiguous (one will recognize the same face regardless of expression, corner of vision, light, shades, etc.) and processing it in such a way would not work. Rather, information has to be selected, compressed and then interpreted. This could be done with the help of several processes: cross-modal influences, action-dependent selection (M. Keestra, personal communication, July 18, 2014), but here we will focus on another: expectations.
A review of recent neurobiological studies on visual expectations by Summerfield and Egner (2009) supports the general idea of top-down influences of expectations on visual perception. They suggest that one way to select, compress and interpret the complex visual information is to rely on previous experiences. They state that perceptual processing might have evolved in such a way that it relies on probabilistic information about future events (broadly speaking: expectations), through priming, context, associations and scene information. Advantages of these expectations include a lower amount of processing for expected objects, as well as enhanced recognition of ambiguous stimuli on the basis of prior likelihood (Summerfield & Egner, 2009). Gilbert and Sigman (2007) confirm this. They argue that expectations play a role in perception modulation, since internal representations of visual objects alter the way scenes are segmented. They claim that this may “represent a form of hypothesis testing, such that before objects are identified, the visual system compares stored representations of object forms against bottom-up information on stimulus characteristics” (Gilbert & Sigman, 2007, p. 679).
But is it plausible that such bottom-up modulation also originates in areas that play a role in expectation forming based on narratives? Areas that showed to supported sequential processing (middle frontal gyrus, with the right inferior frontal gyrus as specialized in sequencing narrative-like events; Crozier et al., 1999; Knutson et al., 2004) have indeed been found to play a role in top-down modulation of visual perception. A network of fronto-pariental network might be the most important candidate of top-down control of colour and motion perception, which includes the middle frontal gyrus, and the right inferior frontal junction (which borders with the inferior frontal gyrus; Zanto,
16 Rubens, Bollinger, & Gazzaley, 2011). In addition, the inferior frontal gyrus was also included in another potential network that was identified as being associated with top-down dimensional orienting (Weidner, Krummenacher, Reimann, Müller, & Fink, 2009)
Empirical Demonstrations of Visual Modulations
So, theoretical support for the idea of top-down visual modulations exists – but can these effects be demonstrated experimentally? Expectations are known to influence responses in perceptual tasks (Chalk, Seitz, & Seriès, 2010). Expecting a certain stimuli increases the discrimination of it. In addition, it results in a reduced reaction time to detect it. Finally, expecting to see something can lead to reduced detection thresholds. It remains unclear, however, whether these altered responses are a result of decision mechanisms (i.e. response biases) or of actual underlying changes in perceptual representations (i.e. genuine perceptual modulation). This issue has to be taken into mind when discussing the implications of studies measuring perception through subjects’ response.
We will first discuss studies that demonstrated how expectations about events in a sequence can modulate the perception of those events. Many of those events are ambiguous visual stimuli, i.e. multi-interpretable events. Multiple strategies can be applied to disambiguate these events, and expectations will show to play an important role here. After this, we will focus on events that have causal links between them; we will see that such a link can result in modulated perception as well.
Sequence. Numerous paradigms were used to demonstrate the perception modulating effects of expectations. A method that is very popular in measuring perception modulation is the use of random dot motion patterns (RDPs; e.g. Chalk et al., 2010; Kok et al., 2013; Summerfield & Egner, 2009). Here, subjects are presented with a group of dots moving coherently in one direction, together with a number of randomly moving dots. The task is to indicate in which direction the overall motion is going. Even when only 10% of the dots are moving coherently, subjects are able to recognize them as a unified moving surface (Felisberti and Zanker, 2005).
Using the RDP paradigm, Chalk et al. (2010) showed that subjects easily form expectations based on average movements, which they expected to occur again when a new pattern was presented. These expectations were found to influence their visual perception of new patterns: the perceived direction of movement of new patterns was biased towards the expected direction.18 These different patterns following each other can be interpreted as a sequence of events – here it is thus demonstrated that expecting an event sequence to continue the way it does, modulates the perception of a new event in this sequence. This was confirmed by Kok et al. (2013), who influenced subject’s expectations implicitly by using an audio cue. Subjects were told to ignore the cue, which secretly gave an indication about the direction of the target dots (those moving coherently). The direction that the subjects reported was again biased towards the implicitly indicated direction.19 In addition, this experiment nicely demonstrates how concurrent fMRI-scans can be used to provide evidence that these results were not based on a response bias. Activity in visual cortex was analyzed using forward modelling, which allowed the researchers to reconstruct the motion direction of the stimuli from the measured neural activity. Sensory representations in visual cortex showed the same bias towards what was expected. This again implies that perception processing is a result of the integration of bottom-up input information and top-down expectations.
Sterzer and Frith (2008) showed that expectations indeed can have a strong effect on the perception of ambiguous visual stimuli. Subject first saw dots that are perceived as rotating in an illusionary 3-d cylinder, alternating in both directions. The researchers influenced subjects’
18
However, this could be a result of response bias.
19
Unfortunately, this bags the question whether this was just a matter of priming the subjects’ focus of attention towards dots moving in the direction secretly indicated by the audio cue. We will discuss attentional effect on visual perception in the next section.
17 perception by giving them 3-d glasses and adding coloured 3-d directional cues, skewing the overall perceived rotation in one direction.20 Finally, when believing to see the same stimuli, subjects saw the first stimuli again. In other words: they were seeing a completely ambiguous stimulus but were expecting to perceive the cylinder rotate in the preceding dominant direction. And indeed, this expectation skewed their perception in this direction. The researchers argue that this result was not based on report bias, because subjects’ confidence (which they had to report throughout all tasks) remained equal between tasks.
A similar result was found using the Necker cube as an ambiguous stimulus (Haijiang, Saunders, Stone, & Backus, 2006). Subjects saw a wire-frame cube rotating, whose perceived direction is again bistable. Here as well, depth cues indicating the moving direction were added, together with a signal corresponding to the direction (+ or –). In the test trials, directional cues were removed but the signal remained; perceived direction of the (know ambiguously rotating) cube was biased towards the direction the signal had represented until then; analogues to the results of Sterzer and Frith (2008).
Causality. The previous results demonstrated how expecting a future event in a sequence to be equal to the preceding one, influences the perception of that future event by skewing it toward what is expected. As we have discussed in the previous chapter, another element supporting expectations about future events is causality. Unfortunately, to our knowledge, fewer demonstrations exist of its effects on perception.
A study by Fernbach, Linson-gentry and Sloman (2006) provides a demonstration of the effect of causal connections on visual perception. They first taught subjects a causal structure of events: moving a slider (depicted on a computer screen) caused other sliders to move as well, following different causal chains. For example, slider B caused slider C to move, which caused slider A to move; or slider C caused both A and B to move simultaneously. When subjects understood the causal connection, three new sliders were presented, which moved either (partly) together or quickly after each other (with an interval of just 100 ms, so that it was hard but not impossible to distinguish an order). The subjects were then asked to indicate the order of the slider movements. This was repeated for 10 different causal relations. The researchers found that the indicated order was biased towards the antecedent causal connection.
As mentioned, there is ample evidence for top-down influences of lower visual processes. More specific evidence for top-down modulation by expectations is sparse, yet we found that expectations that are based on narrative structure – either on sequence of events or on causal relations – can modulate perception in specific cases. Here, perception of future events is bias towards what is expected.
Perception of ambiguous stimuli is in need of disambiguation, which allows applying tricks like in the studies discussed above to see that this disambiguation might be based on previous experience and subsequent expectations. Ambiguous stimuli offer great opportunities to measure perception quite directly. Still, we do not want to argue that perception modulation is limited to ambiguous stimuli. In fact, non-ambiguous stimuli might be interpreted according to elements of narrative as well.
Other Perception Modulating Factors
In the previous section we have seen experiments in which expectations were manipulated directly, showing an effect on perception. However, numerous other perception modulating factors exist.21
20
This change in perception is not an effect of expectation, attention, or the like, but just a visual result of the added 3-d dots perceived through a 3d-glass.
21
For example, emotions, which are found to modulate face perception, visual illusions and contrast detection (Stefanucci et al., 2011). Specifically, the perception of spatial layout might be influenced in different ways,
18 We will limit ourselves to discussing only those that indirectly measure the effect of expectations, or those that might have played a role in the previous experiments. The three modulating effects that we will discuss here are memory, intention and attention.
Memory. It is evident that memory is a factor that modulates perception. For example, the time it takes for subjects to detect which object they see, depends on stored information about that object (Purcell & Stewart, 1991). In addition, recognizing which object one is perceiving facilitates the depth segregation of ambiguous stimuli (Peterson, 1994).
As discussed in the previous chapter, memory mechanisms might also support making predictions about the future. In addition, expectations are always based on prior knowledge: one needs to be familiar with a certain sequence of events to predict what the next event will be, and one needs to remember the causal connection in order to make additional causal inferences.
Intention. As we saw in the first chapter when we discussed the role of causal connections in narratives, motives and intentions are important ingredients that contribute to the causal structures of narrative. Interestingly, knowledge about one’s intentions was found to have its effect on visual perception. First, intentional binding is the subjective compression of the time period between an action and its effect (Moore, Teufel, Subramaniam, Davis, & Fletcher, 2013). Specifically, this effect occurs when the action is perceived to be intentionally: the perceived time interval between intentional movements and their causal effect is perceived as smaller as compared to the time interval between unintentional movements and their effects. In line with other studies demonstrating this effect, Moore et al. (2013) showed participants movies of a finger that was attached to a key on a keyboard; the key could either be moved by the finger or by a mechanism that was attached to the key. Either way, when the key was pressed, this resulted in a delayed sound. Subjects indicated the time delay between the moving key and the sound. They reported smaller time delays when they believed the finger was moving the key (an intentional action) than when it moved because the mechanism made it move.
Second, the intention to act was found to influence space perception. Witt, Proffitt, & Epstein, (2005) asked participants to hold a long stick and estimate the distance to an object just out of reach, finding that distances appeared closer to participants holding the stick, as compared to those whose reach was not extended. This change in perceived distance only occurred when participants intended to use the stick, suggesting that the intention to act rescales space perception for that action. Like in some of the earlier discussed RDP-experiments, however, these results could rely on effects of attention; subjects could have focussed their attention more strongly when intending to act. We will focus our own attention on the perception modulating effects of attention next.
Attention. Attention is an important perception modulating factor, which might have an overarching modulating effect. It is related with memory (since deciding where to focus one’s attention can rely on remembered events), intentions (because one will attend to features that are involved in what one intends to do) and on expectations (paying less attention when the expected thing occurs, or more when one is surprised because one’s expectation was not met). Attention might have its effect by prioritising stimulus processing (Summerfield & Egner, 2009). For example, Felisberti and Zanker (2005) showed that attention can improve transparent motion detection, which occurs when one perceives different motion signals in the same visual region. Felisberti and Zanker tried to investigate whether attention can improve detecting a certain motion direction within a
according to the type of emotion (Stefanucci et al., 2011). Fear appears to result in more extreme spatial perception: people showing higher fear levels overestimated height, and manipulating fear by placing subjects on a skateboard on top of a hill resulted in overestimated slope judgements. Furthermore, when the researchers used sad/happy music to alter emotions, sad subjects overestimated slant more so than happy ones.
