MULTISENSORY INTEGRATION: EFFECTS ON BEHAVIOURAL PERFORMANCE AND OSCILLATORY ACTIVITY IN SENSORY AREAS
Anupama Nair Student number: 11120487
Supervisor: Dr. Conrado Bosman Vittini Co-assessor: Dr. Umberto Olcese M.Sc. in Brain and Cognitive Sciences
Track: Cognitive Neuroscience University of Amsterdam Amsterdam, Netherlands
TABLE OF CONTENTS
Page
1 Introduction . . . 1
1.1 Multisensory integration (MSI) . . . 1
1.2 Oscillatory activity and MSI . . . 3
1.3 TMS and TMS-induced entrainment . . . 5
1.4 TMS in MSI . . . 7
2 Implications of interventional experiments in humans and animals . . . 10
2.1 Animal studies in MSI . . . 10
2.1.1 The contribution of SC multisensory neurons to MSI . . . 10
2.1.2 Cortical contributions to MSI . . . 12
2.2 Human studies in MSI . . . 16
3 Effects of TMS on Oscillatory Activity . . . 19
3.1 Top-down and bottom-up processing . . . 19
3.2 Oscillatory activity in feedforward and feedback processing . . . 23
3.3 Role of TMS in multisensory oscillatory operations . . . 24
4 Predictive Coding and MSI . . . 29
4.1 Predictive coding and neural oscillations . . . 30
4.2 Feedforward and feedback connections in predictive coding . . . 32
4.3 Divisive Normalization and Phase Resetting in MSI . . . 35
4.3.1 Phase resetting (PR) . . . 36
4.3.2 Divisive normalization (DN) . . . 36
4.3.3 Rhythmic vs. continuous processing . . . 38
4.4 TMS, MSI and predictive coding . . . 39
5 Discussion . . . 45
Stimuli perceived in more than one modality, matched in time and space, often benefit from enhanced processing due to our inherent integration capacities, resulting in what is known as multisensory integration (MSI). While abundant research has been carried out in the area of MSI, fewer studies have explored the interplay between MSI processes and how they can be influenced through intervention. The focus of this dissertation is two-fold: first, we review the existing research aimed at uncovering the nature of integration processes, the neurological correlates underlying MSI, the conditions promoting integration and the resulting consequences on our perceptive abilities. Second, we look at how the balance of activity underlying MSI is disrupted through interventional strategies such as Transcranial Magnetic Stimulation (TMS) which offers an effective means to observe the behavioural ef-fects of temporary deactivation of a region. To achieve these objectives, we adopt multiple perspectives while reviewing our literature. We focus on the neuronal level by studying oscillatory activity supporting MSI, receptive field (RF) characteristics of cortical and sub-cortical areas underlying MSI and predictive coding capacities of our sensory systems. We scrutinize models (Bayesian models, divisive normalization model and hierarchical network model) that attempt to explain and describe MSI processes, based on the model’s premise. We also focus on the cognitive and behavioural implications of MSI processes by studying feedforward and feedback interactions between lower and higher-order brain regions, and the behavioural consequences of hindering these interactions.
1. Introduction
1.1 Multisensory integration (MSI)
Much of the processing of the external world calls for synchronous activation of all our sensory capacities, requiring our senses to work together to form a coherent, dynamic rep-resentation of input stimuli. Multisensory integration (MSI) is thus an integral element of perceptual processing. Stevenson et al. (2014) define multisensory integration “... according to the number of action potentials produced in response to a given stimulus.” A statistical account of the phenomenon describes MSI as ” a statistically significant difference between the number of impulses evoked by a crossmodal combination of stimuli and the number evoked by the most effective of these stimuli individually” (Stein & Stanford, 2008). Finally, Talsma (2015) defines MSI as ”the neural process by which unisensory signals are combined to form a new product or representation”. The different definitions highlight the enhancing effect of multimodal integration —in terms of neuronal activity as well as representations at the cognitive level.
MSI often results in a representation that is significantly more enhanced than that of a unisensory stimulus. It also leads to an increase in the level of neuronal firing, indicating in-creased activation for spatiotemporally coinciding multisensory stimuli (Wallace, Meredith, & Stein, 1998; Stein, Laurienti, Stanford, & Wallace, 2000). Multisensory enhancement oc-curs when the neuronal firing rates are higher for multisensory stimuli than for the strongest unisensory stimuli (Stein, Stanford, & Rowland, 2009). In fact, multimodal stimuli coin-ciding in space and time have shown to elicit greater neuronal excitation than the sum of unisensory stimuli activation and this is termed as the superadditivity principle (Meredith & Stein, 1986). This enhancement effect is greatest for weak unisensory stimulus pairs as compared to strong unisensory stimulus pairs (Wallace et al., 1998), in a phenomenon known as inverse effectiveness (Wallace, Stein, & Carolina, 1997; Atteveldt, Murray, Thut,
& Schroeder, 2014; Stein et al., 2000). The enhancement effects found for weak unisensory stimulus pairs presented in unison reflect greater levels of multsisensory integration, than what is commonly found for strong unisensory components. Therefore, the individual effi-cacy of unisensory stimuli dictate the level of integration and enhancement effects observed in a multisensory context (Stevenson et al., 2014). Such enhancement effects occur in the deep Superior Colliculus (SC) layers (layers IV - VII), where the weakest unisensory inputs combine to form a stronger representation of the multimodal stimulus (Wallace et al., 1998; Anastasio, Patton, & Belkacem-boussaid, 2000; Stein et al., 2009). Moreover, it is important to note that the principle of inverse effectiveness is appropriately observed only in multisen-sory conditions -i.e. it manifests to a much lesser degree in the presence of synchronous presentation of same-modality inputs (Holmes, 2007). Conversely, if the multimodal stimuli are discordant in time and space, they are most likely to elicit a weaker response than the strongest unisensory inputs, otherwise known as response depression (Stein et al., 2009)
Effective MSI follows certain principles. An important factor determining the extent of multisensory enhancement is receptive field (RF) characteristics. Multisensory stimuli presented such that each of the component unisensory stimuli fall within the modality-specific RFs will lead to greater enhancement than multisensory stimuli where any one stimulus falls outside of its RF, which would in turn lead to response depression (Stevenson et al., 2014). In fact, response enhancement effects are observed even for stimuli that are not spatially concordant, as long as the individual unisensory components fall within the limits of their respective RFs (Stevenson et al., 2014) .
The magnitude of response enhancement is also contingent upon peak discharge periods of the individual unisensory stimuli. Specifically, higher the overlap in peak discharge periods of component unisensory stimuli, greater the enhancement effects and vice versa (Stevenson et al., 2014) .
The best example of multisensory enhancement includes ambiguous speech stimuli due to noisy environments where weak auditory inputs often benefit from accompanying correspond-ing visual cues, leadcorrespond-ing to better speech comprehension (Grant & Seitz, 2000). Multisensory enhancements are also observed for auditory visual stimuli. Concurrent auditory stimuli
en-hance processing of subsequently presented visual stimuli in the same location, with respect to target detection as well as reaction times (Noesselt et al., 2010; Cappe, Thut, Romei, & Murray, 2010; Brang et al., 2015)
1.2 Oscillatory activity and MSI
To understand multimodal integration, it is important to understand the mechanisms supporting such integration. One such mechanism discussed in this chapter is that of oscil-latory dynamics. Large scale neuronal assemblies fire synchronously to support higher-level cognitive functions, resulting in rhythmic brain activity within an area or over larger net-works. This rhythmic activity is grouped into categories based on the frequency and nature of functions that they subserve. e.g.: delta (0.5 - 3.5 Hz), theta (4 - 7 Hz), alpha (8 - 12 Hz), beta (13 - 30 Hz), and gamma (>30 Hz) are the most frequently studied frequency channels (Engel & Fries, 2010).
Neural oscillations have been studied extensively in recent years due to increasing aware-ness regarding its role in various cognitive and behavioural processes. e.g.: Fries (2006) reported how synchronous cycles of activity facilitate inter-regional communication, through a hypothesis known as ’communication-through-coherence (CTC). This hypothesis posits that rhythmic activity of neural oscillators modulate neuronal excitability, which in turn generate temporal windows allowing for communication. The authors establish such oscilla-tory coherence as being necessary for effective communication, which is rendered ineffective in the absence of coherence (Fries, 2005). The phase of oscillations also contains significant information - especially with respect to perceptual processes. e.g. alpha phase has been strongly correlated with ”phosphene” (illusory flashes) experiences, i.e stimulation at the optimal alpha phase has been found to induce phosphenes in a subject (Dugu´e, Marque, & VanRullen, 2011). In a review by Bosman, Lasink and Pennartz (2014), the authors re-ported findings linking gamma activity to higher-order cognitive functions such as memory encoding, visual processing, decision-making and information processing, amongst others (Bosman, Lansink, & Pennartz, 2014). Given the robust involvement of oscillations in
per-ceptual and sensory processes, it is reasonable to expect oscillations to be an important candidate in furthering our understanding of MSI. Our focus in this section would therefore be to evaluate the role of oscillations in MSI.
Before moving forward, it is necessary to understand concepts that describe changes in oscillatory activity. Some of these include phase-locking and phase resetting mechanisms in response to sensory input. In phase-locked activity, the phase of oscillation is consistently aligned to the onset of the stimulus, and emerges in time and time-frequency averaged activity. It is also referred to as ”evoked” activity. Activity that is time-locked but not phase-locked to stimulus onset is referred to as ”induced” activity and is observed in time-frequency domain averaged activity but not in time-domain averaged activity (or ERPs) (Cohen, 2014). Further details about the two kinds of activity would be introduced in section 1.3 of this chapter. Phase-resetting, on the other hand, occurs when ongoing oscillations of a specific frequency-band undergo a reset in their phases due to presentation of a stimulus. The measure of distribution of phase angles across each-time frequency point is referred to as ”inter-trial phase coherence” and is used to determine the uniformity of phase angle distribution across trials (Cohen, 2014). Oscillatory activity can be the result of natural rhythmic brain rhythms or entrainment via external stimulation which acts by resetting the phase of oscillatory activity or impacting the amplitude of the oscillation. (Thut, Schyns, & Gross, 2011).
Multisensory auditory-visual stimuli rely on oscillatory entrainment mechanisms for in-tegration. Auditory and visual stimuli interact uniquely with each other such that stimuli in either modality have the power to influence processing in the other modality. Audi-tory stimuli benefit from faster cortical processing and are therefore capable of resetting phase of ongoing visual cortex oscillatory activity (Atteveldt et al., 2014). Therefore, a sound co-localized in space with a corresponding visual target enhances the processing of the accompanying visual target by resetting the phase of visual cortex activity (Mercier et al., 2013). Conversely, visual inputs to auditory cortex can reset ongoing auditory cortical activity, enriching the response to the oncoming auditory stimulus (Schroeder & Foxe, 2005)
Entrainment of oscillatory activity can occur due to sensory inputs (rhythmic visual stimulation for instance, resulting in steady-state visual evoked potentials SSVEPs) or due to external force such as pulsed stimulations. (Thut, Schyns, & Gross, 2011).
Based on the understanding of oscillatory dynamics, we will briefly introduce the network mechanisms supporting MSI computations. One such mechanism is that of phase-resetting, which we described earlier (Atteveldt et al., 2014). Divisive normalization is the other mechanism that governs multisensory interactions as occurring on a network level (Angelaki, Gu, & DeAngelis, 2009). The term divisive refers to the process by which a neuronal output is divided by the sum of all outputs (Heeger, 1992). These mechanisms will be discussed in greater detail in section 4.3 of this dissertation.
1.3 TMS and TMS-induced entrainment
Based on Faraday’s observations in 1831 of the capacity of magnetic current to induce electrical activity in a conductor, a similar means to stimulate the cerebral cortex was es-tablished (Huerta & Volpe, 2009). Due to the conductive capacities of the brain cells, an electromagnetic current passed through the scalp can stimulate underlying neuronal ar-eas, which is the main principle underlying the Transcranial Magnetic Stimulation (TMS) technique. Repetitive pulsed stimulation at periodic intervals forms basis for the repetitive Transcranial Magnetic Stimulation (rTMS) technique used to stimulate cortical areas (Thut, Schyns, & Gross, 2011).
To study the impact of TMS on brain dynamics, time-frequency decomposition (e.g.: wavelet transforms, Fourier transforms) is performed to scrutinize stimulation-related changes in oscillatory activity over time, which involves studying either of evoked or induced responses to TMS intervention. An introduction to the two kinds of activity was presented in section 1.2. The distinction between these two kinds of responses is often blurry in literature and is even carelessly interchanged or used incorrectly. Pellicciari et al. (2017) emphasize the need to exercise caution while delineating the two kinds of responses. In a nutshell, evoked response is the average activity across time, time- and phase-locked to stimulation onset.
Thus, an evoked oscillatory response (EOR) emerges after every occurrence of the stimu-lation on every trial and is sometimes observed only after averaging across trials, due to physiological and measurement noise (David, Kilner, & Friston, 2009) . Induced response (IOR), on the other hand, is understood in relation to total oscillatory response (TOR). It is the activity that remains of the total oscillatory activity, when stimulus-locked evoked response and baseline activity has already been taken into account. In other words, TOR includes both phase-locked and non-locked phase locked activity amongst others, and ex-cluding the phase-locked and baseline activity from this response leaves us with the IOR. TMS intervention results not just in generation of phase and time-locked responses but also in some non-stationery, jittered responses that are not phase-locked to the stimulation pulse. These jittered responses, captured in total and induced oscillatory responses, might shed light on the nature of TMS after-effects (David et al., 2009; Pellicciari, Veniero, & Miniussi, 2017).
Effects of TMS can be varied and is contingent on the area of stimulation. The most observable effects are those over the motor cortex, resulting in muscle movements reflected in motor evoked potentials (MEPs) (Huerta & Volpe, 2009). TMS is also used to deactivate cortical areas over short periods by creating ”virtual lesions” - which can in turn be used to investigate the functions of the deactivated region (Pasalar, Ro, & Beauchamp, 2010). These effects are observed immediately as a direct consequence of stimulation by TMS. However, TMS is also known to exert indirect effects on performance of higher cognitive tasks, not via direct stimulation but by mediating the performance of the ongoing task. For example, studies have shown TMS stimulation to interfere with speech production when applied over Broca’s area during a language task (Huerta & Volpe, 2009).
TMS effects also vary as a function of stimulation frequency and subject states. Resting-state activity in the presence of TMS intervention can differ significantly from TMS effects on task-engaged activity. This difference is reflected in TOR and IOR. e.g. TMS pulses in rest-ing state is found to result in spontaneous phase reset vs, in active state, more complicated interactions with TMS pulses emerge (Pellicciari et al., 2017).
TMS effects are generally transient and do not have any long term consequences (Noh, Fuggetta, Manganotti, & Fiaschi, 2012). However, TMS pulses applied at certain frequencies can lead to long-lasting changes. One such application is the Transcranial Magnetic Theta burst stimulation (TBS) that applies electromagnetic pulses in rhythmic ”theta” bursts (Huang, Edwards, Rounis, Bhatia, & Rothwell, 2005), which can lead to long-term synaptic changes (including long-term potentiation (LTP) and long-term depression (LTD)) possibly lasting for days (Huerta & Volpe, 2009). TBS has also been shown to affect oscillatory activity, by synchronizing activity across different frequency bands and modulating power in beta-band frequency, highlighting the mediating effects of TBS on cortical vs. deeper structures (Noh et al., 2012).
As mentioned previously, TMS can affect the phase of oscillation of the stimulated region, entraining rhythmic neuronal activity. Entrainment posits rhythmic TMS stimulation to match the natural rhythm of the stimulated area, enhancing existing oscillatory activity as opposed to driving new oscillatory signatures . This enhancement is a function of pre-stimulation phase of natural, ongoing activity through amplification of the stage of oscillation that coincides with the TMS pulse. Some research posits strong entrainment to arise from TMS pulse interaction with an oscillatory phase angle of 00 and 3600 vs. 1800which typically results in weak entrainment (Thut, Veniero, et al., 2011) Such TMS-induced entrainment activity can regulate perceptual and cognitive performance.
1.4 TMS in MSI
The reason we introduce TMS-induced entrainment is to understand the effects of such external rhythmic intervention on multisensory activity. In the area of perception, TMS stimulation over visual cortical areas has been found to evoke phosphenes under certain conditions. As briefly mentioned earlier, the experience of phosphenes is strongly linked to pre-stimulation alpha frequency phase in combination with TMS-induced cortical excitation. Significant phase-locking activity is observed on occasions where phosphenes are experienced
which significantly differs from phase-locking activity in the absence of phosphenes (Dugu´e et al., 2011).
TMS can exert effects on both audio-visual integration as well as visual-tactile multisen-sory integration. These effects are not limited to multimodal stimulus presentations but also unisensory cross-modal presentations. e.g.: TMS has been found to exert beneficial effects on reaction times (RTs) for auditory stimuli when applied over the occipital pole with short latencies (30-150 ms) but introduce delays (by 60 to 75 ms) in response to unisensory visual stimuli when applied with the same latencies (Amassian et al., 1989). This period of delay (60-75 ms) is also believed to mark the critical time window wherein visual cortex receives feedforward inputs. On the other hand, no effects were observed for multisensory audio-visual stimuli presentations with TMS application over the occipital pole. These differential reactions to visual (disruptive) or auditory (enhanced) stimuli with TMS application over the visual cortex hint at the nature of multisensory integration in the brain. These opposing effects to the two kinds of sensory stimuli can be described in terms of the differential laminar organizations involved in the processing of the two kinds of stimuli. The laminar response profiles evoked in response to any one kind of stimulus could explain the discrepant reaction to TMS application over the visual cortex (Romei, Murray, Merabet, & Thut, 2007).
In a subsequent study, the same researchers found enhanced visual activity (e.g.: as indicated by the experience of phosphenes) with TMS application over the occipital pole but when preceded by an auditory stimulus by 70-120 ms. These results provide evidence for auditory-induced facilitation effects in the visual cortex (Romei et al., 2007).
In a similar vein, TMS also has the power to disrupt visual-tactile multisensory inte-gration. The posterior parietal cortex (PPC) is implicated in visual-tactile multisensory integration, and TMS application to this region with concurrent visual-tactile stimulation tends to dilute these multisensory integration capacities of the PPC (Pasalar et al., 2010). This effect will be discussed in detail in section 3.3 of this thesis.
These studies summarize the effects of TMS intervention on oscillatory activity, and briefly describe the role it plays in MSI. Specifically, we see how TMS can be used to enhance natural oscillatory activity, reset phase of ongoing oscillations and also functionally disrupt
certain areas necessary for integration. However, certain conceptual gaps remain in fully understanding the effects of TMS on multisensory activity and the mechanisms through which it exerts its influence on brain dynamics. These concerns would be addressed in the following chapters.
2. Implications of interventional experiments in humans and
animals
From the previous chapter, we already know about the disrupting effects of TMS on visual-tactile integration and unisensory visual processing via stimulation of PPC and occipital pole, respectively (Amassian et al., 1989; Pasalar et al., 2010). We also know about multisensory neurons in the SC that are essential to successful integration performance (Anastasio et al., 2000; Stein et al., 2009; Wallace et al., 1998). However, we are yet to attain a deeper understanding of the workings of the SC multisensory neurons to know how integration takes place. It would also be interesting to note the effects of experimental intervention on cortical regions implicated in MSI. Specifically, in a multisensory environment, how does such intervention affect neurons in component unisensory areas, multisensory areas of the SC or the multisensory cortex? In this chapter, we will evaluate the impact of intervention on RF characteristics of SC multisensory neurons and relevant cortical areas, and attempt to generalize these findings to the effects of TMS application on regular functioning.
We will navigate this chapter by first focusing on animal studies that would shed light on the role of SC multisensory neurons and cortical areas necessary for integration. We will then discuss findings in human research and attempt to draw inferences from interventional studies exploring multisensory capacities in humans.
2.1 Animal studies in MSI
2.1.1 The contribution of SC multisensory neurons to MSI
Most of our understanding of the layers of SC comes from studies performed on cats. Mul-tisensory inputs converge onto mulMul-tisensory neurons in the cat SC, which mature around 15-20 days post birth (Wallace et al., 1997). With development, the multisensory neurons
in the SC learn to integrate information from multiple modalities. These multisensory neu-rons behave in accordance with the principles of effective MSI described earlier. Therefore, weak pairs of stimuli benefit from greater multisensory enhancement (inverse effectiveness) (Wallace et al., 1998) and stimuli concordant in their presentations with respect to space and time also receive enhanced processing (spatial correspondence) (Stevenson et al., 2014) (see section 1.1).
Figure 2.1.: RF size plotted as a function of post-natal age. The x-axis represents the postpost-natal age in weeks and the y-axis represents RF size as a percentage of adult value, for the three kinds of RFs: somatosen-sory, auditory and visual. As seen in this figure, mat-uration is accompanied by a rapid decline in RF size, typically between four to six weeks after birth. Figure retrieved from Wallace et al., 1997.
This gradual maturation is also accom-panied by changes in RF characteristics (see figure 2.1). The relatively young sensory neurons have large receptive fields, exhibit erratic, weak and slow responses to stimuli and show quick habituation to repeated pre-sentation of stimuli (Wallace et al., 1997). The auditory receptive fields in early neu-rons are especially remarkable with a ten-dency to tune to auditory stimuli presented anywhere in space. These young neurons are incapable of performing MSI. However, with development, these receptive fields fine-tune to stimuli localized in a region of space, al-though some of them may still have large receptive fields in the course of development (Wallace et al., 1997).
Under normal rearing circumstances, these RFs shrink and respond to modality-specific stimuli presented in exclusive spatial
locations. MSI capacities of a neuron are determined by this contraction process (Wallace et al., 1997). Multisensory neurons of the SC consist of multiple receptive fields based on the sensory modality they respond to. Therefore, in the case of multisensory inputs, the manner
in which the component stimuli map onto the RFs determine the extent of multisensory enhancement (Stein et al., 2009)(see section 1.1). Greater the alignment with the respective RFs, greater is the enhancement and vice-versa (Stein et al., 2000). Therefore, the size of the multisensory neurons are linked to their integration capacities, with smaller RF sizes reflecting more nuanced MSI and larger RF sizes reflecting premature or weak integration potential, suggestive of a common factor underlying the two processes. The N-methyl D-asparate (NMDA) receptor is also implicated in the functioning of the sensory neurons of the SC, primarily because of its role in the consolidation of the RFs (Wallace et al., 1997; Binns & Salt, 1996). As further evidence, some research has demonstrated the emergence of NMDA receptors in SC to be linked with the initiation of multimodal integration abilities of the SC (Binns & Salt, 1996).
In this section, we learned how RF properties of the SC multisensory neurons drive the process of MSI. Therefore, disruption of RF properties of these neurons in the SC are likely to disturb normal MSI processes. It would be interesting to investigate these links in human subjects, to determine if this hypothesis holds true even for humans. There is scarce literature investigating this assumption, mainly because of the invasive nature of RF exploratory research. However, as we will see in section 2.1.2, the SC has been implicated in multisensory functions even in humans, which leads one to expect the involvement of SC neuronal RFs in MSI.
2.1.2 Cortical contributions to MSI
The cortex is found to work closely with the colliculus in driving multisensory processing. The SC integrates the unisensory inputs it receives from different sources. However, what is interesting to note is that the SC can only integrate these inputs if at least some of these of these inputs sources lie in a specific part of the cortex, namely the association cortex. Cats in which the cortico-collicular communication was impaired (by disrupting inputs from the association cortex to the SC) had SC multisensory neurons that showed distorted multisensory capacities. In the absence of cortical input, they treated multisensory
stimuli like unisensory stimuli presented in isolation. The most important regions of the association cortex implicated in this function include the anterior ectosylvian sulcus (AES) and the rostral part of the lateral suprasylvian sulcus (rLS) (Stein et al., 2009; Ghazanfar & Schroeder, 2006). Some research has suggested convergence of inputs from distinct primary sensory areas in the AES onto the deeper layers of the SC neurons (Binns & Salt, 1996). In fact, some research has suggested that damage to these association cortex regions in the early developmental years is not compensated for by other brain regions, leaving the AES and rLS to be exclusive in their multisensory functions (Wilkinson, Meredith, & Stein, 1996; Stein et al., 2009)
Some researchers have speculated that the early multisensory neurons of the SC do not receive such inputs from the association cortex, possibly one of the reasons rendering them incapable of MSI. In fact, these researchers conjecture that it may be the cortical inputs to SC neurons that bestow the SC neurons with adult-like integrative capacities (Wallace et al., 1997).
Previous research has also located multisensory neurons in the AES, in regions surround-ing unimodal neurons. The AES is essentially comprised of three regions, respondsurround-ing to each modality type: (a) the visual region (b) the somatosensory region and (c) the auditory region. Areas bordering these regions are found to be responsive to stimuli in more than one modality. These multisensory neurons were distributed in a somewhat systematic fash-ion, most likely bordering neurons encoding for modality-specific stimuli. E.g: multisensory neurons encoding for audio-visual stimuli were most likely to be located along the borders of unimodal auditory and visual regions (Wallace, Meredith, & Stein, 1992). The remarkable fact about these neurons is that they exhibited RF characteristics similar to SC multisensory neurons. Therefore, multi-modal stimuli falling within their respective RFs, presented con-currently, led to response enhancement whereas stimuli falling the outside the RF range led to response depression (Wallace et al., 1992). Such common response characteristics across the SC and AES neurons indicate a link between multisensory processes in different regions in the brain and further point to the regulatory role of RFs in MSI .
In the framework of the findings outlined above, some researchers investigated the cortical microcircuitry involved in the process of MSI. e.g.: Olcese and colleagues (2013) targeted a multisensory area between rostral V1 and caudal S1 (area RL) in mice using optical imaging in order to identify distinct laminar profiles of unimodal and multimodal neurons, across the cortical layers and the cortical surface. The results from their multi-unit cell recordings identified area RL to respond to both tactile and visual sensory inputs. Furthermore, they also found evidence for more robust multisensory enhancement effects in the supragranular layers as compared to the infragranular layers or interneuron clusters. A noteworthy finding discussed in their results describes similarities between MSI (at the synaptic level) in the RL and MSI in the SC of cats, specifically how bimodal integration leads to enhancement in both RL and SC as compared to suppression in primary cortices (Olcese, Iurilli, & Medini, 2013).
Given the findings summarized above, it follows that damage to certain cortical struc-tures (especially the AES in the association cortex), can impair MSI capacities of the SC. This is exactly what was reported by Wilkinson and colleagues (1996) when they temporarily deactivated the AES in cats by injecting lidocaine, and tested their performance on mul-tisensory stimuli (Wilkinson et al., 1996). SC response to unimodal stimuli remain largely unaffected, however, the neurons are no longer able to integrate information from multiple modalities effectively (Wilkinson et al., 1996; Stein et al., 2000).
We do not yet know why inputs from the association cortex are so essential to effective MSI in the SC. However, given the conclusions drawn above, we could probably assume that deactivating the association cortex could somehow impact the RFs of multisensory neurons in the SC. As mentioned before in section 2.1.1, research on RF characteristics of the SC require invasive recordings, which would be difficult in humans. However, there are studies examining the effects of masked (or inhibited) cortical inputs on RF characteristics of cells in animals, that we could deduce from.
For example, one study examined the effects of cortical cooling of area 3b (caudal cuta-neous field, area 1) on RF properties in a symmetrical region of the contralateral hemisphere of flying foxes and macaque monkeys. These researchers were interested in examining the
role of interhemispheric connections in determining neuronal RF properties in the cortex (Clarey, Tweedale, & Calford, 1992). They found that cortical cooling of area 3b introduced changes in neuronal RF properties of the mirror region in the contralateral hemisphere. Specifically, the neurons in these regions showed RF expansion and response enhancement within the original RF (see figure 2.2). Moreover, these effects were found to reverse when the cooling process lasted over 15 minutes, and the effects were found to diminish in inten-sity or disappear altogether if the cooling cycles were repeated multiple times. When the cortex was rewarmed, the expanded RFs of the opposite hemisphere were found to contract, and their responsiveness restored, to initial levels. The researchers justify this finding as a consequence of altered balance between excitatory and inhibitory inputs to the contralateral hemisphere’s neurons that determine its RF size and responsiveness. They also describe the reversal of the effect (with extended cooling period or repeated cooling cycles) as a sign of synaptic plasticity due to changes at the level of the synapse (Clarey et al., 1992). Moreover, they suggest a role for inhibitory interneurons containing GABA (the primary inhibitory neurotransmitter in the cortex) in the suppression of unwarranted inputs to a cell, thereby determining its response characteristics. In the event of cortical cooling, there is a decline in activity of excitatory collicular inputs to inhibitory interneurons that are unable to regulate unnecessary inputs to a cell, ultimately leading to RF expansion of those cells (Clarey et al., 1992).
Figure 2.2.: RF expansion in contralateral hemisphere due to cortical cooling of area 3b. The lighter bar represents initial (pre-cooling) values and the dark bar represents post-cooling values derived from seven flying foxes (FF). The light/dark bar pairs on the x-axis represents data from each FF pre- and post-cooling. The Y-axis scale reflects the maximum change in RF size across the 2 conditions. Most recordings in this figure are multiunit, with the exception of two (marked by a ‘+’) which are isolated. The data from two FF, presented to the extreme left differed in the scale of their RF sizes and are presented separately. As seen from this plot, cortical cooling of a symmetrical region in the contralateral hemisphere leads to RF expansion. Figure retrieved from Clarey et al., 1992.
2.2 Human studies in MSI
We have evidence for the role of SC in MSI in animals. However, further research is required to translate these findings to humans. Specifically, do multisensory neurons exist in human SC that are responsible for MSI?
Maravita and colleagues (2008) attempted to answer this question by delving into the phenomenon of redundant signals effect (RSE), put forth by Miller (1982). The premise of the RSE is that signals presented through two modality channels (redundant signal trials) will lead to faster responses than when stimuli is presented through a single modality
chan-nel (single signal trials) (Miller, 1982). Therefore, shorter latencies (and therefore quicker responses) are supposed for redundant audiovisual stimuli than for unimodal ones (Maravita, Bolognini, Bricolo, Marzi, & Savazzi, 2008). They further speculate that the enhancements take place in the SC. They test this assumption by pairing auditory stimuli with red (which SC neurons respond to) or one of purple or blue visual stimuli (that the SC neurons do no respond to) to determine which of the above pairings elicit shorter reaction times. Moreover, they also test the principle of spatial and temporal correspondence in audio-visual stimulus presentations to see if they hold true for humans, as in cats. In line with their expectations, they found quicker reactions to sounds paired with spatially concordant visual stimuli in red, which would be encoded by SC neurons and no such enhancement effects for stimuli presented in blue or purple, which do not project significantly onto the SC (Maravita et al., 2008).
Given these findings, one can deduce the role of SC neurons in MSI even in humans. Since our focus in this dissertation is on the intervening effects of TMS in particular, it would be interesting to note the effects it has on SC neurons. In section 1.4, we reviewed how low-frequency, short-duration TMS can suppress cortical activity. Furthermore, in section 2.1.2, we established the importance of specific cortical inputs to SC multisensory neurons in op-timal MSI functioning. Therefore, we can assume that functionally disrupting areas of the association cortex (especially the AES) temporarily via TMS could impair MSI capacities in humans due to its effect on SC multisensory neurons. This could be put to test using behavioural measures like the RSE in responding to audio-visual stimuli, following TMS stimulation of the association cortex. If the RSE reflects changes as a result of TMS applica-tion to AES, it could point towards the involvement of the SC in MSI in humans. However, if changes are present, we cannot be certain if they are due to alterations in RF properties of SC multisensory neurons (as deduced in section 2.1.1) or due to another mechnanism not accounted for; further research would be needed to investigate that hypothesis.
From the studies reviewed so far, we can conclude that (1) parts of the association cortex (such as AES) can be temporarily deactivated which in turn suppress inputs to the SC, (2)
inputs from the the association cortex (and especially the AES) are essential for MSI in the SC and (3) RFs of the multisensory SC neurons are implicated in MSI.
Figure 2.3.: Schematic representation of the research focus of the present chapter. Exploring the links between deactivation of important association areas necessary for integration and SC multisensory RF char-acteristics.
To summarize the findings in this chapter, we learned that: (1) RF contraction of SC multisensory neurons and the alignment of stimuli to modality-specific RFs in the SC multi-sensory neurons drive the process of MSI (Wallace et al., 1997; Stein et al., 2009). (2) Inputs from association areas such as AES and rLS are necessary for MSI in the SC (Stein et al., 2009; Ghazanfar & Schroeder, 2006). (3) There is some evidence pointing to the role of SC in MSI functions even in humans, as inferred through the redundant signals effect (RSE) (Maravita et al., 2008). (4) There is also evidence for temporary expansion of the RFs in areas of interest through functional deactivation of target areas (Clarey et al., 1992). Based on the abovementioned summary, we can hypothesize that by temporarily deactivating spe-cific parts of the association cortex (such as the AES) via TMS, we can influence the MSI capacities of SC neurons (possibly by influencing the size of RFs in SC neurons). Specifi-cally, the inhibited inputs from the cortex to the SC could possibly lead to RF expansion of SC multisensory neurons, which as we established in section 2.1.1 was detrimental to MSI. Whether there is a mediating role of inhibitory interneurons in this circuit, we cannot be certain of, yet it is likely they may be involved in regulating the size of SC neuronal RFs.
3. Effects of TMS on Oscillatory Activity
We were introduced to the role of oscillations in MSI in section 1.2. In this chapter, we will delve into the specifics of the nature of connections and oscillatory mechanisms underlying MSI. Specifically, are MSI processes a consequence of feedforward or feedback mechanisms, or a combination of the two? Through what frequency bands do these feedforward and feedback connections, implicated in MSI, primarily operate? How does TMS affect these connections involved in MSI?
3.1 Top-down and bottom-up processing
Top-down processing refers to task-driven processing of input stimuli, where the internal knowledge-based model defines the processing of incoming sensory stimuli. On the other hand, bottom-up processing is driven by salient features in the incoming stimulus, where the properties of the sensory input define and shape the internal model (Talsma, 2015; Buschman & Miller, 2007). Top-down connections are also referred to as feedback or back-ward connections, due to updating of our sensory processing capacities based on predictions by our internal models and bottom-up connections are otherwise known as feedforward con-nections because the sensory inputs have the power to influence our representations of the environment (Talsma, 2015).
How is this relevant for MSI?
Some accounts posit equal involvement of both bottom-up and top-down processes in MSI. The role of each type of process is partly determined by salience of the unisensory component stimuli. If the component stimuli are salient by themselves, then integration mainly occurs by way of bottom-up processes, especially if the stimuli are largely consistent in space and time. If the stimuli are mismatched at the time of presentation, bottom-up processes come into play to resolve integration issues early in the processing stream.
Conversely, in the presence of weak unisensory stimuli, top-down mechanisms are called into action more strongly to support integration (Talsma, Senkowski, Salvador, & Woldorff, 2010).
The timing of integration is also a topic of debate, with some research suggesting that integration takes place before complete processing of the component stimuli has taken place. Such research insinuates that sensory stimuli in any one modality influences processing of stimuli in the other modality, without the need for a new mental representation(Talsma, 2015).
The response enhancement effect of multisensory stimuli can also be explained through top-down and bottom-up processes. Van der Burg and colleagues (2008) found a facilitatory effect of task-irrelevant sounds in detection of visual stimuli in a visual search task, which they labeled the pip and pop effect (Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008) (see figure 3.1). They explain this facilitation as a consequence of heightened attention drawn by the multisensory stimuli. In a later study, the same authors found an early ERP effect corresponding to sound-induced improved performance. Moreover, they also found an ERP component that was observed everytime a visual stimulus was paired with a sound. This specific ERP component, the N2pc, also reflects automatic bottom-up attention related mechanisms which hint at the involvement of bottom-up processes in MSI (Luck & Hillyard, 1994; Van der Burg, Talsma, Olivers, Hickey, & Theeuwes, 2011).
Figure 3.1.: Mean correct reaction time (top) in seconds and mean error-rate (bottom) in percentage as a function of set size and presence/absence of auditory tone (results from experiment I of Van der Burg et al. (2008) study). The numbers above the lines represent the search slope values (in ms/item). The error bars represent the .95 confidence intervals for within-subject designs and the confidence intervals are those for the set size interaction effects. The figure shows the facilitation effect of task-irrelevant sounds on a visual search task. Figure retrieved from Van der Burg et al., 2008.
Studies involving simple audio-visual stimuli usually rely more heavily on bottom-up rather than top-down processes for effective MSI (Talsma, 2015). To support this claim, Giard and Peronnet (1996) found electrophysiological scalp activity corresponding to audio-visual integration as early as 40 ms post stimulus onset in the audio-visual cortex during an object
recognition task, reflecting bottom up processing. Previous electrophysiological studies have shown the earliest activity in the primary visual cortex have been recorded at 45 to 60 ms post stimulus onset. Moreover, the same researchers found audio-visual integration activity to coincide with the emergence of the C1 component of the VEP that is associated with activity reflective of the first impulse to the visual cortex. Such coinciding temporal activation patterns suggests the recruitment of feedforward connections in multisensory processes (Foxe & Schroeder, 2005).
The reliance on any one process over another is further mediated by the role of attention. More complicated tasks requiring greater attention, such as matching visual cues to speech stimuli as observed in the McGurk phenomenon are more likely to rely on top-down pro-cessing mechanisms. Attention interacts uniquely with MSI in that tasks requiring greater attention are more likely to use top-down mechanisms for integration vs. tasks requiring lesser attention that are more likely to employ bottom-up processes (Talsma, 2015).
Additionally, some intercranial studies also provide evidence for sound-induced phase resetting of activity in the visual cortex, similar to what we described in section 1.2 (Mercier et al., 2013). An fMRI study investigating similar questions further claims that in the context of such cross-modal activation, the sounds are processed as always through the auditory pathways and the resulting representations trigger changes in visual cortex activity only through feedback mechanisms. In fact, the visual cortex is found to be sensitive to different kinds of sound stimuli, as evidenced through different activation patterns to the different kinds of sounds (Vetter, Smith, & Muckli, 2014). Furthermore, some anatomical studies posit separate pathways for thalamacortical inputs based on modality; therefore cross-modal influence on the primary cortical areas implies the involvement of top-down processes from higher cortical areas (Konen & Haggard, 2014). Feedback mechanisms are therefore likely implemented in sound-induced visual cortex activation. Moreover, these researchers also found similar activation patterns in multisensory convergence areas, such as the precuneus and superior temporal sulcus which implies that the visual cortex activation is in fact the result of feedback inputs from these multisensory areas (Vetter et al., 2014).
The visual enhancement of touch (VET) effect, where viewing an of the body improves tactile processing as compared to viewing a neutral object, follows the same principle. VET effect is observed when the vision of the body provides no beneficial cues to tactile processing but is still found to improve performance. This improvement cannot be readily explained through multisensory feedforward processes, but is better understood through a feedback account of multisensory interactions. The researchers propose that VET could be the result of visual neural signals from early visual areas being passed on to multisensory parietal areas which then have feedback connections with early somatosensory cortex, finally leading to enhanced tactile performance (Konen & Haggard, 2014).
To summarize, MSI is a combination of bottom-up and top-down factors depending on task demands and stimulus saliency, amongst others. Simple, audio-visual stimuli are more likely to recruit bottom-up processes for integration, especially if each of the stimuli are sufficiently salient or incongruent. On the other hand, weak stimuli pairs or multisensory stimuli drawing heavily on attentional resources employ top-down processes to a greater extent. In addition, cross-modal influences on early cortices are also most likely expained through feedback connections.
3.2 Oscillatory activity in feedforward and feedback processing
Feedforward and feedback connections are distinct in their laminar profiles. Feedforward connections usually have their origins in supragranular layers and target the granular layer; feedback layers on the other hand avoid targeting the supragranular layers and favour origins in the infragranular layers. The visual cortex is hierarchically organized on the basis of these principles (Bastos et al., 2015).
Bastos et al. (2015) tested this hypothesis by collecting intercranial data from ECoG grids placed on macaque monkeys. The ECoG grids were placed over visual areas and they were able to record activity from both superficial and deep layers. They determined the interareal synchronization in frequency bands on the basis of a set coherence metric. To test whether a connection was feedforward or feedback, the researchers used a retrograde
tracer and assessed the resulting connections on the basis of this metric1 - where the level of
supragranular layer activity indicated the likelihood of feedforward/feedback projections. To relate spectral activity to any one process (feedforward or feedback) they adopted an index2
which suggested whether a particular frequency band conveyed feedforward or feedback information. Their results were in line with their hypotheses, i.e. feedforward networks were mediated by theta and gamma-band activity whereas feedback networks, by beta-band. (Bastos et al., 2015). In further support of these findings, some research has reported enhanced bottom-up processing fueled by gamma-band synchronization in attentional tasks, whereas top-down attention increased beta-band synchronization between select brain areas in the same task (Buschman & Miller, 2007). e.g.: In a visual search task performed by monkeys, stimulus driven attention increased activity in the gamma-frequency band whereas endogenously-driven decisions were accompanied by greater beta-band activity (Pesaran, Nelson, & Andersen, 2008). Moreover, some research has reported stimulus-induced gamma activity in superficial layers of the visual cortex, and alpha/beta dominated activity, in response to a stimulus, in the deeper layers of the cortex (Engel & Fries, 2010) Such a distinction in spectral band dominance and laminar organizations, mediating feedforward and feedback networks, could underlie predictive coding mechanisms (Bastos et al., 2015).
3.3 Role of TMS in multisensory oscillatory operations
To review this chapter, we learned that MSI processes are often an amalgamation of feed-forward and feedback connections. Simple integration activities are often handled through feedforward mechanisms early in the processing stream whereas more complicated integra-tion tasks, requiring effort and attenintegra-tion, rely more heavily on feedback mechanisms. We also learned that feedforward and feedback connections are determined by their
hierarchi-1SLN metric: For feedforward projections, the proportion of supragranular labeled neurons (SLN) as com-pared to supragranular plus infragranular labeled neurons is high and SLN is high. For feedback projections SLN is low.
2The Directed Influence Asymmetry index (DAI) calculated as follows: GCI(source → target)−GCI(target → source)
GCI(source → target)+GCI(target → source) where GCI = Granger-Causality Influence.
cal laminar organizations and operate through different frequency bands, in communicating with target areas.
This section will now focus on the effects of TMS on feedforward and feedback multisen-sory connections. Specifically, what are some of the implications for these kind of connections in the presence of TMS stimulation? To answer this question, we will focus on a study in-vestigating the VET phenomenon (see section 3.1) in the presence of TMS stimulation.
A study led by Konen and Haggard (2014) required participants to make tactile discrim-inations following visual stimulus presentation and possible TMS application. The visual stimulus either consisted of a view of one’s own hand or a neutral object. TMS pulses were applied shortly after visual stimulus presentation to one of the following sites: the Sensorimo-tor Hand Area (SMHA), anterior Intraparietal Sulcus (aIPS) or the Posterior Intraparietal Sulcus (pIPS). The control condition subjects did not receive TMS application.
The researchers found a consistent VET effect (improvement in tactile performance by viewing one’s own hand vs. a neutral object) across all conditions except for when the TMS was applied over the aIPS (see figure 3.2). The temporary disruption of aIPS immediately after viewing one’s own hand diluted the enhancement effect observed on viewing a hand over a neutral object. As mentioned in section 3.2, the VET effect is proposed to be a consequence of top-down processing and represents feedback signals from multisensory parietal areas to early somatosensory areas (Konen & Haggard, 2014). The researchers further speculate, based on their findings, that the aIPS is implicated in providing feedback signals to primary somatosensory areas, driving the VET effect. Their study stands as evidence for changes in performance in unisensory areas due to impaired multisensory areas (Konen & Haggard, 2014).
Briefly described in section 1.4 was an overview of TMS-induced disruption of the PPC also leading to disruption of visual-tactile multisensory integration. In this study, the re-searchers used a combination of visual and tactile stimulation presented concurrently. The tactile stimulation (consisting of a light mechanical touch) was delivered in synchrony with visual stimuli which could be congruent to the touch, incongruent to the touch or devoid of information (what they called the no-pointer condition). Subject’s task was to identify the
Figure 3.2.: Effect of TMS on VET. The x-axis represents the different conditions, and the Y-axis repre-sents mean accuracy in percentages. The error bars represent standard errors and the asterisks reprerepre-sents significance in the difference between conditions. In all other conditions except for stimulation of aIPS, per-ceiving one’s own hand improved tactile discrimination performance. TMS application over aIPS disrupted the VET effect. Figure retrieved from Konen & Haggard, 2014.
finger receiving the mechanical touch. During this task, single-pulse TMS was either applied to the PPC, a control site (posterior and ventral to the PPC where TMS was otherwise applied) or not at all (Pasalar et al., 2010).
In conditions where TMS was absent or delivered exclusively to the control site, a mul-tisensory enhancement effect was observed, as evidenced by the shorter RTs (but intact accuracies) in the congruent condition. Performance suffered in the incongruent condition as compared to the congruent condition but still exceeded performance to the no-pointer condition. In conditions where TMS was applied to the PPC, the multisensory enhance-ment effect disappeared. Furthermore, performance on the no-pointer trials was found to be similar for TMS vs. no TMS condition. The researchers use these results to argue for the mediating role of PPC in visual-tactile MSI. A follow-up MRI study was performed to local-ize areas activated during visual and tactile stimulation. The PPC TMS site was found to
be adjacent to areas involved in visual and tactile perception, lending further support to the multisensory capacities of the PPC. In contrast to the previous study however, this study reveals the disruptive effects of TMS application to feedforward multisensory connections (Konen & Haggard, 2014)
TMS has also been found to weaken neuronal responses to sensory inputs (Pasalar et al., 2010). In section 1.2, we reviewed how MSI was subserved by oscillatory processes and in sections 1.3 and 1.4, we reviewed how TMS can entrain oscillatory activity through phase resetting. Pasalar and colleagues (2010) further vouch for the entrainment effects of TMS that may potentially disrupt MSI activities. Based on the conclusions presented in the current chapter in addition to what we already know from the previous chapters, we could infer that TMS affects the oscillatory processes subserved in both feedback and feedforward multisensory connections, depending upon the area of stimulation.
Figure 3.3.: Schematic representation of the research focus of the current chapter. MSI is a function of both feedforward and feedback processes which are subserved by distinct oscillatory channels (Pesaran et al, 2008; Talsma, 2015)Therefore, here we explore if deactivation of target areas that provide feedforward or feedback inputs to a region would also result in a change in oscillatory architecture of target areas.
We could test the conclusion drawn above by relating the findings of Bastos et al.(2015), Konen and Haggard (2014) and Pasalar and colleagues (2010). E.g.: In the case of VET, where the aIPS is posited to exert a causal influence in feedback connections to the early somatosensory areas, we could determine the reliance on feedback activity by investigating spectral band coherence. If the effects are indeed due to feedback influences, higher
inter-areal beta-band synchronization should be observed in target areas. This hypothesis can be further tested by cortical cooling of the aIPS to render it temporarily inactive or through TMS intervention of the mediating area. If the area is functionally disrupted through such procedures, then the feedback influences should decline, resulting in reduced beta-band syn-chronization in target areas.
4. Predictive Coding and MSI
The brain continuously updates its internal model based on the sensory regularities in our environment - not merely in terms of representational content but also in terms of their temporal characteristics. It uses the knowledge stored in this model to predict events and their onset times, as they occur in the environment. This ability of the brain is accounted for by the predictive coding and predictive timing hypothesis respectively - which state that the brain is sensitive to causal relations in sensory inputs, that are often missed by our natural senses (Arnal & Giraud, 2012).
According to Arnal and Giraud (2012), predictive coding is ”the idea that the brain generates hypotheses about the possible causes of forthcoming sensory events and that these hypotheses are compared with incoming sensory information.”
The predictive coding hypothesis is explained by way of Bayesian principles, according to which the brain makes predictions based on probabilistic estimates derived from the environ-ment. The brain stores these predictions as stochastic models that are continually updated based on novel sensory inputs. The updated predictions (or ”priors”) are communicated to the lower sensory regions to modulate the stream of incoming sensory information (Talsma, 2015). These predictive mechanisms operate by influencing activity in different frequency channels. E.g.: predicting the occurrence of an event affects phase of (2-8 Hz) oscillatory ac-tivity before the onset of the stimulus i.e. a predictive coherence of delta-theta bands during conducive phase alignments enhances stimulus detection (Arnal & Giraud, 2012). Atten-tion further interacts with these probabilistic computaAtten-tions, enhancing neural responses to stimuli that fall within its limits.
Based on this understanding of predictive coding, some of the questions that we will raise in this chapter revolve around the mechanisms underlying predictive behaviours in a multisensory context, oscillatory entrainment involved in predictive coding mechanisms, consequences of violations of predictions and effects of TMS intervention on predictive coding.
4.1 Predictive coding and neural oscillations
As described in the introduction of this chapter, predictive behaviours are influenced by delta-theta alignment and resetting of ongoing oscillatory phase in the unisensory cortex. The researchers cite the example of auditory phase reset in the presence of a fast speaker. The fast speech entrains the rhythmic activity in the auditory cortex to match the speech rate. Once this pattern is established, the brain makes probabilistic computations, wherein the regularities in speech rate tune the neuronal excitation in auditory cortex in order to ex-tract relevant information from speech. This tuning, as described above, is largely achieved through a phase reset of ongoing oscillations in the auditory cortex. In a cross-modal con-text, visual and somatosensory stimuli affect auditory cortical processing in a similar fashion, resetting the phase of low-frequency activity to predicted sound modulations (Arnal & Gi-raud, 2012). This cross-modal reset is most distinct in noisy environments, where visual cues support auditory processing of speech. The amount of information contained in visual cues is conveyed in the alignment of delta and theta phases, which most likely enhances auditory speech detection. The researchers also argue for the automatic nature of predictive behaviours (Arnal & Giraud, 2012).
When it comes to predicting the timing of an event, alpha oscillations have been known to play a critical role - specifically, alpha band desynchronization in the cortex of interest has been linked to the onset of a predicted activity (Arnal & Giraud, 2012; Rohenkohl & Nobre, 2011). This desynchronization has found to peak in the interval immediately preceding stimulus onset; moreover, this desynchronization is often coupled with enhanced processing for target stimuli, as evidenced by their ERPs. Alpha band oscillations are primarily implicated in inhibitory mechanisms and are known to filter sensory input in the order of cognitive relevance. Therefore, it could be inferred that a desynchronization of alpha activity releases the inhibitory effect it exerts When a stimulus is marked by temporal irregularities (Arnal & Giraud, 2012).
Beta oscillations (12-30 Hz) also play a role in predictive behaviours, especially with respect to motor actions. Just like alpha-band desynchronization prior to predictable
stim-ulus onset, beta-band desynchronization is also observed, just not to the same levels as alpha-band desynchronization. Specifically, beta-band desynchronization is more transient, unrelated to the stimulus rate and undergoes resynchronization as per the rate of the stim-ulus (beta rebound) following inhibition. Beta-band power is also found to align with the phase of delta-theta oscillations at the time of onset of predicted stimulus, reflecting its reg-ulatory role in synchronizing low and high frequency activity; this alignment is also found to to enhance predictable stimulus processing (Saleh, Reimer, Penn, Ojakangas, & Hatsopou-los, 2010). Therefore, predictable activity is generally a consequence of alignment of low-and mid- frequency activity, coupled with alpha-blow-and desynchronization (Arnal & Giraud, 2012). The delta-theta alignment is largely stimulus driven and therefore feedforward in nature while the regulatory predictive mechanisms driving delta-theta and beta-band co-ordination involve feedback inputs from higher order processes. Engel and Fries (2010) scrutinized the role of beta-band activity in various functions and found that beta activity largely reflected cognitive status quo. Therefore, beta-band activity increased in predictable, unchanging contexts whereas a disruption in status quo due to unpredictable stimuli led to a decline in beta activity and an increase in gamma-band activity, whose role in predictive coding shall be discussed next.
Gamma oscillations (30+ Hz) also come into play while encoding for predictable vs. un-predictable stimuli. Specifically, gamma-band processes are involved in comparing incoming stimuli to internal representational models stored in memory, based on our prior experiences, and react in accordance with the results of this comparison. Such a comparison process is usually followed by a stimulus-appropriate response, updating of memory or reallocation of attention, where the role of gamma activity is implicated (Herrmann, Munk, & Engel, 2004). According to Arnal and Giraud (2012) gamma activity is found to increase with prediction errors and decrease with confirmation of predicted activity such as a correct repetition or omission of predicted activity. Unexpected occurrences or omissions of predicted activity leads to an increase in gamma-activity. Therefore, an increase in gamma-activity is found to correlate with sensory surprise or prediction error.
Beta-band frequency works in accordance with gamma-band activity. As described ear-lier, processing of predicted stimulus is marked by pre-stimulus beta-band desynchronization, followed by beta-band rebound. Beta-band activity is further tied to the gamma-band in that the beta-band rebound is a function of gamma enhancement in response to prediction violation. Therefore, the extent of beta activity post desynchronization is contingent on gamma-band enhancement. Beta-activity signals feedback processing following recognition of prediction error (Arnal & Giraud, 2012)
4.2 Feedforward and feedback connections in predictive coding
Based on the conclusions drawn above, Arnal and Giraud (2012) infer that prediction errors are carried through in a feedforward manner through the gamma-band whereas feed-back predictions are transmitted through activity in the beta-band. This inference stands in agreement to what we reviewed in section 3.2. To reason for the differential involvement of frequency channels in feedforward and feedback processes, the speed of the processes were compared. Predictions are revised slower than prediction errors are communicated. The rapid pace of the process requires a fast frequency channel like the gamma-band to communicate prediction errors. Conversely, the accumulation of prediction errors compose a low-pass filter which necessitate a low-frequency channel like the beta-band to facilitate feedback interactions (Bastos et al., 2015; Friston et al., 2008).
Early support for these conclusions comes from a study of Rao and Ballard (1999) where they presented a unique hierarchical model to describe prediction errors and the specific contributions of frequency channels involved. This model finds its basis in the visual pro-cessing neurons of the visual cortex. The neurons in the cortex are known to respond to stimuli within the limits of their receptive fields. With increasing distance from these lim-its, the response intensity declines and gradually disappears. The researchers describe this phenomenon as ’end-stopping’ i.e. the drop in neuronal firing to a stimulus with increasing distance from the neuron’s classical RFs. They label these effects as ’extra-classical’ RF
effects, which they observe in a number of visual areas such as V1, V2, V4 and MT (Rao & Ballard, 1999).
In their Hierarchical network for predictive coding model, at each level of the hierarchy, feedback loops communicate the predictions of neural activity to the lower-levels and feed-forward inputs communicate the prediction errors to the higher-levels (see figure 4.1). These prediction errors essentially consist of the residual neural activity not accounted for by the feedback loops. This idea is in accordance with Arnal and Giraud’s (2012) theory about predictions and prediction errors propagated through feedback and feedforward circuits re-spectively. Rao and Ballard (1999) attempt to explain the extra-classical RF effects through the hierarchical network model. According to these researchers, predicted neural activity in the visual cortex is communicated through feedback circuits from higher areas to lower areas (e.g. V2 to V1) and conversely, the residual neural activity in V1, not predicted by V2, is communicated back to V2 through feedforward circuits (Bastos et al., 2015).
Figure 4.1.: Schematic representation of the hierarchical network model for predictive coding by Rao and Ballard (1999). At each level of the hierarchy, feedback channels communicate predictions of neural responses to the lower levels and feedforward channels relay information about the residual neural activity, not accounted for by the predictions, to the higher levels. The Predictive Estimator (PE) at each level uses this information (prediction error) to update its next prediction. Figure retrieved from Rao & Ballard, 1999.
They test this model by training the predictive estimators distributed across three hierar-chical levels, on a set of natural images - they draw a comparison between the performance of these predictors to that of the visual neurons, stating that the neurons estimate their responses based on the statistical properties of the images. Given an input image, the model maximizes its response accuracy and minimizes its error by narrowing down on the closest predicted response to the stimulus; this is achieved by subtracting predicted activity from the neuronal response (residual activity) at each level and passing it on to the next level. The neurons at the next level extract relevant information from this residual activity, take the prediction error into account to finally improve its prediction.
The neurons carrying information about prediction errors are compared to the visual cortical neurons showing extra-classical RF effects in response to a stimulus. The source of these neurons estimated to be in layers 2/3 of the cortex that project to higher visual areas (Friston et al., 2008). Therefore, these neurons fire most in the event of prediction errors and are inhibited under normal conditions. Feedback circuits contribute to end-stopping mechanisms in the RFs of neurons i.e. feedback from higher areas help shape the margins of RFs in lower areas. If feedback from higher areas is suppressed or deactivated, layer 2/3 neurons are released from their inhibition, firing vigorously even in the absence of prediction errors. For example, deactivation of higher visual cortical areas in anesthetized monkeys results in otherwise suppressed response patterns to surround stimuli (Rao & Ballard, 1999; Hup´e, James, Payne, Lomber, et al., 1998). This model, along with conclusions derived from studying visual neurons in monkeys, foster our understanding of the role of feedback and feedforward connections in determining sensory predictions, violation of these predictions, shaping of RF margins and extra-classical RF effects.
To review this section thus far, we have evidence for delta-theta coherence in the occur-rence of predictable stimuli, alpha and beta desynchronization preceding predictable stimulus onset, beta-band rebound following beta-desynchronization which in turn is contingent upon gamma-band enhancement in reaction to violation of prediction. We also have evidence for propagation of feedforward and feedback interactions through gamma-band, and beta-band activity, respectively (see sections 3.2 and 4.2). Since bottom-up processes originate in the
superficial layers and are mediated by gamma-activity and top-down processes originate in the deeper layers, mediated by alpha-beta activity, it falls in line with our expectation that prediction errors fed forward by the gamma channel originate in the superficial layers and predictions are transmitted to the deeper layers of the lower sensory areas through the beta channels.
As the next step, we will try to integrate MSI into the realm of these findings. As noted above, expected and unexpected stimuli entrain different oscillatory patterns in the cortex. e.g.: Cats were presented with expected and unexpected stimuli, while activity in their primary visual areas and visual association cortex were scrutinized. Presentation of an expected stimulus precipitated synchronization in the low-frequency band (4-12 Hz) whereas an unexpected stimulus lead to gamma-band coherence (von Stein, Chiang, & K¨onig, 2000), which is in line with our summary above. Such effects are not limited to stimuli presented in the same modality but are observed with cross-modality stimuli as well. Arnal and colleagues (2011) set out to explore similar questions using audiovisual stimuli in an MEG study on humans. In their study, when subjects were presented with visual speech cues that correctly predicted auditory speech signals (congruent stimuli), activity was observed in the 34 Hz frequency band in higher-order speech areas. On the other hand, when there was a discrepancy between the speech and visual signals, they found coupling between the low-beta (14 - 15 Hz) and high gamma activity (60 - 80 Hz) in multisensory areas of the superior temporal sulcus (STS) (Arnal, Wyart, & Giraud, 2011). The high gamma activity observed in higher-order areas is evidence for the role of feedback activity with predicted or in this case, congruent stimuli, even in a multisensory context. Moreover, the beta-gamma coupling observed for incongruent stimuli conditions possibly signifies beta-band rebound, contingent on the magnitude of gamma activity (see section 4.1).
4.3 Divisive Normalization and Phase Resetting in MSI
In section 1.2, we briefly introduced some network mechanisms supporting MSI, which we will elaborate on in this section. According to Atteveldt et al. (2014), two canonical
