Tilburg University
Are alpha oscillations instrumental in multisensory synchrony perception?
Bastiaansen, Marcel; Berberyan, Hermine; Stekelenburg, Jeroen J.; Mathijs Schoffelen, Jan;
Vroomen, Jean
Published in:
Brain Research
DOI:
10.1016/j.brainres.2020.146744
Publication date:
2020
Document Version
Publisher's PDF, also known as Version of record
Link to publication in Tilburg University Research Portal
Citation for published version (APA):
Bastiaansen, M., Berberyan, H., Stekelenburg, J. J., Mathijs Schoffelen, J., & Vroomen, J. (2020). Are alpha
oscillations instrumental in multisensory synchrony perception? Brain Research, 1734, [146744].
https://doi.org/10.1016/j.brainres.2020.146744
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
Contents lists available atScienceDirect
Brain Research
journal homepage:www.elsevier.com/locate/brainres
Research report
Are alpha oscillations instrumental in multisensory synchrony perception?
Marcel Bastiaansen
a,b,⁎, Hermine Berberyan
a,c, Jeroen J. Stekelenburg
a, Jan Mathijs Scho
ffelen
d,
Jean Vroomen
aaDepartment of Cognitive Neuropsychology, Tilburg School of Social and Behavioural Sciences, Tilburg, the Netherlands bBreda University of Applied Sciences, Academy for Leisure, Breda, the Netherlands
cUniversity of Groningen, Institute of Artificial Intelligence and Cognitive Engineering, Groningen, the Netherlands dRadboud University, Donders Institute for Brain, Cognition and Behaviour, Nijmegen, the Netherlands
H I G H L I G H T S
•
We examine the role of ongoing EEG alpha oscillations in multimodal synchrony perception.•
Individual alpha peak frequency correlates with accuracy of simultaneity judgements.•
Prestimulus alpha power was lower when simultaneity judgements were inaccurate.•
Prestimulus alpha phase did not significantly modulate simultaneity judgements.•
EEG alpha oscillations are instrumental in shaping multisensory perception. A R T I C L E I N F OKeywords:
Multimodal integration Simultaneity judgements EEG alpha oscillations Phase dependence Individual alpha frequency
A B S T R A C T
Different inputs from a multisensory object or event are often integrated into a coherent and unitary percept, despite differences in sensory formats, neural pathways, and processing times of the involved modalities. Presumably, multisensory integration occurs if the cross-modal inputs are presented within a certain window of temporal integration where inputs are perceived as being simultaneous. Here, we examine the role of ongoing neuronal alpha (i.e. 10-Hz) oscillations in multimodal synchrony perception.
While EEG was measured, participants performed a simultaneity judgement task with visual stimuli preceding auditory ones. At stimulus onset asynchronies (SOA’s) of 160–200 ms, simultaneity judgements were around 50%. For trials with these SOA’s, occipital alpha power was smaller preceding correct judgements, and the individual alpha frequency was correlated with the size of the temporal window of integration. In addition, simultaneity judgements were modulated as a function of oscillatory phase at 12.5 Hz, but the latter effect was only marginally significant.
These results support the notion that oscillatory neuronal activity in the alpha frequency range, which has been taken to shape perceptual cycles, is instrumental in multisensory perception.
1. Introduction
We perceive the world around us through different senses, each of which is receptive to different aspects of our environment. At the same time, these different modality inputs into our sensory organs have to be integrated into coherent and unitary percepts of objects and events, despite existing differences in sensory input formats, neural pathways, and processing times. The multisensory nature of perception suggests a sophisticated interplay between the different sensory systems. This has received much scientific attention (for reviews, see e.g.Alais et al.,
2010; Calvert et al., 2004; Chen and Vroomen, 2013; Stein and Stanford, 2008; Noppeney and Lee, 2018; Vroomen and Keetels, 2010). Determining whether e.g. auditory and visual signals come from one common or from two different sources is based on prior knowledge about the environment and by bottom-up information such as by spatial proximity and temporal simultaneity (Noppeney and Lee, 2018). Temporal simultaneity refers to the fact that when two sensory inputs come from one and the same multisensory source (say the lip move-ments and articulated sounds from a speaker), they are perceived as being simultaneous, despite the existence of sensory modality-specific
https://doi.org/10.1016/j.brainres.2020.146744
Received 17 November 2019; Received in revised form 21 February 2020; Accepted 26 February 2020
⁎Corresponding author at: Department of Cognitive Neuropsychology, Tilburg School of Social and Behavioural Sciences, Warandelaan 2, 5000 LE Tilburg, the
Netherlands.
E-mail address:bastiaansen4.m@buas.nl(M. Bastiaansen).
Available online 27 February 2020
0006-8993/ © 2020 Elsevier B.V. All rights reserved.
lags. Different neural mechanisms have been demonstrated to subserve multisensory temporal binding, such as the occurrence of a temporal window of integration, andflexible adaptation or recalibration to ex-isting or shifting time lags (for review, seeVroomen and Keetels, 2010). Using simultaneity judgements (SJ), temporal order judgements (TOJ), or related tasks, it has been established that crossmodal stimulus pairs are perceived to be simultaneous when presented within a temporal window of 20–80 ms for simple cross-modal stimuli such as auditory beeps and visualflashes (e.g.Keetels and Vroomen, 2005; Spence et al., 2001; Zampini et al., 2005a,b). This temporal window increases to a few hundred ms for more complex stimuli, such as audiovisual speech (e.g.Stekelenburg and Vroomen, 2007; Van Wassenhove et al., 2007; for review, seeVroomen and Keetels, 2010).
Although the notion of a temporal window of integration is widely accepted, its neurophysiological substrate remains unclear. Neuronal oscillations, reflecting fast periodic fluctuations in neuronal excit-ability, may be one candidate mechanism for implementing this window of integration. Notably, alpha oscillations, with frequencies of between 7 and 13 Hz have a periodicity of ~70 to 140 ms. This is in the same order of magnitude as the temporal windows for multisensory integration that are usually observed. Alpha oscillations are therefore considered to be a prime candidate for subserving multisensory tem-poral binding (Cecere et al., 2015).
Alpha oscillations have been proposed to be instrumental in unim-odal sensory perception, shaping so-called perceptual cycles (VanRullen, 2016). The notion of perceptual cycles implies that sensory processing is not continuous, but rather that perceptual input is being periodically‘sampled’ at a frequency of 7–13 Hz. For the visual mod-ality, this periodicity is reflected in the observations that visual detec-tion thresholds are lower, and visual percepdetec-tion is more accurate, at specific phases of the alpha oscillation than at others. For instance, the detection of visual flashes presented around perceptual threshold (Busch et al., 2009; Mathewson et al., 2009), saccadic reaction times to a peripheral target (Drewes and VanRullen, 2011), and the detection of a TMS-induced phosphene (Dugué et al., 2011) all depend on the phase of occipital alpha oscillations immediately preceding stimulus onset. Further, it has been shown that the temporal resolution of visual per-ception is dependent on the exact frequency of alpha oscillations (Samaha and Postle, 2015). This frequency can be modulated by task demands (Wutz et al., 2018). Thesefindings strongly suggest that visual perception is optimal at some phases of alpha oscillations (VanRullen et al., 2014). While most of the literature on perceptual cycles has fo-cused on the visual modality, evidence has been reported for perceptual cycles in the somatosensory modality as well (Ai and Ro, 2014; Baumgarten et al., 2015). In the auditory modality, results have been less clear. Obleser and colleagues (Strauß et al., 2015) observed phase differences in ongoing alpha power between correct and incorrect lex-ical decisions (word vs. non-word) immediately preceding stimulus presentation in a lexical decision task. However, other attempts at de-monstrating the existence of auditory perceptual cycles have been un-successful (seeVanRullen et al., 2014for review). Only when auditory cortex is being entrained by rhythmic stimuli, clear effects of the phase of the entrained rhythm on auditory perception have been reported (e.g. Henry and Obleser, 2012; Kayser et al., 2016; Schroeder and Lakatos, 2009; for review, seeThorne and Debener, 2014).
The above work on perceptual cycles deals exclusively with unim-odal sensory perception. A small number of studies have demonstrated that alpha oscillations influence cross-modal perception as well. For instance, a sound can reset the phase of rhythmic activity in visual cortex (Lakatos et al., 2009; Romei et al., 2012) and as a result influ-ence subsequent visual perception (Diederich et al., 2014; Fiebelkorn et al., 2011). Conversely, visual stimuli can reset auditory cortex rhythms (Kayser et al., 2008; Perrodin et al., 2015; for review, see Thorne and Debener, 2014).
More pertinent to our proposal that alpha oscillations are instru-mental in the temporal binding of auditory and visual stimuli, are two
recent studies that have examined different aspects of alpha oscillations during multisensory tasks. One of these studies (Cecere et al., 2015) has demonstrated a significant correlation between individual alpha peak frequency (IAF) and the temporal window of integration in the sound-induced double-flash illusion task. Most convincingly, this study ad-ditionally showed that when IAF was either slowed down or sped up through transcranial alternating current stimulation, the size of the temporal window of integration changed accordingly. In another study (Grabot et al., 2017), however, a temporal order judgement task did not yield any correlation between IAF and temporal window of integration. This suggests that the observed correlation may be dependent on the experimental paradigm employed (e.g. illusion-induction vs. temporal order detection). Note that in the latter study, correct order judgements around the point of subjective simultaneity were associated with lower prestimulus alpha power.
However, if we consider that unimodal perception is sensitive to the phase of ongoing alpha oscillations (as expressed in the perceptual cy-cles hypothesis;VanRullen, 2016), this notion may extend to multi-sensory temporal binding. One may argue that, rather than individual alpha peak frequency and prestimulus alpha power, it is the phase of prestimulus alpha oscillations that should be most instrumental in de-termining whether or not multisensory temporal binding takes place. In this view, under difficult discrimination conditions, one would expect perceived simultaneity of multimodal stimuli to be dependent on the stage of the perceptual cycle (as indexed by the phase of the alpha cycle). This is commensurate with claims that alpha phase is related to prioritizing information in time (for review, seeJensen et al., 2012). One direct test of the hypothesis that alpha phase is instrumental in multisensory temporal binding comes from the study of Van Wassen-hove and colleagues (Grabot et al., 2017). However, these authors did not observe systematic differences in prestimulus alpha phase as a function of participants’ temporal order judgements. Another study used simultaneity judgements, with stimulus onset asynchronies (SOAs) chosen such that they induced maximal perceptual uncertainty (Ikumi et al., 2019). In a task in which auditory stimuli preceded visual stimuli (AV condition), the authors found that the phase of prestimulus oscil-lations at around 13 Hz correlated with subjective simultaneity. This occurred in the absence of concurrent changes in EEG amplitude. Re-markably, when visual stimuli preceded the auditory ones (VA condi-tion), which is the more natural situation because light travels faster than sound, the authors did not observe any correlation between prestimulus alpha phase and perceived simultaneity. Although the re-sults from Ikumi et al. (2019)suggest that alpha oscillations are in-strumental in shaping temporal judgements, the fact that a relationship between prestimulus alpha phase and perceived simultaneity was only observed in the AV condition challenges the generalizability of their findings.
To summarize, the existing literature on the role of alpha oscilla-tions in the temporal binding of auditory and visual stimuli is at best inconclusive. We aim to elucidate this role by addressing three open empirical questions. We performed an EEG experiment in which par-ticipants performed SJ of visual and auditory stimuli presented with different stimulus onset asynchronies (SOAs), focusing specifically on visual-first (VA) stimulus pairs. SOAs of 160–200 ms were associated with maximal perceptual uncertainty, as indexed by roughly equal numbers of‘simultaneous’ and ‘not simultaneous’ judgements from the participants. One question we address is whether the phase of occipital alpha oscillations (as a proxy of the stage of the perceptual cycle) is instrumental in determining (or at least influencing) participants’ SJ for VA stimulus pairs (as opposed toIkumi et al. (2019), who only observed this effect for AV stimulus pairs)? A second question is whether the observed relationship between IAF and temporal window of integration is specific to illusion-induction tasks (as inCecere et al., 2015), or whether it can also be observed in a task that involves temporal jud-gements (as opposed toGrabot et al., 2017). And third, does the ob-served difference in prestimulus alpha power between correct and
M. Bastiaansen, et al. Brain Research 1734 (2020) 146744
incorrect temporal order judgements (as in Grabot et al., 2017) re-plicate in a SJ task?
2. Results
2.1. Behavioural data
We were mainly interested in the cognitive and perceptual processes around the PSS, because SJ around the PSS are most difficult, and therefore most likely to be influenced by the state of the sensory cor-tices at the time the first stimulus is presented. Therefore, for each participant we selected the SOA for which trials that yielded a ‘si-multaneous’ judgement (S trials) and trials with a ‘not si‘si-multaneous’ judgement (NS trials) were most evenly distributed. Additionally, we then verified whether the proportion of S and NS trials for this SOA was in between 0.3 and 0.7 (or 0.7 and 0.3). For 22 out of the 33 initial participants, these criteria were met. The EEG data from all other participants and all other SOAs were excluded from further analysis. Table 1 presents the selected SOA, and the distribution of S and NS judgements for the selected SOA for each participant.
Fig. 1shows the proportions of S judgements as a function of SOA. As can be seen fromFig. 1, for the 22 participants that met the selection criteria for the behavioural data, the simultaneity judgements at the three SOAs for which we collected large numbers of trials (160, 180 and 200 ms) are on average around the point of subjective simultaneity (a 0.5 proportion of S judgements).
2.2. Phase modulation of perceived simultaneity
The results of the phase binning analysis (Section 4.7.1) are sum-marized inFig. 2. For frequency bins outside of the alpha range (here
7.5 Hz and 17.5 Hz), we did not observe any clear relationships be-tween (binned) prestimulus phase and perceived simultaneity. At 10 Hz and at 15 Hz there are faint hints at clustering of simultaneity judge-ments around certain phase bins, but the effects are far from reaching significance. At 12.5 Hz however we did observe a clear sinusoidal pattern in the simultaneity judgements as a function of phase bin. Al-though this effect falls into the tail of the null distribution and borders on significance, the p-value is above the alpha level of 5% (p = 0.077). 2.3. Analysis of individual alpha frequency
Mean IAF across participants was 10.34 (SD 0.83, range = 9.3 Hz–11.8 Hz). The correlation between IAF and b1(the
parameter modelling the width of the temporal window of integration, cf.Section 4.7.2) was 0.44, p = 0.038 (Fig. 3). Since higher values for b1represent more narrow temporal windows of integration, the
cor-relation indicates that a higher IAF is associated with a higher sensi-tivity to detect AV asynchrony.
2.4. Time-frequency analysis of power
Fig. 4shows the time-frequency representation of power for S and NS trials (Fig. 4A) and their difference (Fig. 4B), respectively. Occipital alpha power was larger preceding the visual stimulus for S trials than for NS trials (cluster p-value = 0.039).
3. Discussion
In order to further elucidate the role of alpha oscillations (7–13 Hz) in the temporal binding of auditory and visual stimuli, participants performed a visual-auditory simultaneity judgement task. The point of subjective simultaneity occurred at SOAs of 160 to 200 ms. There, si-multaneity judgements were at chance level. For these SOAs, alpha power was lower for trials with ‘not simultaneous’ judgements (NS trials) than for trials with‘simultaneous’ judgements (S trials) preceding the onset of thefirst (i.e., the visual) stimulus. In addition, we found a significant positive relationship between individual alpha peak fre-quency (IAF) and the sensitivity to detect AV temporal differences. Finally, we observed a sinusoidal modulation of the proportion of S judgements as a function of phase bin around 12.5 Hz only. However, this effect failed to reach significance (p = 0.077). At other frequencies than 12.5 Hz there were no indications of a phase-dependency of si-multaneity judgements.
Table 1
Selected SOA, proportion of S and NS trials for this SOA, and inclusion or ex-clusion of participants based on behavioural selection criteria.
SOA with the most equally distributed responses
proportion of S/NS judgements at this SOA
Participant included in thefinal set?
160 47/53 yes 160 41/59 yes 160 50/50 yes 160 53/47 yes 160 50/50 yes 160 58/42 yes 160 40/60 yes 160 43/57 yes 160 51/49 yes 180 51/49 yes 180 51/49 yes 180 61/39 yes 180 48/52 yes 180 49/51 yes 200 47/53 yes 200 57/43 yes 200 60/40 yes 200 50/50 yes 200 54/46 yes 200 49/51 yes 200 67/33 yes 200 63/37 yes 160 78/22 no 160 28/72 no 160 9/91 no 180 77/23 no 200 73/27 no 200 89/11 no 200 86/14 no 200 89/11 no 200 70/30 no 200 71/29 no 180, 200 73/27 no
3.1. Does prestimulus alpha phase modulate simultaneity perception? Although the phase dependence of simultaneity judgements in vi-sual-first (VA) stimulus pairs is clearly observable in the data (Fig. 2), the effect was not significant. Thus our results are largely in line with those fromIkumi et al. (2019), who observed a similar phase depen-dence for auditory-first (AV) stimulus pairs, but failed to find an effect for VA pairs. It is interesting to note that the present effect is observed at 12.5 Hz. This closely corresponds with the frequency of 13 Hz at whichIkumi et al. (2019)observed their phase dependence in AV pairs. Taken together, the results of the two studies suggest that the phase of alpha oscillations is instrumental in multisensory simultaneity
perception, although this has been clearly demonstrated only for AV stimulus pairs. It is unclear why this asymmetry in the results for dif-ferent modality orders occurs. It may be related to the more generally established finding that simultaneity judgements are poorer for VA stimulus pairs (e.g.Cecere et al., 2017; Stevenson and Wallace, 2013). These poorer judgements may partly obscure the delicate relationship between those judgements on the one hand, and alpha phase on the other hand. Another reason underlying this asymmetry may be that VA is the most naturally occurring temporal asynchrony in real life (as light travels faster than sound). Therefore the brain may be better adapted to slight temporal differences in VA synchrony than in AV synchrony, resulting in a smaller influence of alpha phase on simultaneity judge-ments in VA stimulus pairs. In any case, taken together, the study by Ikumi et al. (2019)and the present one are compatible with the notion that oscillatory perceptual cycles at alpha frequencies may constitute the neurophysiological mechanism underlying a window of temporal integration in multimodal perception, although convincing evidence for this phenomenon for VA stimulus pairs is still lacking.
3.2. Individual alpha frequency correlates with the size of the temporal window of integration.
Cecere et al. (2015)employed a double-flash illusion task and de-monstrated that individual alpha peak frequency (IAF) correlates with the individually-determined size of the temporal window of integration. We replicate thefindings fromCecere et al. (2015), as we observed a positive correlation between IAF and a measure that reflects the tem-poral window of integration. Our results thus extend thefindings of Cecere et al. (2015)to the perception of temporal simultaneity. At the
Fig. 2. Results of the phase binning analysis. Left-hand row: grand-average percentage of not-simultaneous (NS) judgements as a function of phase bin, for different frequencies. Right-hand row: frequency distribution offitted cosine amplitudes for 10.000 randomly shifted phase bins, for each of the frequencies considered. These distributions represent the null distributions in the statistical analyses. Red vertical lines correspond to the actually observedfitted cosine amplitude. Numbers refer to the p-values resulting from the statistical analysis (random phase bin shifting,Section 4.7.1) for each frequency.
Fig. 3. Left-hand panel: correlation between alpha power and logistic function parameter b1, which models the width of the temporal window of integration. A
least-squares regression line is added for purposes of illustration. Right-hand panel: Grand-average scalp topography of alpha power at IAF frequency.
M. Bastiaansen, et al. Brain Research 1734 (2020) 146744
same time, our results are in contrast with those of Van Wassenhove and colleagues (Grabot et al., 2017). In the latter study, the authors failed tofind a correlation between perception accuracy and IAF in a temporal order judgement task. It is unclear at present what may have caused this discrepancy in the results between two very similar tasks (simultaneity judgement in the present study, temporal order judge-ment inGrabot et al., 2017). More data points are needed before any firm conclusions can be drawn with regards to this discrepancy. How-ever, combined with those ofCecere et al. (2015)our results indicate that the relationship between IAF and temporal window of integration is a phenomenon that extends beyond illusion-induction tasks. 3.3. Simultaneity judgements coincide with spontaneousfluctuations in readiness of the visual cortex.
We observed alpha power in the time interval immediately pre-ceding the visual stimulus to be smaller on trials with NS judgements. Thesefindings extend those of a previous study which reported similar results in a temporal order judgement task (Grabot et al., 2017). To-gether these studies indicate that the relationship between prestimulus alpha power and a task involving temporal judgements are robust across these two different tasks. These findings are embedded in a larger context of studies that demonstrate a relationship between alpha power and (visual) perception, as occipital alpha power has been re-lated to cortical idling (Pfurtscheller et al., 1996), inhibition (Foxe and
Snyder, 2011; Klimesch et al., 2007) and gating (Jensen and Mazaheri, 2010).
3.4. Alpha oscillations are instrumental in multisensory synchrony perception
It is of interest to note that the observed effects, although all within the alpha frequency range (7–13 Hz), have slightly different peak fre-quencies: The prestimulus alpha power differences are largest between 9 and 11 Hz. Similarly, the IAF ranges between 9.3 Hz and 11.8 Hz across participants. However, the phase dependency effects are ob-served at 12.5 Hz, which is clearly higher than the two other effects. They therefore occur on the descending flank of the alpha peak for individual participants. It is unclear to us at present whether these small frequency differences are a technical issue resulting from differences in the underlying analysis techniques (e.g. frequency smoothing for time-frequency analysis of alpha power, peak-picking for determining IAF, and frequency binning in the phase analysis), or whether there is a more functionally related explanation for it.
Regardless of these small differences in peak frequency, the dif-ferent results from the present study clearly converge onto an instru-mental role for alpha oscillations in multisensory synchrony perception: simultaneity judgements are preceded by lower alpha power, are cor-related with individual alpha peak frequency, and are dependent on prestimulus phase (at least for AV stimulus pairs, with a hint that the
same may be true for VA pairs). Taken together with previous studies (Cecere et al., 2015; Grabot et al., 2017; Ikumi et al., 2019), the available evidence suggests that irrespective of specific experimental paradigms, alpha oscillations are functionally related to multisensory binding, and constitute (part of) the neural architecture that underlies the temporal window of integration in multimodal perception. Thisfits well with the broader notion that oscillatory neuronal activity at around 10 Hz shapes perceptual cycles in unimodal sensory perception (VanRullen, 2016).
4. Experimental procedure 4.1. Participants
Thirty-three healthy students of Tilburg University participated in the experiment (age range:18–25; 24 females). The participants were fully informed about all the procedures regarding the experiment (which was approved by the Ethics Committee of Tilburg University) and gave their written informed consent. All participants had normal hearing and normal/ corrected to normal vision, had no neurological disorders, and did not use psycho-active drugs or medication. Based on behavioural performance criteria (see the section on EEG data selection and processing for a detailed explanation), the results of 11 participants were excluded from thefinal analyses. Thus, the final set comprised 22 participants (age range:18–25, 17 females).
4.2. Stimulus materials
The stimulus materials consisted of a visual and an auditory sti-mulus. The visual stimulus was a centrally aligned white circle (Wingdings 2 font, size 50), presented for 50 ms on a black background, on a monitor with a resolution of 1024 by 768 pixels, and 60 Hz refresh rate, positioned at a distance of approximately 70 cm from the parti-cipant’s head. The auditory stimulus was a pure tone (1000 Hz, 60 dB SPL), presented for 50 ms via two loudspeakers placed to the right and the left of the monitor.
4.3. Experimental design
During the experiment, participants performed a simultaneity jud-gement. To this end, stimuli were either presented simultaneously (SOA = 0 ms) or consecutively, the visual stimulus always being the first to occur. SOAs were, on average, symmetrically distributed around the point of subjective simultaneity (PSS). In pilot studies, the PSS was determined to be at an SOA of 180 ms on average. Therefore, SOAs of 0, 40, 80, 120, 160, 180, 200, 240, 280, 320 and 360 ms were used in the experiment. Each SOA was repeated 30 times during the experiment, except for the three SOAs closest to the PSS (160, 180 and 200 ms), which were repeated 100 times. Thus, in total the experiment consisted of 540 trials. Order of SOAs was randomized for each participant se-parately.
Each trial started with a black screen, presented for 1500 ms. Then, the visual and auditory stimuli were presented for 50 ms each, followed by a response delay interval (black screen) of 1900 ms. The response delay was included in order to minimize motor anticipatory processes in the peri-stimulus time interval, which are visible in the EEG from approximately 1500 ms prior to movement execution (e.g.Pfurtscheller and Lopes da Silva, 1999). At the end of the response delay, a response was prompted by presenting the words ‘synchronous = left button, asynchronous = right button’ on the screen. Participants were in-structed to press one of two buttons on a button box situated on the table in front of them, with the indexfinger of their right hand. Upon the button press, an asterisk was presented at the center of the screen for a random time interval of between 850 and 1150 ms, indicating the inter-trial interval.
4.4. Procedure
Participants were instructed about the EEG recordings and about the simultaneity judgement task. They then gave their informed consent. Participants were prepared for the EEG recordings, and were seated in a dimly lit, sound-attenuating cabin. They performed a short practice block of 14 trials, in order to familiarize them with the EEG recordings and with the task.
The 540 experimental trials were presented in 11 blocks (10 blocks of 50 trials each, and a last block consisting of 40 trials), each lasting approximately 5 min. After each block, participants were allowed a short break, and continued with the next block by pressing one of the buttons. The entire session, including participant preparation and de-briefing, lasted approximately 2 h.
4.5. EEG recordings
The EEG was analogfiltered in a frequency range between DC and 102 Hz, and digitized and recorded with a sampling rate of 512 Hz from 64 locations using active Ag–AgCl electrodes (BioSemi, Amsterdam, the Netherlands) mounted in an elastic cap and including two mastoid electrodes. Electrodes were placed according to the extended International 10–20 system (see Fig. 5). Two additional electrodes served as reference (Common Mode Sense active electrode, placed halfway between PO3 and POz) and ground (Driven Right Leg passive electrode, placed halfway between POz and PO4). Vertical and hor-izontal EOG signals were measured with electrodes above and below the left eye, and on the outer canthi of each eye, respectively. 4.6. EEG data preprocessing
The selected EEG data (seeSection 2.1) were analyzed using Vision Analyzer (Brainproducts, DE), and the Matlab-based open source toolbox FieldTrip (http://www.fieldtriptoolbox.org;Oostenveld et al., 2011). Data were re-referenced to the average of left and right mas-toids. In a few cases, bad channels were reconstructed using a spherical spline interpolation. The raw data was then segmented into 100 epochs ranging from 1200 ms before to 2200 ms after the onset of the visual stimulus. Eye-movement and obvious muscle artifacts were removed from the data using Vision Analyzer’s built-in ICA procedure. Trials with remaining artifacts on the EOG channels exceeding ± 100μV were
Fig. 5. electrode layout. The electrodes marked with a + (P1, P2, P3, P4, P5, P6, Pz, PO3, PO4, PO7, PO8, POz, O1, Oz and O2) were used in the statistical analysis of the time-frequency data.
M. Bastiaansen, et al. Brain Research 1734 (2020) 146744
discarded from further analysis. Subsequently, EEG channels with ar-tifacts exceeding ± 220 μV were automatically detected, visually in-spected, and discarded from further analysis when judged appropriate. In total, 9% of trials were rejected due to artifacts.
4.7. EEG data analysis
For each of the research questions formulated in the introduction, we performed different analyses on the preprocessed EEG data. First, in order to address the question of whether the phase of occipital alpha oscillations modulates perceived simultaneity, we created phase bins based on prestimulus phase, and looked for periodicity in the propor-tion of S and NS judgements across phase bins. Second, to address the relationship between IAF and temporal window of integration, we computed IAFs for all participants and correlated it with model-based estimates of the individual temporal window of integration. And third, we computed differences in prestimulus alpha power between S trials and NS trials. The details of the different analysis procedures are given below.
4.7.1. Phase modulation of perceived simultaneity
In order to determine a possible phase modulation of the perceived simultaneity, for each participant we divided trials into phase bins based on the phase estimate at channel Oz 100 ms prior to the onset of thefirst (i.e. visual) stimulus, as follows: we defined 10 phase bins on the unit circle, with their centers equally spaced between -π and π. To each bin, we assigned the 21 trials in which the prestimulus phases were closest to the center phase of the bin, resulting in partially over-lapping phase bins (seevan Elswijk et al., 2010for a similar procedure). Next, we computed the proportion of NS responses within each phase bin, across trials pertaining to that phase bin, and across participants. In order to determine whether phase (or rather, phase bin) periodically modulates simultaneity perception, we thenfitted a 1-cycle cosine to these proportions. The amplitude of the cosinefit is indicative for the extent to which prestimulus phase modulates simultaneity perception. Note that, in the binning procedure, a single data epoch was typically assigned to more than one bin (twice on average), resulting in over-lapping phase bins, and a partial dependency between consecutive bins. For this reason, in order to determine the statistical significance of the cosine amplitude (reflecting the depth of the phase modulation) we opted for a statistical procedure that is not affected by this partial de-pendence: Phase bins were randomly shifted around for each partici-pant (while preserving bin order, using MATLAB’s built-in circshift function), and grand average proportions of NS responses were com-puted. 10.000 such randomizations created a null distribution of cosine amplitude values. We then verified whether the actually observed co-sine amplitude fell within the upper 5th percentile of this null dis-tribution. In order to be able to test for the frequency-specificity of a potential phase-dependency of simultaneity perception, we performed this analysis for a set of frequencies from 7.5 Hz to 17.5 Hz, in steps of 2.5 Hz.
4.7.2. Analysis of individual alpha frequency
First, in order to determine IAF for each participant, the EEG data recorded during the entire SJ task werefiltered with a 1-Hz high pass filter, down-sampled to 100 Hz, and segmented in epochs of 10 s each. After EOG correction (ICA) and artifact rejection ( ± 150 µV), Fast Fourier Transformation (FFT) was used to estimate spectral power in each epoch, yielding a frequency resolution of 0.1 Hz. Power spectra were averaged across epochs, and subsequently averaged across chan-nels at which alpha power tends to be most prominent (PO7, PO3, POz, PO4, PO8, O1, Oz, O2). IAF was then determined as the peak in power in a frequency range from 7 to 13 Hz. For each participant there was one clearly identifiable peak in the 7–13 Hz frequency band.
Next, a logistic function wasfitted on the individual participants’ response profiles, so as to model the proportion of ‘simultaneous’
responses, where SOA stands for the stimulus onset asynchrony: =
+ b b
proportion of 'simultaneous' responses 1 1 . SOA
0 1
Across participants this function fitted the individual response profiles reasonably well (mean R2= 0.67, SD = 0.29). The parameter
b1in the logistic function mainly represents the steepness of the
func-tion, with higher values of b1reflecting a steeper function. This
para-meter was taken as a measure of the sensitivity to detect temporal asynchrony between the two modalities. Finally, we computed the correlation between IAF and parameter b1.
4.7.3. Time-frequency analysis of power
Time-frequency representations of power were obtained by spectral decomposition with fast Fourier transforms in a time range of 800 ms prestimulus to 1000 ms poststimulus, using an asymmetric taper that emphasizes the contribution of the time points closer to the rightmost edge of the data window to the spectral estimates (as defined in equation 3 fromMitchell et al., 2007), with a time window of 400 ms, in 10-ms time steps. We evaluated the frequency range from 2 to 30 Hz, in 1-Hz steps. Because we were mainly interested in the differences in power before stimulus onset, power was expressed as the absolute change in power relative to a baseline interval ranging from 800 to 600 ms prior to the onset of the visual stimulus. Trials were then averaged separately for S and NS judgements, and differences between the two types of trials were tested using a cluster-based random per-mutation analysis (Maris and Oostenveld, 2007) as implemented in the Matlab toolbox Fieldtrip (Oostenveld et al., 2011). As we were speci-fically interested in differences in occipital alpha power we performed a cluster-based random permutation test in which the frequency range was restricted to 7–13 Hz. The latency range for the analysis was −200 ms to 0 ms prior to visual stimulus onset. The following parietal and occipital EEG channels were entered into the analysis (P1, P2, P3, P4, P5, P6, Pz, PO3, PO4, PO7, PO8, POz, O1, Oz and O2, seeFig. 5). All three dimensions (time, frequency and channels) were used for clus-tering, and the sum of the t-statistics within each cluster was used as a cluster-level statistic. The null distribution was based on 5000 random permutations. Finally, grand-average time–frequency representations were computed for display purposes.
CRediT authorship contribution statement
Marcel Bastiaansen: Conceptualization, Methodology, Data cura-tion, Formal analysis, Writing - original draft, Writing - review & editing. Hermine Berberyan: Methodology, Data curation, Formal analysis, Writing - original draft, Writing - review & editing.Jeroen J. Stekelenburg: Conceptualization, Methodology, Formal analysis, Writing - review & editing. Jan Mathijs Schoffelen: Methodology, Formal analysis, Software, Writing - review & editing.Jean Vroomen: Conceptualization, Writing - review & editing.
References
Ai, L., Ro, T., 2014. The phase of prestimulus alpha oscillations affects tactile perception. J. Neurophysiol. 111, 1300–1307.
Alais, D., Newell, F.N., Mamassian, P., 2010. Multisensory processing in review: from physiology to behaviour. Seeing Perceiving 23, 3–38.
Baumgarten, T.J., Schnitzler, A., Lange, J., 2015. Beta oscillations define discrete per-ceptual cycles in the somatosensory domain. Proc. Natl. Acad. Sci. 112, 12187–12192.
Busch, N.A., Dubois, J., VanRullen, R., 2009. The phase of ongoing EEG oscillations predicts visual perception. J. Neurosci. 29, 7869–7876.
Calvert, G., Spence, C., Stein, B.E., 2004. The Handbook of Multisensory Processes. MIT Press.
Cecere, R., Rees, G., Romei, V., 2015. Individual differences in alpha frequency drive crossmodal illusory perception. Curr. Biol. 25, 231–235.
5274–5287.
Chen, L., Vroomen, J., 2013. Intersensory binding across space and time: a tutorial re-view. Attention, Perception Psychophys. 75, 790–811.
Diederich, A., Schomburg, A., van Vugt, M., 2014. Fronto-central theta oscillations are related to oscillations in saccadic response times (SRT): an EEG and behavioral data analysis. PLoS One 9, e112974.
Drewes, J., VanRullen, R., 2011. This is the rhythm of your eyes: the phase of ongoing electroencephalogram oscillations modulates saccadic reaction time. J. Neurosci. 31, 4698–4708.
Dugué, L., Marque, P., VanRullen, R., 2011. The phase of ongoing oscillations mediates the causal relation between brain excitation and visual perception. J. Neurosci. 31, 11889–11893.
Fiebelkorn, I.C., Foxe, J.J., Butler, J.S., Mercier, M.R., Snyder, A.C., Molholm, S., 2011. Ready, set, reset: stimulus-locked periodicity in behavioral performance demon-strates the consequences of cross-sensory phase reset. J. Neurosci. 31, 9971–9981. Foxe, J.J., Snyder, A.C., 2011. The role of alpha-band brain oscillations as a sensory
suppression mechanism during selective attention. Front. Psychol. 2, 154. Grabot, L., Kösem, A., Azizi, L., Van Wassenhove, V., 2017. Prestimulus alpha oscillations
and the temporal sequencing of audiovisual events. J. Cognit. Neurosci. 29, 1566–1582.
Henry, M.J., Obleser, J., 2012. Frequency modulation entrains slow neural oscillations and optimizes human listening behavior. Proc. Natl. Acad. Sci. 109, 20095–20100. Ikumi, N., Torralba, M., Ruzzoli, M., Soto-Faraco, S., 2019. The phase of pre-stimulus
brain oscillations correlates with cross-modal synchrony perception. Eur. J. Neurosci. 49, 150–164.
Jensen, O., Bonnefond, M., VanRullen, R., 2012. An oscillatory mechanism for prioritizing salient unattended stimuli. Trends Cognitive Sci. 16, 200–206.
Jensen, O., Mazaheri, A., 2010. Shaping functional architecture by oscillatory alpha ac-tivity: gating by inhibition. Front. Human Neurosci. 4, 186.
Kayser, S.J., McNair, S.W., Kayser, C., 2016. Prestimulus influences on auditory percep-tion from sensory representapercep-tions and decision processes. Proc. Natl. Acad. Sci. 113, 4842–4847.
Kayser, C., Petkov, C.I., Logothetis, N.K., 2008. Visual modulation of neurons in auditory cortex. Cereb. Cortex 18, 1560–1574.
Keetels, M., Vroomen, J., 2005. The role of spatial disparity and hemifields in audio-visual temporal order judgments. Exp. Brain Res. 167, 635–640.
Klimesch, W., Sauseng, P., Hanslmayr, S., 2007. EEG alpha oscillations: the in-hibition–timing hypothesis. Brain Res. Rev. 53, 63–88.
Lakatos, P., O'Connell, M.N., Barczak, A., Mills, A., Javitt, D.C., Schroeder, C.E., 2009. The leading sense: supramodal control of neurophysiological context by attention. Neuron 64, 419–430.
Maris, E., Oostenveld, R., 2007. Nonparametric statistical testing of EEG- and MEG-data. J. Neurosci. Methods 164, 177–190.
Mathewson, K.E., Gratton, G., Fabiani, M., Beck, D.M., Ro, T., 2009. To see or not to see: prestimulusα phase predicts visual awareness. J. Neurosci. 29, 2725–2732. Mitchell, W.K., Baker, M.R., Baker, S.N., 2007. Muscle responses to transcranial
stimu-lation in man depend on background oscillatory activity. J. Physiol. 583, 567–579.
Noppeney, U., Lee, H.L., 2018. Causal inference and temporal predictions in audiovisual perception of speech and music. Ann. N. Y. Acad. Sci.
Oostenveld, R., Fries, P., Maris, E., Schoffelen, J.M., 2011. FieldTrip: Open source soft-ware for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci., 156869.
Perrodin, C., Kayser, C., Logothetis, N.K., Petkov, C.I., 2015. Natural asynchronies in audiovisual communication signals regulate neuronal multisensory interactions in voice-sensitive cortex. Proc. Natl. Acad. Sci. 112, 273–278.
Pfurtscheller, G., Lopes da Silva, F.H., 1999. Event-related EEG-MEG synchronization and desynchronization: Basic principles. Clin. Neurophysiol. 110, 1842–1857. Pfurtscheller, G., Stancack Jr., A., Neuper, C., 1996. Event-related synchronization (ERS)
in the alpha band–an electrophysiological correlate of cortical idling: a review. Int. J. Psychophysiol. 24, 39–46.
Romei, V., Gross, J., Thut, G., 2012. Sounds reset rhythms of visual cortex and corre-sponding human visual perception. Curr. Biol. 22, 807–813.
Samaha, J., Postle, B.R., 2015. The speed of alpha-band oscillations predicts the temporal resolution of visual perception. Curr. Biol. 25, 2985–2990.
Schroeder, C.E., Lakatos, P., 2009. Low-frequency neuronal oscillations as instruments of sensory selection. Trends Neurosci. 32, 9–18.
Spence, C., Shore, D.I., Klein, R.M., 2001. Multisensory prior entry. J. Exp. Psychol. Gen. 130, 799.
Stein, B.E., Stanford, T.R., 2008. Multisensory integration: current issues from the per-spective of the single neuron. Nat. Rev. Neurosci. 9, 255.
Stekelenburg, J.J., Vroomen, J., 2007. Neural correlates of multisensory integration of ecologically valid audiovisual events. J. Cognit. Neurosci. 19, 1964–1973. Stevenson, R.A., Wallace, M.T., 2013. Multisensory temporal integration: task and
sti-mulus dependencies. Exp. Brain Res. 227, 249–261.
Strauß, A., Henry, M.J., Scharinger, M., Obleser, J., 2015. Alpha phase determines suc-cessful lexical decision in noise. J. Neurosci. 35, 3256–3262.
Thorne, J.D., Debener, S., 2014. Look now and hear what's coming: on the functional role of cross-modal phase reset. Hear. Res. 307, 144–152.
van Elswijk, G., Maij, F., Schoffelen, J.-M., Overeem, S., Stegeman, D.F., Fries, P., 2010. Corticospinal beta-band synchronization entails rhythmic gain modulation. J. Neurosci. 30, 4481–4488.
Van Wassenhove, V., Grant, K.W., Poeppel, D., 2007. Temporal window of integration in auditory-visual speech perception. Neuropsychologia 45, 598–607.
VanRullen, R., 2016. Perceptual cycles. Trends Cognitive Sci. 20, 723–735. VanRullen, R., Zoefel, B., Ilhan, B., 2014. On the cyclic nature of perception in vision
versus audition. Philos. Trans. R. Soc. B: Biol. Sci. 369, 20130214.
Vroomen, J., Keetels, M., 2010. Perception of intersensory synchrony: a tutorial review. Attention, Perception, Psychophys. 72, 871–884.
Wutz, A., Melcher, D., Samaha, J., 2018. Frequency modulation of neural oscillations according to visual task demands. Proc. Natl. Acad. Sci 201713318.
Zampini, M., Guest, S., Shore, D.I., Spence, C., 2005b. Audio-visual simultaneity judg-ments. Perception Psychophys. 67, 531–544.
Zampini, M., Brown, T., Shore, D.I., Maravita, A., Röder, B., Spence, C., 2005a. Audiotactile temporal order judgments. Acta Psychol. 118, 277–291.
M. Bastiaansen, et al. Brain Research 1734 (2020) 146744
