1
Modulation of beta-band activity in human visual cortex by covert perceptual
decisions
Supervisor: Guido Nolte Co-assessors: Tobias Donner & Niels Kloosterman Brain and Cognition, Department of psychology, University of Amsterdam 50 EC - October 2012- August 2014
Research project 2, MSc Brain and Cognitive Sciences – Cognitive Neuroscience track
Abstract
Perception is a product of sensory input and endogenous processes in the brain. Bistable visual illusions provide an excellent opportunity to study these processes, because in these illusions perception alternates despite constant sensory input. Recently, a non-sensory driven signal was isolated in the visual cortex around perceptual switches triggered by motion-induced blindness (MIB). MIB is a bistable illusion in which a salient target, surrounded by a moving mask, spontaneously disappears from perception, only to reappear after some time. Here, we used magnetoencephalography (MEG) to study the nature of this signal during MIB and several non-illusory control conditions. Consistent with earlier studies, we found a wide-spread modulation of beta band (10-30 Hz) MEG power in the visual cortex, time-locked to the report of perceptual switches. We show that the modulation is also present in the absence of an immediate report of the switches, indicating that this signal is not directly related to the motor act used for report, but instead tracks task-relevant perceptual events. We concurrently measured pupil diameter to assess if the modulation over visual cortex could be explained by increased retinal illumination due to phasic pupil dilation around the time of perceptual switches. We found no evidence for this hypothesis. Taken together, our results suggest that the beta power modulation in visual cortex reflects a top-down signal that transforms a sensory stimulus into a perceptual decision.
Introduction
Perception is not just the sum of sensory input but instead is shaped by endogenous processes that govern the internal state of the brain (Kloosterman et al., submitted; Jack et al., 2006; Wilke et al., 2006; Donner et al., 2008; Cardoso et al., 2012; Choe et al., 2014). Bistable stimuli are particularly suited to disentangle these internally generated brain signals from sensory processing because perceptual experience changes while sensory input remains constant. Although it has long been thought that perceptual bistability results from a combination of antagonistic connectivity and adaptation in sensory brain regions that represent the stimulus, alternative explanations have recently become popular. Instead of competition on a sensory level, processes more closely related to the action selection and decision making are proposed to be the driving factor of bistable perception (Leopold and Logothetis, 1999). These processes are not only engaged during bistable perception, but always influence perception.
Variables that reflect perceptual decision processes have been isolated outside of sensory or motor cortices in monkeys and humans (Gold and Shadlen, 2007; O’Connell et al., 2012), but their effect can also be found in visual cortex. For example, fMRI BOLD responses in early visual cortex were found to relate more closely to reported percepts than to visual input (Polonsky et al., 2000; Ress and Heeger, 2003).
2
Recently, transient modulations in visual cortex in the fMRI-signal (Donner et al., 2008) as well as in MEG power in the “beta” (12-30 Hz) frequency range (Kloosterman et al., submitted) have been reported around perceptual switches during motion-induced blindness (MIB) (Bonneh et al., 2001), a bistable phenomenon in which a salient target disappears from perception from time to time. These transient modulations reflected the content of the percept (target disappearance or reappearance) and also occurred when observers reported stimulus-evoked instead of illusory target disappearances and re-appearances. In contrast, the modulations did not occur when subjects passively viewed the physical stimulus changes and perceptual switches did not have to be reported. Together, these results suggest that the transient modulation in visual cortex reflect a top-down, decision-related signal that transforms a task-relevant sensory event into a report.
Here, we used whole-head MEG recordings to study whether the switch-related the transient modulation of neurophysiological activity indeed reflect such a top-down, decision-related signal. Specifically, we aimed to rule out two important alternative explanations. First, the modulation might be related to the motor response used to report the perceptual switches. Second, perceptual switches during MIB are associated with phasic pupil dilation time-locked to the report (Kloosterman et al., in preparation), thereby eliciting a transient increase of retinal illumination, which in turn might trigger the modulations observed in visual cortex.
The transient modulation in visual cortex has so far only been found around motor reports of perceptual changes and was not found during passive viewing of stimulus onsets and offsets, when no responses were required (Kloosterman et al., submitted; Donner et al., 2008). This suggests that the signal is related to active (motor) report, but also leaves open the possibility that the signal is decision-related, because both behavioral relevance and active behavioral report were manipulated. When subjects needed to report the perceptual switches they had to decide on which percept they experienced, but when they passively viewed the stimulus changes they did not have to make such decisions. If the transient modulations are decision-related, they should also occur in the absence of motor report, as long as the perceptual changes are task-relevant to the subject in some way. In fact, decision variables in non-sensory areas have been found to be unaffected by the lack of motor report (O’Connell et al., 2012). We instructed subjects to either immediately report switches, or silently count them. We found that the transient modulations are also present in the absence of immediate motor responses, indicating that the signal is not related to the motor act used to indicate switches.
The pupil phasically dilates around perceptual switches (Kloosterman et al., in preparation), thereby increasing the amount of light falling on the retina. Sudden differences in retinal illumination cause transient modulations of activity in visual cortex (Rossi et al., 1996; Haynes et al., 2004). It is possible that the transient modulation around the report of perceptual switches is caused by increased illumination of the retina, and not by a decision process. To investigate this possibility, we measured pupil diameter simultaneously with the MEG while subjects experienced the perceptual switches. We did observe a correlation between phasic pupil dilation amplitude and the transient beta suppression over visual cortex, but its positive sign was opposite of what would be expected when caused by illumination differences on the retina.
The observed correlation between pupil dilation and visual cortex activity might be explained by a common input that drives both signals. Central noradrenaline release from the locus coeruleus is a good candidate for this common input; it is tightly linked to pupil dilation and has widespread projections to the cortex (Aston-Jones and Cohen, 2005). Surprise, defined as a violation of temporal expectation, is associated with phasic noradrenaline release (Dayan and Yu, 2006). We manipulated surprise about perceptual switches (i.e. the predictability of the timing of stimulus changes) to investigate whether this affected the transient modulation in the visual cortex. We did not find an effect of surprise (average predictability) on visual cortex activity.
3 Materials & Methods
Subjects
22 subjects participated in the experiment. One subject was excluded due to incomplete data and one subject did not complete the experiment due to bad quality of the pupil data. Thus, twenty subjects (11 female, age range 20 - 54 years, mean age 29.6, SD 10.7) were included in the analysis. All subjects had normal or corrected-to-normal vision and no known history of neurological disorders. The experiment was approved by the ethics committee of the Universitätsklinikum Hamburg Eppendorf, and each subject gave written informed consent.
Figure 1 |Stimuli, behavioral tasks and stimulus response. A. Schematic representation of the stimulus. Left: MIB stimulus. Subjects fixated on the red fixation box in the center of the screen. A salient target (Gabor patch) was surrounded by a moving mask pattern (a rotating grid of white crosses). Perception fluctuates between ‘target present’ and ‘target absent’ (top left). Right: Replay stimulus. The stimulus is the same as during MIB, except the target was (i) flickering at 10 Hz and (ii) was physically removed and replaced on the screen, so subjects reported actual target onsets and offsets. B. Behavioral response regimes. Left: response regime for MIB and Replay-button conditions. Subjects pressed their right index finger and middle finger to report target disappearances and reappearances, respectively. Right: response regime for Replay-count condition. Subjects covertly counted the number of target disappearances and reported the total in a 4AFC-question after the end of the run. C. Hazard functions used for generating percept durations during Replay. The intervals between events were drawn from the probability distributions (not shown) corresponding to the three hazard functions shown here. Because subjects’ expectation of event timing follows the hazard rate, events occurring around the mean interval in both conditions will elicit different levels of surprise. Inset: distribution and corresponding hazard function of disappearance durations in MIB. D. Cortical response to the MIB stimulus during ‘Stimulus on-off’-condition. Scalp maps, topography of 8-25 Hz and 60-120 Hz modulations (0.25-0.75 s after stimulus onset; see dashed outlines on time frequency representations). Fully saturated colors indicate clusters of significant modulation (p < 0.05, two-sided permutation test across subjects, cluster-corrected, N = 17 subjects). Highlighted circles in high frequency scalp map (top right): MEG sensors showing the biggest stimulus response. These sensors are used for the subsequent analyses of overall power modulations.
Paradigm & stimuli
MEG and pupil diameter was measured simultaneously while subjects viewed the continuous presentation of a static target (full contrast Gabor patch; diameter: 2°) and reported target disappearance and reappearance. The Gabor patch contained two cycles. In the Replay conditions (see Behavioral Tasks and Design) the Gabor patch flickered at 10 Hz. Target flicker was implemented by counter-phasing the sinusoid used to generate the Gabor patch. The target
4
was located in either the lower left or lower right visual field quadrant (eccentricity: 5°, counterbalanced between subjects), surrounded by a rotating mask (17°x17° grid of black crosses), and superimposed on a gray background (Figure 1A). The mask rotated at a speed of 160°/s. The target was separated from the mask by a gray “protection zone” subtending about 2° around the target (Bonneh et al., 2001). Subjects fixated on a fixation mark (red outline, white inside, 0.8° width and length) centered on the mask in the middle of the screen. Stimuli were presented using the Presentation Software (NeuroBehavioral Systems, Albany, CA, USA).
Stimuli were back-projected on a transparent screen using a Sanyo PLC-XP51 projector with a resolution of 1024x768 pixels at 60 Hz. Subjects were seated 58 centimeters from the screen in a whole-head magnetoencephalography (MEG) scanner setup in a dimly lit room.
Behavioral Tasks and Design
The subjects participated in various tasks designed to determine the factors driving transient modulations of cortical activity around reports of perceptual switches. The MIB condition and the various Replay conditions were performed by all subjects; the Stimulus-on-off condition was performed by 17 of the subjects.
Stimulus-on-off
17 of the subjects performed the stimulus-on-off condition. In this condition subjects viewed the full MIB stimulus (grey background, fixation, rotating mask and target) for 0.75 s, preceded and succeeded by only the background and fixation. This stimulus duration was too short to induce MIB target disappearance, but sufficiently long to measure the stimulus-induced modulation of cortical population activity (Figure 1D). Subjects were instructed to maintain fixation and passively view the stimulus on- and offsets.
MIB condition
The MIB stimulus (figure 1A) was continuously presented for several runs of 3 min duration each. The subjects’ task was to maintain stable fixation and report the spontaneous disappearance and reappearance of the target by pressing a response button with their right index finger and middle finger, respectively.
Replay conditions
The “Replay” conditions were identical to MIB, except that the target was intermittently removed from the screen physically, so subjects reported veridical instead of illusory target disappearances. The physical offsets and onsets of the target were always instantaneous, mimicking the typically abrupt quality of the perceptual switches in MIB.
The general purpose of the Replay conditions was two-fold: to test (i) whether transient modulations in cortical activity during MIB are motor-related, and (ii) whether these transient modulations are affected by surprise. To test the first question, we manipulated the report method in a Replay-button and a Replay-count condition. To test the second question, we manipulated the timing of the on- and offsets of the target in three levels of surprise (high, medium and low). Both manipulations were implemented in the same experiment, resulting in a 2x3 factorial design (report method x surprise level).
Replay-button: In this condition subjects had to report disappearances and reappearances
in the same way as during MIB, by pressing a button with their index finger and middle finger.
Replay-count: This condition was identical to Replay-button, except for the way subjects
reported the perceptual changes. Instead of pressing buttons, subjects had to silently count the number of target offsets that occurred during the 3 minute run and report the total in a four alternative forced-choice (4AFC) question (Figure 1B). The 4AFC question was prompted on the screen after the run ended. The three incorrect 4AFC alternatives were generated by randomly
5
subtracting or adding 1, 2 or 3 from the actual number of disappearances, under the constraint that the four alternatives were all different from each other. In each run, Replay-button or Replay-count conditions were randomly selected (50% of the Replay-trials we Replay-button). The corresponding instructions were displayed on the screen before the run started. Subjects could only start the next run after they confirmed the instructions to the experimenter over the intercom.
During three different Replay conditions, perceptual surprise (defined as a violation of temporal expectation) was manipulated by sampling interval durations (target present/absent) from duration distributions based on three hazard function that differed in broadness (Figure 1C). A hazard function describes the probability that an event will occur at a particular point in time, given that it has not occurred yet (Luce, 1986). Varying the broadness of the hazard function allowed us to parametrically manipulate surprise about the time of occurrence of an event, since broader hazard functions yield more unpredictable event occurrences. The hazard function is computed as follows (eq. 1):
Eq. 1 where λ(t) is the value of the hazard function at time point t, f(t) is the value of distribution f on time point t, and F(t) is the area under the curve of distribution f from -∞ to time point t. In this experiment, instead of calculating the hazard function from a distribution of intervals, we defined hazard functions and subsequently calculated the corresponding interval distributions (eq. 2).
Eq. 2
High surprise: For the most surprising condition we used a flat hazard function (mean = 6s),
meaning a perceptual event (target disappearance or reappearance) has an equal chance of happening at every point in time, hence surprise is high.
Low surprise: For the least surprising condition we used a narrow Gaussian hazard function
(mean = 2s, SD = 0.2s), resulting in a regular, predictable distribution of interval durations.
Medium surprise: The intermediate conditions also consisted of a Gaussian hazard function,
but with a higher mean and SD (6s and 0.6s, respectively), making it more surprising as the narrow hazard function (Fiorillo et al., 2008).
The experiment consisted of two sessions of ca. two hours. Over these sessions, subjects completed a total of 44 three-minute runs (MIB: 6, Replay: 38). During Replay, subjects completed a total of 16 high, 16 medium and 6 low surprise runs to achieve a roughly equal amount of trials for each of the three Replay conditions. Subjects performed the MIB and High surprise conditions in one session and the Medium surprise and Low surprise in the other session. The two types of runs in each session were presented within two separate blocks to entrain subjects as much as possible to the event durations within the conditions. The order of blocks within a session and the order of sessions were counterbalanced across subjects. The Stimulus-on-off condition was performed at the end of one of the two sessions.
MEG and Eyetracker recordings
Magnetoencephalography (MEG) data were acquired on a CTF 275 MEG system (VSM/CTF Systems, Port Coquitlam, British Columbia, Canada) with a sample rate of 1200 Hz. The location of the subjects’ head was measured real-time using three fiducial markers placed in the both
)
(
1
)
(
)
(
t
F
t
f
t
))
(
1
(
)
(
)
(
t
t
F
t
f
6
ears and on the nasal bridge to control for excessive movement. Furthermore, electrooculography (EOG) and electrocardiography (ECG) were recorded to aid artefact rejection.
The diameter of the left eye’s pupil was sampled simultaneously with the MEG recordings at 1000 Hz with an average spatial resolution of 15 to 30 min arc, using an EyeLink 1000 Long Range Mount (SR Research, Osgoode, Ontario, Canada). This MEG-compatible (non-ferromagnetic) setup was placed on a table under the stimulus presentation screen. The Eyelink system was calibrated before every block of four runs.
Data analysis Preprocessing
Both the MEG data and pupil diameter data were analyzed in Matlab (The Mathworks, Natick, MA, USA) using the Fieldtrip (Oostenveld et al., 2011) and Chronux (Mitra and Bokil, 2007) toolboxes and custom-made software.
Trial extraction
For the Stimulus-on-off condition, we extracted trials of fixed durations, ranging from 0.2 s before to 0.75 s after stimulus onset.
For the MIB and Replay conditions involving subjects’ reports, we extracted trials of variable duration, centered on subjects’ button presses or releases, from the 3 min runs of continuous stimulation. Thus, in the case of MIB, the term “trial” refers to an epoch of constant stimulation and is solely defined based on subjects’ subjective reports of target disappearance and re-appearance. We call this method for trial extraction “response-locked”. The following constraints were used to avoid mixing data segments from different percepts when averaging across trials: (i) The maximum trial duration ranged from -1.5 s to 1.5 s relative to report; (ii) when another report occurred within this interval, the trial was terminated 0.5 s from this report; (iii) when two reports succeeded one another within 0.5 s, no trial was defined; (iv) for the analysis of all Replay-button conditions, we included only those reports that were preceded by a physical change of the target stimulus within 0.2 to 1 s, thus discarding reports following illusory target disappearances.
In an alternative analysis of all Replay conditions, trials were defined in the same way as described above, but now aligned to physical target on- and offsets, eliminating the need to discard illusory target disappearances (“stimulus-locked”). In the Replay-count conditions, no button responses were given during the run, so stimulus-locked trial extraction was the only option. We used this method in any analysis that involved the Replay-count condition.
To align the pupil data exactly with the MEG data for every trial the pupil data was first up-sampled to 1200 Hz to match the MEG sampling rate. Trial extraction was performed with the same method and restrictions as for the MEG data. Then, the timings of reports and stimulus on- and offsets were used to exactly align the pupil channel with the MEG data.
MEG data artifact rejection
All epochs that contained artifacts caused by environmental noise, eye-blinks, muscle activity or squid jumps were excluded from further analysis using standard automatic methods included in the Fieldtrip toolbox. Epochs that were marked as containing an artifact were discarded after every artifact detection step. For all artifact detection steps the artifact thresholds were set individually for all subjects. Both of these choices aimed at optimization of artifact exclusion. Line-noise was filtered out by subtracting the 50, 100, 150 and 200 Hz frequency components from the signal.
7
For all trials that contained an MEG-artifact the pupil data was also discarded. Although the data quality of some of these trials was sufficient to include them in the analysis, we focused on the MEG signal and its relation to pupil diameter in this paper. We therefore excluded all trials that had insufficient MEG-data quality, regardless of the quality of the corresponding pupil diameter data. The pupil data are more elaborately reported in Kloosterman et al. (in preparation).
Pupil data artifact rejection
Eye-blinks were detected using the standard algorithms of the manufacturer of the Eyetracker system. When a trial contained a blink that occurred within 0.5 s of the stimulus on- or offset the trial was discarded. Blinks that were not within 0.5 s of a stimulus event were removed by linear interpolation of the values of blink on- and offset plus 0.1 s padding. During some epochs the pupil signal was of non-acceptable quality due to difficulties specific to pupil measurements (incorrect threshold settings, eye not opened wide enough etc.), while the MEG data was of sufficient quality. In these trials the pupil data was discarded, but the trials were still used for the MEG analyses that did not include pupil data. We baseline corrected each pupil data trial by subtracting a pre-response baseline based on the average in the interval from 1 to 0.6 s before report.
Time-frequency decomposition
We used a sliding window Fourier transform to compute the time-frequency representation for each sensor and each trial of the MEG data. The sliding window had a length of 400 ms and a step size of 50 ms. We used two slightly different methods for low (3-35 Hz) and high (36-200 Hz) frequency ranges to get an optimal balance between signal-to-noise ratio and frequency resolution. For the low frequency range we used one Hanning taper (frequency resolution 2.5 Hz and bin size 1 Hz). For the high frequency range we used a multi-taper method with 5 tapers (spectral smoothing 8 Hz and bin size 2 Hz).
The data was baseline corrected for all analyses by subtracting and dividing the power signal by the baseline for every frequency bin separately. For the Stimulus-on-off condition, used for the sensor selection of later analyses, we used as a baseline the period from -0.25s to 0 s with respect to the time of stimulus onset. Since the main experiment was continuous, and therefore did not have a pre-stimulus period, we used the mean power of the whole epoch, averaged over all trials within one condition, as the baseline.
Quantification of the pupil diameter modulation
To quantify the pupil diameter modulation we used a linear projection method on every single trial, yielding a (positive) scalar measure of how strong the mean time course over all trials (template) was represented in the single trials, eq. 3:
Eq. 3 where Ri is the time course of trial i, and
R
is the average time course of the pupil modulation ofa given subject. The typical pupil response to a stimulus consisted of a dilation that started around report and peaked after about 0.5s after report, after which it would come back to baseline around 2s after report (figure 4A-B). We calculated linear projection of the whole modulation (-1 to 2s from report) to get a comprehensive idea of how the full pupil modulation is related to the MEG time frequency spectrum. Additionally, we computed the mean of every trial as a measure of tonic pupil diameter modulation.
' 2
||
|| R
R
R
A
i i
8 Statistics
Statistics of power-modulations
We used the ‘Stimulus-on-off’-condition to select the 25 sensors that showed the biggest response to the stimulus in the gamma band (60-120 Hz). These sensors are all located over occipital cortex (marked a circles in the top right topographic plot in figure 1D). The mean of all these sensors was used for further statistical analysis. For the analysis of the time-frequency representations we used two-tailed permutation tests (1000 permutations) to test for deviations of the power-modulation from zero in every time-frequency bin (figures 2, 3 and 4), combined with cluster-based method to control for multiple comparisons (Maris and Oostenveld, 2007).
To test the effect of motor response on the power modulation around perceptual switches we separated the data of the Replay conditions in two groups: the Replay-button trials and the Replay-count trials. The cluster-based permutation analysis was performed separately on both of these groups and on the difference between these groups. We could not test for the effect of motor-report during MIB trials, since the perceptual switches in this condition are generated internally so a physical response was always necessary.
We performed a similar type of analysis to look into the effect of surprise on power modulations around perceptual switches, this time not separating button and Replay-count trials, but the three surprise conditions. For every condition, we used cluster-based permutation statistics to test for power modulations significantly different from zero. To test for differences in power-modulation between conditions we compared every combination of two surprise conditions (three comparisons in total) using the same method.
Correlation of power modulation with pupil dilation
We investigated the relationship between pupil diameter and the time-frequency resolved MEG data on a trial-by-trial basis. Specifically, we were interested in a relationship between transient pupil dilation and MEG power modulations over visual cortex around perceptual report. Due to the sluggishness of the pupil signal, modulations in pupil diameter might be contaminated with effects of the previous trial, especially when the previous trial was short. The same could be true for the time-frequency resolved MEG data. We therefore removed the effect of previous trial duration from the pupil and MEG data using orthogonal projection (Donner et al., 2008):
Eq. 4 where y is the z-scored time course of the signal (pupil or MEG) and y* is the residual time course of the signal without the removed component of previous trial duration. r is the unit vector that needs to be removed (z-scored duration of the previous trial). The superscripted T indicates a matrix transpose. The algorithm was applied to both the pupil data and the MEG data.
After removing the effect of previous trial duration from both the pupil and MEG data, we calculated the magnitude of the pupil dilation response on a trial-to-trial basis using the linear projection method described above. We pooled all trials of all conditions together, while keeping disappearance reports separate from reappearance reports. We then correlated the scalar magnitude values of the pupil responses to every time-frequency bin of the corresponding trial. This analysis resulted in two time-frequency spectra of correlation values for every subject, one for target disappearance and one for re-appearance. We then performed group level cluster-based permutation statistics on the r-values to test significance of the correlations between pupil dilation and the MEG power modulation around the report of perceptual switches, as described above.
r
r
y
y
y
*
(
T
)
9 Results
Using MIB, a previous MEG-study has recently discovered a decision-related signal over visual cortex in the beta frequency band (Kloosterman et al., submitted). Here, we aimed to replicate this signal in a different subject group using a different MEG scanner setup. Moreover, we asked (i) whether this signal truly reflects the decision process or the motor act used to indicate perceptual decisions, and (ii) whether the signal could be explained by increased retinal illumination due to phasic pupil dilation around the time of perceptual report. Finally, we investigated (iii) whether surprise about the timing of behaviorally relevant perceptual events influences this decision-related signal.
Figure 2 | Switch-related modulations in visual cortex are independent of motor report. MEG power modulations are shown as time-frequency representations, averaged across trials and subjects. Fully saturated colors highlight clusters of significant modulation (p < 0.05, two-sided permutation test, cluster-corrected). In each panel, the first and middle rows correspond to high- and low-frequency ranges, respectively. Bottom row corresponds to time course of stimulus components and subjects’ reports. Fading indicates variable timing of (instantaneous) stimulus changes with respect to the trigger. Different panels correspond to different experimental conditions and different trigger events. A. MIB, aligned to disappearance report. B. Replay-button, aligned to stimulus offset. Dashed line indicates median reaction time. C. Replay-count, aligned to stimulus offset. D-F. The same conditions, aligned with reappearance report, target onset and target onset, respectively.
Perceptual switches are accompanied by content specific power modulations in the visual cortex
Previous research shows that the report of perceptual switches are accompanied by a transient modulation of beta power in the visual cortex (Kloosterman et al., submitted). This modulation was absent when subjects had to passively view the stimulus and looked radically different from the cortical stimulus response to the physical MIB stimulus. Here, we replicated these findings, using a different MEG system and a different pool of subjects.
10
First, when just presenting the stimulus for short episodes (0.75s, Stimulus-on-off condition) there is a decrease of broadband low frequency power (~8- 40 Hz) and an increase of high frequency power (~60 -120 Hz) over the visual cortex (figure 1D). These effects are significant and highly consistent with previous findings (Kloosterman et al., submitted.; Fries, 2009; Donner and Siegel, 2011). We calculated the mean power increase in the 60-120 Hz frequency range and the 0.2-0.75 s time window after stimulus onset (see dashed box in figure 1D, top panel). We selected the 25 sensors with the highest power increase in this time frequency window for further analyses. All of the selected sensors were located over the occipital cortex (depicted as circles in the topographic plot in the top panel in figure 1D).
Next, we looked at the modulation of oscillatory power around reports of target disappearances and reappearances in the 25 sensors selected based on the response to the stimulus. We found a decrease in power in the beta band (~10-30Hz) around the reports of target disappearances in the MIB condition (figure 2A). The transient modulation of power was specific to this band, in contrast to the cortical response to the mere stimulus, no significant power modulations were found in lower or higher frequency bands. A more or less opposite effect was observed during reappearances to the target in the MIB condition. Although less clear and confined in time, transient beta power over the visual cortex increases around reappearances (figure 2D).
Interestingly, very similar power modulations were observed around the report of target disappearances and reappearances in the Replay conditions (figure 2B and 2E). Again there is a decrease of power around disappearances that centers around the beta band (figure 2B) and an opposite modulation around reappearances (figure 2E). The fact that similar modulations occur during internally generated perceptual switches (in MIB) and perceptual switches determined externally by the stimulus (in replay) tells us that the transient modulations are not the cause of the perceptual switches. Instead, they seem to be an effect of the perceptual switches.
All these results are highly consistent with the previous MEG experiment in our lab (Kloosterman et al., submitted). Since that was the first experiment to show such a power modulation over the visual cortex with electrophysiology, these current results provide an important replication. Not only did we replicate these results using a completely new pool of subjects, we also showed that the results generalize over different MEG setups.
Power modulation over visual cortex is related to perceptual decision, but not motor activity
The beta power modulation over visual cortex has so far only been measured during motor responses. No modulation was present when subjects passively viewed the Replay condition (Kloosterman et al., submitted.; Donner et al., 2008). It could be that the beta power modulation is related to the motor activity of the report, but not to a decision process. However, these results are also consistent with our decision hypothesis. The lack of visual cortex power modulations during passive viewing of the Replay condition could also be because the perceptual switches were not relevant, thus no decision had to be made. To test whether indeed the motor report was not the cause of the power modulations over the visual cortex, we added an extra replay condition (Replay-count) to the experiment. This condition was equal to the Replay-button condition in every aspect except the way the subjects reported the perceptual switches. Instead of pressing a button, subjects counted the number of disappearances and answered a 4AFC-question about the total number of disappearances after every three minute run.
11
Figure 3 | Correlation between pupil dilation and MEG power modulation in visual cortex does not explain the beta power modulation. A. Time-frequency representation of visual cortex power modulation around report of target disappearance (pooled across MIB and Replay-button). Fully saturated colors highlight significant clusters (p<0.05, two-tailed permutation, cluster-corrected). The black line superimposed on the plot is the time course of pupil diameter, averaged over the same trials (see inset in figure 3B for scaling). B. Time-frequency representation and pupil time course locked to report of target reappearance. The vertical black bar depicts the scale of the pupil time course in raw eye tracker units. C. Frequency spectrum of the transient modulations, averaged over -0.3 to 0.3 s with respect to report. Vertical bars on the right depict significant modulation for disappearance (blue), reappearance (red) and the difference between them (black, p<0.05, two-tailed permutation test, cluster corrected). D. Topographical map of transient (-0.3 to 0.3 s to report) beta power (10-30 Hz) E. Time-frequency representation of the correlation between pupil dilation and power modulation in the visual cortex, locked to report of target disappearance. The effect of previous trial duration was removed. Fully saturated colors highlight significant clusters (p<0.05, two-tailed permutation, cluster-corrected). F. The as in figure 3E, but locked to report of reappearance. G. Frequency spectrum of the transient correlation (averaged over significant time window of correlation TFR, figure 3E). H. Topographical map of significant correlation. I. Correlation between baseline pupil diameter (-1 to -0.6 s with respect to report) with phasic pupil dilation. Left: one example subject, dots represent single trials. Right: average over subjects, gray lines indicate single subjects, black line indicates correlation over subjects (r =0.24, p<0.001). J. Correlation between baseline pupil diameter (1 to -0.6 s with respect to report) with previous trial duration. Left: one example subject, dots represent single trials. Right: average over subjects, gray lines indicate single subjects, black line indicates correlation over subjects (r =-0.15, p=0.004). K. Frequency spectrum of correlation between MEG power baseline (-0.8 to -0.7 s with respect to report) in visual cortex and previous trial duration. Horizontal bars indicate correlations significantly different from zero (red bar) or significantly different between disappearance and reappearance (black bar, p<0.05, two-tailed permutation test, cluster-corrected).
12
We found the same significant transient modulation of beta power in the Replay-count condition as we saw during MIB and the Replay-button (figure 2C and 2F). Although no overt response was required in this condition, we found a decrease of beta power around the time of the median response time in the Replay-button condition (figure 2C), and an increase around reappearances (figure 2F). We can therefore rule out the possibility that the power modulation we observed around perceptual switches reflects motor-related activity.
Pupil dilation correlates to power modulation in the visual cortex
The pupil dilates around the report of perceptual changes (Kloosterman et al., in preparation), which causes an increase in light falling on the retina. Changes in retinal illumination can trigger transient modulations in the visual cortex (Rossi et al., 1996; Haynes et al., 2004). Does increased retinal illumination cause the modulation in the visual cortex? If this is the case, trials in which there is strong pupil dilation should also have a strong beta power modulation over the visual cortex. To test this hypothesis we measured pupil diameter simultaneously with the MEG. We correlated a scalar measure of the pupil dilation strength on every trial to the power in every time frequency bin (for -0.5 to 1s around report) on that same trial (see figure 3A-B). We pooled the data together over the MIB and replay conditions, but separated the disappearances from the reappearances. The rationale for this was that the different conditions resulted in highly similar effects, while the power modulation were opposite for disappearance and reappearance reports (Figure 2). This opposite effect was not present in the pupil, where both perceptual switch types were associated with pupil dilation (Kloosterman et al., in preparation).
Pupil dilation is a sluggish signal, typically peaking about 1s after the event that triggered the dilation (Hoeks and Levelt, 1993)(see figure 3A-B, black lines). This means that perceptual switches that follow shortly after each other can have overlapping phasic responses. In other words: the pupil diameter might not have returned to baseline before the phasic dilation starts. When the pupil has not yet returned to its baseline diameter, the subsequent phasic pupil dilation tends to be smaller (De Gee et al., 2014). Our data also show these effects. First, there
is a negative correlation between pupil baseline size and pupil dilation strength (figure 3I, r = -0.24, p<0.001 over subjects). Second, pupil diameter between 1 and 0.6s before report (the
interval we use as baseline) negatively correlates with previous trial duration (figure 3J, r = -0.15, p=0.004). Consistently with these correlations, pupil dilation amplitude correlated positively with previous trial duration (r = 0.08, p = 0.03, data not shown). Although MEG power signals are far less sluggish than pupil diameter, the duration of the previous trial might still affect the activity in the cortex, and thereby overshadow the correlation between phasic pupil dilation and MEG power we were interested in. We indeed found a correlation between previous percept duration and beta band power in the interval before report of perceptual switches. This correlation was significant in the interval from 0.8 to 0.7s before report of disappearances (figure 3K). To exclude effects of previous trial duration, we removed it from both the pupil and MEG data using an orthogonal projection method (see methods, Eq. 4).
If the beta band power modulation over visual cortex would be caused by a transient increase of retinal illumination due to phasic pupil dilation after perceptual changes, greater pupil dilation would be associated with stronger beta suppression in visual cortex around disappearance reports. Thus, a negative correlation would be expected. The retinal illumination scenario predicts a positive correlation around reappearance reports, due to the associated beta power increase.
The results of the correlation between MEG power over the visual cortex and pupil dilation (after removing the effect of previous trial duration) are shown in figure 3E and 3F. There is a significant positive correlation between pupil dilation and ~8-18 Hz power around the report of disappearances (figure 3E, p<0.05, permutation test, cluster-corrected), indicating that stronger pupil dilation is associated with weaker beta suppression (i.e. less negative beta modulation). Such a correlation is not present around reappearance reports, although an opposite trend is
13
visible (blue area around report in the corresponding time-frequency window in figure 3F). These correlations are opposite of what would be expected when retinal illumination would be the cause of the modulation in beta power we see in the visual cortex. These results thus argue against retinal illumination as the cause for the power modulations we see in visual cortex, but there is still a significant correlation between pupil diameter and beta power.
Figure 4 | Beta power is not modulated by global temporal context. A. Time-frequency representation of power modulation in the High surprise condition, locked to target offset. B. Time-frequency representation of power modulation in the Low surprise condition. C. Difference between High surprise and Low surprise. Fully saturated colors indicate significant modulations (p<0.05, two-tailed permutation test, cluster-corrected).
Power modulation over visual cortex is not modulated by global temporal context
Is the beta power modulation affected by other cognitive factors than perceptual decision formation? The possible role of central neuromodulation in causing the modulation inspired us to assess the influence of surprise on the signal. We manipulated the global level of temporal predictability in three replay conditions, thereby introducing differences in the level of surprise (Figure 1C).
We found no significant differences in transient power modulation between these three different surprise levels (figure 4). All three surprise conditions show the previously reported decrease of beta power around reports of disappearances. The transient modulations of the two conditions with maximally different surprise level, the High surprise condition (figure 4A) and the Low surprise condition (figure 4B), were not significantly different in strength (figure 4C). Around reappearance there were also no significant differences (data not shown). Although the transient power modulations might be affected by surprise in some way, our results show no indication that they are affected by the global temporal predictability of a series of trials.
Discussion
In summary, we replicated earlier findings of a transient modulation of beta power over the visual cortex, which occurs around perceptual switches. We found that this modulation is not motor related. Furthermore, we found that phasic pupil dilation is correlated to MEG power around disappearances in more or less the same time frequency window, although the opposite way of what would be expected if retinal illumination would be the cause of the transient beta power modulations. The beta power modulations were not found to be affected by the global temporal predictability of the task. In sum, our results suggest that these modulations reflect a decision related process.
The possibility that the visual cortex modulations are caused by motor-related activity was already partly accounted for by a previous experiment in our lab. In this MEG study, subjects
14
pressed a button for one type of perceptual change (target disappearance or reappearance) and released it for the other (counterbalanced within subjects). Even though these two report actions are motorically different, no differences were found in visual cortex (Kloosterman et al., submitted). However, these results did not exclude the possibility that the beta power modulations in the visual cortex reflected preparatory motor activity, which is known to evoke beta band power modulations in motor cortex (Donner et al., 2009). Here, we went one step further and showed that the modulations also happen in complete absence of motor reports.
The transient beta modulations were stronger when subjects reported the perceptual switches immediately, compared to when they counted the number of offsets (figure 2B-C). There are two possible explanations for this. First, the modulations might be partially motor-related. Although we can rule out that the signal is purely motor-related, part of it might still reflect motor activity. Second, the difference in the strength of the modulation between the Replay-button and Replay-count condition might be due to the difference in the kind of behavioral relevance of the perceptual switches between the conditions. In the Replay-button condition target on- and offsets require an immediate response, whereas in the Replay-count condition there is no such immediate need for action. This difference in urgency might underlie the difference in the strength of the beta power modulation.
The correlation analysis between pupil dilation and visual cortex power modulations rule out that increased retinal illumination, caused by phasic pupil dilation, drives the beta power modulations. However, we did find a significant correlation between strength of the pupil dilation and the power in a time-frequency window that largely overlaps with the window of the switch-related power modulation (compare figure 3E to figure 3A). It is likely that this correlation is due to a common input that drives both signals.
One candidate underlying mechanism is the locus coeruleus-noradrenaline (LC-NA) system, which is known to affect pupil diameter and which directly innervates visual cortex, among many other regions (Aston-Jones and Cohen, 2005; Nieuwenhuis et al., 2011). Moreover, the LC-NA system is increasingly associated with perceptual decision-making, as it has been shown to be able to become phasically activated around the time of decisions (Aston-Jones and Cohen, 2005), in line with the transient modulations of the MEG and pupil signals observed here. However, the same argument that rules out retinal illumination also rules out noradrenaline release as the cause of the beta power modulations: if noradrenaline release caused both the pupil dilation and the visual cortex modulation, stronger pupil dilations should go together with stronger beta power suppression. In other words, a negative correlation would be expected. Given our results, if noradrenaline is involved in the transient power modulation in visual cortex, it would inhibit it, but not cause it.
Alternatively, a different central neuromodulator, acetylcholine, might be the common factor that underlies the correlation. The neurons that innervate the constrictor muscle of the pupil are cholinergic (Huhtala et al., 1976; Loewenfeld, 1993) and acetylcholine antagonists cause pupil dilation (Little et al., 1998). Pupil constriction might be controlled by cholinergic nuclei in the basal forebrain (Yu, 2012). Furthermore, neurons in these nuclei have been found to phasically increase their activity when stimuli are detected in a cue detection task (Parikh et al., 2007), a task that shares many features with the task we used. Because acetylcholine is thought to constrict the pupil (i.e. inhibit pupil dilation), a strong boost of acetylcholine would cause a small pupil diameter. Hence, if acetylcholine release would drive the visual cortex modulation, a negative correlation with pupil dilation would be expected. Although our results corroborate this hypothesis, evidence for a role of central acetylcholine release in both pupil dilation and visual cortex activity is scarce. More research is needed to specifically test this hypothesis, preferably including manipulation of acetylcholine levels with pharmacology.
The beta power modulations in visual cortex were not affected by the level of surprise about the timing of stimulus switches. Surprise about the timing of relevant events has been found to trigger noradrenaline release (Aston-Jones and Cohen, 2005; Dayan and Yu, 2006). We aimed
15
to manipulate the amount of surprise by differing the predictability of the stimulus changes. We reasoned that if this surprise manipulation would affect the transient modulations in the visual cortex, it should be mediated by noradrenaline. We found no evidence for this hypothesis. Together with the relation between pupil dilation and visual cortex activity, this suggests that noradrenaline is not the driving factor of the power modulation in visual cortex.
However, there is an alternative explanation. Surprise in our experiment was defined as the uncertainty about the timing of upcoming events, if an event occurs at an unpredictable time surprise will be high. In other words, subjects expected a high degree of surprise in this condition. Alternatively, surprise could be defined as unexpected uncertainty about the timing of upcoming events given the global uncertainty of the task (Dayan and Yu, 2006). In our High surprise condition the timing of upcoming events was always unpredictable, so the uncertainty in this condition was always expected and thus not surprising by the latter definition of surprise. The level of expected uncertainty depends on the recent trial history. Expected uncertainty is high when the trial history was very unpredictable, when the trial history is regular, on the other hand, expected uncertainty is low. Surprise, by this definition, occurs when the timing of an upcoming event violates the expected level of regularity. It could be that the beta modulation over visual cortex is affected by this form of surprise.
By modeling of the recent trial history, expected uncertainty and the violation of expected uncertainty can be quantified on a trial-to-trial level (Mars et al., 2008). This method provides a sensitive measure of unexpected uncertainty. We plan to use these kinds of models for further analysis in the future.
References
Aston-Jones G, Cohen JD (2005) An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu Rev Neurosci 28:403–450.
Bonneh YS, Cooperman A, Sagi D (2001) Motion-induced blindness in normal observers. Nature 411:798– 801.
Cardoso MMB, Sirotin YB, Lima B, Glushenkova E, Das A (2012) The neuroimaging signal is a linear sum of neurally distinct stimulus- and task-related components. Nat Neurosci 15:1298–1306
Choe KW, Blake R, Lee S-H (2014) Dissociation between neural signatures of stimulus and choice in population activity of human V1 during perceptual decision-making. J Neurosci 34:2725–2743 Dayan P, Yu AJ (2006) Phasic norepinephrine: a neural interrupt signal for unexpected events. Network
17:335–350.
Donner TH, Sagi D, Bonneh YS, Heeger DJ (2008) Opposite neural signatures of motion-induced blindness in human dorsal and ventral visual cortex. J Neurosci 28:10298–10310.
Donner TH, Siegel M (2011) A framework for local cortical oscillation patterns. Trends Cogn Sci 15:191– 199.
Donner TH, Siegel M, Fries P, Engel AK (2009) Buildup of choice-predictive activity in human motor cortex during perceptual decision making. Curr Biol 19:1581–1585.
16
Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11:966–973.
Fries P (2009) Neuronal gamma-band synchronization as a fundamental process in cortical computation. Annu Rev Neurosci 32:209–224
De Gee JW, Knapen T, Donner TH (2014) Decision-related pupil dilation reflects upcoming choice and individual bias. Proc Natl Acad Sci U S A 11:618–625.
Gold JI, Shadlen MN (2007) The neural basis of decision making. Annu Rev Neurosci 30:535–574 Haynes J-D, Lotto RB, Rees G (2004) Responses of human visual cortex to uniform surfaces. Proc Natl
Acad Sci U S A 101:4286–4291
Hoeks B, Levelt WJM (1993) Pupillary dilation as a measure of attention : A quantitative system analysis. Behav Res Methods, Instruments, Comput 25:16–26.
Huhtala A, Tervo T, Huikuri KT, Palkama A (1976) Effects of denervations on the acetylcholinesterase-containing and fluorescent nerves of the rat iris. Acta Ophthalmol 54:85–98.
Jack AI, Shulman GL, Snyder AZ, McAvoy M, Corbetta M (2006) Separate modulations of human V1 associated with spatial attention and task structure. Neuron 51:135–147.
Kloosterman NA, Knapen T, Hillebrand A, Dijk BW Van, Lamme V, Donner TH (submitted) State Change in Human Visual Cortex Predicts the Stability of a Perceptual Illusion.
Kloosterman NA, Meindertsma T, Van Loon AM, Donner TH (in preparation) Pupil dilation reflects specific perceptual alternations in a bistable visual illusion.
Leopold D, Logothetis N (1999) Multistable phenomena: changing views in perception. Trends Cogn Sci 3:254–264.
Little JT, Johnson DN, Minichiello M, Weingartner H, Sunderland T (1998) Combined nicotinic and muscarinic blockade in elderly normal volunteers: cognitive, behavioral, and physiologic responses. Neuropsychopharmacology 19:60–69.
Loewenfeld IE (1993) The pupil: Anatomy, physiology, and clinical applications, 2nd ed. Oxford: Butterworth-Heinemann.
Luce RD (1986) Response Times: Their Role in Inferring Elementary Mental Organization, 1st ed. Oxford University Press, USA;
Maris E, Oostenveld R (2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164:177–190.
17
Mars RB, Debener S, Gladwin TE, Harrison LM, Haggard P, Rothwell JC, Bestmann S (2008) Trial-by-trial fluctuations in the event-related electroencephalogram reflect dynamic changes in the degree of surprise. J Neurosci 28:12539–12545
Mitra P, Bokil H (2007) Observed Brain Dynamics. Oxford University Press, USA.
Nieuwenhuis S, De Geus EJ, Aston-Jones G (2011) The anatomical and functional relationship between the P3 and autonomic components of the orienting response. Psychophysiology 48:162–175. O’Connell RG, Dockree PM, Kelly SP (2012) A supramodal accumulation-to-bound signal that determines
perceptual decisions in humans. Nat Neurosci.
Oostenveld R, Fries P, Maris E, Schoffelen J-M (2011) FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput Intell Neurosci 2011:156869. Parikh V, Kozak R, Martinez V, Sarter M (2007) Prefrontal acetylcholine release controls cue detection on
multiple timescales. Neuron 56:141–154
Polonsky A, Blake R, Braun J, Heeger DJ (2000) Neuronal activity in human primary visual cortex correlates with perception during binocular rivalry. Nat Neurosci 3:1153–1159
Ress D, Heeger DJ (2003) Neuronal correlates of perception in early visual cortex. Nat Neurosci 6:414– 420
Rossi A f, Rittenhouse CD, Paradiso M a (1996) The representation of brightness in primary visual cortex. Science (80- ) 273:1104–1107.
Wilke M, Logothetis NK, Leopold D a (2006) Local field potential reflects perceptual suppression in monkey visual cortex. Proc Natl Acad Sci U S A 103:17507–17512.
