Interbrain synchrony during playing a multi-player
Brain-Computer-Interface game
Report of a Research Internship (42 EC), conducted by
Martine Cederhout
(6056261) Supervised by: Dr. M. Congedo, GIPSA-lab Prof. C.E. Jutten, GIPSA-lab Co-assessed by: Prof. C.M.A. Pennartz, UvA Conducted at GIPSA-lab, Grenoble, France 9 February 2015 – 24 October 2015As part of
MSc in Brain and Cognitive Sciences, University of Amsterdam Track: Cognitive Neuroscience
Abstract
We conducted an EEG Hyperscanning study focused on ERPs. This enabled us to study interbrain synchrony between two subjects that were playing together an ERP-controlled Brain Computer Interface (BCI-) Game. We recorded EEG simultaneously from 18 pairs of subjects and assessed intra- as well as inter-brain measures of synchrony. We hypothesized that synchrony is engendered by an external factor (common stimulation) as well as by a social factor arising from the interaction between the two subjects. Furthermore, we expected that the interbrain synchrony would change as a function of type of social interaction; either cooperating or competing in the game. We found task-related synchrony at 100-200 ms post-stimulus at 5-10 Hz, reflecting the N100-P200 complex of the ERP. This synchrony was not affected by condition. Furthermore, we found that when subjects were playing in competition the phase coherence over trials between their brains in 2-3 Hz increased over a large centro-parietal region in a time-window from 200 to 600 ms. A similar effect was found when the subjects exhibited an ERP simultaneously versus when their ERPs were separated in time. This indicates the interbrain synchrony arises when subjects ‘act’ in the same time and that this is modulated by the type of interaction. However, the intra-subject analysis of phase concentration revealed that such a difference is not found within subjects, i.e., there was no difference in phase consistency over trials within subjects. Except from the instructions and the reward structure the game was exactly the same in cooperation and competition. Taken together, the results indicate that the phase coherence increase is due to an emergent social phenomenon that arises between the players during their interaction, which cannot be found at the intra-subject level. The 2-3Hz inter-brain synchronization we have observed appears related to the P300 response, since it manifests itself in the a space-time-frequency region largely overlapping with the P300. The increase of synchrony during competition as opposed to cooperation might be due to heightened arousal during this condition, which is known to have an influence on the P300.
Introduction
Human kind is an inherently social species. Only in interaction with others we come into being (Hari & Kujala, 2009). In these exchanges with others it is of crucial importance to be able to infer the other’s intentions and internal states. This is achieved by a mechanism which mirrors the other’s action in ourselves. When seeing someone pick up an apple, similar motor patterns in our own brains get activated, as if we are picking up the apple ourselves. This perception-action coupling puts us in the other’s shoes, by which we are able to make a representation of his mental state (e.g., ‘she is going to eat the apple’ or ‘she is hungry’) . The implicit ‘tuning in’ with others does not only happen on the level of actions, but is also observed for other biological functions, such as respiration and cardiac activity (Muller & Lindenberger, 2011), as well as in higher cognitive functions. Recently, there has been a surge of interest in studying the neural mechanisms underlying social interaction as an emergent phenomenon that arises between two individuals. This field of studies, called
Hyperscanning (Montague et al., 2002) takes a new approach, which conceives the brains of
two interacting individuals as a two-in-one-system or a hyperbrain. The central idea is that the behaviour of the hyperbrain cannot be explained by studying the individual brains in isolation. However, this latter has always been the main approach over the last decades. In psychological studies it has been tried to study the behaviour and brain of an individual in isolation, trying to eliminate or control all confounding variables. However, as social interaction is a complex phenomenon, it calls for a more elaborate approach and more naturalistic research setting. Hyperscanning studies have studied brains in interaction by using different paradigms allowing for more naturalistic settings in which subjects were interacting. Findings from these studies suggest that social interaction involves a certain degree of synchronization observable on the neural level.
In these studies, social interaction is perceived in all its variety and not only constricted to verbal communication. For instance, a study conducted by Dumas and colleagues (2010) focused on communication through hand movements. They demonstrated that when subjects freely follow hand movements of the other subject, their brains synchronize: During synchrony of movements there was an increase in interbrain synchrony between right centro-parietal scalp regions in the alpha-mu band (7-12 Hz). Muller and colleagues (2013) investigated neural synchrony between guitarists when they are improvising together. They showed the presence of hyperbrain networks, where the inter-brain connections primarily functioned at low frequencies. Furthermore, it was shown that the empathic abilities of musicians are correlated with the degree of synchronization in the alpha band when viewing their own performance (Babiloni et al., 2012). Other factors that might influence the degree of synchronization might be the type of social interaction. Several Hyperscanning experiments have made use of Game Theory paradigms to study the neural synchrony that is associated with different types of social behaviour (reviewed in Babiloni & Astolfi, 2012). Results showed that when subjects choose to cooperate in a Prisoner’s Dilemma, this is accompanied by statistically significant links between homologous prefrontal areas of the two players, which are absent when they choose to defect (Astolfi et al., 2009). Based on the inter-brain synchrony measures estimated four seconds before a decision, it was actually possible to predict the decision (De Vico Fallani et al., 2010). Even when the social interaction is more implicit a difference can be observed between cooperating and competing on a task. In a NIRS-based (Near-InfraRed Spectroscopy) study conducted by Cui and colleagues (2012) subjects were instructed to either press a button simultaneously after mentally counting ten seconds (cooperation) or to press the button within closest distance from ten seconds (competition). The results showed more synchronous activity in the right
superior frontal cortices of participants while cooperating versus competing, which was also correlated with performance (i.e., more behavioral synchrony).
Together these studies indicate that brain synchrony arises between individuals that are involved in some kind of interaction. This synchrony seems to be a function of the type of interaction, is modulated by empathy and is not confined to brain regions involved in the task at hand. The current study aims to further the understanding of interbrain synchrony, especially the influence of the type of interaction and empathy. In order to do so, we conducted an Event-Related Potential (ERP)-based Hyperscanning study.
ERPs are stereotypical neural responses to internal or external events and have received a lot of attention in neuroscience research over the past decades. Visual stimulation is known to evoke ERPs; the early components (N100-P200-complex) are visible in the occipital cortex and are associated with low-level visual processing. They concern a mere physiological bottom-up respons to the stimuli and show little inter-subject variability (Key, Dove, & Maguire, 2005; Luck, 2005). However, the later the component, the more time there is for top-down influence. For instance the P300 (positive peak around 300 ms), which is classically observed in oddball-paradigms as a response to an oddball among a series of similar stimuli, is shown to be modulated by several factors. These factors can be either natural (e.g., circadian) or induced (e.g., fatigue, caffeine-intake; Ogura, 1996). In an integrative review Polich (2007) captures all these influences by the general term of ‘arousal’. In his framework he posits the P300 as an index of attention allocation. The general level of arousal determines the amount of available processing capacity for attention allocation to the ongoing task. When an oddball stimulus calls for attention, the higher the amount of resources available, the higher the amplitude and the shorter the latency of the P300. The
amplitude is considered as reflecting the amount of neural activity and the latency as reflecting the stimulus evaluation time. As motivation influences the arousal-level, it is also an important modulator of the P300. The P300-component can be subdivided in the P3a, which occurs slightly earlier and more central and P3b which occurs later, has a more parietal scalp topography and is considered as the ‘classical’ P300. The P3a is primarily observed in response to unexpected or novel stimuli, whereas the P3b arises after stimuli that are expected but uncertain (Polich, 2007). As in our paradigm we use familiar stimuli and thus do not expect to find the P3a, we will use the term P300 to refer to the P3b.
The current study builds further on the vast amount of research on ERPs, introducing a new perspective by recording ERPs from two subjects at the same time. We expected to find a task-induced synchronization due to subjects having a simultaneous ERP. Furthermore, we expected to find an additional type of synchronization due to a social component. We expected that this type of synchrony could be modulated by the type of interaction (cooperation versus competition) and to be a function of the empathic abilities of the subjects. Finally, we expected the interaction not only to influence the consistency of the phases of the two brains, but also the consistency in the characteristics of the P300. The latter might be influenced by motivation.
In order to test our hypotheses we designed a multiplayer version of the Brain Invaders game (Congedo et al., 2011). The Brain Invaders game is based on the classification of the P300. Electroencephalographic (EEG) data were simultaneously recorded in 18 pairs of subjects playing the game either in cooperation or in competition. The game would either elicit ERPs simultaneously in both subjects or separated in time. First, we validated the data by conducting time-domain analyses of the ERPs. We analyzed condition-effects on the
amplitude and latency of the P300 and looked for comodulation of these measures between subjects. Second, we applied intra-subject measures in the time-frequency domain to investigate the behavior of phases over trials within individuals. Third, we used inter-subject phase measures to investigate the neural correlates of social interaction in the hyperbrain.
Method
Participants For participation to the screening session we recruited 50 participants. Among
those, 44 were selected to take part in the actual experiment (more details on selection procedure below). None of the participants reported a history of neurologic or neuropsychiatric disorder, a motor or sensory problem or was currently on medication susceptible to modulate neural activity. 22 pairs were formed of which 4 were excluded due to technical problems during the experiment. The final sample (18 pairs) consisted of four female-female pairs, four female-male pairs and ten male-male pairs. The mean (± SD) age of the participant was 23.8 (±3.1) years. Written consent was obtained from all participant. All procedures were approved by the local ethics committee of Grenoble Alpes University (CERNI).
Task-Screening In the experiment participants played a Brain-Computer-Interface (BCI)
game, called ‘Brain Invaders’ (Congedo et al, 2011). This game is based on classification of the P300 component of the ERP, similar to a P300 speller. The experiment consisted of a screening session and an experimental session. In the screening session subjects individually played a standard version of the game. The aim was to destroy the target alien by merely focusing on it. This target alien was indicated by a red circle at the beginning of the level (Figure 1A). Each level of the game consisted of a grid of 36 aliens which were randomly divided in groups of six. The groups of aliens flashed in a random order, such as that after 12
flashes (called a repetition) each group had flashed two times (Figure 1B). After three repetitions the classification of the continuous brain signals by the algorithm resulted in the destruction of the alien with the highest probability of being the target (Figure 1C). If the target (TA) was destroyed the player
continued to the next level, if a non-target (NT) alien was destroyed the player returned to the same level after a short break with feedback (Figure 1D). One game consisted of nine levels and during the screening each subject played three games with an decreasing time-interval between two flashes (i.e., increasing difficulty level). The screening session allowed us to establish the timing parameters at which the average performance (success in destroying the TA) would be around 70%. Furthermore, we used this session to get subjects acquainted with the game, be able to exclude participants with low performance and pair participants for the experiment based on their performance. In making pairs, care was taken that subjects did not know each other, while trying to get an equal gender division in the pairs.
Task- Experiment In the experimental session pairs of subjects played a multi-player version
of Brain Invaders, which was especially designed for this study. Four types of gameplay were introduced in the game, representing two experimental manipulations. The first
Figure 1 Screenshots of the multi-player game, used in the experimental session, indicating the timeline. Screenshots of the target indication period, the flashing of the aliens, the destruction of the aliens and the feedback. For the solo-game played during the screening the timeline is comparable.
manipulation concerned the type of social interaction; subjects would either play in cooperation (CO) or competition (CM). The second manipulation concerned whether subjects would attend to the same target or not, i.e., whether they exhibited a simultaneous ERP (SI) or not (NS). This latter manipulation allowed us to distinguish between merely task-induced synchrony due to having an ERP (which is the case in both SI and NS) and synchrony due to having a simultaneous ERP (which is only the case in SI). The two manipulations resulted in four conditions: COSI, CONS, CMSI, CMNS. The couples played four blocks, each containing the four conditions in semi-random order, ensuring a unique position of each condition in the sequence. The timing parameters in the multi-player game were chosen such that the game would be rapid and there would be no ceiling effects to performance, to keep an appropriate level of motivation. Each flash had a duration of 80 ms with an interstimulus interval drawn from an exponential distribution spaced between 30 and 1000 ms (mean = 100 ms). After the destruction of the alien(s), participants were shown a feedback-screen indicating the amount of points they earned (4000 ms) (Figure 1D). This was also a moment during which participants were allowed to talk and move a bit. In order to minimize overlapping of target epochs, the minimum interval between two targets was 500 ms for each subject. The awarded points depended on the performance and on the condition. In cooperation, each player in the couple would receive the same amount of points, meaning they would receive points even when only one of the two succeeded. The maximum number of points would be awarded when both participants succeeded to destroy the target. In competition, the award-structure was altered to invoke competition: when one failed he did not receive any points and the highest amount of points could be obtained by destroying the target while the other player failed. During the instruction the idea of playing
in a team versus playing for one’s own profit was emphasized. However, besides instruction and feedback the cooperation and competition condition were identical.
Questionnaires After the task subjects filled in a questionnaire assessing their motivation for
the different conditions and a validated French version of a questionnaire assessing empathy (Favre, Joly, Reynaud, & Salvador, 2009).
Data acquisition & Preprocessing From each participant EEG signals were recorded using 32
active electrodes and two USBamp 3.0 g.Tec amplifiers (Guger Technologies, Graz, Austria) at 512 Hz. Included electrodes (Fp1, Fp2, AFz, F7, F3, F4, F8, FC5, FC1, FC2, FC6, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, P7, P3, Pz, P4, P8, PO7, O1, Oz, O2, PO8, PO9) were placed according to the 10-10 system. Fz was used as ground electrode and the right earlobe was used as a reference (Figure 2). The raw EEG signals were acquired and processed by OpenVibe 0.12 software (Renard et al., 2010). To minimize jitter, the four amplifiers were linked and events were tagged by using an analog channel connected directly to the amplifier. After recording bad channels were removed based on visual inspection of the raw EEG signals. Further preprocessing depended on the analysis and was done using custom-made scripts in MATLAB (The Mathworks, Massachusetts, United States, version 2011b).
Figure 2 Experimental set-up. Subjects are seated next to one another and watching the same screen. EEG is recorded by using 32 active electrodes each. Each set of 32 electrodes is connected to two pre-amplifiers, which are connected to two amplifiers. There the signals are synchronized between the subjects and the game.
Data analysis
The recording of EEG signals from two brains at the same time allowed us to study not only intra-subject measures, but also inter-subject measures. Intra-subject analysis is common practice and entails analyzing the data from each subject individually. The inter-subject measures, at the other hand, only assess what is common between the signals of the two brains. Analysis was carried out in the time-domain and in the time-frequency domain.
Time-domain analysis
Intra-subject ERP analysis
For the analysis of the ERP after TA flashes (N=480 per subject) data were downsampled to 128 Hz and bandpass-filtered from 0.1 to 20 Hz, using a 2nd order Butterworth filter with a linear phase response. One-second epochs were extracted, starting at stimulus-onset. ERPs were computed and baseline-activity (500-0 ms before stimulus onset) was subtracted from the signal. Data were smoothed in the time-domain, using a 5-point Hamming-window, to get a more robust estimate of the P300-peak. The amplitude of the P300 was defined as the maximum positive peak between 300 and 500 ms post-stimulus. The latency was defined as the interval between stimulus onset and the peak amplitude. For each subject the ERPs were inspected and only in case of a clearly visible P300 data were included for analysis, yielding a sample of 25 subjects. Then the maximum responsive electrode was determined for each subject and in these electrodes the amplitudes and latencies of the P300-peaks were compared over conditions by a non-parametric Friedman test.
Cooperation/empathy The correlation between the P300 amplitude and latency in
Inter-subjects ERP analysis
The differences in P300 amplitude and latency within pairs was analyzed for pairs in which both subjects had a clear P300 (N=10 pairs). The differences in cooperative setting were compared to the differences in competitive setting with a signed rank test. Furthermore, the differences in P300 within pairs during cooperation were correlated to the mean empathy score of the group with a Spearman correlation.
Time-Frequency analysis
Before extracting time-frequency information the artifacts were removed from the data by visual inspection. The data were downsampled to 256 Hz and bandpass-filtered from 1 to 40 Hz, using a 4th order Butterworth filter with a linear phase response. Then the EEG data were decomposed into their time-frequency representation by use of a wavelet-transform. A set of ten Morlet-wavelets was used, logarithmically increasing from 2 to 20 Hz, with a variable number of cycles (logarithmically increasing from 3 to 5). The wavelets were constructed by multiplying a complex sine wave (left part of the equation) with a Gaussian window (right part of the equation), using the following formula
with
, ,
where n indicates the number of wavelet cycles. The frequency decomposition yielded a complex representation containing both power and phase information for each channel (C) and trial (K) at each time-frequency (TF) point.
Trial definition For the study of interbrain synchrony special care had to be taken in defining
windows for analysis. Between the conditions Simultaneous ERP (SI) and Non-simultaneous ERP (NS) there is a crucial difference with respect to when the data epochs are recorded for
the subjects. For SI the TA-epochs are recorded simultaneously (Figure 3A), whereas for NS the TA-epochs are recorded at different times (Figure 3B). In the contrast of these two conditions, this introduces the possibility of finding more synchrony in SI due to any type of environmental or instrumental factors, picked up by the recording devices for both subjects. In order to investigate this possibility we also made a contrast between SI and NS of data epochs which were
all recorded non-simultaneously: We swapped the data of one subject of the pair for each two consecutive TA-trials. In SI this resulted in epochs that were paired between subjects that were not recorded at the same time, but during which the recorded ERPs were elicited at the same time as the other subject. For NS this procedure just rendered the time-interval between the paired epochs comparable to that in the SI condition. For the contrast between cooperation and competition we only used data that was recorded simultaneously (SI) for both subjects.
Figure 3 Illustration of trial definition. Displayed are the raw EEG traces for both subjects, the upper 32 traces belong to subject 1 (S1) and the lower 32 traces to subject 2 (S2). Epochs of one second are extracted after a Target-flash. In (A) the simultaneous TA condition is illustrated; both subjects attend to the same TA, leading to simultaneously recorded TA-epochs. In (B) the non-simultaneous TA condition is illustrated; each subject attends to his personal TA, leading to non-simultaneously recorded TA-epochs. In the latter condition there is no synchrony in ‘noise’ coming from anything but the brain, because the epochs for both players are not recorded simultaneously.
Intra- and Inter-subjects time-frequency analysis
Measures On the intra-subject level we analyzed Phase Concentration, given by
where k:{1,..., K} indexes the trials and is the complex wavelet coefficient. This is a measure of the consistency of phase angles over trials for each TF-point, for each channel. On the inter-subject level we used the measure of Phase Coherence (also known as Phase Locking Value or Inter Site Phase Clustering), given by
where and represent the phases of the signals in the CTFK-point of interest for
player 1 and 2, respectively. This measure is comparable to phase concentration but now calculated over the relative phases between the subjects. Both of these measures only take into account phase information and ignore amplitude information. Because there were no clear hypotheses about the time or frequency of interest we used a non-parametric cluster-based permutation test from the Fieldtrip toolbox (Oostenveld, Fries, Maris, & Schoffelen, 2011) to test condition differences in the described measures in the whole time-frequency plane and for all electrodes simultaneously.
Cluster-based permutation test The output for each measure was smoothed over time using
a 5-point Hamming-window, downsampled to 16 Hz and put into the cluster-based permutation test with 5000 permutations (Maris, Schoffelen, & Fries, 2007). The cluster threshold was set to 0.05 and the significance of the identified clusters was tested using α =0.05.
Correlation phase coherence and empathy In order to test the hypothesis regarding the
influence of empathy on neural synchrony, the Spearman correlation between the phase coherence differences with the average score on the empathy questionnaire for each pair was computed.
Results
Behavioural Results
The game was designed so as to allow a success rate of destroying the target of about 70%. The actual average (±SD) performance was 74.0% (±18.7%). There was a significant effect of session, F(1,42)=10.645, p=0.002, which was due to a significant lower performance on the last session (M=68.1, SD=4.5 ) as compared to session 1 (M=76.6, SD= 3.1, p=0.001), session 2 (M=73.8, SD=3,9, p=0.016) and session 3 (M=75.1, SD=3.8, p=0.002; Figure 4)
.
There were no significant differences in the performances between conditions (F(1,40)=1,344 , p=0.253). In general the motivation for performing well in the game as reported by the participants, was high (4.4/5) and there was no difference in motivation for either cooperation or competition (2.9/3). Time-domain analysisIntra-subject ERP analysis
In order to validate the data the grand average ERP over the 38 included subjects was calculated for Target and Non-Target trials (Figure 5A). The early peaks of the ERP (N100, P200) are bigger in amplitude in response to TA-flashes, but can also be found in response to
Figure 4 Average performance over sessions. Performance is measured as the success rate in destroying the target. The performance during the fourth session was significantly lower than in each of the other sessions (p<0.05).
Figure 5 (A) Grand average ERP for Target (TA) and Non-Target (NT) trials over electrodes CP1, CP2, P3, Pz and P4 (N=38). For the ERP peaks the respective scalp topographies are displayed (C). Peak latencies for TA-trials are: P1 at 117 ms, N100 at 156 ms, P200 at 234ms, N200 at 289 ms and P300 at 359 ms. (B) Grand Average ERP (N=38) for each of the four conditions, averaged over electrodes CP1, CP2, P3, Pz and P4. There were no significant differences in P300 amplitude or latency for the different conditions.
topographies, we observe that the P100 and N100 are focused around the occipital regions (Figure 5C). The P200 has a fronto-central distribution and the P300 has a slightly more parieto-central distribution.
Condition-differences in P300 latency and amplitude There was no significant condition
effect found on the P300 peak latency or amplitude in the maximum responsive electrode after target flashes, χ 2(3)=1.36, p=0.714 and χ2(3)=3.34, p=0.343, respectively (Figure 5B). The latency was similar for the conditions; COSI (M=368 ms, SD=7), CONS (M=367 ms, SD=6), CMSI (M=368 ms, SD=7) and CMNS (M=370 ms, SD=6). Also the amplitudes did not differ; COSI (M=4.02, SD=0.39), CONS (M=4.24, SD=0.40), CMSI (M=4.07, SD=0.37) and CMNS (M=4.10, SD=0.36).
Cooperation and empathy For the correlation between P300 amplitude during cooperative
play and the score on the empathy questionnaire, we observed a trend towards significance, r=0.369, p=0.069. This positive correlation indicated that the more empathic subjects were, the higher the amplitude of their P300 during cooperation. The latency of the P300 however, was not related to empathy, r=-0.101, p=0.632.
Inter-subjects ERP analysis
The differences in latencies between participants of the same group between cooperation (M=6.64 , SD=11.7) and competition (M=8.98 , SD= 12.9 ) did not differ (Z=12.5, p=0.539). Also for amplitude there was no difference, in cooperation (M=0.55, SD=0.67) versus competition (M=0.58, SD=0.51; Z= 23, p=0.413). The correlation of the difference in P300 amplitude between players and their empathy score yielded a significant trend, r=0.61, p=0.062. This suggests that a bigger difference between the amplitudes in cooperation could be related to a higher empathy score. The P300 latency did not correlate with the empathy score, r= -.470, p=0.170. (Amplitude and latency differences were trending to be inversely correlated between each other, r=-0.558, p=.094).
Time-Frequency analysis
In the following section we will discuss the findings in the time-frequency domain. We will first present the results from the intra-subjects analysis which will facilitate the extension of the time-domain findings towards the inter-subjects time-frequency domain findings.
Intra-subjects time-frequency analysis
Phase concentration in Target-trials In Figure 6 the average phase concentration for TA-trials
is displayed. There are two distinguishable clusters of phase concentration; one at 100-200 ms, centered around 5-10 Hz, which is spreading into the other cluster around 200-400 ms at
Figure 6 (A) Phase Concentration in all Target-trials for all electrodes, showing two distinguishable clusters. One between 4-7 Hz around 100-200 ms, displayed in scalp distribution in (B). The other cluster is found at 2-2.5 Hz around 300 ms (C).
2-3 Hz (see Figure 6B and 6C for the respective scalp distributions). No significant difference was found by the cluster test in cooperation versus competition.
Inter-subjects time-frequency analysis
Simultaneous Target-trials The cluster-based permutation test revealed a significant
difference in Phase Coherence between the two subjects during simultaneous target flashes when they are playing in cooperation as compared to competition. The time-frequency maps show that the difference involves the two lowest frequency bins (2 and 2.6 Hz) in almost all electrodes, but is significant in a cluster covering the left and right central-parietal electrode sites (FC1, C3, P3, C4, P4) at 375 and 438 ms. The contrast indicates that during competition there is more phase coherence between the two players. Furthermore, the general trend in the TF-maps (Figure 7AB) looks similar to that for phase concentration (Figure 6A), only the magnitude is smaller.
Figure 7 Phase Coherence values in cooperation (A) and competition (B) for all the electrodes. (C) T-values of cooperation-competition. (D) Scalp distribution of Phase Coherence values in Cooperation at 2-2.5 Hz at 0.375 and 0.4375 s. (E) Scalp distribution of Phase Coherence values in Competition at 2-2.5 Hz at 0.375 and 0.4375 s. (F) Scalp distribution of t-values for the contrast cooperation-competition. Stars indicate significant clusters from the cluster-based test at α=0.05.
There was no significant correlation between the phase coherence differences between the conditions at the timepoints of interest (375 ms and 438 ms) at 2 Hz at the electrodes showing an average absolute t-value higher than 2 (F4, FC1, C3, C4, CP6, P3, P4, O2, PO8) and the average empathy score of both players (r=0.058, p=0.818).
To further explore the finding, we also calculated the Spearman correlation between self-reported relative motivation for competition and phase coherence differences at the selected electrodes/frequencies/timepoints of interest. This did not yield any significant
results (r=0, p=1). Also there was no significant correlation between the difference of performance during cooperation versus competition and the phase coherence differences in the selected electrodes (r=-0.16, p=0.524). Finally, self-reported motivation did not correlate with performance differences between the conditions (r=-0.109, p=0.668).
Simultaneous versus non-simultaneous Target-trials When contrasting simultaneous versus
non-simultaneous target-trials the cluster-based permutation test indicated significant differences between the two conditions. Primarily in the lower frequencies (2-4.3 Hz) large clusters of condition differences are found spanning between 250 and 750 ms (Figure 8).
Figure 8 (A) Phase Coherence in Simultaneous Target-trials, (B) in Non-Simultaneous Target-trials and (C) the t-values of the differences of A – B. (D) Phase Coherence in simultaneous TA-trials at 2 Hz at 0.375 and 0.4375 s. (E) Phase Coherence in non-simultaneous TA-trials at 2 Hz at 0.375 and 0.4375 s. (F) Two representative scalp distribution of t-values for the contrast D-E. Stars indicate significant clusters from the cluster-based test at α=0.001.
At 2 Hz large clusters of condition differences are found between 250 ms and 940 ms, shifting from a centro-parietal topography before 650 ms to fronto-central and occipital topography after 650 ms at α=0.01. At 2.5 Hz significant clusters are found between 250 and 750 ms with a centro-parietal distribution at α=0.01.
We also investigated another contrast of SI and NS, in which all the data was recorded non-simultaneously (see Methods for details). The results (Figure 9) show that in this case there is no condition differences between trials in which ERPs occurred simultaneous (but were analyzed over non-simultaneous trials) or non-simultaneous; there were no significant clusters found.
Figure 9 (A) Phase Coherence in Simultaneous Target-trials, (B) in Non-Simultaneous Target-trials with the shuffled data and (C) the t-values of the differences of A – B of non- simultaneously recorded data. (D) Two representative scalp distributions similar to those found in Figure 8D for Simultaneous Target-trials. (E) The same scalp distributions for Non-Simultaneous Target-trials. (F) Scalp distribution of t-values for the contrast Simultaneous versus Non-Simultaneous Targets.
Discussion
We conducted an EEG Hyperscanning study involving ERPs, which is the first of its kind, to the best of our knowledge. We studied interbrain synchrony between pairs of subjects that were playing an ERP-controlled BCI-game. The advantage of focusing around ERPs is that it allows us to assess the appropriateness of the synchrony measures, because the ERPs are known to be exhibited by both subjects at approximately the same time. Therefore a baseline-level of merely task-induced synchrony must be observable. Indeed, the results showed task-induced phase coherence between the subjects, apparent in the spectral representation of the ERP-components. Furthermore, this phase coherence appeared to be a function of social interaction, as we found a significant difference between playing in competition and cooperation. Next we will discuss these findings in more detail.
More phase coherence during competitive play
The main finding of this study is significantly increased phase coherence around 400ms after target stimulus in the low frequency band (2-3Hz) between subjects when they play in competition as compared to cooperation. The direction of this effect is opposite to what we expected to find. We tried to further investigate the finding by correlating the phase coherence differences to performance or self-reported motivation, but this did not yield any significant results. It seems that the increased phase coherence in competition cannot be explained by an increase in performance or self-reported motivation in the competition condition. However, as performance and self-reported motivation are also not correlated between one another, their reliability for inferring the subject’s actual state during the experiment is questionable. The self-reported motivation for the conditions was only assessed after the experiment, which might have made it difficult for the subjects to accurately report their relative motivation during the game.
No condition difference in phase concentration
By using the intra-subjects measure of phase concentration we did not find a similar effect. Phase concentration (intra-subjects) and phase coherence (inter-subjects) are independent measures. Whereas the first is sensitive to consistency in phases within a subject across trials, the second is sensitive to the consistency of the relative phases between subjects across trials. This means that phases within subjects can be random, but as long as they are consistent with the phases of the other subject, the phase coherence can be high. The other way around, if the two subjects have high internal phase consistency over trials, the chances that their respective phase coherence is high as well, will be increased. In a way then, phase concentration can indirectly influence phase coherence. The phase concentration values we have observed are generally higher (± 0-0.2) than the phase coherence values (± 0-0.1). The main pattern in phase concentration and phase coherence is similar. There are two distinguishable clusters: One at 100-200 ms around 5-10 Hz mainly in parieto-occipital regions. This is the spectral representation of the N100-P200 complex with an occipital topography. The other main cluster is found at 200-400 ms around 2-3 Hz with a more widespread scalp distribution, involving parietal and central regions. This cluster corresponds to the P300. The early N100-P200 complex reflects low level visual processing of the stimulus. The P300, as a later component, can be modulated by top-down influences (Polich, 2007).
Findings from the time-domain analysis give a complementary view on the data. As expected we observe the N100-P200 complex in both target and non-target trials, as it reflects low-level visual processing. In target-trials the peaks are larger amplitude, which can be explained by a sensory gain mechanism. This attention-mediated mechanism increases the sensory responses to attended target location versus attended locations (flashes of
non-targets) (Luck, Woodman, & Vogel, 2000). The P300 shows up as the classical response to an oddball stimulus, with a midcentro-parietal scalp distribution. It is observable after target stimuli and absent after non-target stimuli.
The finding of a significant trend of a correlation between empathy and P300 amplitude during cooperation could be due to highly empathic subjects being more motivated and aroused during cooperating in the game, leading to a higher P300 amplitude (Polich, 2007). Other correlations with the empathy measure were not found.
Furthermore, there was no general effect of condition on the P300 characteristics. This finding is in line with the absence of a self-reported motivation differences between the conditions and also the absence of phase concentration differences in the time-frequency domain. This means that within subjects there are no condition differences in phase-locked activity.
Phase coherence due to social interaction
The fact that the condition difference between cooperation and competition in low frequencies was only found on the inter-subject level and not on the intra-subject level indicates that the influence of either playing cooperation or competition is only evident on the level of the hyperbrain. In this case, studying the brains of the two subjects in isolation is not sufficient to understand the emergent phenomena that arise because of the social interaction and are different for the conditions. Except for the instruction and the feedback the actual task during cooperation and competition did not differ. Therefore, the explanation for the effect must be sought in any possible differences in the social interaction between the two conditions.
A possible explanation for this effect is that during competition the subjects could be more aroused. Although subjects did not report to be more motivated during competition, in general competition is known to increase motivation, especially in men (Gneezy, Niederle, & Rustichini, 2003). Since two-third of our sample consisted of men (24/36 subjects), this might have influenced our result. Arousal levels during competition might have been elevated by the increased motivation. This arousal might set the stage for interbrain synchrony to arise. The heightened level of engagement in the competition condition might trigger a synchronous state.
Phase coherence during simultaneous versus non-simultaneous ERPs
The results discussed so far concerned the differences between different types of social interaction during simultaneous ERPs. In order to investigate what actually happens when two brains ‘act’ in the same time, i.e., have a simultaneous ERP (SI), versus when they ‘act’ at different times, i.e., having a non-simultaneous ERP (NS), we introduced this as another experimental manipulation. The contrasting of simultaneous ERPs (simultaneously recorded EEG for both players) and non-simultaneous ERPs (non-simultaneously recorded EEG) leads to the finding of large-scale broadband synchrony in the lowest frequencies between 250 and 750 ms at centro-parietal regions. The space-time-frequency region of this synchronization is similar to that of the spectral representation of the P300. This suggests that there is more phase coherence on the level of the P300 when both subjects simultaneously exhibit an ERP as compared to when the ERPs are separated in time. This finding is in line with our expectations. This effect could be due to alignment of brain responses due to social interaction. However, there might be a bias due to non-cerebral magnetic fields which could introduce synchrony in the EEG recordings of both players. We tried to overcome the potential influence of confounding factors by manipulating the epochs
in such a way that in both conditions the data would not be recorded simultaneously. Significant findings of brain synchrony in this contrast would indicate that the phase coherence that subjects display while watching the same TA-flashes would be independent of time, i.e., the phases of both subjects would align with one another in a similar fashion over trials. However, in this contrast we found no significant differences. This does not exclude the possibility that having a simultaneous ERP introduces brain synchrony between the two subjects of a pair. Still it can be that phases are only aligned at each trial individually and are not consistent within subjects over trials. This is in line with the finding that the phase concentration within subjects is rather low (up to 0.2). To further investigate the difference between simultaneous and non-simultaneous ERPs future research could try only using data that is simultaneously recorded in both conditions. To achieve this, subsequent TA and NT trials could be collapsed by using multivariate regression average estimation to generate epochs containing TA-ERPs and multiple NT-EPRs for each subject.
Future directions
The analysis conducted so far on this dataset can be regarded as exploratory. The amount of data collected in this study is vast and so are the different kinds of analysis that can be performed. The intersubject measure of phase coherence that we have used is only sensitive to phase consistency between the two subjects in a couple over trials at exactly the same location, frequency and time. This specificity should be kept in mind when appreciating the results. The most important future direction is to move from the electrode level to source space. At the electrode-level group-level inferences suffer from a large inter-subject variability due to its different source localization and orientation, as well as possible slight differences in electrode cap positioning. Analysing the data in the source space using joint blind source separation (Congedo, Phlypo, & Chatel-Goldmann, 2012) would circumvent
these difficulties and would at the same time reduce the number of statistical tests in the analysis. This would reduce the amount of correction needed to account for multiple testing. Another important feature of our analysis is that we computed phase coherence over trials, rendering it sensitive to similarities in phase configuration over trials. Therefore, findings based on this method provide strong evidence, but might miss smaller effects. Another method for assessing phase-based coherence is analyzing the phase angle differences over time instead of over trials (Lachaux, Rodriguez, Martinerie, & Varela, 1999). The advantage of this method is that it is insensitive to phase angle differences being different over trials. Therefore it will capture both phase-locked and non-phase-locked coherence. But, this comes at the cost of lower temporal precision when analyzed over entire trials (Cohen, 2014). Still, it might pose an interesting direction for future research. With the use of a sliding time-window the loss in temporal precision might be partially avoided.
Furthermore, the scope of the current study could be widened by not only focusing on ERPs after target-flashes, but to also investigate the response to the feedback. After each trial feedback on the performance is given by the destruction of an alien. Negative feedback might elicit an error-related negativity (Ruchsow, Grothe, Spitzer, & Kiefer, 2002). This part of the game is the point where subjects not only receive feedback on their own performance, but also on the performance of the other player. With respect to that, the social component during this timeframe might be even greater than during the flashes. Other approaches for research on the collected data could be investigating induced activity on the trial level or in the continuous EEG-signal.
Conclusion
In this study we investigated phase coherence phase-locked to target-stimuli between subjects that were involved in an ERP-based BCI-game. We showed that during playing in competition the phase coherence in 2-3 Hz was increased over centro-parietal regions as compared to playing in cooperation. A similar cluster of increased phase coherence was found for simultaneous ERPs as compared to non-simultaneous ERPs. This cluster of activity is corresponding to the P300 that was elicited by the target-stimuli in the task. Together this indicates that an increase in brain synchrony can be found between subjects when they have an ERP at the same time, which is modulated by whether they play in cooperation or competition. Furthermore, as we did not observe differences in phase concentration on the intra-subject level, this underlines the emergent nature of the phenomenon due to the social interaction. This finding is new and deserves replication and further investigation.
Acknowledgements
This research was conducted in close collaboration with Louis Korczowski and Florent Bouchard under supervision of dr. Marco Congedo and prof. Christian Jutten. Technical support was provided by Anton Andreev. The research is part of the CHESS-project, funded by the ERC grant of prof. Christian Jutten.
Literature
Astolfi, L., Cincotti, F., Mattia, D., De Vico Fallani, F., Salinari, S., Marciani, M. G., . . . Babiloni, F. (2009). Estimation of the cortical activity from simultaneous multi-subject recordings during the prisoner's dilemma. Conf Proc IEEE Eng Med Biol Soc, 2009, 1937-1939. doi:10.1109/iembs.2009.5333456
Babiloni, C., Buffo, P., Vecchio, F., Marzano, N., Del Percio, C., Spada, D., . . . Perani, D. (2012). Brains "in concert": frontal oscillatory alpha rhythms and empathy in professional musicians. Neuroimage, 60(1), 105-116. doi:10.1016/j.neuroimage.2011.12.008
Babiloni, F., & Astolfi, L. (2012). Social neuroscience and hyperscanning techniques: past, present and future. Neurosci Biobehav Rev, 44, 76-93. doi:10.1016/j.neubiorev.2012.07.006
Cohen, M. X. (2014). Analyzing Neural Time Series Data; Theory and Practice. Cambridge, MA: MIT Press.
Congedo, M., Goyat, M., Tarrin, N., Ionescu, G., Varnet, L., Rivet, B., . . . Jutten, C. (2011, 2011-09-22). "Brain Invaders": a prototype of an open-source P300- based video
game working with the OpenViBE platform. Paper presented at the 5th International
Brain-Computer Interface Conference, Graz, Austria.
Congedo, M., Phlypo, R., & Chatel-Goldmann, J. (2012). Orthogonal and Non-orthogonal
Joint Blind Source Separation in the Least-Squares Sense. Paper presented at the 20th
European Signal Processing Conference, Bucharest, Romania.
Cui, X., Bryant, D. M., & Reiss, A. L. (2012). NIRS-based hyperscanning reveals increased interpersonal coherence in superior frontal cortex during cooperation. Neuroimage,
59(3), 2430-2437. doi:10.1016/j.neuroimage.2011.09.003
De Vico Fallani, F., Nicosia, V., Sinatra, R., Astolfi, L., Cincotti, F., Mattia, D., . . . Babiloni, F. (2010). Defecting or not defecting: how to "read" human behavior during cooperative games by EEG measurements. PLoS One, 5(12), e14187. doi:10.1371/journal.pone.0014187
Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., & Garnero, L. (2010). Inter-brain synchronization during social interaction. PLoS One, 5(8), e12166. doi:10.1371/journal.pone.0012166
Favre, D., Joly, J., Reynaud, C., & Salvador, L. (2009). Empathie, contagion émotionnelle et coupure par rapport aux émotions : validation d’un test pour repérer et aider des élèves at risque. 59(3), 211–227. doi:10.1016/j.erap.2009.03.004
Gneezy, U., Niederle, M., & Rustichini, A. (2003). Performance in Competitive Environments: Gender Differences. The Quarterly Journal of Economics, 118(3), 1049-1074. doi:10.1162/00335530360698496
Hari, R., & Kujala, M. V. (2009). Brain basis of human social interaction: from concepts to brain imaging. Physiol Rev, 89(2), 453-479. doi:10.1152/physrev.00041.2007
Key, A. P., Dove, G. O., & Maguire, M. J. (2005). Linking brainwaves to the brain: an ERP primer. Dev Neuropsychol, 27(2), 183-215. doi:10.1207/s15326942dn2702_1
Lachaux, J. P., Rodriguez, E., Martinerie, J., & Varela, F. J. (1999). Measuring phase synchrony in brain signals. Hum Brain Mapp, 8(4), 194-208.
Luck, S. J. (2005). An Introduction to the Event-Related Potential Technique. Cambridge, MA: MIT Press.
Luck, S. J., Woodman, G. F., & Vogel, E. K. (2000). Event-related potential studies of attention. Trends Cogn Sci, 4(11), 432-440.
Maris, E., Schoffelen, J. M., & Fries, P. (2007). Nonparametric statistical testing of coherence differences. J Neurosci Methods, 163(1), 161-175. doi:10.1016/j.jneumeth.2007.02.011
Montague, P. R., Berns, G. S., Cohen, J. D., McClure, S. M., Pagnoni, G., Dhamala, M., . . . Fisher, R. E. (2002). Hyperscanning: simultaneous fMRI during linked social interactions. Neuroimage, 16(4), 1159-1164.
Muller, V., & Lindenberger, U. (2011). Cardiac and respiratory patterns synchronize between persons during choir singing. PLoS One, 6(9), e24893. doi:10.1371/journal.pone.0024893
Muller, V., Sanger, J., & Lindenberger, U. (2013). Intra- and inter-brain synchronization during musical improvisation on the guitar. PLoS One, 8(9), e73852. doi:10.1371/journal.pone.0073852
Ogura, C., Koga, Y., & Shimokochi, M. (1996). Recent Advances in Event-related Brain
Potential Research: Elsevier.
Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J. M. (2011). FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data.
Comput Intell Neurosci, 2011, 156869. doi:10.1155/2011/156869
Polich, J. (2007). Updating P300: an integrative theory of P3a and P3b. Clin Neurophysiol,
118(10), 2128-2148. doi:10.1016/j.clinph.2007.04.019
Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., . . . Lécuyer, A. (2010). OpenViBE: An Open-Source Software Platform to Design, Test, and Use Brain– Computer Interfaces in Real and Virtual Environments. Presence, Teleoperators and
Virtual Environments, 19(1), 35-53. doi:10.1162/pres.19.1.35
Ruchsow, M., Grothe, J., Spitzer, M., & Kiefer, M. (2002). Human anterior cingulate cortex is activated by negative feedback: evidence from event-related potentials in a guessing task. Neurosci Lett, 325(3), 203-206.
