University of Groningen
Detecting Mind-Wandering with Machine Learning
Jin, Christina
DOI:
10.33612/diss.171835555
IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from it. Please check the document version below.
Document Version
Publisher's PDF, also known as Version of record
Publication date: 2021
Link to publication in University of Groningen/UMCG research database
Citation for published version (APA):
Jin, C. (2021). Detecting Mind-Wandering with Machine Learning: Discovering the Neural Correlates of Mind-Wandering Through Generalizable Machine Learning Classifiers with EEG. University of Groningen. https://doi.org/10.33612/diss.171835555
Copyright
Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).
Take-down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum.
5
SUMMARY AND DISCUSSION
The general goal of the thesis was to determine the neural correlates of mind-wandering by training machine learning classifiers with EEG. Ideally, such a classifier should perform above chance level when transferred to a different task, group of participants, or study. We arrived at such a classifier in a series of steps. 5 . 1 O V E R A L L S U M M A R Y
First, in Chapter 2, Experiment 1, we tested multiple EEG markers of mind wandering: single-trial P1, N1 and P3 from four selective channels representing key frontal and parietal-occipital positions, as well as power and phase coherence information at the alpha (8.5-12Hz) and theta (4-8Hz) bands, to train intra-subject SVM classifiers. We found that individual model accuracies varied in the range between 50-85%. Moreover, when testing the classifiers trained on one task on data from the other task, they still achieved an accuracy of 60%. This accuracy is significantly above chance level, indicating that the classifiers were task-general. We found the alpha power was the most predictive EEG marker of mind-wandering among all the computed markers. These findings support the perceptual decoupling theory.
Following up on that, we aimed in Chapter 3 to differentiate mind-wandering from two closely-associated phenomena – being in a situation of low task demands and being in a state of low vigilance. All three are reported to have a similar effect on task performance and lead to reduced sensory processing of external stimuli. We found that the EEG classifier that was effective at predicting task demands or vigilance could not predict mind-wandering. Meanwhile, similar to what we reported in Chapter 2, an effective mind-wandering classifier could predict across tasks, even while in this chapter we changed the modelling from an intra-subject level to an inter-subject level. Moreover, after examining the importance of the source localized ICs in the alpha band, we found that different neural correlates underlying the three phenomena explained the vigilance/demands classifiers’ failures in predicting mind-wandering. This experiment demonstrated how it is
106 S U M M A R Y A N D D I S C U S S I O N
possible to uncover the neural mechanism of these processes by using EEG source localization combined with machine learning.
In Chapter 4, we sought to increase the accuracy of the mind-wandering predictions by training deep neural networks on the data we collected in Chapter 2. Crucially, we tested the obtained classifiers on the data of Chapter 3, making this an across-study prediction. We used CNN instead of SVM as the training algorithm. We used the global raw EEG, stERPs, and power/phase clustering from a broad frequency range, including theta, alpha, beta, and gamma bands as inputs instead of the few hand-crafted EEG markers in Chapter 2. We found that a subset of channels for each type of inputs is sufficient to train a mind-wandering classifier. The across-study prediction accuracy was .68 on average. Furthermore, we noticed that the balancing state of the original training dataset had an impact on the detection biases among the classes even after proper balancing procedures were performed during training. Specifically, overall testing accuracy dropped but the accuracy of the minority class increased when training with a better-balanced subset of the datasets.
Across the thesis, we evolved the validation method hierarchically (Table 5.1) to expand the generalizability of the classifiers. In Experiment 1, we made across-task predictions. In Experiment 2, we moved from intra-subject modelling to inter-subject modelling. In Experiment 3, we tested the classifiers on the data from an independent study (experiment). The final classifier we arrived at could predict across tasks (of the same experiment), participants, and experiments. The accuracy we achieved increases from .6 in Experiment 1 and 2 with SVM classifiers to .68 in Experiment 3 with CNNs.
Table 5.1 Hierarchical validation methods
across-tasks (within the same
study) across-individuals across-studies
Chapter 2 x
Chapter 3 x x
We also explored different types of EEG as the features or inputs for the classifiers. In Chapter 2, we found that sensory-decoupling-related EEG markers like stERPs and alpha power best predicted mind-wandering. In Chapter 3, we improved the spatial precision of the alpha oscillations by performing the ICA and the dipole fitting analysis to obtain the source-localized EEG ICs as classifier features. In Chapter 4, we surprisingly found that raw EEG as input performed best with CNN compared to the more processed features used in the preceding chapters. Altogether, it seems that handcrafted EEG markers P1, N1 and P3, parietal-occipital alpha power, and source-localized EEG signals are the best features for training traditional machine learning classifiers like SVMs. In contrast, CNNs are more capable of taking advantage of the raw EEG; this provides convenience when the models are trained in situations where there is little prior neural or cognitive knowledge.
Finally, our studies highlight a hitherto underestimated factor – the class ratio – for training EEG classifiers. The validation accuracy was influenced by the original balancing state of the training data. Even with balancing procedures performed during the training, we still found a strong positive correlation between class accuracy and the class size (Chapter 2 and 4). This leads to risky situations when testing the classifiers in real-life applications since higher accuracy in the majority class often contributes to a higher overall accuracy. Whether to further training the classifiers by collecting more minority cases or stay with the current class ratio should be up to the specific context. For example, if mind-wandering detection were to be applied in a normal office environment, the detection accuracy of mind-wandering (minority) would be lower while the accuracy of the detection of on-task states would be high. This could be acceptable since mind-wandering during office work often has few serious consequences and might even benefit the mental health or may satisfy personal long-term goals. Besides, too many warnings being created in the office might cause unnecessary stress. However, if the detection is applied to a working environment that requires intense sustained attention, for example, urban driving, the accuracy of mind-wandering detection should be guaranteed at a high level since attentional lapses might cause serious consequences like a road crash. In such cases, collecting more data to obtain more mind-wandering examples might be a solution, which may not always be feasible. Moreover, determining exactly how much data is enough is challenging given the large individual differences in mind-wandering rates.
108 S U M M A R Y A N D D I S C U S S I O N
5 . 2 M O D E L L I N G M I N D - W A N D E R I N G
In this thesis, I have focused on measuring the occurrence of mind wandering with EEG classifiers. These findings could inform cognitive theories of the underlying mechanisms of mind-wandering.
The traditional concept of mind-wandering describes it as attentional disengagement from the primary task (Cheyne et al., 2009). This views mind-wandering as a state diametrically opposite to task engagement. However, other accounts suggest the distinction between mind-wandering and being on task is more gradual, with mental resources alternating between being externally- and internally directed (Smallwood & Schooler, 2015). Usually, performing tasks requires attentional focus to the task-related stimuli by activating the sensory cortex. When mind-wandering occurs, part of the cognitive resources shifts from the external stimuli to the internal self-retrieved task-irrelevant memories (Christoff et al., 2016; Ellamil et al., 2016). When memory-related structures are recruited during mind-wandering, sensory processing is reduced, which is referred to as sensory/perceptual decoupling (Kam & Handy, 2013). In this sense, mind-wandering is a facet of the dynamics of the attentional fluctuations, which when turned inwards are associated with increased automatic memory retrieval processes that generate the material for mind-wandering and reduced sensory processing that allow the attention to focus on this internal stimulus material. One of the computational models that best catches this dynamic is to view mind-wandering as a cognitive function that is dual tasking with task-related cognitive functions. In this dual-tasking framework, participants have two goals to attend to. One is performing the task; the other is to engage in self-generated thoughts. Initially, the tasking goal activates higher than the mind-wandering goal. This represents that people start working with a focused mind state. The activation of the task goal gradually declines until it drops below the mind-wandering goal, causing the attentional focus to shift from outside to inside and a habitual response will be made instead of a controlled response. After realizing that it is mind-wandering – for example after an error – the goal activation of task-attending bounces back, indicating a re-engagement with the task (van Vugt et al., 2015). An account of the switch between mind-wandering and on-task focuses on the transition moment (Mittner et al., 2016). They combined the neuromodulation evidence of the locus coeruleus norepinephrine (LC-NE) system and proposed an
off-focus state before on-task changes to mind-wandering, supported by an above-optimal amount of tonic LC-NE activity. During the period of increased neural gain, the off-focus state is reflected by the connectivity between the DMN and multiple distinct neural networks, including the dorsal attention network (DAN) and the medial temporal lobe (MTL).
Neural models of mind-wandering stress that the connectivity between multiple networks decides the form of thoughts. In the review of Christoff et al. (2016), they put forward the concept of the source of variability as the main content constituting thoughts, and two types of constraints – automatic and deliberate constraints – controlling the forms of thought. The neural basis of the source of variability includes the sensorimotor cortex, using outside information to form the thought, and the MTL, retrieving memories to form the thought. The automatic constraints are thought to be enforced by the salience network, DAN and the DMN cores (amPFC and PCC). The function of the automatic constraint is to limit the thought to a restricted set of information, especially those of high emotional or sensory salience. The deliberate constraint is enforced by the frontoparietal control network (FPCN), which implements cognitive control. Mind-wandering in this framework is defined as thoughts generated from MTL with low levels of deliberate and automatic constraints. Mind-wandering is thereby contrasted with other modes of thinking, such as rumination, which are thoughts under strong automatic constraints, or intentional mind-wandering, which are thoughts under strong deliberate constraints.
The behavioral results in the current research support the dual-tasking framework, in which mind-wandering is considered one of the tasks carried out simultaneously. In Chapter 2, mind-wandering is only apparent in the behavior as an impairment on the visual search task but not on SART performance. In Chapter 3, we did not find any difference in behavioral performance between self-reported mind-wandering vs. on-task. This demonstrated that mind-wandering is not a mental state that is completely opposite from focusing on the task. While people do reduce the amount of mental effort in dealing with the outside world, they still keep part of their sensory processing functioning at a minimal level to sustain their performance in the current environment.
As people switch from the task to the mind-wandering state, a process of sensory decoupling takes place to facilitate the internal focus of mind-wandering. Our EEG findings support the presence of sensory decoupling during mind-wandering. In Chapter 2, we observed reduced P1, N1 and P3 while participants were
mind-110 S U M M A R Y A N D D I S C U S S I O N
wandering, in combination with enhanced alpha power. In Chapter 3, we replicated this increased alpha power during mind-wandering, which was localized at the precuneus. Moreover, our results in Chapter 3 were consistent with the idea that mind-wandering involves memory retrieval, since we observed a source at the STG – a neural structure involved in self-referential processing and semantic processing that predicted the presence of mind-wandering.
My thesis further suggests that any cognitive or neural models of mind-wandering should take individual differences into account. It was obvious in Chapter 2 and 4 that people engaged in widely differing amounts of mind-wandering when performing the same tasks (Figure 2.5, 4.3). Individual cognitive capacity, personality, or pre-task state might have influenced their mind-wandering frequencies or properties during the experiment (as reviewed in ‘Contributing factors’, Chapter 1). Future experiments could measure some of the possible influencing factors and incorporate these measurements as a parameter during modelling (McVay & Kane, 2009). For example, working memory capacity might be fitted to the slope of the decline in activation of the task goal, which allows mind-wandering to emerge. Participants with higher working memory capacity may be able to maintain their task goal longer, and this would decrease the slope of the task goal activation.
5 . 3 F U T U R E D I R E C T I O N S
Given these results, how could we best conduct studies that aim to train efficient EEG classifiers of mind-wandering using EEG?
At the experimental design level, we suggest to include participants with higher self-awareness or selecting them based on an assessment of their self-awareness level using a relevant measure. A possible reason that our testing accuracy could not surpass 70% may be that people are not very good at distinguishing their own mental states, which creates substantial noise in the signal. As the self-reports still serve to be an important tool to label the data currently, a higher level of self-awareness would be vital to ensure the labelling quality (Fleming & Dolan, 2012; Tan et al., 2014). For example, descriptive experience sampling (DES) has been proposed as a reliable way to label the inner experience (Fernyhough et al., 2018; Hurlburt & Heavey, 2002).
Apart from this, more mind-wandering dimensions could be measured during the probe presentations, for example, asking participants to report their temporal orientation, emotional valence, and the stickiness level of the thought, etc. This information could provide extra information to study the subtypes of mind-wandering, like maladaptive vs. adaptive mind-wandering. We suggest this, because the current off-task category might be too broad to derive useful neural correlates. Including more precise information on the type of mind wandering is likely to lead to a more precise result as it has a more homogeneous brain state. It would also help study the dynamics of mind-wandering since our mental changes are probably more like a fluctuating wave instead of jumping at two ends of the dichotomy. Learning more about the specific features of mind-wandering could help discriminate the possible microstates.
At the EEG recording level, we propose to include resting-state EEG as an extra measure and examine whether it can be predictive of participants’ overall mind-wandering rate or frequencies of attentional shifts during the experiments since many ‘stable’ factors like personality traits or cognitive ability are relevant (see ‘influential factors’ in Chapter 1) and resting-state EEGs are associated to those factors (Jach et al., 2020; De Pascalis et al., 2020).
At the input or feature level for the classifiers, we think the connectivity between the EEG sources is also a promising direction to explore since it connects the EEG to neuroimaging studies that have reported similar correlations (Christoff et al., 2016). Such connectivity has only been connected with mind-wandering on average, but as Mittner et al. (2014) have demonstrated, it can also be used to predict mind-wandering on single-trial levels. Future machine learning studies could further explore whether classifier performance may be optimized by improving spatial resolution by techniques such as spatial filters or by using source-localized EEG, like we did in Chapter 3.
5 . 4 C O N C L U S I O N
In conclusion, the current thesis confirmed the possibility of training efficient mind-wandering classifiers using neurophysiological measures. First of all, these EEG findings promote the understanding of mind-wandering by largely supporting the perceptual decoupling and the thought generation accounts of mind-wandering. Secondly, we achieved classifiers that can be generalized across
112 S U M M A R Y A N D D I S C U S S I O N
tasks, participants and studies, making it more likely that mind-wandering detectors on the basis of neurophysiological data can be developed for daily-life or therapeutical purposes in the future.
