Multivariate pattern analysis for scientific inquiry in cognitive neuroscience: what does the brain tell us?

(1)

Multivariate pattern analysis for scientific inquiry in

cognitive neuroscience: what does the brain tell us?

Student:

Ron van de Klundert 11059915

Course code: 5244LTB12Y

Abstract

A fundamental topic in cognitive neuroscience is information representation; what is represented? Where in the brain is it represented? How is it represented? And how do these representations change during cognitive processing? This topic has traditionally been approached by performing univariate analysis, assessing different levels of activity in the brain. However this analysis approach ultimately obscures information in the patterns of neural activity. In contrast, multivariate pattern analysis (MVPA) provides a means to gain insights into these patterns of neural activity and their informational content. This study explores how MVPA contributes to the discipline of cognitive neuroscience. We examine some of the fundamental differences between univariate and multivariate analysis that lead to some of the core arguments in favor of the use MVPA, elaborate on the key approaches and what insights they provide for the discipline, followed by discussing some of the new advances made in this methodological framework and current highlights in MVPA research. Furthermore, we discuss contemporary challenges that come with the use of MVPA and their proposed solutions and wrap-up with some examples of how MVPA is being applied for practical aims. This study concludes that MVPA has proven to be a versatile and pow-erful tool for scientific inquiry in cognitive neuroscience. It has revolutionized the research questions that the discipline can investigate and provides a novel and unique way to analyze information representation and processing in the brain.

(2)

1 Introduction

1.1 Traditional analysis in cognitive neuroscience

Humans seem to effortlessly engage with information; thought processes and perception can seemingly flow without the need for active maintenance of these mental phenomena. Familiar faces are recognized in an instance and during our conversations language is processed mean-ingfully without actively ascribing semantics. In addition to the seemingly effortless mental processes, there are some that demand some mental “gymnastics”. Mental arithmetics or trying to remember a name of a person with a slightly familiar face might require some mental effort, for example. If we deem tasks important we sometimes try to actively direct attention to them and sometimes get passively diverted by external or even internal distractors. Most mental op-erations seem indispensable and in some sense shape how humans engage in their everyday life. This intrinsic recognition of oneself within these processes is exactly what sparks interests about them. How do we recognise faces, how does meaning get connected to words, how do we direct attention and how can all these mental operations be realized by the physical structure of the brain and its neural processing architecture? Cognitive neuroscience is the research discipline that tries to uncover exactly these types of burning questions.

A fundamental topic in this discipline is information representation; what is represented? Where in the brain is it represented? How is it represented? And how do these representa-tions change during cognitive processing? These quesrepresenta-tions have traditionally been approached by adopting an activation-based framework for scientific inquiry. The analysis within this frame-work is mainly occupied with assessing different levels of activity in the brain. In this sense, researchers relate a cognitive process under study to an overall higher firing rate of neurons, an increased BOLD response or deflections in an EEG signal. Different levels of activity are then in-terpreted as different levels of engagement with the cognitive process under study. Such a causal statement is argued as a valid interpretation when there is an absence of alternative explana-tions. (Weichwald et al., 2015). This approach has resulted in some of the most well-established theories in cognitive neuroscience. For example, a famous study by Kanwisher, McDermott and Chun (1997) demonstrated that a certain area in the fusiform gyrus was on average significantly more activated when subjects were presented with pictures of faces than when they were pre-sented with pictures of objects. The authors conclude that this area is selectively involved in face perception. This area is now well known as the Fusiform Face Area (FFA). The activation-based framework has been tremendously beneficial to answer some of the basic research questions in cognitive neuroscience. However, getting to the overall activity, the method at heart of the tradi-tional analyses, requires averaging across measurement units (e.g. voxels in a ROI or electrodes). This then only enables us to correlate this average activity with a tested condition. This ap-proach is limited in a sense that it ultimately obscures the information that could be present in the pattern of activation across these measurement units.

In contrast to traditional analysis, multivariate analysis provides a means to gain insights to the information present in the patterns of activation. As an example, some of these studies were able to classify the identity and location of the presented face based on the spatial activity pattern in the FFA (Anzellotti, Fairhall & Caramazza, 2014; Haxby et al., 2001). These studies were thus able to unveil the informational content that is discriminable from just the activity in the FFA during face perception, something that would not be possible by examining the pooled activity for face perception. Interestingly, these studies also revealed that this information is not exclusively discriminable from the FFA, but can also be discriminated from other regions. This refuted the idea of one dedicated area that represents faces, while proposing the novel idea of more distributed representations in the ventral temporal cortex.

1.2 What is multivariate pattern analysis (MVPA)?

The analysis technique employed by the face classification studies mentioned above is known as multivariate pattern analysis (MVPA). MVPA is a set of methods or tools to investigate the

(4)

patterns of neural activity, instead of the average neural activity. MVPA originally referred to multi-voxel pattern analysis, but since the emergence of the technique it has been applied to a broader range of different measurements such as EEG, MEG, ECoG and single-cell recordings. This makes the former terminology preferable to acknowledge the fact that its applications go beyond that of fMRI (Haxby, 2012). In contrast to traditional analyses, multivariate approaches adopt an information-based framework for scientific inquiry. The approach emphasizes on the informational content present in a pattern of neural activity. MVPA works by transforming the neural patterns associated with the conditions to vectors in a high-dimensional representational space (figure 1). The dimensions of this space are then N×M with N conditions (e.g. cogni-tive processes, stimuli categories) and M neural features (e.g. voxels, weight of electrodes). This higher-order representational space can then be further investigated by several MVPA methods.

The most commonly used application of MVPA is multivariate decoding. Decoding in this context is a synonym for classification in machine learning terms. Multivariate decoding is used to demonstrate that different stimuli or cognitive processes are subject to distinct underlying patterns of neural activation, by discriminating between them. The goal of multivariate decoding is to find a decision boundary that enables discrimination between the pattern vectors for each condition in the high-dimension representational space. This approach starts by separating neural data in independent training and test sets. The training set is then used by a classifier to determine, or “learn”, the decision boundaries for each stimuli or cognitive process under study based on the associated pattern of activity. The classifier is then assessed on classification performance with the remaining test set (figure 2). If the neural patterns are actually distinct for the conditions of study in such a way that they can be separated by the decision boundary, then classification of the test set should reach above chance performance.

A different application of MVPA is the representational similarity analysis (RSA). While mul-tivariate decoding is able to demonstrate the presence of distinctive neural patterns or represen-tations for conditions, RSA focusses on how these represenrepresen-tations are structured (Kriegeskorte, Mur & Bandettini, 2008). The goal in RSA is to determine the relative proximity of the pattern vectors for each condition in this high-dimensional representation space, unveiling the geometric representation of this information. This method affords to visualize the representational space where the relative proximity between conditions is a proxy for how similar their underlying rep-resentations are (figure 3). For example, faces and animate objects are relatively ‘closer’ in this space than faces and inanimate objects, indicating a higher representational similarity between faces and animate objects (Hanson et al., 2004; O’Toole et al., 2005). This provides more infor-mation than a simple multivariate decoding analysis, which would only indicate that these three categories are separable by distinct neural patterns.

Since their initial applications in cognitive neuroscience, these methods had time to advance and prosper as central tools for analyzing neural data, providing new insights on neural processing and functioning. However, how exactly does MVPA change the scientific inquiry of cognitive neuroscience and its exploration of the brain and mind? The aim of this literature thesis is to examine how MVPA contributes to the discipline of cognitive neuroscience. Below, we will delve deeper into some of the core arguments in favor of the use of MVPA (section 2), elaborate on the key approaches and what insights they provided for the discipline (section 3), followed by discussing some of the new advances made in this methodological framework and current highlights in MVPA research (section 4). Furthermore, we will discuss contemporary challenges that come with the use of MVPA and their proposed solutions (section 5) and conclude with some examples of how MVPA is applied for practical aims (section 6).

(5)

Figure 1: Representational space. MVPA works by transforming the neural patterns associ-ated with the experimental conditions to vectors in a high-dimensional representational space. The dimensions of this space are then N×M with N conditions (e.g. cognitive processes, stim-uli categories) and M neural features (e.g. voxels, weight of electrodes). This figure shows the pattern vectors for three conditions ranging from M1 to Mn (middle). The corresponding

(6)

2 Arguments for the use of MVPA

Before delving deeper into the topic of what MVPA has taught us in cognitive neuroscience it’s important to get a grasp on some of the main differences that results in arguments in favor of the use of multivariate analyses compared to classical univariate analyses. In this section some of the core arguments on this topic in the literature will be discussed.

2.1 The informational benefit

One of the major advantages of MVPA is the information-based framework. As stated in the introduction, traditional univariate analyses have dominated the field of cognitive neuroscience with an activation-based framework. The analysis at heart of this framework works by looking at the average activity across neural measurement units and correlating it with a cognitive process under study. Different levels of average activity are then interpreted as different engagement levels with the cognitive process under study. This approach is effective in a sense that it helps inferring which regions of the brain are likely to be engaged with a cognitive process, but the analysis procedure is somewhat problematic. Taking the average activity reduces the effective resolution of the neural measures and ultimately obscures the information present in the pattern of activity across the measurement units (Kriegeskorte & Bandettini, 2007). Additionally, one could argue that the average neural activity, like pooled BOLD response, is unlikely used by the brain itself to encode information. In this sense, the acquired measure has no clear functional significance for the processing architecture of the brain and thus, no clear functional relation to cognitive processes. MVPA on the other hand adopts an information-based framework. The analysis does not simply correlate average activity to a cognitive process, but it focuses on the statistical dependency between the patterns of neural activity and the cognitive process under study (Ritchie, Kaplan, & Klein, 2019).

MVPA achieves this by considering neural data in a principally different way than univariate analysis does. Firstly, MVPA takes each neural measurement unit into account, resulting in an increased sensitivity compared to only considering the pooled activity of these units. Sec-ondly, MVPA considers the data in a non-uniform way; both increased and decreased activity in a set of measurement units have value in the analysis of informational content. In contrast, univariate analysis only interprets activity in a uniform way; it assumes a process is subject to a uniform activity and the direction of activity is taken as an important factor for a cognitive process under study. Lastly and most importantly, MVPA is directly concerned with the possible informational content present in the neural data, instead of the activity present, resulting in an increased specificity. This last principle becomes clear with an example of inactivity. Within the activation-based framework of univariate analysis a brain region would be interpreted as unre-lated to a condition, if this region shows no activity given this condition. However, if this region becomes selectively inactive given this condition, this region would be considered informative about this condition within the information-based framework of MVPA. To give an example of the other principles once again consider the FFA in an fMRI study. Different facial expressions could be associated with different combinations of increased and decreased activity of voxels, which result in the same pooled activity. Considering only the pooled activity, as in univariate analysis, does not unveil any difference in activity between different facial expressions in the FFA. Only when all measurement units are taken into account it is possible to identify that different facial expressions are associated with distinct activity in the FFA.

The increased sensitivity and specificity of MVPA results in an informational benefit that revolutionized the research questions that cognitive neuroscience could ask. Instead of question-ing which regions are engaged durquestion-ing a cognitive process, the discipline is now able to question if and how information structure is related to cognitive processes. Directly investigating at the information level of analysis is also more likely to yield a measure with greater functional signif-icance, as the brain is commonly theorised to be a complex system that processes information that uses neural populations to encode the information (Panzeri, Macke, Gross, & Kayser, 2015; Pouget, Dayan, & Zemel, 2000). Simplified, this theory states that the brain receives information as input (e.g. sensory information) which the brain processes and encodes into a usable format.

(7)

Once this information is in the right format it can be further transformed to guide behaviour. This active transformation of information refers to cognitive processing. Neural populations pro-vide a means to encode this information through different combinations of distributed activity in this theory. The foundations of neural populations encoding information dates back to the early discussions of the infamous “grandmother cell”. This discussion resulted in a strong argument against single neurons encoding complex concepts and favoured the idea of information being encoded by ensambles, or populations, of neurons (Barlow, 1972; Gross, 2002).

2.2 Conceptual benefits

Having read Bredo (2015) I would argue that working within the information-based framework of MVPA has the additional benefit of being less prone to conceptual errors when investigating information processing. A conceptual error here refers to using the wrong conceptualization to approach a problem. In this case, I would propose that approaching, or investigating, in-formation processing with an activation-based framework is more prone to conceptual errors than approaching it with an information-based framework. As discussed in the previous section, MVPA is directly concerned with the informational content in neural measures. MVPA there-fore seems to directly operate at the information level of analysis, providing a presumably more correct conceptualization for the process under study. In contrast, univariate analysis operates within an activation-based framework that is concerned with the pooled activity of neural mea-sures. Approaching information processing by measuring activity levels is arguably a less correct conceptualization of the process under study, since it seems to require thinking about informa-tion in terms of a completely different concept, namely activity. This approach is not necessarily wrong, but it could result in less appropriate insights about how information processing is solved by the physical structure of the brain.

However, as Bredo (2015) stated, conceptual errors are incredibly difficult to recognize. De-termining whether one approach results in less erroneous conceptualizations than another may be ambiguous. Nevertheless, I think it is an overlooked yet important research practice to con-sider if our propositions or conceptualization under concon-sideration are warranted and related to the process under study. For this reason, I provided my own considerations to this matter, but I would like to emphasize that these ideas have not to be shared.

Besides considering whether our methodological framework provides a conceptually correct approach it is important to consider whether our assumptions about the process under study (i.e. cognitive concepts) are correct. Bredo (2015) argues that cognitive concepts used to explain be-havioral phenomena are commonly assumed to be concrete entities within the subject that cause the behavior. This assumption is partly problematic, as the cognitive concepts used might not be inner entities that underlie the behavior, but are merely accurate descriptions of the behavior. This practice could become even more problematic when cognitive concepts are based on vague theoretical postulates, making their formalisation challenging. As shown by Francken and Slors (2014), the translations between cognitive concepts and brain activity are often implicit and un-systematic, partly due to the lack of clear formalisation leading to ambiguous operationalization. Breaking down complex mental mechanisms into cognitive concepts that correctly formalise their operations is thus a challenging task at hand for cognitive neuroscientists.

MVPA could prove an important auxiliary role for this task because the methods afford to analyze how cognitive processes are structured with fewer assumptions than univariate analyses. Some literature argues that with MVPA we could rely less on conventional assumptions about cognitive architecture than with univariate analyses. This follows from the fact that MVPA methods afford to reveal how the brain structures these processes, obviating the need for strong a priori assumptions about cognitive structure (Carlson, Goddard, Kaplan, Klein, & Ritchie, 2018). An example of this would be visualizing the representational structure of object recognition with RSA. Instead of assuming which objects or visual features are similarly represented, we analyze how their representations are structured. This approach would not solve all problems with the conceptualization of cognitive processes, but might prove a step forward in the right direction.

(8)

2.3 Interim conclusion

In conclusion, the main argument for using MVPA over classical univariate analysis is that it provides an informational benefit. This informational benefit follows from MVPA taking each unit of measurement (e.g. activity in a voxel) into account when analyzing neural data, increasing the sensitivity compared to looking at the pooled activity of these units. Moreover, MVPA looks at data in a non-uniform way; both increased and decreased activity in a set of measurement units have value in the analysis of informational content. The direction of the neural responses is not important, but the pattern of the responses is at the heart of MVPA. In contrast, univariate analysis only interprets activity in a uniform way; it assumes a process is reflected by an uniform increase of activity. This direction of uniform activity is taken as an important factor for a cognitive process under study in classical univariate analysis. Most importantly, MVPA operates in a framework that is directly concerned with the possible informational content present in the neural data, instead of the activity present. This information-based framework has an increased specificity compared to the activation-based framework of classical univariate analysis. As we will see below in the following chapter, this increased sensitivity and specificity broadened the research questions that could be investigated in cognitive neuroscience. Additionally, an information-based framework arguably provides a conceptually more appropriate approach when investigating information processing. MVPA could also provide a more objective perspective on cognitive architecture by analyzing how the brain structures cognitive processes, obviating the need for strong a priori assumptions about the underlying cognitive structure. In a sense, we can let the brain ‘tell’ us how it structures.

3 General application

The next important question for this thesis is to answer how MVPA methods have been applied generally and what kind valuable insights it has given about cognition and the brains underlying neural mechanisms.

3.1 Multivariate decoding

As introduced before, multivariate decoding has been the most commonly adopted MVPA method in cognitive neuroscience. Readymade toolboxes to apply decoding for neural data with relatively straightforward output are now easily accessible (Hanke et al., 2009; Treder, 2017) , making its practice more attractive for researchers. To reiterate, the goal of multivariate decod-ing is to find the decision boundaries in the high-dimension representational space that enables discrimination between the pattern vectors for each experimental condition (figure 2).

One of the first studies that adopted a simple multivariate decoding approach was that of Haxby et al. (2001). They investigated how faces, cats and several objects were represented in the ventral temporal cortex. fMRI responses were measured while subjects viewed images of the experimental categories. They analyzed how each of the experimental categories was associated with patterns of responses in the fMRI data (i.e. voxels). Their hypothesis was simple and straightforward: If a stimulus of a certain category is reflected in a pattern of responses in the fMRI signal, then this response pattern should be more similar to independent measures made for that same category than to independent measures made for a different category. By using the correlation between patterns as a measure of pattern similarity they were able to demonstrate that each of the experimental categories were accompanied by a distinct pattern of responses for that category. Interestingly, their study demonstrated that these distinct patterns of responses for each category were independent of the regions that evoke maximal activation for these cate-gories. For example, even after removing the fusiform face area from their analysis, the patterns were still discriminative of face stimuli. These results were in contrast with the then generally accepted strong modular hypothesis, which assumed the presence of highly-specific regions in the brain, dedicated to representing only one category due to their biological significance. Instead, these results suggest that the representation of these categories are distributed across these re-gions and can be overlapping and that submaximal responses across rere-gions are an integral part of representing the information. This study sparked new interest in the idea of distributed neural

(9)

Figure 2: Multivariate decoding. Multivariate decoding is used to demonstrate that different stimuli or cognitive processes are subject to distinct underlying patterns of neural activation, by discriminating between them. A. This approach starts by separating neural data in independent training (above) and test sets (below). B. The training set is used by a classifier to determine, or “learn”, the decision boundaries for each stimulus based on the associated pattern of activity. The representational space for the training set is visualized for the first two dimensions of M with the corresponding hypothesized decision boundaries. C The classifier is then assessed on classification performance with the remaining test set. Each pattern vector in the test set is classified with the use of the learned decision boundaries from the training set. The representa-tional space for the test set is visualized for the first two dimensions of M with the corresponding classification of the test set based on the hypothesized decision boundaries.

(10)

population encoding category information as patterns of their activity, rather than highly-specific modules that encode information of one category as peak activity. Reanalysis of this dataset with more sophisticated decoding methods also confirmed the intuitions of the initial study (Hanson, Matsuka, & Haxby, 2004; O’Toole, Jiang, Abdi, & Haxby, 2005).

Not much later two papers that used multivariate decoding to assess information of edge ori-entation in the fMRI measures of the early visual cortex received a lot of attention. Haynes and Rees (2005) used multivariate decoding to provide the first direct measurement of unconscious orientation selective processing in the early visual cortex. Earlier studies demonstrated that visual gratings that do not reach awareness still produce orientation-selective aftereffects, sug-gesting that there is unconscious orientation-selective activation in the cortex (He & MacLeod, 2001; Rajimehr, 2004). However, as discussed previously classic univariate analysis uses aver-age signals, obscuring possible population encoded information about the orientation of gratings in the early visual cortex. With the use of linear discriminant analysis (LDA) classification, a classifier used to execute multivariate decoding, Haynes and Rees were able to classify which orientation was perceived from the associated fMRI measurements of V1, V2 and V3. This clas-sification was still possible when the orientation gratings were rendered invisible, but only from the fMRI measurements of V1. These results demonstrated that information about the orienta-tion of a grating can be represented in V1, even when participants have no conscious access to this information.

Concurrently, a study by Kamitani and Tong (2005) demonstrated that it was possible to decode the subjectively perceived orientation from fMRI measures of the early visual cortex. In this case, subjects were presented with overlapping orthogonal gratings and had to focus on either of the two orientations. With the use of a support vector machine (SVM), a different classifier used for multivariate decoding, this study was able to classify which of the two orien-tations was attended by the subject. Their results indicate that these subjective states of the mind are coupled with distinct patterns of neural activity. Furthermore, they are one of the earliest demonstrations that a cognitive process, specifically the target of visual attention, can be decoded from the patterns of neural activity with MVPA.

The large amount of attention these paper received also raised awareness of the value of multi-variate analyses, resulting in a series of studies that apply multimulti-variate decoding and papers that systematically explain the principles of multivariate decoding (Haynes & Rees, 2006; Norman, Polyn, Detre, & Haxby, 2006; O’Toole et al., 2007). The better understanding of multivariate decoding and more readily accessible software to conduct these analyses resulted in a multitude of new decoding studies that tackled novel research questions regarding the representation of information in the brain. For example, subsequent studies demonstrated that besides visual as-pects, auditory aspects such as sound categories, speech content and speaker identity are encoded through distributed neural populations in the auditory cortex (Formisano, De Martino, Bonte, & Goebel, 2008; Staeren, Renvall, De Martino, Goebel, & Formisano, 2009), as well as showing that observed actions (e.g. grasping or touching an object) can be reliably distinguished from fMRI patterns in the parietal and premotor regions with an invariance to changes in lower-level visual features (Ogawa & Inui, 2011). Work on cognitive aging has shown that the distributed patterns associated with visual categories are less distinctive in older adults, supporting the idea of dedifferentiation with age (Carp, Park, Polk, & Park, 2011). Another study was able to use multivariate decoding to demonstrate a neural distinction between visual working memory and attention (Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012). The authors showed that short-term retention of information does not require an active maintenance of the initial neu-ral representation of that information. Rather, only information that stays inside the focus of attention maintains an active representation and refocusing attention on previously unattended information restores the initial representation of this information.

3.2 Representational Similarity Analysis

As discussed above, representational similarity analysis (RSA) is another popular MVPA method that focuses on determining the relative proximity of the pattern vectors for different information

(11)

in the high-dimensional representational space. One of the advantages of RSA is that it affords to analyze if the same set of conditions differ in their representation structure between certain brain regions, while the multivariate decoding accuracy might not differ between those regions (Connolly et al., 2012; Kriegeskorte, Mur, & Bandettini, 2008). Another important advantage is that the conversion of pattern vectors in a representational space to a set of relative proxim-ities of those vectors results in a feature independent space (figure 3). All relative proximproxim-ities taken together form the representational dissimilarity matrix (RDM). Conversion to RDM af-fords to compare the same information in different representational spaces, such as comparing the representational geometry of the same set of visual stimuli in the human IT to a trained deep convolutional neural network (Khaligh-Razavi & Kriegeskorte, 2014).

The first instance of analyzing the similarity of neural representations was by a study of Edelman, Grill-Spector Kushnir and Malach (1998). This study used multidimensional scaling (MDS) to visualize the relative proximity of pattern vectors in the higher-order representational space associated with object categories. This visualization demonstrated that the categories form separate clusters in the representational space, most presumably driven by similarities within and dissimilarities between categories. Indeed, two reanalyses of the data from Haxby et al. (2001) that included RSA supported this idea of representation clusters, as they demonstrated a large distinction between the representations of animate and inanimate categories (Hanson et al., 2004; O’Toole et al., 2005).

Work by Kiani, Esteky, Mirpour and Tanaka (2007) took this a step further by analyzing the population responses of single neuron recordings to a multitude of visual stimuli in the IT cortex of monkeys. With RSA they were able to show that, just as in humans, there is a large distinction between the representations of animate and inanimate categories. Additionally, they showed that there were clear distinctions between the representations of other animal species, suggesting that monkeys possibly possess knowledge about species. As an addition to this study, another study showed that the representational structure in the human ventral temporal cortex is extremely similar to that in the monkey IT cortex (Kriegeskorte, Mur, Ruff, et al., 2008). This comparison further demonstrates the versatility of RSA, as a meaningful comparison between fMRI measures in humans and single neuron recording in monkeys was only possible by converting both to a feature independent space.

3.3 Searchlight analysis

Another important functionality of MVPA is the informational mapping technique, otherwise known as searchlight analysis. This is a method, specifically for fMRI, to localize informa-tive regions with more efficacy than mass-univariate analysis (Kriegeskorte & Bandettini, 2007; Kriegeskorte, Goebel, & Bandettini, 2006). Informative in this context refers to how distinctive the patterns of activity are for different experimental conditions or cognitive processes of study, based on multivariate decoding. This technique assigns an informational value to every voxel by analyzing that respective voxel and a small area surrounding it (the “searchlight”). Once every voxel has been assigned an informational value the result is a map that reflects how informative each region of the brain is about the experimental conditions. This technique affords to do whole-brain analysis, obviating the need for a priori assumptions about which regions are of interest (Etzel, Zacks, & Braver, 2013). After the identification of these informative regions one could proceed with RSA to unveil the geometric structure of the information in the representational space.

Above described a data-driven way to conduct a RSA, but searchlight also affords model-driven analyses. The model-model-driven approach tries to identify regions that have a neural RDM that can be predicted by a RDM of a model of choice. This model of choice could for example be a cognitive model based on subjective perception, a computational model of V1 cortical responses or deep convolutional network visual models (Haxby, Connolly, & Guntupalli, 2014; Khaligh-Razavi & Kriegeskorte, 2014). This model-driven approach was introduced in a study by Connolly et al. (2012), in which they validated the approach. They further showed that the neural RDM of the lateral occipital complex (LOC) showed high similarity with the behavioral RDM based on similarity ratings of the shown animal classes, whereas the neural RDM of the

(12)

Figure 3: Representational Similarity Analysis. RSA is a method that determines the relative proximities between the pattern vectors of each experimental condition in the higher order representational space. This figure shows the pattern vectors for three conditions (left) and visualizes the corresponding representational space for the first two dimensions of M (middle). The black dotted lines represent the relative proximities of the pattern vectors. The total set of relative proximities result in a feature independent space, unveiling the geometric representations of the three conditions (right).

(13)

medial occipital lobe showed high similarity with the computational RDM based on a V1 cortical responses model. With these results they demonstrated that neural activity in the LOC is likely to reflect the semantic structure, while activity in the early visual area is best explained by low-level image features. Their study further emphasizes how the MVPA methods are complementary in the searchlight analysis and how RSA is able to unveil the geometric structure of information that would remain implicit in more simple multivariate decoding approaches.

3.4 Interim conclusion

This chapter examined how some of the MVPA methods have been deployed in cognitive neuro-science research and what we have learned from their usage. Multivariate decoding has primarily been used to simply demonstrate the presence of distinct neural patterns of activation for a set of experimental conditions (e.g. stimuli, cognitive states). This technique provided convincing results against the strong modular hypothesis. Results of several studies suggest that percep-tual categories are encoded by neural populations that are distributed across the brain and that submaximal responses are an integral part of representing this information. Decoding was furthermore able to demonstrate that certain representations change in VWM when subject to cognitive processes such attention. In essence, multivariate decoding affords to analyze which information is discriminable and thus distinguish between brain states in different situations. RSA on the other hand is a more sophisticated MVPA method that looks at the structure of these brain states. Studies have used this method to dissect the geometrical structure of visual categories and showed that the representations of these categories cluster together based on the similarity between the categories. The results acquired with RSA support the idea that simi-lar features of visual stimuli are represented by overlapping neural populations, while distinct features are represented by more distinct neural populations. With the use of this technique it was even possible to demonstrate that the geometrical structure of animate and inanimate stim-uli are extremely similar between human and monkeys. RSA affords to analyze the structure of information that would be implicit in multivariate decoding. In addition, it enables us to compare the structure of information between brain areas, species and even computational and cognitive models; connecting the disciplines within cognitive neuroscience (Kriegeskorte, Mur, & Bandettini, 2008). Lastly, both multivariate decoding and RSA can complement each other in order to localize regions that are informative about our experimental conditions and dissect how this information is structured or to localize regions that are similar to the structure we obtained from models.

In conclusion, each of these methods taught us something about which information is rep-resented, where it is represented and how it is represented. More specifically, they taught us that categorical information of objects is likely represented in the VT and LOC in a distributed fashion, while low-level visual features are likely represented in early visual areas in a modular way. Object categories form separate clusters in the representational space, where categories with more similar features are ‘closer’ to each other than categories with more dissimilar features.

4 New advances and highlights in MVPA research

It has become clear that MVPA had a significant impact on the field of cognitive neuroscience, changing the questions we can ask and adjusting our knowledge about information representation. The next section will discuss some of the new advances in MVPA, discussing some of the new highlights in MVPA research and exploring novel ways to employ MVPA analysis.

4.1 Temporal generalization

The studies in the previous sections were mostly concerned with fMRI analysis and the spatial organization of information, however, cognitive tasks are often seen as sequences of operations that are not captured by a single instance of information representation. King and Dehaene (2014) describe two elegant ways of applying multivariate decoding to time-resolved neural mea-sures (e.g. EEG, MEG) to unveil the temporal dynamics of information representation, which

(14)

were first applied by Cichy, Pantazis, and Oliva (2014). The first method they describe is time-course decoding of which the principles are relatively straightforward. A different classifier is trained for each timepoint in the time-resolved signal and then tested on decoding performance for that specific time point. The output of this analysis is decoding performance as a function of time (figure 4), enabling us to identify when certain mental representations emerge in neural measures with high temporal resolution. With searchlight analysis as a method for identifying distinct neural patterns in the spatial domain, one could by analogy see time-course decoding as a method for identifying distinct neural patterns in the temporal domain.

Figure 4: Time-course decoding. Time-course decoding is a method that identifies distinct neural patterns in the temporal domain. This figure shows a hypothetical time-course decoding result. Some time after stimuli presentation the neural patterns associated with them become distinct and reach decoding accuracy significantly higher than chance level, indicating when and how long the representations of these stimuli are present.

The other method they describe is the temporal-generalization method, which essentially is an extension of the time-course decoding method. The principle is still somewhat straightfor-ward: A different classifier is trained for each timepoint in the time-resolved signal, but now each classifier is tested on decoding performance at all timepoints in the time-resolved signal. This essentially checks how well a certain classifier generalizes to other time points. For example, if a classifier is able to decode information from a distinct neural pattern at time point t and time point t’, it would imply that the distinct neural patterns at time point t recurred at time point t’. The output of this analysis is a temporal generalization matrix in which each row corresponds to the time point a classifier was trained on and each column to the time point a classifier was tested on. The decoding performance for each train-test combination can then be color coded to see how well classifiers were able to generalize over time. The temporal generalization ma-trix is able to provide insights into the underlying temporal processing architecture, by looking at how representations are dynamically transformed across time. The authors simulated seven hypothetical temporal processing architectures and provided the time-course decoding output and the temporal generalization matrix for these architectures (figure 5). As demonstrated by

(15)

this simulation, different underlying processing architectures could result in the exact same time-course decoding output. The temporal generalization method is therefore an essential tool for unveiling the underlying processing architecture the neural representations are subject to.

A study by Rajaei, Mohsenzadeh, Ebrahimpour, and Khaligh-Razavi (2019) used this tempo-ral genetempo-ralization method to characterize the processing architecture for object recognition under occlusion. Occlusions refers to part of the object being blocked, or occluded, from the perspective of the subject. In this study they used black circles to occlude objects to different degrees. They showed that the temporal generalization matrix for objects without occlusion consisted mainly of a diagonal pattern without off-diagonal generalization, indicating that objects that are not oc-cluded are processed in a sequential manner. In contrast, when objects were 60% ococ-cluded, some off-diagonal generalization emerged in the temporal generalization matrix, indicating a process-ing architecture with recurrent interactions. When recurrent processes were disrupted through visual-backward masking these off-diagonal generalizations disappeared, paired with a decline in recognition performance when objects were occluded. Taken together their results indicate that recurrent processes are necessary for the recognition of occluded objects. This study used the temporal generalization method as a means to unveil the underlying processing architecture for object recognition in simple and more challenging conditions.

Figure 5: Temporal generalization analysis (figure from King & Dehaene, 2014). The seven simulated hypothetical temporal processing architectures by King & Dehaene (2014) shown at the top of the figure. For each of these architectures the respective time-course decoding pat-terns and the time generalization matrices are shown. The time-course patpat-terns demonstrate that comparable decoding patterns can be caused by substantially different processing architectures. The architectures are only revealed examining the temporal generalization matrices.

4.2 A novel framework for unconscious processing

The introduction of MVPA furthermore opened up the possibilities of investigating more specific information processing in the brain. For this reason, Soto, Sheikh, and Rosenthal (2019) advo-cate the use of MVPA to investigate unconscious processing in a novel brain-based framework. In their paper they address numerous challenges with behavioral tests for unconscious processing, specifically the use of subjective or objective measures of awareness. They argue that the care-ful implementation of signal detection theory (SDT) sensitivity measures can be used to assess awareness unambiguously. Null sensitivity then reflects an objective measure of having no aware-ness. The problematic part of this implementation is that null sensitivity for any experimental condition is likely to produce weak to no effect on behavior. This absence of behavioral effects or behavioral differences between conditions is then interpreted as the absence of unconscious

(16)

or differential unconscious processing. The brain-based framework is introduced to obviate this interpretation and use MVPA to investigate the patterns of neural activity associated with ex-perimental conditions at null sensitivity. Even in the absence of behavioral effects MVPA could unveil if and how unconscious information is represented. In line with this framework there have been studies on semantic processing and short-term memory that reported no behavioral differences between unconscious experimental conditions, but did demonstrate distinct neural patterns for these conditions (Axelrod, Bar, Rees, & Yovel, 2015; Bergstr¨om & Eriksson, 2018).

4.3 Hand use predicts the structure of representations in the

sensori-motor cortex

Now that cognitive neuroscientists have become more familiar with the application of MVPA it has been deployed in more sophisticated manners to unveil how the psychical structure of the brain relates to information representation. One of those recent studies used MVPA to uncover how the representations of hand movements are structured in the brain and provided a novel and compelling explanation (Ejaz, Hamada, & Diedrichsen, 2015). Prior to this study it was known that the primary motor cortex (M1) does not have a characteristic somatotopic organization for individual finger movements. The body of literature preceding this study suggested that M1 neurons are tuned to combinatorial movements of all fingers with an associated overlapping distribution of activity in M1 for the movement of fingers. Using fMRI, this study also found no characteristic somatotopic organization for single finger movement and demonstrated that single voxels in M1 showed activity for movement of every finger. Additionally, the spatial activity pattern for finger movement in M1 was highly variable across subjects. To investigate whether this high variation in spatial patterns across subjects was a random variation or if it still reflects a common structure in representations they conducted RSA. The RSA unveiled that the representational structure of finger movement is actually invariant across subjects. This suggests that the variable spatial pattern of activity still reflects a common structure in representing single finger movement that is preserved across subjects. The study went a step further and checked whether the representational structure of finger movement in M1 could be explained by natural statistics of hand use. An existing dataset of muscle activity for each finger during daily life activity was used to calculate the distance between each muscle activity pattern between each finger (i.e. the RDM for natural hand use statistics). The relative distance between each finger in the hand use model correlated highly with the relative distance between each finger in the neural representations.

Taken together, the common representational structure in the variable neural activity pat-terns in M1 seems to be shaped by a learning process related to the common way individuals use their hands in everyday life. Importantly, this study demonstrates that different physical brain architecture can underlie similar representational structure across humans and suggests a clear relation with human behaviour. This is an elegant study that provides interesting insights to how the physical architecture of the brain is related to similar ‘mental’ representations and behaviour. Furthermore, the authors state that examining these invariant representational struc-tures could prove a vital step for gaining a better understanding into the changes in underlying brain architecture related to learning, aging or even pathological processes.

4.4 Task-dependent representational structure in semantic processing

A study by Wang et al. (2018) investigated how activity in the visual word form area (VWFA), a crucial region involved when reading, is modulated by semantic processing. Prior to their study it was assumed that the VWFA plays an important role in mapping meaning onto word forms for reading comprehension. However, how the VWFA is modulated in semantic tasks that require subjects to relate words by their semantic meaning was still relatively unfathomed. In the study by Wang et al. (2018) they investigated this matter by letting subjects perform two semantic tasks during which they had to categorize words either by their taxonomic (based on shared features) or thematic (based on frequent, congruent co-occurrence) relations while measuring BOLD responses in the VWFA. The authors used two different possible relations based on their hypothesis that different relations might be associated with different representational

(17)

structures. Importantly, in both tasks participants were asked to categorize the same words, only the relation with the categories was different between tasks. Comparison of the neural RDMs with the behavioral semantic RDMs from both tasks showed that the representational structure of the VMFA was task-dependent. This means that behavioral RDMs from the taxonomic related words only correlated significantly with the neural RDMs from the taxonomic task, while behavioral RDMs from the thematic related words only correlated significantly with the neural RDMs from the thematic task. Or put into simpler words, the neural patterns of taxonomically related words are more similar during the taxonomic task than during the thematic task and vice versa. These results suggest that the representational structure of words in VWFA depends on the semantic relations being judged. The authors propose that it is possible that VMFA encodes both taxonomic and thematic relations through neural population codes, where attention modulates the representational structure to correspond with the task at hand. An important insight of this study is that the structure of information can dynamically change based on the specifics of the task being performed. With these results the authors emphasize that this dynamic nature should be considered when studying neural processing to acquire meaningful results.

4.5 Interim conclusion

Since MVPA methods had time to prosper as scientific tools in cognitive science the applications of it have become more refined. As seen in the proposed method of King and Dehaene (2014), these methods can be applied to see how the information in neural signals are dynamically transformed across time and how these could relate to different underlying information processing architectures. Furthermore, MVPA affords a novel framework to investigate the processing of unconscious stimuli, even in the absence of behavioral differences. This opens up new doors for investigating how processes that do not guide behavior operate and possibly unveil which differences in neural patterns are necessary to reach awareness. Novel studies that employ MVPA seem to get more refined as well. A common pattern in current MVPA studies is that they don’t simply demonstrate distinct patterns of activity for experimental conditions, nor just construct their respective representational structure. They are focused on unveiling how the information in the neural patterns relate to cognitive models based on human behavior and thus strengthen the link between the identified representations and behavior. This practice is also of significant importance, as will be argued in the next chapter.

5 Contemporary challenges in MVPA

It has become clear that MVPA has altered the field of cognitive neuroscience and the questions it is able to investigate. The framework of MVPA has benefitted the field substantially and holds promising future applications. However, pursuing our voyage to understand the mechanisms of the mind using MVPA methods requires us to consider the conceptual and methodological challenges that come with its use.

5.1 Ambiguity of the source

The first major challenge in MVPA occurs when interpreting multivariate decoding results. Mul-tivariate decoding rests on the assumption that whenever a classifier is able to identify distinct patterns of activity for experimental conditions, it is using the underlying neural population encoding of this information to achieve this. However, it is rarely the case that the actual source of the information the classifier uses is known. It is arguable that without this knowledge mul-tivariate decoding does not deepen our understanding of neural processing and how the brain encodes information.

Let us again consider the earlier mentioned studies that were able to decode information about edge orientation from activity in the early visual areas (Haynes & Rees, 2005; Kamitani & Tong, 2005). It was already well known that V1 neurons are tuned to information about edge orientation (Hubel & Wiesel, 1968), which might make it seem trivial that these studies were able to decode orientation-related information from this region. However, the fascinating

(18)

question that remains unanswered how the classifier was able to pick up on this information represented in V1. The fMRI resolution used by these studies was much coarser (3 x 3 x 3 mm) than the columns in V1 that represent orientations (0.5 mm). Since multivariate decoding is only working with the information at the voxel level, it remains unclear how it is able to extract information that is represented at the level of these orientation columns.

This raised a debate as to how this was possible resulting in competing hypotheses that try to explain the possibility to decode orientation information. The corresponding authors of the papers propose that the proportion of neurons tuned to a certain orientation in a voxel could be biased, which multivariate decoding is able to exploit. Another possible explanation of successful decoding is that the orientation tuned neurons are unequally distributed across V1 (Freeman, Brouwer, Heeger, & Merriam, 2011; Sasaki et al., 2006). A more recent explanation is given by (Wardle, Ritchie, Seymour, & Carlson, 2017), arguing that the edges of orientation gratings could provide a source for successful orientation decoding. Ultimately, each of these explanations demonstrate the problem of source ambiguity. These three possible sources of information are all byproducts or artefacts that multivariate decoding is able to exploit for successful orientation decoding. In this sense, any information that can be used to distinguish between experimental conditions is likely to be picked up by multivariate decoding, even when this information is an artefact. Therefore it is necessary to identify the true source of information used by multivariate decoding to get to a meaningful interpretation.

5.2 Biological plausibility of MVPA

Following the challenge of source ambiguity is the question to what extent MVPA is biologically plausible. MVPA methods are advocated to unveil the informational content and representa-tional structure of the brain by many studies, but this interpretation might be ill-considered. The biological plausibility of MVPA relies on the assumption that the brain is an information-processing system that uses neural populations in a linear fashion to encode information. Using linear classifiers as a surrogate for the brain itself to decode this information seems reasonable under this assumption. In contrast, the use of non-linear classifiers are already deemed biologi-cally implausible by several authors, as they would be able to exploit too much information that is probably not exploited by the brain itself (Kamitani & Tong, 2005; Naselaris, Kay, Nishimoto, & Gallant, 2011). However, as argued in the previous chapter, the source of information used by linear classifiers is still opaque and can be artefacts of information that are probably func-tionally unavailable to the brain itself. So although the classifiers used by MVPA methods are able to successfully discriminate experimental conditions based on the measured neural patterns of activity, it remains unclear if the information in these patterns exploited by the classifier is actually used by the brain itself to represent these experimental conditions.

This is not only problematic for multivariate decoding, but also for RSA. One could construct the representational structure of experimental conditions from the measured neural patterns, but it remains unclear whether these representational structures are actually used by the brain.For example, even if neural population codes carry information that is accurately exploited by a classifier, the information in these neural codes might not be used to guide behavior. An exam-ple of this is the fact that perceived visual illusions can be decoded from V1 (Haynes & Rees, 2005). However, it is unlikely that conscious perceptual information is exclusively represented in the early visual areas. V1 probably carries a lot of information about visual stimuli, which a classifier can exploit, but it is unlikely that all the information in V1 is actually used to guide behavior. It is more likely that this information is forwarded for downstream processing to make it functionally usable. In this case, it is not the case that V1 lacks the information, but that this information is in the wrong format for the brain to actively use it (DiCarlo & Cox, 2007).

To move beyond the aforementioned challenge it is necessary to demonstrate a stronger link between the information used by MVPA and the underlying neural representations. Ritchie et al. (2019) suggests that the model-driven RSA approach where the RDM of a cognitive model, derived from subject behavior, is directly related to the RDM of neural measures. In this case it is more plausible that the underlying neural representations are in a usable format for the brain, as the representational structure is directly predicting subject behavior in a way that is plausible

(19)

in cognitive terms.

5.3 Interim conclusion

It becomes clear that MVPA results are hard to interpret without knowing the source of in-formation exploited by the methods. Successful decoding of experimental conditions does not directly enable the inference that this information is truly represented in the patterns of activity and RSA does not directly enable inferences about the true representational structure of experi-mental conditions. This section emphasizes that caution is necessary when drawing conclusions from MVPA studies and sheds light on the limitations of investigating the informational content with MVPA in general. Although these arguments seem to leave the methodology in a gloomy position, it does not discredit the approach. Rather, it is necessary to augment the methods in a way that strengthens the inferences possible with regards to neural representations.

6 MVPA for practical aims

Everything discussed about MVPA so far focused on using this methodology with the aim of explaining and gaining a better understanding of how the brain operates. This is however not the only way this versatile tool can be employed. Cognitive neuroscience has been cooperating with multiple disciplines, such as the clinical field, neuromarketing and engineering, to achieve novel ways to approach problems. It is no surprise that that MVPA has also found its way beyond the fundamental questions about the brain. Its predictive power makes MVPA an ideal candidate for studies with practical aims. In these studies the goal is to achieve maximum decoding accuracy to predict certain mental states, while the source of the information is not of interest. This obviates the need for the tedious conceptual challenges as mentioned in the previous chapter, with the only aim being to reliably predict mental states.

6.1 Prediction of medical scans

A common problem with progressive neurodegenerative disorders is that their respective diag-nosis mostly only happens years after the onset of the neurodegenerative processes. This is because the behavioral and physiological phenotype associated with the disorders only start to emerge after these degenerative processes have progressed beyond a certain point. Prediction, or diagnosis, of progressive neurodegenerative disorders at an early stage is highly desirable to start early intervention and improve the prognosis for the patients. Studies therefore use mul-tivariate decoding of medical brain scans to identify whether there are early biomarkers that indicate the presence of neurodegenerative disorders as means for diagnosis. An example of such a study was that of Garraux et al. (2013) were they used multivariate decoding to dissociate between Parkinson’s disease (PD) and atypical parkinsonian syndromes (APS). In this study they looked at 120 patients that exhibited Parkinson-like symptoms, but were outside the per-ceptions of a clinical diagnosis at the time of the medical scan. The medical scans that they analyzed were all conducted between 1993 and 2009, and all consisted of positron emission to-mography (PET) scans of fluorodeoxyglucose (FDG) uptake patterns. FDG PET was thought to reflect specific differences between PD and APS by capturing abnormalities associated with each disorder. Years after the medical scans all of the patients received their medical diagno-sis, which also enabled them to see how well multivariate decoding could predict the diagnosis from FDG PET scans years before the diagnosis. Dissociating between PD and APS based on their FDG PET scans with binary classification reached a predictive sensitivity accuracy of 90%, however, dissociating between PD and three APS classes did only reach classification accuracies ranging from 45-62%. With these results the authors demonstrated that multivariate decoding can be used as a potential diagnostic tool to differentiate between PD and APS at an early stage.

Another study looked into the possibilities of identifying early neural changes in Hunting-ton’s Disease (HD) with multivariate decoding (Rizk-Jackson et al., 2011). HuntingHunting-ton’s Disease already has a known genetic marker, the huntingtin gene. Although it is known that all patients

(20)

that carry this gene will eventually develop the clinical image of HD and the associated neu-rodegeneration, early estimation of how far patients have progressed in the disease is desirable to for example assess the effectivity of neuroprotective trials. Clinical diagnosis of HD is based on the expression of motor symptoms, but neurodegenerative processes precede the expression of these motor symptoms (Beglinger et al., 2005). With the use of multivariate decoding it was possible to distinguish the fMRI and structural scans of pre-symptomatic HD and controls with accuracies reaching up to 76%. This accuracy is not extremely high, but the goal of this study was to assess early disease progression and not disease classification, as HD already has a pene-trant genetic marker. With the use of multivariate methods they were able to get measures that correlate strongly with other known measures of disease progression, such as years to clinical onset. This demonstrates that multivariate analyses can be used to assess disease progression, presumably by MRI measures that reflect neurodegeneration.

6.2 Classification of mental states in Unresponsive Wakefulness

Syn-drome

Unresponsive wakefulness syndrome (UWS) is a disorder of consciousness (DOC) where patients show no response to commands but do exhibit some reflexive movements. UWS is more generally known as a vegetative state, but UWS was proposed as a new term for this neurological disorder to preserve dignity for the patients suffering (Laureys et al., 2010).

fMRI studies with these patients have demonstrated that some patients show signs of con-sciousness. However, the inconvenience of fMRI combined with the high costs make it an unfeasi-ble cost-effective method to assess the level of awareness for many patients. For this reason Cruse et al. (2011) aimed to assess the possibility of awareness detection through EEG multivariate decoding. They included 16 diagnosed UWS patients in their study and each patient went to a motor imagery procedure during which they were instructed to either perform right-hand moto imagery or toe motor imagery. They used multivariate decoding to see whether EEG pattern signals for both of these motor imagery instructions were distinguishable in these patients. It was possible to classify which motor movement was imagined for three out of the 16 patients, indicat-ing that they exhibited awareness of the task instructions. This was possible even though these three patients were diagnosed as entirely unaware through several behavioral assessments by specialists. These findings showed that EEG can be a cost-effective method to assess awareness in UWS patients.

6.3 Classification for neural commands in Brain-Computer Interfaces

The possibility to classify self-generated mental states has huge implications for the development of Brain-Computer Interfaces (BCI). BCIs are systems that try to translate brain signals into digitalised commands that are used to direct certain devices. These devices could be anything from cursors on personalised computers to robotic prosthetic devices. The development of BCIs is mainly focused on improving the quality of life for those that are unable to interact with their environment through typical means, like people that suffer from quadriplegia. With the introduction of MVPA a multitude of studies aimed to use multivariate decoding as a tool to classify neural “commands”. Neural commands are neural responses that are evoked by individ-uals by engaging in mental operations, such as motor imagery. A study by Sun, Xiang, Sun, Zhu, and Zeng (2010) showed the potential use of EEG for a non-invasive BCI. Three participants performed motor imagery of left-hand, right-hand and foot movement while EEG measurements were recorded. A classifier was trained to distinguish between the EEG patterns associated with the three motor imagery conditions, corresponding to three neural commands. These commands were then used as three degrees of freedom to control a simulated man in a video game envi-ronment, with the highest classification accuracy being 86.3%, 91.8%, and 92% for the three participants respectively. The BCI system used in this study was also able to update during gameplay to improve classification performance, while participants also showed some learning. These results showed that EEG signals associated with motor imagery can be reliably decoded with MVPA and thus reliably used as neural commands for a non-invasive BCI system. With this, the authors demonstrated a BCI system to train and effectively use motor imagery as neural

(21)

commands.

It is important to note that BCI systems are already being deployed in several case-studies to improve the quality of life of these patients with relative great success. These case studies mostly involve an invasive electrode that measures neural activity with greater specificity than EEG, thus allowing for greater control of more degrees of freedom with the neural commands. Vansteensel et al. (2016) report about a woman with locked-in syndrome (LIS) that is able to fully control a BCI system for communication. The system works with a subdural electrode strip placed on the hand region motor cortex. After several training sessions this woman was able to control the system through hand movement imagery, which worked in conjunction with her eye-tracking device. Another clinical study reported about a patient suffering from quadriplegia that was able to control a high-performance prosthetic arm (Collinger et al., 2013). This prosthetic arm was controlled through a BCI system that uses intracortical microelectrodes implanted in the motor cortex. Using only neural commands this patient was able to control the prosthetic arm with seven degrees of freedom, corresponding to orientation and grasping movements. Success rate in a task where the patient had to reach a target was 91.6%. Both of these studies concluded significant improvement in the individuals capacity to engage with the external environment through the use of BCIs.

6.4 Decoding for neuromarketing: subjective naturalistic experience

through words

Besides prediction for clinical purposes MVPA is being deployed to predict economic behaviour from neural measurements. Neuromarketing uses MVPA as a tool to decode mental states that provide insights into the appraisal of a product, the effectiveness of commercials or predict consumer behaviour. Literature suggests that neural data can be a more reliable predictor for consumer behaviour than subjective reports. For example, multivariate decoding has proven to be an effective tool for predicting the purchase behaviour of music and ratings of television content (Berns & Moore, 2012; Dmochowski et al., 2014). In a recent study Nishida and Nishimoto (2018) propose an interesting framework that could aid neuromarketing. They were able to decode the naturalistic experience of perceptual content through word representations from fMRI BOLD measurements. To achieve this they had subjects annotate several naturalistic videos and projected the words used to annotate the videos into a word vector space based on natural language statistics, resulting in a set of scene vectors. Then they proceeded by training a classifier to estimate the scene vector representations from the neural responses associated with watching those scenes. This classifier is then used for word-based decoding of naturalistic scenes. The exact details of their procedure go beyond the scope of this literature thesis, but with this method they were able to decode words that were likely to reflect the perceptual experience from neural activity. The content they were able to decode from the neural activity reflected perceived actions, objects and impressions. Moreover, the decoded words reflected inter-individual variability of the perceptual experience. The results of this study suggests that the word-based decoding framework is able to unveil the subjective content of the perceptual experience of subjects when watching naturalistic videos. This opens up ways to retrieve subjective perceptual information from individuals without them having to report it actively. The authors suggest that especially the ability to decode subjective impressions from neural data could provide a powerful addition for predicting and assessing economic behavior.

6.5 Interim conclusion

MVPA has proven to provide substantial value for practical aims in cognitive neuroscience. Specifically multivariate decoding proves to be a powerful for early, pre-symptomatic identifica-tion of neurological disorders. The principle of multivariate decoding as a diagnostic tool can be seen as the accurate and precise identification of deviating neural patterns in patients with neurodegenerative disorders. This identification seems to be possible long before clinical symp-toms emerge. In addition, MVPA seems to provide a feasible way to determine the presence or absence of consciousness in patients with unresponsive wakefulness syndrome, even when trained

Multivariate pattern analysis for scientific inquiry in cognitive neuroscience: what does the brain tell us?