Locating and decoding decisions in the human brain

Locating and decoding decisions in the human brain

Author: Valentijn M. T. de Jong Student number: 10120998

Beta-Gamma, track Brein & Cognitie Tutor: Tobias Donner

Being able to predict a person’s future decisions is of magnificent importance to law makers, economists, advertisers and many others. If one could do so, one could know whether a certain advertisement causes people to buy a certain product, or even whether a criminal will behave in criminal activities again. Whereas most research in this field is still of more philosophical importance, the field is rapidly evolving and will become increasingly important. Practical applications have been achieved in the practice of lie detection. This paper focuses on finding out how and where decisions are represented in the human brain and which areas encode which part of the decision making process. Some recent game-changing studies involving decoding intentions and decisions, using the latest techniques such as multivariate analyses of fMRI data, are discussed.

Already in the 1980s, Libet and his colleagues (cited by Haynes, 2011) investigated brain activity, recorded with an electroencephalogram (EEG), preceding motor activity. The subjects were asked to move a finger whenever they wanted to, while watching a rotating clock. After pressing the button, they would report at which time they first felt the urge to do so. A few hundred milliseconds before the subject wanted to move the finger, based on the subjects’ reports, a deflection in the EEG signal was recorded, which was termed the readiness potential (RP). This RP was seen as evidence that upcoming motor actions could be predicted based on brain activity, even before the subject had consciously decided to do so. However, this temporal difference could also be caused by the subjects’ inability to read the clock accurately.

For the past decades, neuroscientists have been searching for neural correlates of various higher order processes. Using increasingly accurate imaging techniques, such as fMRI, the neural correlate of decision-making is being unraveled too. By comparing the level of activity of brain regions across conditions, and using lesion studies and studies in monkeys, researchers have found several regions to be active during decision-making, specifically the SMA and the pre-SMA but also regions in the parietal cortex and more in the frontal cortex (Haynes, 2011). Knowing a brain structure is active during the decision-making process is valuable; however, knowing how it is involved is more complicated. If a subject is scanned by an fMRI again, the activity of these regions cannot be used to determine what the outcome of the subjects decision is, only that a decision is being made. To decode a decision, one must look at what brain areas are active in one choice, and which are active in the other. This activity pattern can then be used to decode decisions from brain activity. Moreover, these techniques can be used to predict future decisions, by analyzing the brain activity pattern

preceding a decision. This paper focuses on identifying which brain areas encode for which specific decision and how each area takes part in these decisions.

One of the main problems in cognitive neuroscience is that of causality. When a correlation is found between two variables, this only means they covary together. It does not mean that one variable is causing the other to change. Therefore, when decoding decisions from brain activity, one can’t know what exactly is going on. Even though the decisions can be predicted, one can’t know whether the measured brain activity is causing the decision to be made, or perhaps other brain activity is causing this. Also, one can’t know what caused the brain activity. Only when the experimenter causes the independent variables to change, one can speak of a causal connection. In experiments where the subjects discriminate between stimuli, this is the case, as the experimenter can change the stimuli. On the contrary, in experiments where no cues are provided to the subjects, causality cannot be inferred. As will be revealed below, in these studies priming effects can cause problems. Also, there is the criterion of precedence, which means that the cause has to occur before the effect (Haynes, 2011). Different techniques to circumvent this problem will be discussed. Primarily,

researchers have focused on predicting decisions before they have been made. The question that still remains is how this translates to the real world. This paper also aims to ascertain whether the measured variables can also be used to predict real-world decisions or whether they are they tied to the specific protocols used in the lab. Before exploring how future decisions can be decoded, one must first know which areas are active during decision-making.

Identification of decision-making brain areas

Traditionally, when investigating which brain areas are involved in a specific process, researchers select several brain regions depending on previous research, and compare their amplitudes of activity between conditions. However, in the past years, multivariate

techniques, such as pattern classification, have made the decoding of decisions more accurate than before (Gallivan, McLean, Valyear, Pettypiece & Culham, 2011). Whereas in the older methods the amplitudes of voxels, brain areas, were analyzed separately, these multivariate techniques analyze activity from voxels together. As multiple brain areas work together to achieve the same goal, they might all encode for a small process necessary to achieve that goal (see Figure 1). Analyzing them together allows one to predict specific decisions (Davatzikos et al., 2005, Haynes, 2011). In other words, higher decoding accuracies are

achieved by multivariate techniques, because they can take the interactions between brain regions into account (Davatzikos et al., 2005).

Figure 1. Fictional example of multivariate spatial pattern analysis. When each area is

analyzed separately, each encodes for one decision. When analyzed together, a higher

decoding accuracy is achieved for both decisions. Adapted from (“Outer surface of the human brain,” n.d.) and Haynes (2011). The decisions together comprise a specific decision.

As decoding brain intentions requires choosing certain areas to decode, it is important to know beforehand what brain areas are most active during decision-making. Cunnington, Windischberger, Deecke & Moser (2002) investigated this by scanning subjects with an fMRI while they performed button pushes. More precisely, the subjects were asked to do three rapid button presses, following an auditory cue after a variable interval in the externally triggered condition, or any moment they liked after a variable interval in the internally triggered condition. As the supplementary motor area (SMA) is often associated with internally

triggered movements it was surprising that little difference was found between the groups’ activity in the SMA, nor in the anterior cingulate cortex (ACC), primary motor cortex (PMC), insular cortex or superior parietal cortex. These areas were equally active in the conditions. However, they did notice that the pre-SMA activity started earlier in the internal compared to the externally triggered condition. This early activity could be the pre-SMA planning a

voluntary movement (Cunnington et al., 2002), and could be the readiness potential Libet and his colleagues (cited by Haynes, 2011) also found in their experiments.

The brain is considered to be constantly identifying, analyzing and comparing multiple sources of evidence. As this happens through time, the evidence accumulates towards one choice or the other. If enough neurons are firing due to the accumulation of evidence, this could be recorded with an EEG or fMRI. Gluth, Rieskamp and Büchel (2012) intended to investigate this accumulation of evidence in the brain. In their experiment, subjects were free to buy or sell fake stocks any time they wanted. Their brain activity leading to each decision was tracked with an fMRI. A model was used to predict the subjects’ actions, with an

accuracy of 62.6%, where the chance level was 14.2%. Activity in the pre-SMA, caudate nucleus and anterior insula increased throughout the experiment. Also, high brain activity was associated with responding more quickly. Gluth et al. (2012) conclude that the increasing activity in the pre-SMA and caudate nucleus represents the general willingness to respond, as this is commonly concluded in neuroscientific research. In contrast, the insular activity however would represent the accumulation of evidence, or perhaps the tendency to take risks.

The research of Heekeren, Marrett, Bandettini & Ungerleider (2004) shows there is more that can be decoded from brain activity. In their study, subjects were to visually discriminate between faces and houses, while being scanned with an fMRI. Although their aim was not to decode brain activity, their results were promising. They found a positive correlation between the difficulty of the task, and the activity in the frontal eye field (FEF), the supplementary eye field (SEF) and the intraparietal sulcus (IPS). Apparently, these areas had to work harder, to distinguish between the stimuli. In contrast, the activity in the higher-level decision-making areas, including portions of the dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC) and the superior frontal gyrus, was lower during difficult decisions than during easy decisions (Heekeren et al., 2004).

In a follow up study, the authors worked together with Ruff, to investigate how these different brain areas worked together in decision-making, and why some correlated positively and others negatively with the difficulty. They found that activity in several areas, including the FEF and SEF, was positively correlated with reaction time. As these areas were highly

active during difficult decisions, the authors suggest they are part of the attentional network. On the other hand, the activity in the DLPFC, left SFS and left middle temporal gyrus (l-MTG) was low during difficult decisions and correlated negatively with reaction times. These areas could therefore be part of a default mode network (DMN) (Ruff, Marret, Heekeren, Bandettini, Ungerleider, 2010). These results show not only that it may be possible to decode from brain activity how difficult it is for a subject to make a decision, but also put the

scientist’s aim to predict decisions based on brain activity in a different perspective. If a DMN is active during easy decisions, and an attentional network during difficult decisions, the difficulty of the decision may define which brain area should be used to decode the decision.

Also, as Haynes (2011) notes, in the premotor cortex, DLPFC, posterior parietal cortex and frontopolar cortex and subcortical areas, a readiness potential can also be measured, and they are involved in the planning of actions. Therefore, these regions might be useful in decoding decisions as well. Depending on the exact task a person performs, one brain area might encode for a future decision or not. For instance, areas involved in planning may encode for future motor actions, whereas areas involved in visual discrimination may encode for the choice during visual discrimination tasks.

Decoding of specific decisions

The goal of Gallivan et al. (2011), was to be able to predict upcoming hand

movements with an fMRI, based on activity in brain areas that had not yet been associated with hand movements, and discriminate between different hand movements. In their study, subjects performed a delayed object-directed movement task. Firstly, they were shown the target object, secondly they were instructed what task they were to perform later, either reach, grasp or touch an object. And finally, they performed the action. This approach allowed the researchers to separate and contrast the brain activity related to the different cognitive functions; object recognition, motor planning and motor execution (Gallivan et al., 2011).

Their multivariate approach appeared effective. Whereas they found no difference in amplitude of activity between the three movement conditions, the activity patterns in several regions of interest in the parietal and premotor cortex predicted grasp and reach movements. More specifically, the lateral anterior precuneus (L-aPCu), lateral superior parieto-occipital cortex (L-SPOC), the lateral mid-intraparietal sulcus (L-midIPS), the lateral pre-SMA and the lateral SMA coded for planned grasp versus planned touch movements. Also, the anterior IPS (aIPS) and the post-aIPS encode for grasp versus reach movements. This shows technological

advances have made it possible to decode small differences in planned action intentions (Gallivan et al., 2011).

In the study of Soon, Brass, Heinze & Haynes (2008), subjects watched a slideshow and were told to press either the left or right button as they wished. After pressing, they were instructed to select the slide they saw when they decided which button to press. A new slide showing a single letter appeared every 2 seconds. The researchers used an fMRI to measure local activity patterns in the brain, to predicted upcoming decisions. They were able to predict what button the subject would press with up to 60% accuracy. Most significant was the lateral frontal polar cortex (lFPC), though the medial frontal polar cortex (mFPC) and an area in the precuneus and the posterior cingulate cortex also encoded for this. Already 7 seconds before the decision reached the awareness of the subject, it could be predicted in the lateral

frontopolar cortex. Whereas the results of Libet’s experiment may have resulted from inaccuracies of reading the clock, that can’t be the case here, as the decisions could be decoded 7 seconds before they reached consciousness. Interestingly, there was no increase in activity when averaged over the entire frontopolar and precuneus and posterior cingulate area. Instead, the motor planning was encoded in the local spatial pattern. However, there might have been a RP, but it could not be measured due to the experimental setup (Bode et al., 2011, Soon et al., 2008).

Not only could Soon et al. (2008) predict which action would be taken, they could also predict when it would be taken. Using the activity pattern of the pre-SMA and the SMA, they could predict this 5 seconds ahead of time. On the contrary, activity in the frontopolar and parietal cortex was of no use for predicting when the decision would be taken. They also went on to investigate externally triggered actions. Now, the subjects would still decide to choose left or right themselves, but do so at a time specified by the researcher. In this setup, the activity in the frontopolar cortex was predictive during response selection. On the other hand, the activity in the precuneus was only predictive after the response selection. The authors suggest that the frontopolar cortex is involved in the decision making process, whereas the precuneus would merely be involved in storing it (Bode et al., 2011, Soon et al., 2008) As the decisions could be predicted at least 5 seconds before they were made, this is strong evidence they could already decoded before they reached awareness.

Astonishingly, in the study of Hollmann et al. (2011), while subjects engaged in the ultimatum game, where two subjects together decide how to divide a sum of money between the two of them, the researchers predicted their decisions with up to 70% accuracy before these actions had happened in real time. Moreover, the anterior insula, ventral striatum, and

lateral orbitofrontal cortex appeared to be good predictors of overt behavior later in the game. However, more detailed analysis of the data shows that the model based its prediction on perceptive and evaluative brain activity, rather than a decision making process.

Hampton and O’Doherty (2007) used an fMRI to predict evidence based responses. In their study, choosing one of two stimuli provided on average a larger reward than the other, though this alternated every few trials. This choice was predicted once the previous decision had been made. The difference between the brain activity when a negative outcome elicited the subject to switch to the other stimulus on the next trial and when a negative outcome did not elicit such a switch appears to be a good predictor. The brain regions best used for such a prediction are the anterior cingulate cortex (ACC), medial prefrontal cortex (MPC) and the ventral striatum. These areas are commonly associated with reward related learning and with changes in behavior. Additionally, Hampton and O’Dogherty (2007) ran a multivariate analysis on the activity of brain areas, smaller than 8 mm, to decode decisions. This provided higher accuracies than the individual brain areas. The local activity pattern in the ACC was responsible for this increase.

As these studies differ in what cues were used and what action the subjects performed, and different brain areas can be used to decode these decisions, it can be discerned that each brain area has its own specific part in the decision-making process. Whereas the research of Soon et al. (2008) provides further evidence that the pre-SMA and the SMA encode for timing of actions, the research of Gallivan et al. (2011) shows that they can also encode for the type of action that is planned. Gallivan et al. (2011) also found that various regions in the parietal cortex, including the precuneus, encode for specific arm movements, which expands the evidence that these areas are involved in the planning of arm movements. However, the research of Bode et al. (2011) and Soon et al. (2008) shows that the precuneus is merely involved in the storing of action plans. This does not contradict the results of Gallivan et al. (2011) as in that study the subjects might have action plans stored in the precuneus throughout the entire study, and only when cued to respond the action plan would be performed. Also, the frontopolar cortex encodes for decisions of free will (Bode et al., 2011, Soon et al, 2008). Finally, the ACC, MPC and ventral striatum encode for choices related to reward related learning (Hampton & O’Doherty, 2007). Together, this shows that a distributed network is responsible for the decision-making process, and that this is an emergent property. When trying to decode a decision, the brain area chosen for this should depend on the decision to be made. As will be revealed below, decoding decisions of free will is harder than it previously seemed.

Decision-making without evidence

Sometimes, the possibility to decode subjects’ decisions is based on artifactual evidence produced by the experimental setup. In the study of Bode et al. (2012), subjects performed a visual discrimination task. An EEG was used to decode the subjects’ choices when showing them a discriminable stimulus, either a chair or piano, and when showing them a stimulus that wasn’t discriminable, being noise. In both conditions, the stimulus was masked with noise immediately after it disappeared. The former produced the now obvious result that a decision based on accurate information can easily be decoded after the stimulus is presented, but not before it is presented. In the case without discriminable information, however, already before the stimulus, the choice could be predicted based on the previous stimulus. The brain was actively deciding what to choose on the next trial. If the stimulus contained no

discriminative information, this activity was predictive for the choice the subject would make. Apparently, the brain was already busy deciding what to choose, but if discriminative

information appeared, any progress was discarded.

The authors followed these results up with analyzing at what point in time the activity was predictive. They found a relationship between the time at which the activity was

predictive, and what decision was made. In trials where the same decision was made as the previous trial, activity was predictive earlier. They also found that in non-discriminable trials, subjects tended to repeat their answer of the previous trial, regardless of the motor action associated with the response. Moreover, subjects were faster to repeat their decision of the previous trial than they were to switch to a different answer. Bode et al. (2012) considered this strong evidence towards the priming view. Each trial would prime the subject to respond the same on the following response. When the stimulus contains no discriminable information, the prime has a relatively large effect. But when it did, the subjects made the right decision regardless of a prime. This has a major implication for what exactly researchers decode when monitoring brain activity. If subjects are primed, artifactual results of decoding decisions appear. However, just because priming effects have caused the decoding effects in this study, doesn’t mean they also do in other studies.

Another explanation for how decisions that are not based on evidence can arise is given by Rolls and Deco (2011). They made two artificial neural network decision-making models of a human brain, using the principle that neurons that fire together, wire together. One model contained 500 neurons, the other 4000. For every four excitatory neurons, there

was one inhibiting neuron. As the models analyzed new information, random fluctuations appeared before a decision was made, or even before a cue was applied. These fluctuations were analyzed by the authors, and allowed them to predict with 68% accuracy the upcoming decision, still before the cue was applied. In the 500 neuron model this was approximately 300 ms, and in the larger model 750 ms. The authors suggest that if some neurons are more active than others at one point in the simulation, they are likely to also be more active later on. Even if the original activity wasn’t based on any evidence, the excitatory and inhibitory nature of the neurons causes the increased activity to persist. If this activity remains when the cue is applied, this increases the odds of reaching the threshold of making a decision.

The same thing could be going on in experiments with human subjects. In the

experiments of Soon et al. (2008) and Bode et al. (2011) subjects freely chose to push one of two buttons. Bode et al. found no priming effects caused by previous trials. Apart from waiting a few seconds, Soon et al. (2008) took no extra care to prevent carry-over effects. Therefore, random fluctuations in the activity of some neurons could have made the subject more prone to decide one way or the other, even before a cue was applied and before they know what they’re going to make a decision about. However, in the second part of their experiment, the subjects received external cues, and their model could predict the time at which a decision would be made. As priming effects were absent in the discriminable condition in Bode et al. (2012), and were discarded once a cue was applied in the model of Rolls and Deco (2011), it is unlikely that the results were artifacts produced by the

experimental setup, when an external cue was applied.

Summary

In this paper, different methods of decoding intentions and decisions from brain activity have been reviewed. fMRI studies have produced increasingly accurate mappings of what areas are involved in the decision-making process, and can be used to decode it. Lately, multivariate techniques, analyzing patterns of activation instead of voxels, have revealed that many brain areas are involved in the decision-making process together, and have reached predictive accuracies of up to 63%.

As many brain areas are involved in a single process, and one brain area can be involved in many processes, different brain areas can be used to decode decision-making, dependent on what decision is made. The pre-SMA appears to be the best area for decoding internally triggered movements. Accuracies of 60% have been achieved, using this area alone.

Activity in the SMA and pre-SMA can be used to predict when a decision is taken at least 5 seconds before that decision is taken. Activity in the SMA, however, doesn’t prove well for predicting what future action is taken. This is explained by the notion that the SMA is primarily involved with the general action preparation rather than the selection of specific action plans. In order to predict these, other brain areas must be analyzed.

The frontopolar cortex appears to be a good predictor for what action is chosen in experiments where a simple left/right decision must be taken, as well as a small area in the area precuneus and the posterior cingulate cortex. Interestingly, overall activity here does not appear to increase or decrease during task performance. Instead, the small areas here interact to aid the decision making process. In accordance, Gallivan et al. (2011) showed that small areas in the frontal and parietal lobe together code for what action the hand should perform. The general conclusion here is that the frontal and parietal lobe work together in the selection of specific action plans.

Causality remains a problem in this line of research. Just because a decision can be decoded before it is taken, doesn’t mean the brain activity that elicited the decoding is causing the decision to be taken. Still, the question that remains is what increased levels of brain activity represent. Claims are often made that this is the accumulation of evidence. However, as seen in the visual discrimination tasks of Heekeren et al.(2004) and Ruff et al. (2010), sometimes the opposite is true. Their research suggests that increased levels of brain activity mean very different things in different brain areas.

The study of Bode et al. (2010) shows that carry-over effects can prime the subjects toward one option or the other. After choosing one option on one trial, subjects were primed to pick it again in the next. However, in the study of Gallivan et al. (2011), external cues prevented such priming effects, yet still decisions could be decoded. Though, in studies where subjects could freely choose, priming effects remain a problem. The random fluctuations account demonstrates how. Computer models show that a random fluctuation in the activity of groups of neurons can reinforce itself and inhibit the other neurons. Thus, the random

fluctuation would persist until a threshold is reached and the subject decides what action to take. When no clue is provided as to which action to take, random fluctuations may cause the subject to pick randomly. Or, when primed by previous trials, the subject will be biased toward the prime. Therefore, one should remain skeptic towards claims of being able

Discussion

Despite many different motor responses have been shown to be predictable by brain activity, there is still an infinite amount of responses possible. Decoding of decisions has shown that many brain areas that have been associated with certain tasks don’t decode for these tasks, and thus might not be as involved in these tasks as previously thought. The decoding of more decisions and intentions will provide more knowledge on the functional architecture of the human brain, and its decision-making processes. Soon, He, Bode and Haynes (2013) have already shown that more abstract decisions, like the choosing whether to add or subtract two numbers can also be decoded at least 4 seconds before they are made. Here too, the medial frontal polar region and an area bordering the precuneus and the posterior cingulate encoded for this decision, similar to the results of Bode et al. (2011) and Soon et al. (2008). Important in their results was that the spatial activity pattern that decoded for these decisions, was similar to that of the DMN, the spatial activity pattern active during idling. Although in other studies researchers relied on subjects’ reports to determine when a decision reached conscious awareness, this provides evidence that a network outside of conscious awareness was indeed involved in the decision-making process.

Whereas the purpose of the research discussed above is mostly scientific and the social and practical applications remain unclear, some researchers use these decoding techniques to detect lies; a feature which can prove valuable in court. For instance, Davatzikos et al. (2005) have trained a computer program to detect lies, by using analyzed brain activity. They gave subjects several cards, and asked them to decide when to tell the truth or lie about having certain cards, while being scanned by an fMRI. By classifying multivariate non-linear spatial patterns of brain activity, the authors could discriminate between truth and lie. By cross-validating, leaving the subject in question out, they achieved an overall accuracy of 88%. The brain regions that were most predictive were the right inferior frontal gyrus, right superior frontal gyrus, bilateral superior temporal and bilateral inferior parietal gyri, which have been previously associated with error monitoring and response inhibition. Both these tasks are obviously necessary when lying.

Furthermore, there is more to decode regarding decisions than which decision is taken and why. Heekeren et al. (2004) and Ruff et al. (2010) have shown that there is a correlation between the brain activity and the difficulty of taking a decision. It seems several sensory areas had to work harder in order to distinguish between stimuli, whereas several higher order processing areas lacked the evidence to make a decision, yet these hypotheses require more

evidence. Using multivariate spatial pattern analysis, one might be able to predict the

difficulty of a decision regarding visual discrimination. This could be expanded to economic paradigms, such as the prisoner’s dilemma, where players repeatedly decide to cooperate or defect. As the optimal score of a player depends on the strategy of the opponent, it is

interesting to predict how difficult it is for a player to choose one strategy or the other.

In research regarding internally triggered actions, or actions of free will, causality will always remain a major problem. However, there are possibilities to prevent priming effects. For instance, one could design an experiment where the subjects can freely choose between three or more options, except that the option chosen on the previous trial is not available. This should prevent the previous trial from priming the current trial. Therefore, what is then decoded cannot be a priming effect. However, whatever measure is taken, evidence for causality will be hard to accumulate, if not impossible.

References

Bode, S., He A.H., Soon, C.S., Trampel, R., Turner, R., & Haynes, J-D. (2011). Tracking the unconscious generation of free decisions using ultra-high field fMRI. PLoS ONE, 6, 6. Bode, S., Sewell, D. K., Lilburn, S., Forte, J. D., Smith, P. L., & Stahl, J. (2012). Predicting perceptual decision biases from early brain activity, The Journal of Neuroscience, 32, 36, 12488-12498

Cunnington, R., Windischberger, C., Deecke, L., & Moser, E. (2002). The preparation and execution of self-initiated and externally triggered movement: a study of event-related fMRI,

NeuroImage, 15, 373-385

Davatzikos, C., Ruparel K., Fan, Y., Shen, D.G., Acharyya, M., Loughead, J.W., & Gurb, R. C. (2005). Classifying spatial patterns of brain activity with machine learning methods: Application to lie detection, NeuroImage, 28, 3, 663-668

Gallivan, J.P., McLean, D. A., Valyear, K. F., Pettypiece, C. E., & Culham, J. C. (2011). Decoding action intentions from preparatory brain activity in human parieto-frontal networks.

The Journal of Neuroscience, 31, 26, 9599-9610

Gluth, S., Rieskamp, J., & Büchel, C. (2012). Deciding when to decide: time-variant sequential sampling models explain the emergence of value-based decisions in the human brain, The Journal of Neuroscience, 32, 31, 10686-10698

Hampton, A. N., & O’Doherty, J. P. (2006). Decoding the neural substrates of reward-related decision making with functional MRI, Proc Natl Acad Sci USA, 104, 4, 1377-1382

Haynes, J-D. (2011). Decoding and predicting intentions, Ann. N. Y. Acad. Sci., 1224, 9-21 Heekeren, H. R., Marrett, S., Bandettini, P. A., & Ungerleider, L. G. (2004). A general mechanism for perceptual decision-making in the human brain, Nature, 31, 859-862. Hollmann, M., Rieger, J. W., Baecke, S., Lützkendorf, R., Müller, C., Adolf, D., &

Bernarding, J. (2011). Predicting decisions in human social interactions using real-time fMRI and pattern classification. PLoS ONE, 6, 10

Outer surface of the human brain. (n.d.) Wikipedia, retrieved from:

http://commons.wikimedia.org/wiki/File:PSM_V46_D167_Outer_surface_of_the_human_bra in.jpg

Soon, C. S., Brass, M., Heinze, H-J., & Haynes, J-D. (2008). Unconscious determinants of free decisions in the human brain. Nature Neuroscience, 11, 5, 543-545

Soon, C. S., He, A. H., Bode, S., & Haynes, J-D. (2013). Predicting free choices for abstract intentions. PNAS, 110, 15, 6217-6222

Rolls, E. T., & Deco, G. (2011). Prediction of decisions from noise in the brain before the evidence is provided. Frontiers in Neuroscience, 5, 33

Ruff, D. A., Marret, S., Heekeren, H. R., Bandettini, P. A., & Ungerleider, L. G. (2010). Complementary roles of systems representing sensory evidence and systems detecting task difficulty during perceptual decision making, Front. Neurosci., 4, 190

Decoding confidence in decisions using spatial

pattern mapping and comparing to reported

confidence

Research Proposal

Author: Valentijn M. T. de Jong Student number: 10120998

Beta-Gamma, track Brein & Cognitie Tutor: Tobias Donner

Summary

Convictions in court often rely on witnesses’ reports, which vary in accuracies. If the accuracies of these reports could be decoded using brain activity, this could reduce the amount of false convictions or acquittals. Although decoding the accuracies of eyewitness’ reports may be still be unattainable, it may be possible to do so for simpler decisions.

In this research, the subjects will perform tasks of varying difficulty, and state their confidence in their decisions. This confidence score will be used to predict the accuracy of the decision made in the task. Using spatial pattern mapping, their brain activity as measured by an fMRI will be used to predict the accuracy of each decision. By analyzing the probable correlation between the confidence scores and the predicted accuracies, the relation between the brain activity and the subjects’ confidence in decisions will be revealed. Also, the two methods will be compared in order to determine which is the most accurate. This will provide answers to the questions of whether brain activity can be used to decode the accuracy of a decision, and whether this method is more accurate than subjects’ reports.

Research question

The research of Heekeren, Marrett, Bandettini, and Ungerleider (2004) showed that there are differences in the pattern of brain activity during decision-making in visual discrimination tasks between easy and difficult decisions. Although the authors did not specifically try it, but using the same paradigm, it should be possible to decode the difficulty of the decision from the brain activity. Rolls, Grabenhorst and Deco (2010) made a neural network computer model, and suggest that the activity that encodes for the difficulty of the decision reflects the brain’s confidence in the decision. Their view supports the accumulation of evidence account. As evidence accumulates towards one hypothesis, the amount of neurons firing should increase in areas that are responsible for gathering evidence.

Bode, Bogler, Soon & Haynes (2012) used the brain activity in a visual discrimination task to predict the content of the stimuli. They found that sensory regions were predictive for the content of clearly visible stimuli, and parietal regions were predictive for the choices following noisy stimuli. Therefore, the sensory regions were responsible for discriminating between visible stimuli, whereas the parietal regions were responsible for randomly choosing when a noisy stimulus was seen. Using the same paradigm, but using a variety of stimuli that vary in noisiness, one may be able to predict the accuracy of the decision based on the activity

patterns during various conditions difficulty. In other words, the question is whether it is possible to decode the brain’s confidence in a decision, based on its activity.

The second question here, is whether the decoded confidence reflects the confidence a person feels, or at least reports feeling. If these two concepts do not match, it is still interesting to find out which of the two predicts the accuracy of the decision the best.

Methods

A visual discrimination task will be used, in which subjects first view one image (training phase) and a mask, and then view two images and are to recognize the image previously seen (discrimination phase) and then a mask again. There will be five types of image in the training phase. In the first type, only noise will be shown. In the second, a clear image of a piano will be shown and in the third a blurred image of a piano. In the fourth a clear image of a chair, and in the fifth a blurred image of a chair. Chairs and pianos have been chosen, because these images were effective in the study of Bode et al. (2012).

Data gathering

An fMRI scanner will be used to measure the blood-oxygen-level-dependent (BOLD) contrast, to determine brain activity. A spatial pattern analysis will be used to identify which areas encode for the accuracy of the decision. After each trial, the subjects will state how confident they are in their decision.

Participants

Twenty subjects will be tested, because in the research of Bode et al. (2012), this appeared to be enough to decode the confidence in decisions. All should be right-handed, to avoid differences in right-, left- or lateral-sided brain activity. Data from subjects who performed at chance level in the medium and/or the easiest difficulty condition will be discarded.

Data analysis

The spatial pattern analysis will be used to decode the accuracy of the decision, after selecting regions of interest using a searchlight analysis. The confidence scores will be used to predict the accuracy of the decision using linear regression. The correlation between the predicted accuracies by the pattern analysis and the confidence scores of the subject will reveal whether this activity indeed reflects the confidence in the decision, and to which

degree. Also the accuracies of the two methods will be compared, to determine which predicts the accuracy of the decision the best.

Interpretation of possible results

Whereas previous studies have shown that differences in brain activity can be measured between easy and difficult decisions, that the difficulty of a decision in visual discrimination can be predicted (Heekeren et al., 2004), and that the difference in confidence between easy and impossibly difficult decisions can be predicted (Bode, Bogler, Soon & Haynes, 2012), it remains unclear whether the confidence in real life decisions can be predicted, and whether these predictions are more or less accurate than the subjects own confidence in their decisions.

If the difference in accuracies between the noise condition and the clear stimulus condition cannot be decoded, something has gone wrong and the other results cannot be interpreted, because Bode et al. (2012) have shown that this is possible. If it does, but the difference in accuracies between the noise condition and the blurred stimulus condition cannot be decoded, or the difference in accuracies between the blurred stimulus condition and the clear stimulus condition cannot be decoded, too few participants have been tested. This because these both rely on the same method as the first analysis, except that the difference in brain activity would be smaller.

If a correlation is found between the decoding of the accuracy and the confidence of the subjects in the decisions, this would support the hypothesis of Rolls et al. (2010) that the activity in certain neurons concerned with decision-making reflects the consciously aware confidence of the person in the decision. However if no correlation is found, but decision accuracies can be decoded, this would mean that the activity in these neurons reflects the confidence in the decision that the person is not consciously aware of.

Finally, if accuracies can be predicted using brain activity and this prediction is more accurate than the confidence of the subjects, this has implications for the concepts of

consciousness and confidence as it would mean the conscious “part” of the brain does not have full knowledge of how accurate its decisions are. If accuracies can be predicted using brain activity, but this prediction is less accurate than the confidence of the subjects, this would simply mean that more powerful imaging or data-analysis techniques are necessary to achieve similar results. Given that fMRI scanners can only analyze activity in groups of

neurons, and not individual ones, it is probably not possible to reach the same accuracies achieved by confidence reports.

References

Bode, S., Bogler, C., Soon, C. S., & Haynes, J-D. (2012). The neural encoding of guesses in the human brain. NeuroImage, 59, 2, 1924-1931

Heekeren, H. R., Marrett, S., Bandettini, P. A., & Ungerleider, L. G. (2004). A general mechanism for perceptual decision-making in the human brain, Nature, 31, 859-862.

Rolls, E. T., Grabenhorst, F., & Deco, G. (2010). Decision-Making, Errors, and Confidence in the Brain. Journal of NeuroPhysiology, 104, 5, 2359-2374