TOWARDS A COMPLETE UNDERSTANDING OF DECISION-MAKING
model types of decision-making
structured within three levels of understanding
A literature thesis written by
REBECCA SIER
as part of the final requirements for fulfillment of the Master of Science in Brain and
Cognitive Sciences at the Universiteit van Amsterdam
Exam date: September 30, 2014
Student ID: 5893011
Study track: Cognition
Supervisor: Dr. Leendert van Maanen
Co-assessor: Prof. Dr. Birte U. Forstmann
2
TABLE OF CONTENTS
Abstract 4
Acknowledgements 5
Introduction
6
Marr’s theory: levels of understanding
10
The computational level of understanding: sequential sampling models
13
Theory on sequential sampling models 13
Early decision-making theory and experiment 13
Horse race model 14
Diffusion model 15
Neurobiological evidence 17
Single-unit recordings 18
Model-based fMRI recordings 19
Example uses of sequential sampling models 20
The origin of age-related difference in posterror slowing 20
How prior knowledge biases decisions 21
Sequential sampling models and the computational level of understanding 22
The representational/algorithmic level of understanding: neural network models
23
Theory on neural network modeling 23
Feed-forward inhibition 24
Mutual inhibition 25
Neurobiological evidence 27
Example uses of neural network models 27
Neural network models and the representational/algorithmic 29 level of understanding
3
The implementational level of understanding: biophysically plausible
30
decision models
Theory on biophysically plausible modeling 30
The basal ganglia model 30
The attractor model 33
Neurobiological evidence for biophysically plausible modeling 34
Basal ganglia pathways and the DA system 34
Decision-making on the neuronal level 35
Example uses of biophysically plausible models 36
Reward-guided learning in relation to age and Parkinson’s disease 36
Finding decision-making areas in the brain 37
Biophysically plausible models and the implementational level of understanding 37
Discussion
39
Levels of understanding and model types of decision-making 39
Dependencies between levels of understanding 40
4
ABSTRACT
In the literature on decision-making the decision-making process is simulated with models that range from the psychological to the neurobiological type. According to David Marr (1982), a complex information processing device can only be understood completely when combining all three independent levels of understanding: the computational, representational/algorithmic and implementational levels. Here, it is argued that three decision-making model types – (1) sequential sampling, (2) neural network and (3) biophysically plausible models – can be classified into these three levels. In addition, it is argued that a complete understanding of the process of decision-making not only requires all of the logically independent levels of understanding, but also an awareness of how the levels depend on one another in a causal, structural and evolutionary way.
5
ACKNOWLEDGEMENTS
This thesis was written under supervision of Dr. Leendert van Maanen and was co-assessed by Prof. Dr. Birte U. Forstmann, as part of the final requirements for the master of science in Brain and Cognitive Sciences. I would like to thank Leendert for his very helpful advice on useful literature, the thorough commentaries on each version of this thesis and his patience.
6
INTRODUCTION
Decision-making is a cognitive process that results into the choosing of one option over others. Obvious are the cases in which to decide upon bigger questions in life, such as marriage and war. More common are the day to day decisions about which shoe to put on first or what to have for breakfast. Clearly, these examples differ in the types and amount of alternatives that can be chosen from (a “yes” or “no” in marriage and war, against “left” and “right” shoes and multiple breakfast alternatives in the case of a filled fridge). But, however broad the range of decision types, a feature shared by all of them is the selection of an option out of many, guided by evidence from the outside world.
Starting from these common features, many attempts have been made to answer what decision-making is, why and how it works and how it is implemented in the brain. Here, the brain is assumed to be a computer; a piece of hardware that computes outputs in response to incoming information. Over the years, modeling how such input-output relations can result in decision-making showed to be a fruitful approach, as a means to test hypotheses and gain new insights. Along with the variance in questions that are asked about decision-making, a range of decision-making models emerged to answer them.
What decision-making entails was initially investigated through analysis of behavior. Psychologists found how decision behavior driven by perceptual events depends on (1) the quality and quantity of sensory evidence that supports one of the choice alternatives, and (2) the time that is available to make a decision. In pondering, pro's (“I feel like having pancakes”) and con's (“I should watch my weight”) for either of the options are gathered, until one of the options wins and gets to be chosen.
This notion of decision-making as an information accumulation and outweighing process is
translated into sequential sampling models (e.g., Bogacz, Usher, Zhang, & McClelland, 2007; Logan & Cowan, 1984; Ratcliff & Rouder, 1998; Usher & McClelland, 2001; Wang, 2008). The models use decision parameters like drift rate and a decision threshold to predict reaction time (RT) distributions for correct and incorrect decisions, as well as accuracy (Hick, 1952; Wickelgren, 1977). The parameters formed new steps in understanding what decision-making is and why features like information accumulation exist. The success of the sequential sampling model type
7 also shows in neurological evidence in the form of single-cell recordings from behaving monkeys (Shadlen & Newsome, 1996, 2001).
With the rise of graph theory, neural network models formed a new step in understanding decision-making. In the network models, successful aspects from sequential sampling models are embedded in an algorithm; nodes represent input units and accumulators, with edges indicating the excitatory or inhibitory connections between them. For example, it was found how an inhibitory effect
between choice alternatives results in RTs and accuracies that are comparable to those from sequential sampling models (Usher & McClelland, 2001). Next to being influenced by sequential sampling models, the network models are constrained by neurobiological findings on decision-making. The neural network algorithms need to reflect neurobiological knowledge. Therefore, the degree to which the algorithms can be used in combination with neurobiological evidence serves as a test to decide between different algorithm hypotheses: in model-based functional magnetic resonance imaging (fMRI) the model predictions are combined with images of neural activity, to reveal the relative success of the algorithms (Forstmann, Wagenmakers, Eichele, Brown, & Serences, 2011).
More sophisticated are biophysically plausible decision models. These take into account what is physically possible within the brain, to answer how the decision-making algorithm is implemented in this piece of hardware. For example, neural pathways in the basal ganglia (BG) are shown to be of specific importance, along with neurotransmitters like dopamine (DA) and glutamate (Frank, 2005). Mathematical models of interconnected integrate-and-fire neurons use what is known about membrane potentials, gating variables and the like to simulate the effect of action potentials on neural networks (Bogacz, 2007). Interestingly, the features that define what decision-making is emerge from dynamics inherent to these mathematical representations (Deco, Rolls, Albantakis, & Romo, 2013; Wang, 2002).
Each of the above presented model types of decision-making answers different questions on the matter: either what decision-making is, how it works algorithmically or how it can be implemented in the brain. According to Marr (1982) the answers to these questions form three separate levels of understanding, respectively called the computational, representational/algorithmic and
implementational levels. Marr (1982) stated that all three levels are needed to “understand an information processing device completely”.
8 Although according to Marr (1982) each of the three levels of understanding forms a piece of the puzzle to understand the process of decision-making, in the literature most commonly either one of the three presented model types or a combination of two is mentioned or used (e.g., Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006; Bogacz & Larsen, 2011; Frank, Seeberger, & O’Reilly, 2004; Hunt et al., 2012; Jocham, Hunt, Near, & Behrens, 2012; Ratcliff & Frank, 2012). In addition, oftentimes a reasoning behind the choice for a model lacks. It seems that an awareness of the levels within which the models operate, with each having its own explanatory force, is lacking. Research on decision-making might miss out on benefits that a leveled understanding could provide. Or, in the words of Marr (1982), in this practice decision-making is never “understood completely”.
Here, it is argued that understanding decision-making means to understand and combine the three mentioned decision-making model types within Marr’s (1982) concept of levels. In addition, it is argued that a “complete understanding” of the process of decision-making not only requires all of the logically independent levels of understanding, but also an awareness of how the levels depend on one another in a causal, structural and evolutionary way. To advance research in
decision-making, it is important to position new knowledge within these levels and thereby contribute to the bigger picture. Such an interdisciplinary, mechanistic integration of mathematical models for decision-making could lead to increasingly streamlined and focused research, facilitating collaboration among researchers.
This paper aims to pave the way towards a complete understanding of decision-making, by investigating the position of the existing model types of decision-making within the three levels described by Marr (1982). First, Marr’s (1982) and related theories will be reviewed, after which the basic theory and development of sequential sampling, neural network and biophysically plausible models will be considered, including neurobiological evidence for each of them. In addition, examples of studies that use the respective model types are reviewed. This will serve to appreciate which model type is commonly used for particular features of decision-making, whether other model types could have been useful too and how the example studies might benefit from Marr’s (1982) leveled understanding. Ultimately, a discussion serves to indicate how the reviewed model types fit within Marr’s (1982) levels, how these levels contribute to a complete
9 It is not the purpose of this paper to overview all current research models of decision-making, nor all variations of the addressed model types. However, it does review main model types that have successfully been used in the past and present, and aims to use this knowledge to obtain the means for creation of a general framework of computational models of decision-making.
10
MARR’S THEORY: LEVELS OF UNDERSTANDING
To “completely understand” an information processing device like the central nervous system (CNS), Marr (1982) stated that there are three levels of understanding one needs to take into account. For example, in the case of a system as complex as the visual system, it does not suffice to explain the elementary components of neurons and neural networks; one also needs an
understanding of how the visual system functions and how it can be implemented in the brain.
The computational and first level is the most abstract of Marr’s (1982) levels. It communicates what the information processing device does and why. This includes a detailed description of the output that the device generates upon receiving input information, the goal of this input-output mapping and its appropriateness and adequacy for the task at hand.
Secondly, the representational/algorithmic level describes how the information processing device works. It gives an algorithmic description of how the input is transformed into a particular output, depending on the representation that is used for inputs and outputs of the process. Both the algorithm and representation are constrained by the computational description in the first level, such that it should work with the same inputs, outputs and other abstract properties of the
mapping. Within these constraints there usually are several optional algorithms that fulfill the same process.
A second set of constraints to the algorithm comes from the third, implementational level, that states how the algorithm can be implemented physically. For example, to understand the visual system, neurons and neural networks provide the hardware in which the algorithm and the representations of input and output information should be able to work.
Marr and Poggio (Marr & Poggio, 1976; Marr, 1982) stressed the importance of the division into levels with the claim that these are only loosely related. Although the levels constrain and thereby influence one another, they each have their own, separate area of explanation. For example, to understand the phenomenon of after-images, one should reach into the biophysics of the visual system and investigate how after-images are represented on a neuronal level, i.e. the
implementational level. Moreover, an explanation for the ambiguity of the Necker cube can be found only at the computational level: it is due to the functioning and purpose of the visual system itself
11 that enables two different interpretations of the picture of a cube. Since these two interpretations do not have different effects on a mechanistic, neuronal level that is limited to the visual system, it would lead to confusion to use the implementational level of understanding for investigation of the Necker cube phenomenon. The levels of understanding therefore serve to avoid this kind of
confusion.
Rumelhart and McClelland (1985) proposed an adaptation to Marr's and Poggio's levels. According to them, the algorithmic level should be separated into two. On the one hand, a detailed
connectionist account of the algorithm gives rise to phenomena that on the other hand are approximated in an information processing account. The hierarchy resembles that of quantum mechanics (lower, detailed level) and Newtonian physics (higher, approximated level): properties of the approximation in the higher level can only be accounted for by understanding the lower level dynamics (Bechtel, 1994). Since the information processing account is merely an approximation of the connectionist account of the algorithm, the accounts are inherently dependent.
The dependency between the two algorithm types by Rumelhart and McClelland (1985) contrasts with the near independency between the levels that Marr and Poggio advocated. A second
difference lies in their claims of the order in which the levels should be analyzed to arrive at a complete understanding of the system at hand (Griffiths, Chater, Kemp, Perfors, & Tenenbaum, 2010). Rumelhart and McClelland used a bottom-up or “mechanism-first” strategy, heading from the system’s detailed neural processes towards higher-level explanations. Often, the latter do not have an independent validity, but are approximations of the fundamental mechanism or emerge from it. In contrast, Marr (1982) gave high importance to the nature of the computations that underlie the system: the algorithm can only be figured out when first understanding the system at the
computational level. He therefore used a top-down or “function-first” approach to understand the visual system, descending from the purpose and meaning of the visual system (computational level), via a plausible algorithm (representational/algorithmic level), towards its implementation in the brain (implementational level).
According to Bechtel (1994) the differences between the two leveled systems of analysis are due to their different goals. Rumelhart and McClelland were concerned with levels of structure in nature, and since these structures are built upon others (i.e. bottom-up) they are necessarily dependent. However, Marr’s aim was to create levels of analysis that are causally and logically related, but
12 inherently independent, meaning that one can understand parts of a complex system within their own levels. When relating the levels to one another, Marr prefers a top-down approach, since the realizations of both the representational/algorithmic and implementational levels can be best understood when starting from the computational level.
In the following it is argued that a complete understanding of the complex system of decision-making needs both of the described approaches. To provide the first step towards this goal, the process of decision-making will be decomposed into three loosely related levels in the sense of Marr and Poggio, descending from the highest to the lowest level. Secondly, when the levels within the process of decision-making are clearly stated, they will be connected in a bottom-up fashion, giving insight to emergent phenomena, as proposed by Rumelhart and McClelland. Both parts will be covered through an analysis of the existing theoretical models of decision-making.
13
THE COMPUTATIONAL LEVEL OF UNDERSTANDING:
sequential sampling models
Marr’s (1982) computational level of understanding aims to answer what a complex system does and why. In the case of decision-making the level must describe what decision-making implies, what the aim of decision-making is and why it exists. This chapter serves to show that these questions can be answered through a behavioral analysis of decision-making, captured in the sequential sampling model type.
Theory on sequential sampling modeling
Early decision-making theory and experiment
Reaction time (RT) experiments were the first to characterize how sensory evidence leads to a response. Using Shannon’s (1948) conception of information, it was found that the rate of gain of information during the making of a decision is on average constant (Hick, 1952; Logan & Cowan, 1984; Wickelgren, 1977). Therefore, the time it takes for a decision to be made is, on average, proportional to the amount of sensory evidence that has been processed by the decision maker.
Following the findings of RT experiments, sequential sampling models were among the first to describe decision-making. These models come in great variety, as all sorts have been developed from the 1950s to the present. What connects them is the hypothesis of evidence accumulation to a threshold, with the aim to predict behavior, i.e. subjects’ RTs and the accuracy of their decisions (Deco et al., 2013; Ratcliff & Smith, 2004). Moreover, the tradeoff that was found to exist between speed and accuracy (speed-accuracy tradeoff; SAT) should be described by the model.
In sequential sampling models, input evidence is stochastically accumulated and integrated over time at a certain speed or ‘drift rate’, until the level of a decision threshold is reached, and upon which the decision is made (e.g., Deco et al., 2013; Laming, 1968; Ratcliff & Smith, 2004; Stone, 1960; Vickers, 19710; Wang, 2008). Figure 1 gives a schematic depiction of this accumulation process, exampling the simple horse race model.
14 Evidence accumulation is assumed to be stochastic to reflect
how input evidence is inherently variable or noisy, making it harder to discern the correct stimulus, and thus possibly leading to an undesired response. Accumulation of successive samples of evidence is therefore used to account for this stochastic delusion, and consequently increase the probability of giving a correct response (Ratcliff & Smith, 2004)
Horse race model
An early and influential type of sequential sampling models was the recruitment (LaBerge, 1962) or horse race model (Logan & Cowan, 1984). Figure 1 is It assumes two or more
sets of information accumulators, each corresponding to a response option. Whichever accumulator first reaches a decision threshold ‘wins’ the ‘race’, upon which the corresponding action will be executed. This is a ballistic process, in the sense that once the threshold has been crossed, the response process runs to completion before it stops.
To exemplify the horse race model, Logan and Cowan (Logan & Cowan, 1984) used it to model response inhibition, where two accumulators respectively correspond to execution and inhibition of a response. Thus, if the inhibition-accumulator reaches threshold first, then the response is
inhibited; if the go-accumulator finishes first, then the response is executed until completion, no matter what other threshold may be reached in the meantime.
The earliest form of the horse race model was limited, since it could not model quantitative aspects of RT experiments, especially aspects of the SAT. For example, the mean RT for decisions that resulted in a correct response was experimentally found to be smaller than the mean RT of an erroneous response (Ratcliff & Smith, 2004; Wang, 2008). However, according to the horse race model two accumulators that receive evidence with equal accumulation rates and equal decision thresholds lead to the same mean RT, independent of accuracy.
Fig. 1. A schematic depiction of the horse race
model, an early and simple sequential sampling model. Two accumulators stand for two choice alternatives. Evidence for each of the alternatives is accumulated from the starting point with drift rates 𝑎 and 𝑏. The accumulator that first reaches the decision threshold “wins”. It means that the decision is made in favor of the choice alternative corresponding to the filled accumulator. Decision threshold Starting point a b
15 A second discrepancy with experimental results is that accuracy can grow without bound in the horse race model (Ratcliff & Smith, 2004). The model allows for the decision threshold to be raised or the starting point to be lowered to increase the period over which the model accumulates evidence and thereby increase the accuracy of decisions. However, when mathematically increasing the distance between starting point and decision threshold without a bound, it would wrongfully suggest the possibility of an “infinite accuracy”.
Several suggestions in the form of new or adapted models have been made to solve both the SAT and the infinite accuracy problems of the horse race model. The diffusion model is one, successful example.
Diffusion model
The first attempt in modeling effects of the SAT resulted in the diffusion model. Here, a single accumulator adds evidence in favor of one choice alternative, and subtracts evidence in favor of a second. Decision thresholds exist in both directions, each corresponding to one of the response options.
Several decision variables of the diffusion model can be adjusted to model speed-accuracy effects. First, when moving the decision threshold away from the starting point, it takes more evidence and thus more time for decisions to be made. As a result, noisy input has less influence on the result, increasing response accuracy (Ratcliff, 1998).
A second variable is the drift rate, which determines the speed with which incoming sensory information is integrated (Ratcliff & Rouder, 1998; Ratcliff & Smith, 2004). Drift variation within a decision trial causes incoming sensory evidence to contribute relatively more or less to the evidence integration process. This mimics how stochastic noise influences the accumulation of sensory evidence in either way. Variation in (mean) drift rate across trials merely indicates how memory and learning influence the making of a decision. For example, learning which of two choice alternatives has a higher probability of being correct induces a larger drift rate in favor of that alternative.
16 As shown in Figure 2A, drift variability
across trials can explain slow errors, compared to fast rewards. Starting with evidence accumulation at point 𝑧 =𝑎
2, the time it takes to accumulate
evidence towards 𝑎 (decision
threshold for the correct response) is longest in the case of the small drift rate 𝑣2. Also, the small drift rate lowers the probability of reaching the threshold for the correct response, since the longer RT provides a larger window for noise to have an effect than in the case of a larger drift rate 𝑣1. As a result, the smaller drift
induces both a larger RT and a smaller probability to end up at the rewarding decision. Conversely, the small drift rate raises the probability of reaching the erroneous threshold at 0 (on the vertical axis of figure 1A), as
compared to the large drift rate 𝑣1, while having the same RTs as in the rewarding case. Therefore, the weighted mean RT results in slow errors and fast rewards.
A third variable in the diffusion model is the starting point (Ratcliff & Rouder, 1998; Ratcliff & Smith, 2004). As in the case of a larger drift rate, a starting point closer to the correct response boundary can be induced by past learning and memory. However, in contrast to the slow errors in drift rate variability, starting point variability can explain fast errors, as illustrated in Figure 2B.
A final variable in the diffusion model represents nondecisional components of processing that add to the RT. Examples are the time it takes to execute the motor act of response execution, or encoding of the stimulus (Ratcliff & Smith, 2004). Since the standard deviation of the distribution of the other
Fig. 2. Illustration of how variability in drift rate and starting point in the diffusion
model respectively leads to (A) slow and (B) fast erroneous decisions. Adapted from (Ratcliff & Rouder, 1998).
17 decision variables is larger than that of the distribution of nondecision times the nondecision time has less effect on the shape of the RT distribution (Ratcliff & Tuerlinckx, 2002).
To solve for the race model’s infinite accuracy, a parameter was added to the diffusion model, resulting in the Ornstein-Uhlenbeck (O-U) model (Bogacz et al., 2006). Through this parameter the rate of change in accumulation depends on itself. As a result, the accumulation rate either decreases or increases with each additional amount of accumulated evidence. Therefore, the accumulated evidence over time ends up in an asymptote instead of in infinity (Ratcliff & Smith, 2004), correctly reflecting a finite accuracy in decision-making.
The diffusion model is successful in several aspects. It is able to fit a great variety of experimental data sets, accounting for characteristics of the SAT, and in the case of the O-U model without the problem of infinite accuracy (Ratcliff & Rouder, 1998; Ratcliff & Tuerlinckx, 2002). In addition, the diffusion model was found to be optimal, meaning that its RTs are the shortest for a given level of accuracy (Bogacz, 2007). However, it is still debated whether either separate accumulators as in the race model, or a difference in evidence as in the diffusion model gives a better prediction of
experimental data (Deco et al., 2013). While the race model can easily be extended to more choice possibilities by adding more accumulators, the use of the diffusion model for multiple-choice decision-making is less straightforward and debated (Bogacz et al., 2006; Roger Ratcliff & Starns, 2013).
Neurobiological evidence
Only in the past decade research started to use computational models of decision-making to actually understand neural activity (Bogacz, Wagenmakers, Forstmann, & Nieuwenhuis, 2010; Forstmann et al., 2011). Moreover, combining mathematical models with not only behavioral, but also neurobiological evidence, serves as a test for the plausibility of model features such as starting point and decision threshold.
18
Single-unit recordings
Electrophysiological single-unit recordings from monkeys performing decision-making tasks provide convincing evidence for the main features of sequential sampling models (Bogacz, 2007; Wang, 2008). From this, RTs were found to be related to neural activity at the single-cell level (Roitman & Shadlen, 2002).
As a first step in the process of decision-making, perceptual input activates neurons in sensory areas that are responsive to critical aspects of the stimulus. For example, neurons in the medial temporal (MT) area are used for motion processing, with each neuron having a ‘preferred’ motion direction (Bogacz, 2007). The more evidence coming in for leftward motion, the higher the firing rates of neurons sensitive to this particular input (Britten, Shadlen, Newsome, & Movshon, 1993). The time it takes to collect and process the sensory evidence is part of the non-decision time in sequential sampling models.
Secondly, the brain should compute which of the sensory populations has the highest activation, indicating the correct stimulus that will lead to its related response (Gold & Shadlen, 2001; Gold, Shadlen, & Sales, 2002). Neural correlates of evidence accumulation have been observed in areas associated with the response modality – for example the lateral intraparietal (LIP) area, the frontal eye field (FEF) and superior colliculus for visual response tasks, and the pre-motor cortex in motor response tasks. During the decision process, neurons in these areas that are selective to the
incoming evidence, gradually increase their firing rates (e.g., Kiani & Shadlen, 2009; Roitman & Shadlen, 2002; Schall, 2001; Shadlen & Newsome, 2001). Thus, corresponding to information integration in sequential sampling models, evidence for the stimulus is accumulated over time, to decrease the effect of noise and increase confidence in one of the responses.
With monkeys performing the random dot motion (RDM) task, neural correlates of the drift rate and decision threshold were observed. During the task monkeys have to decide with a saccade on the main direction in which a cloud of dots moves: either in the left or the right horizontal direction. The difficulty of the task is manipulated by a coherence parameter: with less coherence more of the dots move in a direction that differs from the main direction, which functions as noise in favor of the incorrect choice alternative. The difficulty of the task, i.e. the quality of the incoming sensory
19 ramping slope or drift rate of activity in LIP is smaller (Shadlen & Newsome, 1996, 2001).
Furthermore, the monkeys were found decide upon reaching a certain level of activity of the LIP neurons selective for that choice response. This level was independent of coherence and response time, which coincides with the definition of the sequential sampling models’ decision threshold (Wang, 2008).
Model-based fMRI recordings
On a larger scale, functional Magnetic Resonance Imaging (fMRI) studies combined with sequential sampling models show how variation in activity in specific brain regions account for the SAT and influence model parameters. fMRI studies reveal that speed cues increase baseline activity in several decision-related associative and pre-motor areas, while having no effect on sensory areas (e.g., Bogacz et al., 2010; Forstmann et al., 2008; Ivanoff, Branning, & Marois, 2008; van Veen, Krug, & Carter, 2008). Therefore, it is assumed that the variables influencing speed and accuracy in decision-making find their source in changes in decision-related areas of the brain, rather than sensory areas.
For example, model-based fMRI revealed that interindividual RT and accuracy differences in response inhibition were located in the right inferior frontal cortex (rIFC) (Forstmann et al., 2008). Differences between trials of the same individual have been linked to corticobasal ganglia
functioning (van Maanen et al., 2011). Emphasis on accuracy triggered changes in different parts of the corticobasal ganglia than emphasis on speed. It indicates that accuracy and speed adaptations are induced by different mechanisms, as can be expected from the different latent variables in sequential sampling models.
A sequential sampling model called the Linear Ballistic Accumulator (LBA) was used to find neural substrates underlying the distance between starting point and decision threshold. Instead of a starting point and decision threshold, the LBA uses a variable that states the distance between the two. Therefore, the changing of either starting point and decision threshold is mathematically equivalent when using this model, and the SAT can be caused by changing either of them
(Forstmann et al., 2010). Model-based fMRI analysis was used to find neural substrates underlying the distance over which evidence needs to be accumulated. For example, the instruction of making a
20 decision with speed emphasis results in an increase of sustained activity in several decision related associative and pre-motor areas, among which the striatum (Bogacz et al., 2010). Next to that, upon the instruction to be fast participants show increased sustained activity in the pre-supplementary motor area (pre-SMA) (Forstmann et al., 2008). Both the striatum and the pre-SMA are therefore expected to influence the distance between starting point and decision threshold.
In model-based studies using MRI and probabilistic tractography, it was shown how individuals’ ability to control speed and accuracy is related to the available cortico-striatal connections in the brain (Forstmann et al., 2010). The adjustment of either baseline or decision threshold is therefore dependent on how the cortex is connected to the striatum. According to the striatal hypothesis it decreases the decision threshold or increases the starting point by increasing activation from cortex to striatum and thereby releasing the brain from inhibition.
Example uses of sequential sampling models
Recent investigations used variants of sequential sampling models to understand specific processes in the brain. To better understand why sequential sampling models were used in these specific cases two examples will be given and assessed.
The origin of age-related difference in posterror slowing
Dutilh et al. (Dutilh et al., 2012; Dutilh, Forstmann, Vandekerckhove, & Wagenmakers, 2013) used a simplified diffusion model to assess the psychological processes responsible for age-related
differences in posterror slowing (PES). The results of both an RDM task and a lexical decision task show that the RT of a decision increases when the preceding decision led to an error. In older participants this effect is more pronounced. The diffusion model serves to examine from which psychological processes the PES originates, and what causes the difference between younger and older participants.
A simplified version of the diffusion model was necessary due to a restricted amount of
EZ-21 diffusion model (Wagenmakers, van der Maas, & Grasman, 2007) are that it is especially useful when fitting the data from individuals , and it needs only behavioral speed-accuracy data to fit just a few parameters (drift rate, distance between starting point and decision threshold, and bias
parameter).
The model served to test and distinguish between several psychological explanations of PES: (1) attentional distraction due to the error, (2) time wasted on irrelevant processes, (3) a bias against the response made in error, (4) increased variability in a priori bias, or (5) increase of response caution after the error. According to Dutilh et al. (2012) the diffusion model serves well to
dissociate between the optional explanations, since it is able to isolate and interpret psychological processes that are responsible for PES. The results showed that PES is caused by a combination of response caution and a change in response bias.
How prior knowledge biases decisions
Mulder et al. used a diffusion model combined with fMRI to investigate through which mechanism prior knowledge biases decisions. Two hypotheses tell that prior knowledge respectively changes the starting point or the drift of evidence accumulation, causing the observed bias. Since behavioral results on mean RT cannot distinguish between these two options, the behavior needs to be
analyzed with a mathematical model. Furthermore, the authors aim to find a common neural substrate for the bias by application of the model to fMRI data.
Two types of prior knowledge were given during experiment: (1) a prior probability telling the subject which alternative has a higher likelihood of being correct, and (2) an indication that one of the alternatives has a larger potential payoff. For both conditions, modeling results showed that the biasing effect was primarily due to change in the starting point of evidence accumulation.
Accordingly, imaging results show a change in baseline activity in regions of the frontoparietal network in both prior knowledge conditions.
In this example, the diffusion model primarily serves to find which decision variables are affected by the different prior knowledge conditions. The simplicity of the model, needing only data on
22 model parameters are applied to each individual’s fMRI data, to learn about the effect of prior knowledge on neural activity. Interestingly, model-based fMRI studies show the great advantage that comes with mathematical modeling of behavioral data (Mulder, van Maanen, & Forstmann, 2014). Without the modeling step in between behavioral experiment and fMRI analysis, the latter is less capable of producing useful results.
Sequential sampling models and the computational level of understanding
In sum, sequential sampling models describe the process of decision-making by assuming an accumulation of information and decomposing it into a range of abstract, latent variables. The models thereby serve to answer what decision-making is: through an analysis of observed decision behaviors, i.e. accuracy and RT, variables like drift rate and decision threshold were found to be relevant parts of decision-making. The question why the decision-making process exists could be translated to why an accumulation of information and the decision variables are necessary. Finding that noise is an inevitable part of evidence accumulation, the purpose of accumulation of evidence over time clearly lies in its evolutionary advantage: evidence accumulation to a threshold decreases the effect of noise and increases the likelihood of making the right decision.
Since sequential sampling models explain what decision-making is and why it exists in the first place, this model type fits within Marr’s (1982) computational level of understanding. Sequential sampling models show what is known about decision-making on this abstract level, and constrain the options for representations and algorithms in Marr’s second level of understanding, which is discussed in the next part.
23
THE REPRESENTATIONAL/ALGORITHMIC LEVEL OF UNDERSTANDING:
neural-network models
Marr’s (1982) representational/algorithmic level of understanding aims to answer how a complex system works and what representations should be used to describe the inputs and outputs of the system. In the case of decision-making the level must describe how decision-making works algorithmically, and what expressions should be used to represent incoming evidence, final decisions and intermediate decision processing. This chapter serves to show that these questions can be answered through neural network models.
Theory on neural network modeling
Like sequential sampling models, neural network or connectionist models are abstract descriptions of how a function like decision-making might work. The neural networks consist of nodes and edges, which represent neurons and their interconnections respectively. By representing input sensory evidence as incoming firing rates, the nodes can activate and transfer their activation to the rest of the network. Ultimately, the decisional network model should be able to exhibit similar activity as found in the brains of humans performing the same decision task.
The neural network models that are given most consideration in the literature take off where sequential sampling models end. Each of the successful aspects of the latter has a representation in the network algorithms (Mazurek, 2003). Incoming sensory information that serves as evidence for the decision to be made is represented by activity in a first set of nodes. A second ensemble of nodes represents the decision variable through accumulation of input from the first stage. Thirdly, the decision threshold is represented by a particular amount of activation of this second set of nodes, upon which the decision is made and the process ends.
In addition to implementing the successful parts of sequential sampling models, neural network models should be able to account for their shortcomings as well. In particular, the diffusion model asks for a revision in which both its optimal performance and its ability to explain experimental findings of the SAT remain, while adding the feature of an easy extension from two to multiple choice alternatives, as can be done in the horse race model. To achieve this, several methods were
24 proposed in which inhibition plays an important role (Bogacz et al., 2006). The way in which this inhibition is implemented characterizes the two most used and discussed neural network models: the feed-forward inhibition model and the mutual inhibition model.
Feed-forward inhibition
The architecture of the feed-forward inhibition model for two-choice decision tasks is illustrated in Figure 3a (Bogacz et al., 2006; Ditterich, Mazurek, & Shadlen, 2003). Two or (in the case of multiple-choice decision tasks) more input nodes receive sensory information consisting of perceptual input 𝐼𝑖 for choice alternatives 𝑖 = [1,2] and some noise ±𝑐 (Mazurek, 2003; Shadlen & Newsome, 2001). The resulting input activation gets passed on to the corresponding response units 𝑦𝑖, which
integrate the obtained information over time. Simultaneously, the input nodes inhibit all other response units, i.e. the response units that do not correspond with the input’s choice alternative. The activity in the response units therefore reflects the difference in input information over time from opposing input units. The decision is made once the activity of one of the response units exceeds a decision threshold.
As in the diffusion model, several parameters in the feed-forward inhibition model are variable. Baseline activity of response units, as well as the drift rate can be adjusted to fit experimental
Fig. 3. Architectures of neural network decision models. Arrows denote excitatory connections; lines with filled circles denote
inhibitory connections. (a) Feed-forward inhibition model (simplified from Ditterich, Mazurek, & Shadlen, 2003). (b) Mutual inhibition model (simplified from Usher & McClelland, 2001). Adapted from (Bogacz et al., 2006).
25 results (Mazurek, 2003). For example, raising the accumulation starting point corresponds to a higher baseline activity, decreasing the distance to the decision threshold and thereby biasing the decision.
In addition, the variable 𝑢 describes the weight of feed-forward inhibition; the higher the weight, the stronger the inhibitory effect of inputs on opposing response units. When 𝑢 = 1, the inhibition and excitation of an input equal one another. As a result, the response units’ activation is exactly the difference between activities of the input units, which is equivalent to dynamics of the diffusion model. On the other hand, when 𝑢 = 0 no inhibition is present, reducing the feed-forward inhibition model to the race model. The degree to which input nodes inhibit response nodes therefore
distinguishes this neural network model from sequential sampling models. It is the creation of an algorithm that inspires the creation of this type of new variables which were not found through behavioral analysis.
In the case of multiple alternatives the feed-forward type of inhibition can become problematic (Usher & McClelland, 2001). When all of the 𝑁 input units receive a more or less equal amount of input evidence, each of the response units gets inhibited by 𝑁 − 1 nodes while receiving excitation from only one node. As a result, the activity of all response units can be negative, making them unable to cross a decision threshold. The problem can be solved by decreasing the inhibition weight 𝑢 with a growing amount of choice alternative. However, it implies that the amount of inhibition caused by one amount of input evidence can be more or less depending on the amount of options to choose from. A more plausible way to solve this problem is the use of lateral inhibition, as in de case of the mutual inhibition model.
Mutual inhibition
Figure 3b illustrates the architecture of the mutual inhibition model or Leaky Competing Accumulator (LCA) (Bogacz et al., 2006; Usher & McClelland, 2001). As in the feed-forward inhibition model, sensory evidence is accumulated over time by response units that receive lower level input activity. However, instead of bottom-up inhibition, in mutual inhibition models the response units inhibit one another, laterally. The response unit that is most consistent with the input evidence will receive most excitation, and thereby have the strongest inhibitory influence on
26 other response units. It results in a winner-take-all dynamics, always ending up with one response unit that crosses the decision threshold.
To prevent infinite accuracy by endless evidence accumulation, the mutual inhibition model adds stochastic leakage or decay of information accumulation to the mutual inhibition architecture (Usher & McClelland, 2001). Similar to the O-U model, the accumulated information reaches an asymptotic level once the accumulation of new evidence and the loss due to leakage balance each other out. In addition, phase plane analysis shows that a balanced mutual inhibition model, in which decay equals inhibition, is equivalent to the diffusion model, given that both inhibition and decay take high values relative to noise strength (Bogacz et al., 2006, 2007).
The mutual inhibition model, like the feed-forward inhibition model, uses basic features and parameters from sequential sampling models. The equivalence between sequential sampling and algorithmic models is evident: considering that sequential sampling models provide a
‘computational’, defining understanding of the process of decision-making they necessarily constrain models that provide a lower, structural level of understanding. However, what separates sequential sampling and neural network models into two levels of understanding is their approach to elucidate features of decision-making. The algorithmic architecture of neural network models enables to investigate structural features like different types of inhibition, while sequential sampling models are restricted to explaining basic, functional principles of decision-making.
Additional features of the mutual inhibition model show how the representational/algorithmic level of understanding is not only constrained by the computational level of understanding. Research on a biophysical level resulted in the addition of recurrent self-excitation of the network nodes (Usher & McClelland, 2001). It allows the model to exhibit neural attractor dynamics, where activity of the “winning” accumulator node reaches a stable equilibrium (Bogacz et al., 2007). The resulting persistent activity can explain how decisions are stored in working memory, and are also thought to underlie phenomena in perception and language processing.
27
Neurobiological evidence
Neurobiological evidence for neural network models lies in the neural activity that made scientists develop the models in the first place (Mazurek, 2003). Originally, the activation of input nodes was designed to represent activity of neurons in sensory area MT. Accumulator or response units should exhibit activity similar to that of neurons in LIP, which is expected to facilitate evidence integration.
Ditterich, Mazurek and Shadlen (2003) studied how the activity of neurons in MT visual area are interpreted by the rest of the brain. While performing an RDM task, direction-selective neurons in the MT area of rhesus monkeys received microstimulation, causing the monkeys to choose the stimulated neuron’s preferred direction more often. In addition, it quickened the making of the decisions in favor of the preferred direction, while slowing decisions in favor of the opposite direction. The microstimulation was therefore interpreted as additional information for the decision, even when ending up with a decision different from the stimulated neuron’s preferred option. Finally, it was concluded that the MT area is part of the decision-network, in that it delivers information on sensory evidence used in the decision process.
As for the LIP, several mentioned experimental results indicate this region to be a plausible region for the integration of evidence activity. Single neuron recordings in rhesus monkeys show how neurons in area LIP are selective for specific motion strengths and directions in visual space: the more and the stronger a dot in the RDM task moves towards the neuron’s receptive field (RF), the higher its activation, while movement away from the neuron’s RF results in a suppression of ongoing activity (Shadlen & Newsome, 2001). Over time, the neurons show an increase of activity corresponding to differential sensory activity as predicted by neural network models.
Example uses of neural network models
A study by Schurger, Sitt and Dehaene (2012) shows how neural network models not only serve as a testable algorithm for specific phenomena in decision-making, but can also inspire the
interpretation of these phenomena. A leaky competing accumulator model (Usher & McClelland, 2001) was used to model the neural process that leads to the decision of when to move, in a task in
28 which the only instruction is to make a movement within a range of seconds. The question to
answer is what triggers the movement or the decision to move, if it’s not external evidence.
With the model, the interpretation of the readiness potential (RP) is revisited. This mounting neural activity preceding spontaneous movements was commonly interpreted as preparation for execution of the motor act. However, Schurger et al. (2012) suggested that the RP is caused by the decision process, acting as accumulator or response nodes in the LCA. The lack of external evidence for the decision results in accumulation of internal physical noise only. The actual decision and the following movement are made once this noise randomly crosses the decision threshold.
The authors do not explain why the LCA was used to model the RP, instead of other neural network models or instead of the diffusion model. It was stated that the model needs to account (1) for the shape of neural activity that reflects evidence accumulation before making a movement, and (2) the distribution of RTs. The model was fitted to behavioral data, after which it should be able to predict neural activity reflecting the RP that was measured by means of electroencephalography (EEG). According to the study, the LCA is found to be “conceptually sufficient to account for (…) behavioral and EEG data” (Schurger et al., 2012). However, a diffusion model should be able to fit behavioral and electrophysiological data as well. This shows how sequential sampling models and neural network models overlap in their abilities to predict aspects from the decision-making process. Both model types are equally eligible to make similar or the same predictions with. This creates two possible implications for Marr’s system of levels of understanding.
First, if sequential sampling models and neural network models occupy different levels of understanding, than these levels need to overlap where both model types are able to explain the same phenomenon. It would indicate that Marr’s levels of understanding are not as independent as he proposed.
Second, from a different perspective, sequential sampling models might not only exist in the computational level of understanding, but also provide an intuition on the
representational/algorithmic level of understanding decision-making. The same counts for neural network models, which provide an understanding on the computational level as well as the representational/algorithmic level. Therefore, in the case that levels of understanding are independent, the models that occupy these levels build bridges between the levels.
29
Neural network models and the representational/algorithmic level of understanding
In sum, neural network models describe how the process of decision-making can be described by an algorithmic architecture. The models consist of two sets of nodes, interconnected by edges, in which the successful latent variables from sequential sampling models can be recognized. Sensory input is received by the first set of nodes and represented as the node’s activation, with one node for each choice alternative. Every node projects input activity to its corresponding response unit, which accumulates input evidence to a decision threshold. Drift rate is determined by the strength of these connections, and a baseline activity in the response units influences the accumulation starting point. Importantly, neural network models include either lateral or feed-forward inhibition of the response units. As a result the units accumulate the differential evidence for their choice
alternatives. In specific cases of the models’ variables they can be reduced to two- or multi-alternative diffusion models.
Since neural network models explain how decision-making works algorithmically and provides representations of inputs and outputs within the process, this model type fits within Marr’s (1982) representational/algorithmic level of understanding. Neural network models are constrained by the meaning of decision-making as stated by sequential sampling models, but also need to be easy to implement in their piece of hardware, the brain. Therefore, a second set of constraints comes from biophysical knowledge on the brain, as described by Marr’s third level of understanding.
Although neural network models have features that cannot be found in sequential sampling models, the predictions of both model types overlap for certain parameters. This indicates that Marr’s levels of understanding are not as independent as he proposed, but are actually linked due to how both model types can occupy more than one level of understanding.
30
THE IMPLEMENTATIONAL LEVEL OF UNDERSTANDING:
biophysically plausible decision models
Marr’s (1982) implementational level of understanding aims to answer in which way the algorithm of a complex system can be implemented in its hardware. In the case of decision-making the level must describe how the process that is described in the first and second levels of understanding is implemented in the brain. This chapter serves to show that biophysically plausible decision models provide an elaborate theory on the implementation of decision-making in neural networks of which the brain consists.
Theory on biophysically plausible modeling
Since the brain is investigated on levels ranging from the micro- to the macroscopic scale, biophysically plausible models come with the same variety. In the case of decision-making, it is interesting to look at how it amounts to activation of macroscopic brain areas, as well as to information processing on a cellular, microscopic scale. To cover both levels, two currently much investigated biophysically plausible models of decision-making will be discussed here. On a
macroscopic level, Frank et al. introduced a decision-making model involving the basal ganglia (BG) as a gate for execution or inhibition of several action alternatives (Frank & Kong, 2008; Frank et al., 2004; Frank, 2006; O’Reilly & Frank, 2006; Ratcliff & Frank, 2012). The mesoscopic level is covered by pooled inhibition models (Wang, 2002), which use electrophysiology of single neurons within larger pools that exhibit attractor dynamics. On the same, microscopic level, variability in structure of single neurons was used to model probabilistic computations like evidence integration, of which the details can be found in (Ma, Beck, & Pouget, 2008). Ultimately, a combination of the
biophysically plausible models on levels that range from the micro- to the macroscopic scale should give an overview of the implementation of decision-making algorithms in the brain.
The basal ganglia model
The basal ganglia model for decision-making (Bogacz & Gurney, 2007; Frank & O’Reilly, 2006; Frank et al., 2004; Frank, 2005, 2006) predicts neural activity at the level of the BG, including circuits that
31 connect the BG nuclei to one another and to the cortex. According to the model, frontal cortical units generate candidate motor actions in response to a stimulus (Bogacz & Gurney, 2007; Roger Ratcliff & Frank, 2012). Decision-making boils down to choosing one of these competing actions. The BG are supposed to have a modulatory role in decision-making, acting as a gate that either facilitates or inhibits neural activity in favor of the decision alternatives. As in both sequential sampling and neural network models, once activity corresponding to one of the decision alternatives crosses a decision threshold the decision is made in favor of that alternative.
The gating function of the BG is initialized by input information on the decision problem at hand, provided to the striatum, the BG’s main input nucleus (Bogacz & Gurney, 2007; Frank & O’Reilly, 2006; Roger Ratcliff & Frank, 2012). In the case of a visual motor task, the striatum receives stimulus information and
information on the possible response actions through connections with the sensory cortex and the presupplementary motor area (preSMA) respectively. As shown in Figure 4, the neurons in the striatum are divided into two main populations by a segregation of two kinds of dopamine receptors: D1 and D2. The populations form the starting points of two pathways through
the BG, which have a differential effect on further processing of the input activity.
Activation of the striatal “Go” population inhibits units in the globus pallidus internal segment (GPi), which has the effect of disinhibiting the thalamus, to which the output nuclei of the BG project (Bogacz & Gurney, 2007; Frank & O’Reilly, 2006; Roger Ratcliff & Frank, 2012). The thalamus in itself is connected to the preSMA. As a result of its disinhibition, the thalamus can be activated by preSMA units which correspond to the optional motor action that initially activated the “Go”
population. Reciprocally, the active thalamus amplifies preSMA activity, increasing the likelihood for the active motor response to first cross the decision threshold and thereby be executed.
Fig. 4. Schematic representation of the cortico-striato-thalamo-cortical loops that
include the Go and NoGo pathways in the BG. GPi: internal segment of globus pallidus; GPe: external segment of globus pallidus; SNc: substantia nigra pars compacta; SNr: substantia nigra pars reticulata; VTA: ventral tegmental area. Taken from (Frank, 2005).
32 Different from the Go-pathway, the striatal “NoGo” population projects to the globus pallidus
external segment (GPe) first, before arriving at the GPi. Activation of the NoGo-pathway therefore inhibits the GPe, and subsequently excites the GPi (Bogacz & Gurney, 2007; Frank & O’Reilly, 2006; Roger Ratcliff & Frank, 2012). This further inhibits the thalamus, and thereby reduces the likelihood of the current optional motor action to be executed.
Input evidence might activate the cortical response units enough for them to cross the decision threshold and lead to a response, without any gating by the BG. Especially when the cortical response units do also inhibit one another laterally, as is the case for mutual inhibition neural network models. However, the addition of BG gating adds selective bottom-up input to one of the cortical response units. Thereby the corresponding response action receives a big advantage over the other choice alternatives, nonlinearly influencing the decision process. BG gating combined with lateral inhibition of the cortical response units is therefore able to speed the decision process up and cause a winner-take-all dynamics.
Another purpose of the BG in decision-making is proposed to lie in the ability to learn from previous decision-making. Positive and negative reinforcement following a decision respectively results in positive and negative phasic dopamine changes (Frank & O’Reilly, 2006; Frank et al., 2004; Frank, 2005). These changes induce synaptic
plasticity in connection weights of the Go- and NoGo-pathways, due to long-term
potentiation (LTP) and long-term depression (LTD). The resulting differences in synaptic strengths and the therefrom following input to the two pathways causes larger or smaller effects of future response evidence to the activity of cortical response units. These learning effects thereby either facilitate or suppress some responses over others when being confronted with the same response alternatives.
Fig. 5. Schematic representation of the attracor model for
decision-making. Stimulus input provides stimulus evidence for choice alternatives 1 or 2. Two neuron pools are selective to the choice alternatives and receive stimulus input current. One neuron pool is inhibitory and inhibits the excitatory neuron pools. Neurons within one pool are connected recurrently. All neurons receive background noise. Adapted from (Wong & Wang, 2006).
33
The attractor model
Xiao-Jing Wang (2002) created a model of the basic cellular and synaptic mechanisms of a decision-making circuit. Importantly, he used the finding that neurons in the LIP and other areas that
correlate with information accumulation and decisional activity, all show persistent activity during a delay period between stimulus and response. It resembles how information of a sensory cue is held in working memory, found in animals (Hernández, Zainos, & Romo, 2002; Romo, Merchant, Zainos, & Hernández, 1997). Wang therefore based his model on attractor networks, which can hold persistent activity over time, as used in working memory models. He hypothesized that an attractor network can provide a theoretical framework for decision-making. Therefore it should integrate stimulus information, choose one decision option over others, and hold on to that decision during a delay period, while agreeing with biophysical features of the neural circuit of decision-making. The attractor model for two-choice decision-making consists of one inhibitory and three excitatory pools of recurrently connected neurons (Wang, 2002). Each choice alternative is represented by a pool of excitatory neurons, which means that an increase in the excitatory pool’s activity increases the likelihood for its corresponding choice alternative to be chosen. A pool of interneurons receives excitation from the three excitatory pools, upon which it inhibits the same three excitatory pools. Thus, with increasing activation of one selective pool, all three pools receive a larger amount of inhibition. Together with the recurrent, excitatory connections within the neuron pools, the overall inhibition creates a winner-take-all dynamics, in which one selective neuron pool is bound to be the most active, limiting the activation of its competitor.
Stimulus input activity (plus an amount of noise that accounts for the stochasticity of a decision) is assumed to be equal for both selective neuron pools (Soltani & Wang, 2006; Wang, 2002). However, the selective neuron pools are not equally excited by the same input. The degree to which a
selective neuron pool is excited by afferent neurons is determined by the pool’s synaptic strength: the fraction of a pool’s synapses that is in the potentiated state. The larger the amount of
potentiated synapses, the stronger the pool’s excitation. Therefore, two pools with different synaptic strengths receive a different amount of excitation caused by the same stimulus. As in the BG model, synaptic plasticity by LTP or LTD changes the effect of stimulus input on the two selective neuron pools by a change in their synaptic strengths. In other words, the Hebbian synaptic plasticity
34
caused by rewarding (LTP) or non-rewarding (LTD) feedback results in learning which decision to make in response to the stimulus.
Neurobiological evidence for biophysically plausible modeling
Basal ganglia pathways and the DA system
The pathways into which the BG are divided according to the BG model, were constructed through analyses of the general circuitry within the BG (Utter & Basso, 2008). Most BG nuclei strictly project to other BG nuclei, creating a vast subcortical network on its own. Characteristics of the nuclei that form the in- and outputs of the BG inspired the creation of pathway models.
The striatum forms the main input nucleus of the BG, with direct input coming from virtually the entire cerebral cortex (Nambu, Tokuno, & Takada, 2002). Within this cortical input, five parallel information processing circuits were identified: a motor circuit, an oculomotor circuit, a
dorsolateral prefrontal circuit, a lateral orbitofrontal circuit, and an anterior cingulate circuit (Utter & Basso, 2008). In addition, some of the cortical regions that provide input to the striatum are also targeted through the output of the BG (the GPi and substantia nigra pars reticulata (SNr)), via the thalamus, which indicates how the BG form an intermediate processing hub for these areas. The diversity of the input connections to the BG shows its involvement in the processing of a broad range of functions that are all associated with decision-making.
A second source of input to the striatum is the ventral midbrain, as part of the dopaminergic system (Utter & Basso, 2008). The striatum contains two types of DA receptors on which DA coming from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) has different effects. An increase of DA level (e.g. following a reward) excites striatal neurons that have D1 receptors and inhibits striatal neurons with D2 receptors. In opposite direction, a decrease in DA level (e.g. following a punishment) results in inhibition of striatal neurons with D1 receptors, while it excites the neurons with D2 receptors.
Importantly, it was found that the D1 and D2 receptors are distributed differentially on the dendrites of striatal neurons (Gerfen et al., 1990). While D1 receptors are found on dendrites that directly project from the striatum to the BG’s output nuclei, dendrites with D2 receptors pass
35 through other BG nuclei before arriving at its output. This insight led to the creation of the direct and indirect (or Go and NoGo) pathways. Although more recent evidence suggested that two receptor types are not entirely separated into two pathways (Aizman et al., 2000), the D1 and D2 receptors are overall found to be more pronounced in the direct and indirect pathways. Therefore the phasic changes in DA level has different effects on the BG. A positive DA change excites the “Go” pathway, enabling the corresponding choice alternative to be executed, while a negative DA change has the opposite effect.
Decision-making on the neuronal level
Decision-making at the level of individual neurons and neural networks was first investigated in the past two decades, using physiological and psychophysical experiments. With a random dot motion paradigm, in which participants have to decide with a saccade on the direction in which a cloud of dots moves, Shadlen and Newsome (1996, 2001) found several areas in the monkey’s brain that show activity correlated to parts of the decision process.
Notably, activity in the lateral intraparietal area (LIP) was found to slowly increase when a stimulus is presented. Activity of the LIP is correlated with both the stimulus input and the monkey’s
decision. In addition, the LIP projects to the prefrontal cortex and visual motor areas MT and MST (for saccade responses) (Kim & Shadlen, 1999), and to the medial premotor cortex (for a
vibrotactile discrimination task; (Hernández et al., 2002; Romo et al., 1997). As a result, the LIP is considered to play an important role as part of the decision-making neural circuit: it accumulates sensory information on the stimulus and signals which of the optional responses is to be executed. Therefore, the activity of the LIP and its implicated role in decision-making inspired the creation of the attractor, pooled inhibition model for decision-making (Wang, 2002).
Not only did neurobiological findings on decision-making activity in the brain inspire the creation of the attractor model for decision-making, but the model itself provided a deeper understanding of the decision-making process as well (Wong & Wang, 2006). A phase-plane analysis, depicting the activation levels of the two selective neuron pools over time, shows how the activity of one of the two selective neuron pools is bound to end up in an attractor, at the cost of activity in the other selective neuron pool. This winner-take-all dynamics is therefore inherent to the configuration of
36
the neural network. In other words, decision-making naturally emerges from the neural dynamics once a stimulus is provided.
Another result that was found with the attractor model is how different receptors influence a decision. In an attempt to reduce the attractor model to an approximated version with less
parameters, the effects of inclusion or exclusion of different receptor types was investigated (Wong & Wang, 2006). Due to its slow synaptic time constant, the synaptic dynamics of NMDA receptors turned out to be significant in creating a slow temporal integration, leading to accurate decisions. When substituting NMDA receptors by faster AMPA receptors at the recurrent synapses, the decision-making network was found to become highly unstable, leading to results that are not comparable with experimental findings.
Example uses of biophysically plausible models
Reward-guided learning in relation to age and Parkinson’s disease
The BG model for decision-making was used to model why elderly people and patients suffering from Parkinson’s disease learn less from rewarding decisions and more from non-rewarding decisions compared to younger, healthy controls (Frank & Kong, 2008; Frank et al., 2004). Both groups were found to have a lower tonic DA level than younger healthy controls, and an increase of DA level through medication resulted in normal learning behavior. The level of tonic DA is therefore implicated to influence reward-guided learning, and plays a major role in the mechanism that underlies reward-guided learning as proposed by the BG model for decision-making.
Due to the different effects that DA has on D1 and D2 receptors, the level of DA influences which BG pathways in the model are either activated or deactivated (Frank, 2005). A rise in DA level activates the Go pathway and deactivates the NoGo pathway, while a decay of the DA level has the opposite effect. A phasic DA change that is large enough has longer lasting effects through Hebbian learning and synaptic plasticity. In the case that one receives positive feedback DA level is supposed to rise enough for the synapses in the Go pathway to be potentiated (long term potentiation; LTP), while the synapses in the NoGo pathway are depressed (long term depression; LTD). Again, a DA level that is low enough due to negative feedback has the opposite effect.
