Cholinergic interneurons in the dorsomedial striatum: effect of modulation on performance in a second-order schedule of reinforcement

(1)

Cholinergic interneurons in the dorsomedial striatum:

effect of modulation on performance in a second-order

schedule of reinforcement

Interneurons migrating to the striatum (Verona Villar-Cerviño)

Erik de Keijzer

Supervisor: Ingo Willuhn

i.willuhn@nin.knaw.nl

Co-assessor: Carien Lansink

C.S.Lansink@uva.nl

Netherlands Institute for Neuroscience

(2)

Instrumental learning is the acquisition of an instrumental behaviour in order to obtain a reward. A reinforcer drives instrumental behaviour like pushing a button, flipping a switch or pressing a lever. Performance of instrumental behaviour involves the striatum as well as other subcortical basal ganglia structures. Different types of instrumental learning are thought to involve different parts of the striatum in distinct functional striato-thalamo-cortical loops. The dorsomedial striatum (DMS) is part of a functional circuit associated with action-outcome learning and goal-seeking behaviour. The acquisition of the action-outcome association in the DMS is thought to involve the medial prefrontal cortex, as well as VTA dopaminergic reward signalling. Cholinergic interneurons in the striatum are capable of modulating the function of this network by potently inhibiting striatal medium spiny neurons (MSNs). The exact role of cholinergic interneurons in the acquisition and maintenance of instrumental behaviour is nonetheless unknown. In the current investigation, cholinergic interneurons in the DMS are modulated through Designer Receptors Exclusively Activated by Designer Drugs (DREADDs). Both inhibition and excitation of cholinergic interneurons led to an increase in instrumental behaviour for a food reinforcer. Increased lever pressing in a second-order schedule of reinforcement was observed across both interneuron-inhibiting and interneuron-exciting groups, but in different stages of training. Excitation of cholinergic interneurons in the DMS during ‘early’ training led to increased instrumental behaviour, whereas inhibition of these interneurons in later stages of training had the same effect. The results are consistent with the view that dorsomedial and dorsolateral parts of the striatum compete for control over behaviour. Whereas the dorsomedial striatum is critically involved in acquisition of a goal-directed behaviour, well-established responding under second-order schedules of reinforcement is under dorsolateral striatal control.

Keywords

(4)

Introduction

Decision making in the striatum is important for the appropriate selection and execution of motor actions (Balleine et al., 2009). Rewarding stimuli in the environment can trigger dopamine release in the striatum. Dopamine has a reinforcing effect on behavior performed to acquire the reward. Instrumental behavior involves the mutation of a manipulandum in order to obtain a reward (eg. pushing a button, flipping a switch, pressing a lever). The reinforced behavior in this case is the instrumental behavior to obtain the reward. Instrumental learning involves the encoding of associations between an action and its outcome or between a stimulus and a learned response in the striatum. For instrumental behavior, these associations are thought to be encoded in the dorsal striatum (Yin et al., 2006; Murray et al., 2012).

Many inputs from the environment as well as other brain areas converge in the striatum. The striatum plays a central role in integration of these stimuli and selection of the appropriate (re)action. The principal cells of the striatum are GABA-ergic medium spiny neurons. These receive glutamatergic inputs from the cortex and thalamus and project to the globus pallidus interna and substantia nigra pars reticulata or the globus pallidus externa and subthalamic nucleus, in the direct or indirect pathway, respectively (Calabresi et al., 2014). The abundant subcortical cell population in the striatum also comprises GABA-ergic as well as cholinergic interneurons interconnecting medium spiny neurons, interneurons, and dopaminergic and glutamatergic axon terminals (Lim et al., 2014). Cholinergic interneurons supply the striatum with the highest concentration of acetylcholine of any area in the brain (Schulz et al., 2013). Furthermore, cholinergic interneurons have large somata (>15 microns), are aspiny and tonically active (Lim et al., 2014). They display a distinct pattern of activity, concurrent with dopaminergic signaling (Morris et al., 2004). Accordingly, concurrent cholinergic and dopaminergic modulation of medium spiny neurons in the striatum is hypothesized to be of great importance for reinforcement learning (Schulz et al., 2013; Witten et al., 2010). However, the exact role of cholinergic interneurons in this mechanism remains unknown.

The medial and lateral parts of the dorsal striatum compete for control over behavior (Everitt et al., 2013; Willuhn et al., 2012). Over time, behavior that was guided by dorsomedially stored action-outcome associations can become controlled by dorsolateral stimulus-response driven behavior. Phasic reinforcing dopamine signals shift from the dorsomedial to the dorsolateral striatum along with control over behavior (Yin et al., 2006; Willuhn et al., 2012). This mechanism is implicated in the formation of habits as well as in the emergence of maladaptive forms of habitual behavior, like obsessive compulsive disorder and addiction.

The dorsomedial- and dorsolateral striatum are part of distinct striato-thalamo-cortical functional loops and under differential dopaminergic control (Yin et al., 2006). The dorsomedial striatum (DMS) is innervated by dopaminergic neurons originating in the VTA, whereas the dorsolateral striatum (DLS) is innervated by substantia nigra pars compacta dopamine neurons (Everitt et al., 2013). Dysfunctions in the dopaminergic modulation of striatal circuitry can lead to motor impairments as well as cognitive deficits. Well-known illnesses involving striatal dysfunction are Parkinson’s disease, Huntington’s disease and Tourette syndrome.

In turn, instrumental behavior can become maladaptive in addiction, when pursuance of rewards becomes habitual and impedes normal behavior. Additionally, after repeated pairing with a reward, cues indicating an upcoming reward can acquire reinforcing properties in itself (Everitt et al., 2000). In drug addiction, these environmental cues that have previously been paired with drugs of abuse can drive maladaptive behavior (Everitt et al., 2000; Di Ciano et al., 2005).

(5)

Second order reinforcement schedules

Second order reinforcement schedules (SO-schedules) capitalize on the reinforcing properties that can be obtained by a cue. Well-established instrumental behavior under a SO-schedule is driven by the presentation of a cue indicating an upcoming reward (Everitt et al., 2000; Everitt et al., 2013). This cue, which has previously been paired with the reward, is now presented dependent on instrumental responding, but instrumental behavior is only occasionally reinforced by delivery of the (first-order) reward. Thus, the first-order reward (paired with the cue) can sustain behavior, despite only being delivered infrequently. The second order cue will guide behavior between reward delivery intervals (Everitt et al., 2000; Di Ciano et al., 2005). SO-schedules are especially suited for the investigation of reward-driven instrumental behavior in combination with drugs of abuse. SO-schedules can be used to prolong a subject’s reward seeking behavior before the delivery of a drug reward. The use of a SO-schedule ensures the reward seeking behavior can be prolonged during the first interval, before the first drug infusion. A distinction can subsequently be made between behavior in the (first) drug-free interval and the drug-influenced behavior thereafter. The conditioned cue, signaling the upcoming reward, reinforces the behavior in the absence of the reward (Everitt et al., 2000; Di Ciano et al., 2005). This allows for the studying of goal-directed behavior in the absence of confounding effects of the drug (Yin et al., 2006). For instance, in a trial in which rats have to press a lever during a 15 minute period to obtain a reward, every 10th_{press is accompanied by a cue (this would be}

depicted as FI15(FR10:S)). The cue, which has previously been paired with the reward, is thought to act as a conditioned reinforcer (Everitt et al., 2000). In this example, in the first 15

minutes the behavior will be guided only by the conditioned cue indicating an upcoming reward, independent of confounding effects of administration of the drug. Thus, SO- schedules can be used to induce cue-driven behavior. Over time, the instrumental behavior will be performed habitually, driven by the second order cue (Everitt et al., 2000; Murray et al., 2012).

Control over behavior during the formation of a habit shifts from the dorsomedial to the dorsolateral striatum (Yin et al., 2006; Willuhn et al., 2012). The DMS is thought to encode action-outcome associations, whereas the DLS is associated with stimulus-response behavior (Yin et al., 2006; Everitt et al., 2013). Instrumental behavior for a drug reward comes under dorsal striatal control after extended training (Yin et al., 2006), as does responding for ingestive rewards, after overtraining (Balleine et al., 2009). In addiction, instrumental behavior to obtain a drug reward becomes habitual. Devaluation of the outcome, or degredation of the contingency between reward and outcome does not affect instrumental performance. Thus, the dorsomedial striatum is implicated in the performance of goal-directed behavior, whereas the dorsolateral striatum governs habitual behavior (Everitt et al., 2013; Willuhn et al., 2014).

The role of the dorsomedial and dorsolateral striatum in SO-schedules has so far only been examined in combination with a drug reinforcer (Murray et al., 2012). In the same SO-schedule as used here, a

Notation of second order schedules of reinforcement

Second order schedules are denoted using parentheses. Between the brackets is the number of actions to be performed before the presentation of a stimulus occurs. Outside the parentheses is the criterion that has to be met before delivery of a reward. Depending on the type of schedule, this is the amount of time that has to pass in minutes, or the number of times the stimulus has to be presented, before delivery of the reward. The endpoint of most studies using a second order schedule of reinforcement is an FI15(FR10:S) schedule. A stimulus will be presented every time after a fixed ratio (FR) of 10 presses; a reward will be delivered after a fixed interval (FI) of 15 minutes has passed after the first press. To reach this schedule, animals can be trained on either interval or ratio schedules. Interval schedules increase the time between the first press and delivery of the reward. Ratio schedules increase the number of presses before presentation of a stimulus. Every nth

stimulus presentation is subsequently rewarded according to a (fixed) ratio schedule. Ratio schedules are denoted as e.g. FR10(FR10:S), which can be interpreted as: 10 stimulus presentations before delivery of a reward outside the brackets and in parentheses the effort needed to be made to obtain a cue (10 lever presses).

(6)

dopamine transmission blocker was infused in the DMS or DLS before trails at different time points in training. It was found that blocking dopamine transmission in the DMS during ‘early’ FI15(FR10:S) testing, and the blockade of dopamine signaling in the DLS during ‘late’ FI15(FR10:S) performance dose-dependently reduced instrumental behavior. These findings are consistent with the notion of a shift in control over behavior from the DMS to the DLS in the formation of a habitual behavior (Yin et al., 2006; Willuhn et al., 2012; Murray et al., 2012) and indicate cue-driven behavior during well-established FI15(FR10:S) training is under dorsolateral control. The DMS, on the other hand, is important for the acquisition of the reward seeking behavior in a FI15(FR10:S) second order reinforcement schedule (Murray et al., 2012).

In the current investigation, a SO-schedule is used in combination with a food reinforcer. The goal-seeking behavior is measured and the effect of the conditioned cue is examined. As SO- schedules are usually used in combination with a drug reinforcer, little is known about the use of a food reinforcer in these schedules. One of a few papers pairing SO-schedules with a food reward is Giuliano et al. (2012). In this study, the combination of a food pellet reward with a SO-schedule was used to investigate the effect of a new drug on the motivational stimuli that elicit binge-eating. As the object of the current study is cue-elicited reward-seeking instrumental behavior for a food reward as well, an adaptation of their training was used.

Cholinergic interneurons

Giant aspiny cholinergic interneurons (CINs) are interspersed throughout the striatum. Their extensively branching collaterals innervate large parts of the striatum(Lim et al., 2014). CINs receive afferent glutamatergic input from the rostral pedunculopontine nucleus of the thalamus, as well as the sensorimotor cortex (Wang et al., 2006; Lim et al., 2014). In turn, they innervate other (GABA-ergic) interneurons; have an effect on glutamatergic and dopaminergic axon terminals, as well as a direct influence on the principal striatal medium spiny neurons (MSNs) (Lim et al., 2014). The tonically active CINs are the main source of acetylcholine in the striatum and usually tonically inhibit striatal MSNs. Although comprising only 1% of the cell population in the striatum, CINs are known to be very potent inhibitors of MSNs (Lim et al., 2014; Schulz et al., 2013; Witten et al., 2010). The exact role of cholinergic interneurons in striatal decision making circuitry is nonetheless unknown.

Activity of CINs in the striatum displays a distinct spiking pattern, comprising a pause and rebound in activation (Schulz et al., 2013). A salient incoming stimulus can excite cholinergic interneurons through afferent projections from the intralaminar nucleus of the thalamus. This glutamatergic excitation is followed by a slow calcium-activated potassium current resulting in an afterhyperpolarization(Schulz et al., 2013). Thus, excitation of cholinergic interneurons is followed by a characteristic pause in activity due to an intrinsic cellular mechanism resulting in hyperpolarization. The pause and rebound in activity of CINs coincides with dopaminergic signaling (Morris et al., 2004). In macaques, activity of CINs pauses, relieving cholinergic tone on i.a. MSNs, during the concurrent occurrence of a phasic dopamine signal. This dopaminergic signal originating from the substantia nigra pars compacta (SNc), followed a classical value-encoding activity pattern. Dopaminergic neurons in the SNc would show a phasic burst of activity after either delivery of a reward, or already at the presentation of a reward-predicting cue after repeated pairings of cue and reward. Furthermore, a small burst could occur at the cue and a large burst after delivery of an unexpected reward, depending on the probability of the reward (Morris et al., 2004). Phasic dopamine signalling in the striatum encodes the reinforcing value of a stimulus and has a well-established role in action selection(Balleine et al., 2009; Morris et al., 2004; Willuhn et al., 2014). The reinforcing effect of dopamine signalling is essential for instrumental learning (Steinberg et al., 2013).

Dopamine is thought to have its reinforcing effect through phasic neuronal firing during the pause in tonic cholinergic spiking (Morris et al., 2004). Acetylcholine signaling is particularly suited for this

(7)

purpose as acetylcholine is rapidly degraded by AChE, whereas removal of dopamine from the synaptic cleft is less efficient (Morris et al., 2004). Temporarily lowering cholinergic tone in the striatum is proposed to be necessary for a decrease in activity of nicotinic acetylcholine receptors. When activated nicotinic receptors suppress phasic dopaminergic firing in the dorsal striatum. Nicotinic receptor inactivation boosts dopaminergic firing especially in response to burst firing. Thus, release of the tonic acetylcholinergic inhibition of the striatal medium spiny neurons likely permits the phasic dopamine signal to have its reinforcing effect (Schulz et al., 2013). This way, the cholinergic en dopaminergic systems are proposed to work together, wherein the pause in cholinergic interneuron firing acts as a time-window for dopaminergic influences on plasticity.

The pause in tonic cholinergic interneuron activity is terminated by a phase of increased spiking resulting from a second set of excitatory inputs from the thalamus and cortex (Schulz et al., 2013). This rebound in activation is a result of a cyclic nucleotide-activated cation current. The cation current marks the end of the hyperpolarized state and drives the tonic interneuron firing (Matsumoto et al., 2001). The excitatory input causing the rebound in activity is thought to be context dependent and encode the reaction to a stimulus. The effect of the rebound in CIN activation on local circuitry is the maintenance of a selected action (Schulz, Reynolds 2013).

In the current investigation, we modulate CINs during instrumental behavior using Designer Receptors Exclusively Activated by Designer Drugs (DREADDs). Cholinergic interneurons were pharmacogenetically excited and inhibited during an instrumental learning task. A SO- reinforcement schedule induced goal-directed behavior under the influence of a conditioned cue.

Two groups of genetically modified ChAT::Cre rats (Witten et al., 2011) were infected with hM3Dq or hM4Di DREADDs which could be activated by intraperitoneal injection of CNO, an otherwise biologically inert designer drug. Activation of hM3Dq resulted in a Gq protein coupled activation of the cell, resulting in neuronal excitation (Wess et al., 2013). Alternatively, Gi protein activation through coupling with activated hM4Di led to inhibition of neuronal activity and silencing of cholinergic interneurons. The exact consequence of excitation or silencing of cholinergic interneurons on the balance between the dopaminergic and cholinergic systems, competing dorsal striatal areas or local circuitry is however unclear.

Methods

Animals

Lister hooded ChAT positive rats (BioXpert, Schaijk, The Netherlands) n=19, 9 male/ 10 female were single housed. During the course of the experiment, food deprivation ensured a bodyweight between 85-90% of non-restricted weight.Animals were kept under a normal day/night-cycle (lights out at 7 p.m.). Experiments were approved by the Animal Ethics committee of the Netherlands Institute for Neuroscience (NIN 14.59.3c/d) and were conducted in agreement with Dutch laws (Wet op de dierproeven, 1996) and European regulations (Guideline 86/609/EEC).

Stereotaxic surgery

Lister hooded ChAT::Cre rats (Witten et al., 2011) were anaesthetized with isoflurane (1-2%) and lidocaine for injection with an AAV vector containing a hSyn-DIO-hm4D-mCherry or hSyn-DIO-hm3D-mCherry construct (Krashes et al., 2011). 50 nl virus per injection was administered bilaterally in the dorsolateral striatum at different depths (coordinates, Bregma: anterior-posterior -0.4; lateral +/-2.6; dorsal-ventral -4.7 (2x), -4.6, -4.5, -4.4, -4.3, -4.2(2x)). After surgery, rats received Metacam analgesia for 1 week and were allowed to recover for a total of 4 weeks before start of any in vivo experiments.

(8)

Behavioral training

The behavioral set-up consisted of six skinner boxes (Med Associates, St. Albans, USA), each encased in a sound-proof cabinet and equipped with a speaker playing white-noise throughout the experiment to mask any background noises. Every box was equipped with two levers alongside a magazine in one wall, dispensing single sucrose pellets in variable amounts. A ‘cue light’ was fitted above each lever, as well as a ‘house light’ in the opposite wall. Illumination of the house light would indicate the start of a trial; pressing the ‘active lever’ resulted in delivery of reward; this would always be the right lever. Animals were allowed to acclimatize in the testing room for half an hour before onset of a trial.

The training schedule used was adopted from Giuliano et al. (2012), first training rats (n=19) on increasing fixed interval schedules. The interval in this fixed interval (FI) schedule increased from 1 min (FI1) to FI2, FI4, FI8, FI10, FI15 over six sessions, followed by an additional three sessions of FI15. Reward delivery after the fixed interval was accompanied by the contingent illumination of the cue light above the active lever for 20 seconds. The amount of pellets per reward increased over the six sessions from 2 to 15 pellets per reinforced event. Upon reaching the final FI15 stage, training was continued for an additional three trials until performance was stable. After 4 sessions on FI15 a second order cue was introduced in the form of a 1 second cue light presentation every press (FI15(FR1:S)). The amount of lever presses to obtain the conditioned cue was then increased to ten, resulting in a FI15(FR10:S) second order schedule of reinforcement.

The result of the above mentioned procedure is a fixed interval training up to FI15(FR1:S), followed by extensive training under a FI15(FR10:S) schedule of reinforcement. The FI15(FR10:S) training was followed by twelve omission trials, in which the second order cue was omitted, resulting in a FI15 training. The omission trials were followed by three trials of FI15(FR10:S), in effect reintroducing the second order cue.

Immunochemistry

Immunohistochemistry was performed on 50μ slices (Bregma +2,28mm to -0,6mm), obtained from quick-frozen brains after perfusion and fixation in paraformaldehyde for 24h. Brains were stored in 30% sucrose solution at -20°C for at least 5 days before slicing. Slices were stained for mCherry and choline acetyltransferase (ChAT). The slices were incubated overnight at 4°C with primary antibodies Goat anti ChAT (1:500) and Rabbit anti mCherry (1:1000). Secondary antibodies Donkey anti Goat alexa 488 (1:1000) and Donkey anti Rabbit alexa 594 (1:1000) were used to stain ChAT positive neurons and enhance the mCherry reporter fluorescent signal, respectively. The presence of DREADDs was confirmed by staining for the mCherry tag. Co-staining for ChAT was used to confirm co-localization of DREADDs and cholinergic interneurons.

Results

During the course of this pilot experiment, rats were trained on the FI15(FR10:S) schedule for (almost) six consecutive weeks. CNO was administered during the FI15(FR1:S) trials as well as both

Table 1: Overview of different stages in training. Subsequent stages of training are depicted as blue arrows, details of training during different stages are given in blue boxes. See text for details.

(9)

early and late FI15(FR10:S) training. Injection of CNO took place half an hour prior to the start of the test trials, after which the animals were left in the testing room for their regular acclimatization period of 30 minutes. In the hM3Dq rats (n=9), 1 mg/kg CNO was injected 30 min before the fourth FI15(FR1:S), and before the seventh and twenty-fourth FI15(FR10:S) trial. A higher dose of 7,5 mg/kg was injected 30 min before the tenth FI15(FR10:S) trial. In the hM4Di rats (n=10), a dose of 1mg/kg was injected before the fifth FI15(FR1:S), and before the tenth and twenty-fourth FI15(FR10:S) session. Both groups were pre-fed twice during FI15(FR10:S) training, to examine the effect of devaluation of the reward.

Nine animals infected with hM3Dq, which, when activated, causes an excitation of cholinergic interneurons, show significant increase in lever pressing after injection with CNO during ‘early’ FI15(FR10:S) training (Figure 1). After 1 mg/kg CNO injection 30 minutes before the 7th _FI15(FR10:S)

training a significant increase in lever pressing on the active lever was shown (Wilcoxon signed rank test: Z = -2.666, p = 0.008 compared to day before). Another trial with 7,5 mg/kg CNO injection prior to the 10th_{FI15(FR10:S) trial also showed a significant increase in instrumental behavior}_(Wilcoxon

signed rank test: Z = -2.547, p = 0.011).

A second group of ten animals infected with hM4Di, for which injection with CNO results in the opposite, inhibitory effect, shows a significant increase in lever pressing during the FI15(FR1:S) and the ‘extended’ FI15(FR10:S) schedule (Figure 1). Injection of 1 mg/kg before the 24th_{FI15(FR10:S) trial}

resulted in a significant increase in well-established instrumental behavior (Wilcoxon signed rank test: Z = -2.090, p = 0.037). Injection of CNO in the same concentration before the 4th_{FI15(FR1:S) resulted}

in a similar increase in active lever presses ( Z = -2.499, p = 0.012 compared to day before).

The effect of administration of CNO on instrumental behaviour appears to be unequally distributed in time over the two groups. Injection of CNO leads to an increase in active lever presses In the hM3Dq, CIN activating group, during the second and third CNO trial (7th_{and 10}th_{FI15(FR10:S) trial). In}

Reintroduction second order cue CNO injection 4th FI15(FR1:S) CNO injection 5th FI15(FR1:S) CNO injection 10th FI15(FR10:S) 7,5 mg/kg CNO injection 7th FI15(FR10:S) CNO injection 10th FI15(FR10:S) CNO injection 25th FI15(FR10:S) CNO injection 24th FI15(FR10:S) Pre-feeding 1 17th FI15(FR10:S) Pre-feeding 1 19th FI15(FR10:S) Pre-feeding 2 28th FI15(FR10:S) Pre-feeding 2 28th FI15(FR10:S) Ommission training

Table 2 (timeline): Overview of different manipulations in both the hM3Dq and hM4Di group. Animals in both the hM3Dq (CIN activating) and hM4Di (CIN inhibiting) groups were injected with CNO before onset of trials on different time points in training. Animals in both groups received access to regular chow on two occasions before onset of a trial, to devalue the food reward. Only animals in the hM3Dq group received a single omission trial during the FI15(FR10:S) stage of training in which the second order cue was omitted. The blue arrow is to scale, descriptions of training stages correspond to table 1. The empty square denotes FI15(FR1:S) training as part of training under second order schedules of reinforcement.

Figure 1: Overview of the effect of CNO administration on different time points in training. Mean number of active lever presses (±

SEM) on the day before and after administration of CNO. Top row corresponds to the hM3Dq group, bottom row to the hM4Di group.

Animals in the hM3Dq group received CNO before the 4th

FI15(FR1:S) trial and before the 7th

, 10th

and 25th

FI15(FR10:S) trial. Animals in the hM4Di group received CNO before the 5th_{FI15(FR1:S) trial and before the 10}th_{and 24}th_{FI15(FR10:S) trial. * p < 0,05 compared to}

(10)

contrast, in the hM4Di, CIN inhibiting group, CNO injection during these time points did not result in a significant difference. Rather, administration of CNO before the fourth test trial (24th_{FI15(FR10:S))}

resulted in a significant increase in active lever presses, as did the same before the fifth FI15(FR1:S) session (Figure 1). Under no circumstance did the administration of CNO lead to a significant decrease in instrumental behaviour.

In both groups, the animals received pre-feeding twice, before the onset of FI15(FR10:S) trials. After access to regular chow for one hour before transportation to the testing room, the animals subsequently showed a significant decrease in instrumental behaviour. Devaluation of the reward by pre-feeding the animals before both the 17th_{and 28}th_{FI15(FR10:S) trial in the hM3Dq group resulted}

in a significant decrease in active lever presses (Wilcoxon signed rank test: Z = -2.666, p = 0.008 on both days)(Figure 2a). Devaluation of the reward before the 28th_{FI15(FR10:S) trail in the hM4Di group}

also resulted in a significant decrease in lever pressing ( Z = -2.701, p = 0.007, Wilcoxon signed rank test). The first pre-feeding session (19th_{FI15(FR10:S)) in the hM4Di group however did not result in a}

significant decrease in active presses (Figure 2a).

One animal (3438) showed an increase in active lever presses from 293 presses the day before, to 1058 presses during the first pre-fed trial. All other animals in the same group showed a clear decrease in instrumental behaviour; a Grubbs’ test identified the corresponding data point as an outlier. If this animal were to be excluded from the test, the effect of pre-feeding was significant (Wilcoxon signed rank test: Z = -2.668, p = 0.008)(Figure 2b).

Under no circumstances did manipulations lead to a significant increase in inactive lever presses. (See supplementary data) A decrease in inactive lever presses was observed during the initial ‘lever training’ phase of the experiment (not shown) indicating a successful acquisition of the instrumental

Figure 2: Effect of pre-feeding in both the hM3Dq and hM4Di group. (a) Mean number of active lever presses (± SEM) in the pre-fed

trials compared to baseline measurement one day before. Top row corresponds to hM3Dq group, bottom row to hM4Di group. Animals in the hM3Dq group received access to regular chow before the 17th_{and 28}th_{FI15(FR10:S) trial. Animals in the hM4Di group received}

access to regular chow before the 19th_{and 28}th_{pre-fed trial. Pre-feeding was significant in all trials except the first pre-fed session in the}

hM4Di group. ** p < 0,01 compared to day before. (b) Pre-feeding session 1 without subject 3481 (identified as an outlier) was

(11)

behaviour. Neither the administration of CNO, nor the pre-feeding of animals before onset of a trial resulted in increased activity on the inactive lever.

Only in the hM3Dq group, a single ‘omission training’ was conducted instead of the 15th_FI15(FR10:S)

trial. During this session, the second order cue was omitted, effectively resulting in a FI15 training. The omission of the second order cue during this single trial did not result in a significant difference in instrumental behaviour.

After 28 trials under the FI15(FR10:S)

SO-schedule, the second order cue was omitted in both groups,

for a total of 12 trials. For the last FI15(FR10:S) trial was proceeded by access to regular chow, all subsequent comparisons are made to the 27th_FI15(FR10:S)

training. No significant difference in instrumental behaviour was observed between the last regular (27th_{) FI15(FR10:S) trial and the twelfth FI15 trial in either group (Figure 3a). In the hM3Dq group, a}

significant decrease in active lever pressing was observed after omission of the second order cue in the first FI15 trial (Wilcoxon signed rank test: Z = -2.429, p = 0.015)(Figure 3a). Instrumental behaviour after the initial FI15 trial was not significantly different (Figure 3b). The omission trials were followed by another three FI15(FR10:S) sessions, in which instrumental behaviour after reintroduction of the second order cue was compared to the last FI15 trial. In the hM3Dq group, a significant increase could be observed after reintroduction of the cue, but only in the second and third FI15(FR10:S) trial after reintroduction (Wilcoxon signed rank test: Z = -2,666, p = 0.008 for both trials) (Figure 3a). In the

Figure 3: Omission of the second order cue in a single trial in the hM3Dq group. Mean number of active lever presses (± SEM) on the day before and the omission training session.

(12)

hM4Di group, neither a decrease in instrumental behaviour after omission of the second order cue, nor an increase in active lever presses after reintroduction of the cue could be discerned.

There was no significant increase in inactive lever presses during omission or reintroduction of the second order cue. (see supplementary material)

Discussion

Striatal cholinergic interneurons are well-suited candidates to have an important role in the integrative function of the striatum. Cholinergic interneurons receive a wide variety of inputs from various brain regions, including glutamatergic cortico-striatal inputs, dopaminergic nigro-striatal inputs as well as glutamatergic inputs from the thalamus (Lim et al., 2014). The integration of these different inputs results in the selection of a single action (Balleine et al., 2009).

The action-selection function of the striatum is thought to involve distinct functional circuits(Haber et al., 2000; Yin et al., 2006). The DMS is involved in an associative striato-thalamo-cortical loop, whereas the DLS is part of a loop associated with sensorimotor control. The dorsomedial- and dorsolateral striatum compete for control over behavior (Balleine et al., 2009). Modulation of these distinct functional circuits by the dopaminergic (Yin et al., 2009; Willuhn et al., 2012) and acetylcholinergic(Lim et al., 2014; Morris et al., 2004) systems influences decision making, as has been shown in the formation of habits (Willuhn et al., 2012).

The results of the current investigation show an increase in instrumental behavior after modulation of CIN’s in the dorsomedial striatum on different time points in training on a SO-schedule. Presses on the active lever after administration of CNO increased in both groups; both inhibition and activation of CINs in the dorsomedial striatum leads to an increase in instrumental behavior. Administration of CNO did however not have this effect on the same time points in both groups. Inhibition of CINs only led to an increase after extended training, whereas excitation of CINs only led to an increase during

early training.

The current results are in accordance with a study conducted by Murray and colleagues (Murray et al., 2012), who, using the same SO-schedule, administered a dopamine transmission blocker in the dorsomedial- as well as the dorsolateral striatum at different time points. Infusion of α-flupentixol in the DMS during early stages of training on a FI15(FR10:S) schedule reduced lever pressing for cocaine, whereas infusion of α-flupentixol in the DLS causes a reduction in instrumental behavior only during late stages of training. These results are consistent with the notion that dopaminergic control over behavior shifts towards the DLS after extended training under a FI15(FR10:S) schedule(Murray et al., 2012). Modulation of the CINs in the DMS could facilitate a shift between the competing dorsomedial and dorsolateral loops in the striatum.

Inhibition of CINs in the hM4Di group could lead to a stronger dorsolateral control over behavior. During later stages of training, performance on SO-schedules of reinforcement becomes more habitual, shifting control over behaviour away from the dorsomedial striatum (Di Ciano 2005). Inhibition of CINs in the DMS in the hM4Di group during the late stages of training might have impaired local dorsomedial circuitry in favour of the competing DLS loop.

Activation of cholinergic interneurons in the DMS in the hM3Dq group on the other hand led to increased instrumental behaviour only during ‘early’ FI15(FR10:S) training. Excitation of CINs in the DMS during the goal-directed early phase of training might have facilitated DMS local circuitry to gain

Figure 4: Effect of omission and reintroduction of the second order cue in both groups. ( a) Mean number of active lever presses (±

SEM) on the 27th_{FI15(FR10:S) trial compared to the first and last FI15 trial after omission of the second order cue, or on the last FI15}

trial compared to the first and last FI15(FR10:S) trials after reintroduction of the second order cue. Top row corresponds to the hM3Dq group, bottom row to hM4Di group. (b) Detail of omission of the second order cue in the hM3Dq group. A significant effect of omission of the second order cue is confined to the first FI15 trial after omission of the cue. * p < 0,05 compared to 27th_{FI15(FR10:S). ** p < 0,01}

(13)

more control over behaviour. This account would explain both the increase in instrumental behaviour in both groups and the temporal differences between both groups.

Inhibiting cholinergic interneurons during the 5th_{FI15(FR1:S) trial also led to a significant increase in}

active lever presses. As the inhibition of the DMS supposedly led to an increase in control over behaviour of the dorsolateral striatum, this would imply that the performance under this SO-schedule was already under dorsolateral control after 5 trials. On one hand, fixed interval schedules are known to induce habitual behaviour (Everitt et al., 2000). Moreover, the dorsolateral striatum is known to be involved in a sensorimotor loop associated with stimulus-response behaviour (Yin et al., 2006). Training of the animals under a fixed interval schedule until a stable baseline was reached and subsequent introduction of a cue that indicated the upcoming reward might have evoked this stimulus-response type of behaviour. It is not improbable that the combination of fixed interval training and the introduction of a high number of second order cues could have led to behaviour under dorsolateral striatal control already after 5 trials.

The above described account does however not account for the specificity of our method. Using DREADDs we were able to modulate specifically the cholinergic interneurons in the DMS. Inhibition of the inhibiting CINs in the DMS would have been expected to relief inhibitory cholinergic tone on principal MSN and aid activation of MSNs in this brain area. Accordingly, excitation of CINs would have been expected to result in more inhibition of the DMS. These expected effects of specific modulation of CINs have been observed in previous research by Witten et al. (2010). Using optogenetics, excitation of CIN’s in the nucleus accumbens did lead to a general decrease in activity of neighbouring MSNs (Witten et al., 2010). Correspondingly, inhibition of CINs led to more activity of striatal MSNs. These manipulations however took place on a far smaller timescale and in a more ventral area of the striatum. The exact consequences of DREADD activation on CINs in the local circuitry of the DMS remain unknown. All in all, these contradictory results illustrate that the role of CINs in the complex interplay between the dopaminergic and cholinergic systems in the striatum is not yet fully understood.

In the future, it would be interesting to examine the effects of DREADD activation in other brain areas, especially the DLS. Furthermore, it would be interesting to assess instrumental behaviour under different schedules of reinforcement. Especially performance under a seeking-taking chained schedule, in which pressing a seeking lever results in the possibility to obtain a reward by pressing a taking lever, would allow a distinction to be made between reward seeking and reward taking behavior. Moreover, the degree of habituation to the seeking-taking schedule could be tested by devaluation of the taking lever.

The current investigation was a pilot study. The experimental design was under constant revision. Several components of the design could be improved in future. For instance, animals in the hM3Dq group were trained under various fixed ratio schedules of reinforcement before onset of the above described increasing fixed interval training. Differences between both groups at the end of the experiment should most likely be attributed to these pre-training trials in the hM3Dq group.

Secondly, the animals were, before acquisition of the SO-schedule of reinforcement, trained under a fixed interval schedule. In subsequent studies, training up under a fixed ratio schedule might lead to more experienced contingency between the cue and the reward (Everitt et al., 2000). Increased contingency between cue and reward could lead to better performance under the cue-driven SO-schedule and more effect of omission of the second order cue.

Furthermore, it is at this point unknown whether there is sufficient reason to exclude animal 3438 from the data of the first pre-feeding trial. This animal was clearly identified as an outlier, with

(14)

performance after pre-feeding in the other group, or of other animals in the same group, or of this animal during the second pre-fed trial, always showing a clear, significant decrease. It is however at this point unknown what caused the markedly increased instrumental behavior of this animal during the first pre-feeding session. At last, one more FI15(FR10:S) session after the last (28th_{) pre-feeding}

trial would allow for a better statistical analysis of the effect of omission of the second-order cue. Moreover, this was one of very few studies using a SO-schedule of reinforcement in combination with a food reinforcer. Using a SO- schedule of reinforcement, the instrumental behavior was expected to become cue-elicited. From studies using a drug reinforcer, SO- schedules are known to induce instrumental behavior that is cue driven(Everitt et al., 2000; Everitt et al., 2013). The influence of the cue was tested by devaluing the reward and omitting the second order cue. Unfortunately, the omission of the cue after extended FI15(FR10:S) training did not result in a decrease in instrumental behavior in the hM4Di group. Therefore, the behavior of these animals cannot be said to be cue-driven.

Devaluation of the reward by pre-feeding on the other hand did result in a solid, significant decrease in lever pressing in both groups. Devaluing the reward by pre-feeding the animals before the onset of the trial decreased instrumental behavior. Therefore, the lever pressing can be said to have been goal-directed.

In contrast, omission of the cue in the hM3Dq group did lead to a significant decrease in instrumental behavior. This decrease in active lever presses was however confined to the first FI15 trial after the SO-schedule. After this initial trial, performance in the presence or absence of the cue was not significantly different. As the effect was only observed in one group and the test trial was preceded by a pre-fed trial, it cannot be excluded this effect has to be attributed to other influences (e.g. as a consequence of the preceding pre-feeding trial).

Subsequent reintroduction of the second order cue in the hM3Dq group also resulted in a significant increase in instrumental behavior, but only after the first reintroduction trial. This effect was again observed only in the hM3Dq group. The differences of omission and reintroduction of the second order cue between both groups of animals are likely a result of differences in training before onset of the above described experiment.

An absence of effect of omission of the second order cue could indicate the instrumental behavior had become habitual (after extended training). However, the design of the second order training does not allow for a reversal test, where refraining to perform the instrumental behavior leads to reward. An increase in lever presses under an SO-schedule after extended training seems to reflect a DLS controlled, stimulus response type of instrumental behavior. However, an increase in instrumental behavior after extended training, reflecting a dorsolaterally controlled stimulus-response (re)action, to a second order cue indicating an upcoming food reward, might not be detectable using current behavioral essays, as food may not have the same reinforcing properties as drugs of abuse. As food reinforces may not induce a decrease in dorsolateral striatal dopamine release (Willuhn et al. 2014), the DMS might always regain control over even habitually performed behaviors (Balleine 2009). Familiar tests of habitual behavior, outcome devaluation and contingency degradation, might not qualify stimulus-response behavior for a food reinforcer as a habit. Thus, performance under an SO-schedule for a food reinforcer might not become habitual.

The increase in lever presses was always confined to the active lever. The amount of inactive lever presses did not show any significant increase after administration of CNO. In general, dopamine release in the striatum induces an increase in motor activity(Willuhn et al., 2014). In our subjects, injection of CNO might have altered dopaminergic transmission, resulting in an overall increase in activity. However, as there is no significant increase in inactive lever presses observed after CNO

(15)

injections, the effect of CNO can be said not to rely on an overall increase in locomotor activity. Instead, the increase in behavior is a specific effect of modulation of the dorsomedial striatum. An alternative explanation for the absence of a decrease in active lever presses after omission of the second order cue is provided by Wilson, Bowman (2004), who observed a pause in lever pressing just after delivery of a cue. Every cue delivery was followed by a short pause in instrumental behavior. The pause in lever pressing caused a reduction of the response rate, actually resulting in increased number of presses when the cues were omitted. It is at this point unknown whether lever pressing by our animals also pauses shortly after cue delivery.

A further account for the seemingly counterintuitive observation that activation of inhibiting CINs in the DMS during early training leads to increased instrumental behavior under DMS control, could be given by a CNO induced increase in behavioral rigidity. As the rebound in activity of CINS is associated with the maintenance of a selected action (Schulz et al., 2013), increased activity of CINs after CNO administration might have resulted in an increased rigidity in maintaining the selected instrumental behaviour. Further research will be needed to investigate whether the increased performance during early, goal-directed behavior under DMS control could be the result of CIN induced behavioral rigidity. A possible mechanism for the interplay between the dopaminergic and cholinergic systems is postulated to involve cholinergic innervation of dopaminergic axon terminals on MSNs. The inhibiting effect of CIN activation in the dorsolateral striatum has been proposed to involve GABA release from dopaminergic terminals. A subset of dopaminergic neurons produce and co-release GABA (Tritsch et al., 2012). Cholinergic interneurons can drive release of GABA from dopaminergic terminals in the DLS through activation of nicotinic acetylcholine receptors (Nelson et al., 2014). Optogenetic activation of CINs in the DLS microcircuitry has been shown to result in co-release of GABA from nigrostriatal dopamine neurons resulting in fast inhibitory post-synaptic potentials in MSNs. This mechanism has however only been described for nigrostriatal DLS projecting dopamine neurons.

Finally, in the further interpretation of these findings it should be taken into account that the Chat::Cre rat line used here has been reported to overexpress vesicular ChAT packaging gene (vChat) (Crittenden et al., 2014). The insertion of a BAC construct with an inactivated ChAT-gene, but unaltered vChAT gene into the ChAT locus leads to overexpression of vChAT. Homozygous transgenic rats that have been selected for high expression levels of the BAC construct are likely to have extra exogenous vChAT genes and increased stimulated acetylcholine release. This has been reported to be the case in a ChAT-ChR2-EYFP mouse line as well as a B6.eGFP-ChAT BAC mouse line, and resulted in functional and behavioral differences compared to wild-type, including more behavioral stereotypy (Crittenden et al., 2014).

(16)

Conclusion

In the current investigation, activation of CINs in the DMS leads to increased instrumental behavior only during early performance under an SO-schedule, when behavior is thought to be under DMS control. Inhibition of CINs in the DMS leads to increased instrumental behavior only after extended training, when performance on the SO schedule is thought to be under DLS control. This is possibly a result of a cholinergic influence on a shift in control over behavior from the DMS to the DLS, as has previously been reported for dopaminergic modulation of these striatal areas. The mechanism by which activation or inhibition of CINs leads to an increase in instrumental behavior remains to be elucidated.

For future studies, the effects of modulation of CINs in other striatal areas, especially the DLS, could be examined. If the increase in instrumental behavior during different time points is the result of a shift in control over behavior, opposing effects of activating or inhibiting CINs in the more lateral dorsal striatum compared to the more ventral dorsal striatum would be expected. As dopamine neurons in the DLS have been reported to co-release GABA (Tritsch et al., 2014), and cholinergic interneurons can drive GABA-release (Nelson et al., 2014), it will be interesting to measure dopamine release or MSN activity during modulation of striatal CINs, to examine the effect of modulation of CINs on local circuitry in vivo. Behavioral analysis using a seeking-taking chained schedule would allow tests to distinguish goal-directed (DMS) instrumental behavior from more habitual (DLS) instrumental behavior.

Finally, vChAT overexpression has been reported in mouse lines generated using the same BAC-construct. It will be important to exclude the possibility that increased instrumental behavior is the result of unwanted behavioral stereotypy due to increased stimulated acetylcholine release (e.g. increased stereotypy caused by stress due to administration of CNO). vChAT overexpression is likely to alter the balance in neurotransmitter release in the lateral striatum, which should be taken into consideration before further conclusions from these results can be drawn.

(17)

Acknowledgement

I want to thank Ingo for his patience and for being an example of an excellent neuroscientist.

References:

Balleine, B. W., Liljeholm, M., & Ostlund, S. B. (2009). The integrative function of the basal ganglia in instrumental conditioning. Behavioural brain research,199(1), 43-52.

Calabresi, P., Picconi, B., Tozzi, A., Ghiglieri, V., & Di Filippo, M. (2014). Direct and indirect pathways of basal ganglia: a critical reappraisal. Nature neuroscience, 17(8), 1022-1030.

Crittenden, J. R., Lacey, C. J., Lee, T., Bowden, H. A., & Graybiel, A. M. (2014). Severe drug-induced repetitive behaviors and striatal overexpression of VAChT in ChAT-ChR2-EYFP BAC transgenic mice. Frontiers in neural circuits,8.

Di Ciano, P., & Everitt, B. J. (2005). Neuropsychopharmacology of drug seeking: Insights from studies with second-order schedules of drug reinforcement. European journal of pharmacology, 526(1), 186-198.

Everitt, B. J. & Robbins, T. W. (2000). Second-order schedules of drug reinforcement in rats and monkeys: measurement of reinforcing efficacy and drug-seeking

behaviour. Psychopharmacology, 153(1), 17-30.

Everitt, B. J., & Robbins, T. W. (2013). From the ventral to the dorsal striatum: devolving views of their roles in drug addiction. Neuroscience & Biobehavioral Reviews, 37(9), 1946-1954.

Giuliano, C., Robbins, T. W., Nathan, P. J., Bullmore, E. T., & Everitt, B. J. (2012). Inhibition of opioid transmission at the μ-opioid receptor prevents both food seeking and binge-like

eating. Neuropsychopharmacology, 37(12), 2643-2652.

Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. The Journal of neuroscience, 20(6), 2369-2382.

Krashes, M. J., Koda, S., Ye, C., Rogan, S. C., Adams, A. C., Cusher, D. S. & Lowell, B. B. (2011). Rapid, reversible activation of AgRP neurons drives feeding behavior in mice. The Journal of clinical

investigation, 121(4), 1424.

Lim, S. A. O., Kang, U. J., & McGehee, D. S. (2014). Striatal cholinergic interneuron regulation and circuit effects. Frontiers in synaptic neuroscience, 6.

Matsumoto, N., Minamimoto, T., Graybiel, A. M., & Kimura, M. (2001). Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. Journal

of Neurophysiology,85(2), 960-976.

Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004). Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron, 43(1), 133-143.

Murray, J. E., Belin, D., & Everitt, B. J. (2012). Double dissociation of the dorsomedial and dorsolateral striatal control over the acquisition and performance of cocaine seeking. Neuropsychopharmacology,

37(11), 2456-2466.

Nelson, A. B., Hammack, N., Yang, C. F., Shah, N. M., Seal, R. P., & Kreitzer, A. C. (2014). Striatal cholinergic interneurons drive GABA release from dopamine terminals. Neuron, 82(1), 63-70.

(18)

Schulz, J. M., & Reynolds, J. N. (2013). Pause and rebound: sensory control of cholinergic signaling in the striatum. Trends in neurosciences, 36(1), 41-50.

Steinberg, E. E., Keiflin, R., Boivin, J. R., Witten, I. B., Deisseroth, K., & Janak, P. H. (2013). A causal link between prediction errors, dopamine neurons and learning. Nature neuroscience, 16(7), 966-973. Tritsch, N. X., & Sabatini, B. L. (2012). Dopaminergic modulation of synaptic transmission in cortex and striatum. Neuron, 76(1), 33-50.

Vanderschuren, L. J., Di Ciano, P., & Everitt, B. J. (2005). Involvement of the dorsal striatum in cue-controlled cocaine seeking. The Journal of neuroscience,25(38), 8665-8670.

Wang, Z., Kai, L., Day, M., Ronesi, J., Yin, H. H., Ding, J. & Surmeier, D. J. (2006). Dopaminergic control of corticostriatal long-term synaptic depression in medium spiny neurons is mediated by cholinergic interneurons. Neuron,50(3), 443-452.

Wess, J., Nakajima, K., & Jain, S. (2013). Novel designer receptors to probe GPCR signaling and physiology. Trends in pharmacological sciences, 34(7), 385-392.

Willuhn, I., Burgeno, L. M., Everitt, B. J., & Phillips, P. E. (2012). Hierarchical recruitment of phasic dopamine signaling in the striatum during the progression of cocaine use. Proceedings of the National

Academy of Sciences, 109(50), 20703-20708.

Willuhn, I., Burgeno, L. M., Groblewski, P. A., & Phillips, P. E. (2014). Excessive cocaine use results from decreased phasic dopamine signaling in the striatum. Nature neuroscience, 17(5), 704-709. Wilson, D. I., & Bowman, E. M. (2004). Second-order stimuli do not always increase overall response rates in second-order schedules of reinforcement in the rat. Psychopharmacology, 174(3), 430-437. Witten, I. B., Lin, S. C., Brodsky, M., Prakash, R., Diester, I., Anikeeva, P. & Deisseroth, K. (2010). Cholinergic interneurons control local circuit activity and cocaine conditioning. Science, 330(6011), 1677-1681.

Witten, I. B., Steinberg, E. E., Lee, S. Y., Davidson, T. J., Zalocusky, K. A., Brodsky, M., ... &

Deisseroth, K. (2011). Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron, 72(5), 721-733.

Yin, H. H., & Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nature Reviews

(19)

Supplementary material:

Inactive lever presses after the ‘lever training’ stage (see table 1) were always significantly lower than active presses. Different manipulations (see Table 2) did not affect inactive lever presses.

Figure S1: Effect of different manipulations on inactive lever presses. Mean number of inactive lever presses (± SEM) compared to active presses during the same trial. Top three rows depict manipulations in the hM3Dq group. Bottom three rows depict manipulations in the hM4Di group. See table 2 for a timeline of different manipulations.

(20)

Ommision or reintroduction of the second order cue after extended training on an FI15(FR10:S) schedule did not affect inactive lever presses.

Figure S2: Effect of omission and reintroduction of the second order cue on inactive lever presses in both groups. Mean number of inactive lever presses (± SEM) on the 27th_{FI15(FR10:S) trial compared to the first and last FI15 trial after omission of the second order}