The Neural Mechanisms underlying Iconic Memory
Behavioural evidence revealing that there is more to iconic
memory than retinal afterimages
Marleen van de Beek
Student number: 10179283
Supervisor: Dhr. dr. I.G. Sligte
Second assessor: Dhr. dr. Y. Pinto
University of Amsterdam
Department of Psychology
July 22, 2016
Abstract
Iconic memory is known for its high capacity, yet it only lasts for 0.5 seconds. As earlier studies have provided evidence that iconic memory is partially related to retinal afterimages, we aimed to quantify how much of iconic memory is due to retinal aftereffects and how much is due to visual cortex mechanisms. This study used two adaptations of a change detection task in which binocular information is required. Since binocular information fuses in the early visual cortex, this enabled us to control for afterimages. Firstly, to rule out the contribution of retinal afterimages, 3D stimuli were used that could only be perceived binocularly. We found that the high capacity of IM is still observed when using these stimuli. Secondly, in a binocular masking paradigm, we found that interfering displays only interfered with iconic memory when they were presented at a location congruent to the probed location. It did not matter whether the interfering display was presented to the same or the other eye compared to the memory display. These experiments provided converging evidence for the idea that IM uses a binocular and cortical representation of stimuli, instead of just a retinal percept. Therefore, we conclude that the mechanisms underlying iconic memory can be traced down to the early visual cortex.
Keywords. Iconic Memory, Visual Short-Term Memory, Retinal Afterimages, Stereoscopic
Contents
1 Introduction 3 2 Experiment 1 4 2.1 Method . . . 4 2.2 Results . . . 6 2.3 Discussion . . . 8 3 Experiment 2 8 3.1 Method . . . 8 3.2 Results . . . 9 3.3 Discussion . . . 10 4 Experiment 3 10 4.1 Method . . . 11 4.2 Results . . . 11 5 General discussion 13 References 15List of Figures
1 Presentation of memory display through stereoscope . . . . 5
2 Design Experiment 1 A. Sequence of displays B. High contrast black-white
stimulus producing strong afterimage C. Isoluminant red-grey stimulus produc-ing weak afterimage D. 3D stimulus producproduc-ing no afterimage E. Iconic-cue condi-tion measuring iconic memory F. Retro-cue condicondi-tion measuring fragile memory
G. No-cue condition measuring working memory. . . . 7
3 Results Experiment 1 & 2 A. The results on training task show the typical
IM-FM-WM differentiation B. Results on Experiment 1. 3D stimulus class shows lower capacity compared to high-contrast (HC) and isoluminant (ISO) stimulus classes. No difference is apparent between IM and FM over all stimulus classes.
C. Results on Experiment 2. After some adjustments to the experimental task,
the difference between IM and FM is apparent again. Capacity in 3D stimulus class is lower compared to other stimulus classes. . . 9
4 Design Experiment 3 A. Memory display is presented to one eye B.
Interfer-ing displays. 1: same eye, congruent location, 2. same eye, incongruent location, 3. different eye, congruent location, 4. different eye, incongruent location, 5. no mask presented C. Working memory conditions, cue presented with test display
D. Iconic memory and fragile memory conditions, cue presented before test
dis-play E. Iconic-cue condition measuring iconic memory F. Retro-cue condition
measuring fragile memory G. No-cue condition measuring working memory. . . 12
5 Results Experiment 3 Memory capacity is lower when IDs are presented to
the location congruent to the probed location (solid lines). No difference was found for the eye to which the ID was presented (red vs black lines). . . 13
1 Introduction
Imagine yourself enjoying a beautiful view from the top of a mountain. You seem to be completely aware of your surroundings. How-ever, when you close your eyes, your represen-tation of the world rapidly fades away. How many trees were next to you? What colors are the houses on the horizon? Suddenly it be-comes hard to report of what was completely clear to you just a second ago. This phe-nomenon is an example of our rich but brief representation of our surroundings in our vi-sual short-term memory (VSTM).
Recent studies have distinguished between three kinds of VSTM: iconic memory (IM), fragile memory (FM), and working memory (WM) (Pinto, Sligte, Shapiro, & Lamme, 2013; Sligte, Scholte, & Lamme, 2008; Sligte, Van-denbroucke, Scholte, Lamme, et al., 2010). IM is the initial form of VSTM, having high ca-pacity (up to 30 items) but a brief (<.5s) life-time (Sligte et al., 2010; Sperling, 1960). FM has a longer duration (<4s) and can contain up to 15 items (Pinto et al., 2013). Finally, WM contains a maximum of 3-4 items over a longer period of time (Luck & Vogel, 2013; Sligte et al., 2010). At a glance, the capacity of memory-storage becomes more limited as the lifetime of storage becomes longer.
While the neural mechanisms of WM have been thoroughly explored, the neural under-pinnings of both IM and FM remain specu-lative at best. WM depends on attentional mechanisms that are distributed in a wide neural network (Courtney, Ungerleider, Keil, Haxby, et al., 1997; Postle, 2015). Typically, during WM retention sustained activity in sev-eral higher-order areas like posterior parietal cortex, prefrontal cortex and the frontal eye fields is observed (Haxby, Petit, Ungerleider, & Courtney, 2000; Postle, 2015; Todd & Marois, 2004). In the case of FM maintenance, evi-dence suggests the necessity for activity within V4 (Sligte, Scholte, & Lamme, 2009). In con-trast to FM and WM, there is no consen-sus about the neural mechanisms underlying IM. Nonetheless, we do have some starting points for formulating hypotheses on the
neu-ral source of IM. Typically, only features in-stead of integrated objects are maintained in IM (Sligte et al., 2010). This would suggest that higher-level brain regions beyond V1-V3 are not involved in IM, as these regions are linked to processing of integrated objects and not merely features (Landman, Spekreijse, & Lamme, 2003). As such, our hypothesis is that IM finds its source in the early visual areas.
Congruent with this hypothesis, a recent study provided evidence towards a link be-tween IM and early visual areas (V1-V3) (Sligte, Scholte, van Loon, Vandenbroucke, Koopman, Shapiro & Lamme, forthcoming). Firstly, individual differences in IM capacity seemed to be related to grey matter density in the primary and secondary visual cortex. Sec-ondly, individuals with more excitatory neu-rotransmitters (glutamate) in the visual cor-tex showed higher IM capacity. Lastly, IM ca-pacity could be modulated by electrical stimu-lation applied to visual cortex (Sligte et al., forthcoming). Together, this demonstrates that the neural source of IM can be traced to the early visual cortex.
While this study provided preliminary evi-dence in favour of the cortical brain hypothe-sis, we still need to rule out an alternative ex-planation - namely, the role of retinal afterim-ages in IM. Earlier studies have argued that IM representations depend on positive afterimages of a previously shown image that consists of prolonged activation on the retina after stim-ulus presentation (Coltheart, 1980; Sligte et al., 2008). As such, IM is at least partially re-lated to a retinal afterimage that persists after stimulus disappearance. However, it is not yet clear whether the high capacity of IM can still be observed when we completely remove the possibility that retinal afterimages contribute to IM capacity. If IM’s high capacity is still observed when controlling for retinal afterim-ages, this is evidence for the cortical brain hy-pothesis of IM.
This study ruled out the possibility that reti-nal afterimages influence IM capacity by using stimuli that require binocular depth percep-tion. Depth perception is processed at its ear-liest in layer 2 of the primary visual cortex
(V1) (Qian, 1997). Within this layer, informa-tion from both eyes comes together in binocu-lar cells (Qian, 1997). In our first two exper-iments, we used three dimensional (3D) stim-uli that could only be seen binocularly. Since binocular information was necessary, a percept of these stimuli could not be created earlier in the visual hierarchy than in V1. As the per-cept could only be present at a cortical level, the influence of a retinal afterimage of stimuli was ruled out. When the high capacity of IM is present within these 3D stimuli, this supports the idea that IM is created by the visual cortex instead of being dependent on a retinal after-image. A total of three stimulus classes were used: 3D, high-contrast and isoluminant stim-uli. When high-contrast stimuli are perceived, a percept is created on the retina that persists for quite some time after stimulus disappear-ance because of the strong retinal afterimage it produces. In contrast, isoluminant stimuli also generate a retinal percept, but these stimuli do not produce (long-lasting) afterimages (Sligte et al., 2008). By comparing these three stimu-lus classes, it is possible to quantify the influ-ence of retinal afterimages on IM capacity. We found that the high capacity of IM remained when no afterimage was present, yet only when a non-masking cue was used. This provides ev-idence in favour of the cortical brain hypothe-sis.
The third experiment used binocular mask-ing to provide additional evidence for the cor-tical basis of IM. In this experiment, we inves-tigated whether IM used a cortical or a retinal representation of stimuli by checking for binoc-ular cross-eye interference. When IM uses a cortical representation, information from the opposite eye should influence iconic memory, since at this level information from both eyes is already fused. If IM uses a retinal represen-tation, however, information from the opposite eye should not interfere with IM. We tested this with a change detection task, that was only presented to one of the two eyes, while an interfering display (ID) was presented to either the same or to the other eye. Furthermore, the location of the ID could be congruent or incon-gruent to the probed location. We found that
the capacity of IM deteriorated when the ID was presented at the probed location, but not when the ID was presented at a different lo-cation. Moreover, we found that the eye to which the IDs were presented did not make a difference in memory capacity; presentation to both eyes equally lowered the capacity. This is evidence for the cortical brain hypothesis, as IM made use of the binocularly (and thus cor-tically) merged image instead of the separate images from the two eyes.
To summarize, this study was aimed at set-tling the debate on the neurobiological under-pinnings of IM. We aimed to quantify how much of IM is due to retinal aftereffects and how much of it is due to visual cortex mecha-nisms, by using stimuli that require binocular information.
2 Experiment 1
2.1 Method
Subjects In total 29 students from the
Uni-versity of Amsterdam participated in this ex-periment (7 male, age range 19-28 years, mean age = 22.6 SD = 2.7). All subjects gave their written informed consent to participate in the study, which was approved by the local ethics committee of the University of Amsterdam. Subjects received course credits for participa-tion. All subjects had normal or corrected-to-normal vision and no colour deficiencies.
Task & stimuli During the first
experi-ment, two versions of the change detection task (CDT) were performed; a basic training CDT and an experimental version of this task.
Training CDT The training task was
presented with Presentation (Neurobehavioral Systems) on a 23 Inch LCD display, at a refresh rate of 120Hz. Subjects were seated 57 cm from the display (viewing angle 28.7¶x 50,9¶).
The three levels of VSTM were measured with a partial report CDT (Luck & Vogel, 1997). In this task, a memory display is presented for 500ms. The memory and test displays con-tained 8 white rectangles (2.7¶x 0.6¶) on a
black background placed on a imaginary circle (6¶eccentricity), having horizontal, vertical or
oblique (45¶or 135¶to horizontal) orientations.
The cue consisted of a 3px line, pointing to the relevant location. After the cue display, the test display appeared. In 50 % of the tri-als, the cued rectangle in the test display had a 90¶change in orientation. Subjects had to
indi-cate whether the cued rectangle had the same orientation (no change) or a different orienta-tion (change). The test display was presented until the subject gave a response, with a max-imum of 4 seconds response time.
The CDT uses the partial report technique to measure IM, FM and WM capacity. In the IM and FM conditions, a cue is presented 30 ms (IM) or 1000 ms (FM) after the memory display had disappeared. This cue pointed to the location of the relevant rectangle. It is believed that when the cue is presented shortly (30ms) after the memory display was presented, information about the memory ar-ray is still available in IM (Sperling, 1960). When the cue was presented 1000 ms after the memory array, information should be rep-resented in FM (Sligte et al., 2008). When the cue was presented at the same time as the test display, WM was measured. Subjects received immediate feedback on the correctness of their response through either a high pitch sound (right answer) or a low pitch sound (wrong an-swer).
Experimental CDT The task was
pre-sented with Presentation (Neurobehavioral Systems) on a 22 inch CRT monitor (1600 x 1200, 75 hz). Subjects viewed the task through a four-mirror stereoscope at viewing distance of 40 cm. To make the task visible through the stereoscope, displays were presented on each side of the visual field. For mirror adjustment and binocular alignment, a textured frame sur-rounded the displays (Figure 1). The binocu-lar depth was generated by an 8 pixels dis-placement of the stimuli at the right side of the visual field.
The experimental version of the CDT is an adaptation of the training CDT. In this version of the CDT, the cue consisted of four triangles,
A.
Figure 1: Presentation of memory display
through stereoscope
positioned in each corner of the placeholder of the rectangle (four-triangle cue, figure 2A). Furthermore, in the test-display, only one of the eight rectangles was presented. This way, no cue had to be presented in the WM condi-tions, since it was already clear what the rele-vant rectangle was.
Most importantly, this study used three dif-ferent stimulus classes to control for percept creation. The first stimulus class was high-contrast (HC) stimuli. In HC conditions, white rectangles were presented on a black back-ground (Figure 2B). This way, both rod and cone systems are activated. Rod receptors con-tinue to respond after stimulus off-set and in-tegrate information over longer periods after stimulus presentation, causing stronger after-images than cone receptors (Banks & Barber, 1977). Therefore, high-contrast stimuli cause both a retinal percept, a retinal afterimage as well as a cortical representation.
In isoluminant conditions, the rectangles had the same subjective luminance as the background which causes only the cone sys-tem to be activated (Adelson, 1978). Since the rod receptors were not activated by these stim-uli, only weaker afterimages were generated. Thus, isoluminant stimuli cause a percept on the retina, a weak afterimage and also a cor-tical representation is present. In this study, we used red rectangles on a grey background (Figure 2C). The luminance of the red and grey colours were subjectively adjusted with a flicker-test. In this flicker-test, two red and two grey boxes alternate in position, reversing colour polarity at 10 Hz. The luminance of the red boxes was fixed and the subjects could
adjust the luminance of the grey box to mini-mize perceived flickering (Accornero, Gregori, Pro, Scappini, & La Riccia, 2008; Knapen, Kanai, Brascamp, van Boxtel, & van Ee, 2007; Serences & Yantis, 2006; Shady, MacLeod, & Fisher, 2004). The subjectively perceived iso-luminance was used to adjust the grey scale of the background in the isoluminant stimuli.
The last stimulus class is three-dimensional (3D). It is known that the perception of depth is obtained in the early visual cortex. This study used stimuli that are only visible binoc-ularly, to be sure that cortical activity is re-quired and retinal afterimages do not influ-ence capacity. Therefore, this stimulus class only generates a cortical representation and no stimulus afterimage or percept on the retina. The stimuli consisted of a background that is made up of 20000 lines (width 1px), with random length, random orientation and ran-dom grey-scaled colour, placed on top of each other (Figure 2D). The stimuli are rectan-gles (51px by 13px) made up from those same lines, placed on top of the background (Fig-ure 2D). The rectangles are not visible with-out the stereoscope, because the lines fade with-out into the background. To make the stimuli vis-ible through the stereoscope, binocular dispar-ity was created. This was done by projecting the displays twice, at each side of the visual field, with the left projection laterally shifted 8 pixels towards the middle. This way, the stimuli looked like as if they ’pop out’ from the background.
Together these three stimulus classes and the three levels of VSTM made up nine different conditions (memory (IM,FM,WM) x stimulus-type (high-contrast, isoluminant, 3D)). Subjects were tested on all conditions, which were randomly intermixed during test-ing.
Procedure Subjects completed the
experi-ment in three sessions. In the first session, sub-jects were screened on three vision tests. First, colour vision was tested using an Ishihara test. Second, visual acuity was measured with a reg-ular vision board consisting of Landolt rings. Last, the visual screening involved a stereotest
that tested whether subjects were able to per-ceive the 3D-stimuli through the stereoscope. Displays in this stereotest were similar to the 3D-displays, only now 1 to 4 rectangles ap-peared on the screen for a variable time (200, 300, 400 or 500 ms). Subjects had to indicate how many rectangles they saw appearing on the screen. Like in the 3D-conditions, the rect-angles were only visible binocularly while look-ing through the stereoscope. Subjective isolu-minance was measured with a flicker-test. Fur-thermore, subjects were trained on the training change detection task. This training consisted of a practice block of 30 trials, followed by a test session of 4 blocks of 60 trials. This test-session served to exclude unmotivated subjects or subjects who fail certain vision tests, in ad-dition to training subjects for the actual exper-imental session. In the second and third ses-sion, subjects performed a total of 12 blocks of 144 trials each on the experimental CDT (1728 trials in total, 192 trials in each condi-tion). The sessions were conducted in a dark room, to enhance the strength of afterimages. After each session, a control questionnaire was conducted. This questionnaire was used to col-lect information on age, handedness, hours of sleep, alcohol, coffee and drug consumption.
Data Analysis Memory capacity was
es-timated using Cowan’s K, which is a linear transformation of performance on the CDT into capacity ((hitrate≠.5+correctrejection≠
.5) ú setsize) (Cowan, 2001). A 3
(mem-ory (IM,FM,WM) by 3 stimulus class (high-contrast, isoluminant, 3D)) repeated measures analyses with repeated measures on both fac-tors was computed. Mauchly’s test of spheric-ity indicated that the assumption of sphericspheric-ity had been violated, therefore degrees of freedom were corrected using the Greenhouse Geisser correction.
2.2 Results
In this experiment, we aimed to discover whether IM is a cortical process by ruling out that retinal afterimages contribute to IM ca-pacity. We achieved this by using 3D
stim-500 Memory
750 1750 2250
Test array + respons
Cue
F. Fragile Memory Trial
Test Display Respons up to 4s 500 Memory 750780 1280 1780 Test display Cue
E. Iconic Memory Trial
Respons up to 4s
500 Memory
750 1750
Test display G.Working Memory Trial
Respons up to 4s Time Change or no change? Memorize A. B. C. D.
Figure 2: Design Experiment 1 A. Sequence of displays B. High contrast black-white stim-ulus producing strong afterimage C. Isoluminant red-grey stimstim-ulus producing weak afterimage
D. 3D stimulus producing no afterimage E. Iconic-cue condition measuring iconic memory F.
Retro-cue condition measuring fragile memory G. No-cue condition measuring working mem-ory.
uli, that create cortical representations with-out a retinal percept or afterimage. Thus, we expected the capacity of 3D stimulus class to be lower compared to high-contrast (HC) and isoluminant (ISO) stimuli, since afterimages could not benefit the capacity of IM in 3D-conditions. However, since we hypothesized that IM is not merely dependent on afterim-ages, we expected that even without the
pres-ence of an afterimage, IM has a high capacity. Therefore, we expected that IM is higher com-pared to FM and WM over all stimulus classes. In the training session, subjects needed to score 75% correct averaged over all conditions. Three out of 29 subjects did not meet this criterion and were excluded for further test-ing. Additionally, seven subjects were ex-cluded from further testing, since they did not
pass the stereotest. Lastly, one subject was excluded because of bad performance on the experimental task, due to drug and alcohol-use the night before testing. The results are based on the remaining 18 subjects.
The results on the training CDT were as ex-pected. Significant differences were found be-tween IM, FM and WM [F (1, 8) = 161.01, p <
.001], as IM capacity was higher than FM, and
FM was higher compared to WM (Figure 3A). The experimental data showed a different pattern (Figure 3B). Subjects could report on average 5.1 items in the IM conditons (81.8% performance), 5.3 items from FM (83.1%) and 3.4 items from WM (70.1%) across all stimulus classes. Repeated measures ANOVA revealed that these capacity scores were significantly different from each other, [F(2, 34) = 71.8, p <
.001, ÷2p= .809], as capacity was lower in WM conditions than in the FM conditions. In con-trast to the training session and against our expectations, IM capacity was equal to FM capacity over all stimulus classes [mean differ-ence = .202, p = .840].
Furthermore, capacity scores differed among stimulus classes [F(1.5, 24.6) = 38.9, p <
.001, ÷2p = .696], as capacity in 3D conditions was lower than capacity in HC and ISO condi-tions for all memory types. No significant in-teraction was found between stimulus class and memory type [F(2.8, 47.8) = 2.7, p = .062].
2.3 Discussion
Based on the results of Experiment 1, one might conclude that there is no difference be-tween IM and FM, regardless of stimulus class. On the one hand, this would support the idea that IM is merely a retinal afterimage, since the high capacity of IM deteriorated when no afterimage was present. On the other hand, high-contrast stimuli did not show higher IM capacity either, whereas these stimuli should cause a strong afterimage. If afterimages con-tribute to the high IM capacity, stronger after-images should create higher IM capacity. How-ever, our results do not support this inference. This finding is remarkable, especially since the difference between IM and FM was apparent
in the training sessions. To further investigate this effect and control for possible methodolog-ical influences, we conducted an adapted ver-sion of the CDT in Experiment 2.
3 Experiment 2
A few limitations of the previous experiment were addressed in our second experiment. First of all, although the HC and ISO con-ditions did evoke a percept on the retina, it might be that they were influenced by the depth perception caused by the displacement of the displays. It could be that the depth perception somehow interfered with the effect of IM. To control for such a possible effect, HC and ISO conditions were presented without displacement. The condition with HC stim-uli and displacement remained in the experi-ment for comparison reasons. In Experiexperi-ment 1, subjects reported difficulty when switch-ing between different stimulus class conditions. Therefore, Experiment 2 presented each stim-ulus class in separate blocks. Lastly, it is pos-sible that IM was masked by our four-triangle cue. Previous research has shown that when a cue similar to ours is presented within close temporal proximity, it could act as a mask and thereby reduce target discriminability (Enns & Di Lollo, 1997). Therefore, Experiment 2 used a non-masking line cue to control for masking effects.
3.1 Method
Sample In total 35 students from the
Uni-versity of Amsterdam participated in this study (7 male, age range: 19-29), nine of whom had also participated in Experiment 1. All subjects gave their written informed consent to participate in the study, which was approved by the local ethics committee of the Univer-sity of Amsterdam. Subjects received course credits or money for participation. All sub-jects had normal or corrected-to-normal vision and no colour deficiencies.
Task & Stimuli The task was identical to
2 3 4 5 6 7 8
Items reported (Cowan's K)
Memory capacity during training
Iconic memory Fragile memory Working memory 2
3 4 5 6 7 8
Items reported (Cowan's K)
Effect of different stimulus classes on capacity
3D HC ISO
Iconic memory Fragile memory Working memory
A B 2 3 4 5 6 7 8
Items reported (Cowan's K)
Effect of different stimulus classes on capacity
3D HC ISO HC-disp
Iconic memory Fragile memory Working memory
C
Figure 3: Results Experiment 1 & 2 A. The results on training task show the typical IM-FM-WM differentiation B. Results on Experiment 1. 3D stimulus class shows lower capacity compared to high-contrast (HC) and isoluminant (ISO) stimulus classes. No difference is ap-parent between IM and FM over all stimulus classes. C. Results on Experiment 2. After some adjustments to the experimental task, the difference between IM and FM is apparent again. Capacity in 3D stimulus class is lower compared to other stimulus classes.
following changes. First, instead of randomly intermixing the different conditions, the ISO, 3D and HC conditions were presented in blocks of 44 trials. Second, to check whether the dis-placement of the displays in the high-contrast and isoluminant conditions caused an active cortical process that suppresses IM-capacity, this experiment consisted of 4 stimulus classes: high-contrast with displacement (HC-disp), high-contrast without displacement (HC), iso-luminant without displacement (ISO) and 3D. Third, the same cueing line as in the training task was used in the experimental task, to pre-vent possible masking effects. Fourth, to make the task more similar to the training task, now eight rectangles were used in the test-display.
Procedure Experiment 2 follows the same
screening procedure as Experiment 1. The perimental procedure differs from the first ex-periment in the following: since an extra con-dition was added, one block now contained 192 trials. Subjects performed 6 blocks of testing (1152 trials), divided over 2 sessions.
Data Analysis Cowan’s K was calculated
to estimate VSTM-capacity. A 3 (memory) by
4 (stimulus class) repeated measures ANOVA was conducted.
3.2 Results
Like in the Experiment 1, subjects were only included if they passed the screening session, which included a stereotest, a training CDT and some visual tasks. A total of 14 subjects were excluded of further testing, since they did not pass the stereotest. Furthermore, one sub-ject had to be excluded since he did not pass the colourblindness test. The results of Exper-iment 2 and 3 will be based on the remaining 20 subjects.
See figure 3C for an overview. Capacity scores again differed among memory types [F(2, 38) = 166.2, p < .001]. In contrast to the previous experiment, now IM capacity was significantly higher compared to FM [IM = 7.4 items, FM = 6.7 items, mean difference = .748, p < .001], and FM was higher com-pared to WM [WM = 4.2 items, mean differ-ence = 2.473, p < .001].
Furthermore, a main effect was found for stimulus class [F(3, 57) = 10.0, p < .001], as capacity in 3D conditions was lower compared to all other conditions. As apparent in
Fig-ure 3C, the ISO, HC and high-contrast with displacement conditions showed a similar pat-tern. Against our hypotheses, no difference be-tween ISO and HC conditions was found [mean difference = .114, p = .125]. This indicates that HC stimuli did not produce a stronger afterimage compared to ISO stimuli. Further-more, no difference was found between high-contrast stimulus class with or without dis-placement, suggesting that the displacement in non-3D conditions did not cause the unex-pected results from Experiment 1 [mean differ-ence = .067, p = .107].
3.3 Discussion
In contrast to Experiment 1, Experiment 2 provides stronger evidence towards the idea that IM is cortical. We found that the high ca-pacity of IM was still observed when no after-image was present. The contrasting results be-tween these two experiments are probably a re-sult of the masking effect of the cue. We ruled out that the displacement of non-3D stimuli had influenced Experiment 1, by comparing HC conditions with and without displacement. Furthermore, we did not find any differ-ences between capacity in HC and ISO stimu-lus classes. This suggests that either the differ-ences in afterimages did not influence capacity, or there was no difference between the after-image generated by HC and ISO stimuli. We suggest the latter to be the case, since some subjects reported that they experienced strong afterimages in ISO stimulus class. Overall, capacity scores in Experiment 2 were some-what higher compared to Experiment 1. This is most likely resulting from the minor alter-ations in the experimental design. For exam-ple, in the second experiment eight rectangles were presented in the test-display, compared to one in the first experiment. This way, subjects could rely on a memorized pattern of rectan-gles instead of just one rectangle. Lastly, in both Experiment 1 and 2 the capacity in 3D stimulus class is lower compared to HC and ISO, even within FM and WM. Not only could this effect be a result of a non-existent afterim-age, stimulus complexity is of influence as well.
The perception of 3D stimuli involves greater information load. As such, fewer items can be retained in memory (Alvarez & Cavanagh, 2004).
Taking the findings of Experiment 1 and 2 together, we conclude that IM seems to be a cortical process, yet it can be masked by a four-triangle cue when presented within close tem-poral proximity.
4 Experiment 3
To further discriminate between retinal and cortical contributions to IM, a binocular mask-ing experiment was conducted. The goal of this experiment was to see whether cross-eye interference influences IM capacity. Cross-eye interference can only happen at its earliest in the visual cortex, since binocular information merges at its earliest in V1. Therefore, if IM is influenced by cross-eye interference, this proves that IM can be traced down to the early visual cortex.
In this experiment, the CDT was only pre-sented to one of the two eyes, while an inter-fering display (ID) was aimed to mask the ef-fect of IM and FM by presenting it to either the same or different eye. Furthermore, the location of the mask could be congruent or in-congruent to the probed location. As an ear-lier study has suggested that FM is location-specific, we expected to find the same results for IM (Pinto et al., 2013). Therefore, we ex-pected the capacity of IM to deteriorate when the ID is presented at the probed location, yet not when the mask is presented at a differ-ent location. Most importantly, we expected no difference of ID presentation (right or left eye) in memory capacity; because binocular information comes together at its earliest in the primary visual cortex. Therefore, if the mechanism underlying IM resides within V1-V3, cross-eye interference can take place and thereby interfering displays from the opposite eye could also interfere with IM. This would mean that there is no difference between ca-pacity when IDs are presented to the same or to the other eye. However, if IM is merely reti-nal, ID presentation on the opposite eye would
not interfere with IM capacity since the images are not fused at a retinal level. This would mean that capacity is lower when presented at the same eye, yet when presented to the other eye, capacity remains high.
4.1 Method
Sample The same sample as in Experiment
2 participated in this experiment.
Task, Stimuli & Procedure The design is
adapted from the experiment of Pinto et al. (2013). Again, a change detection task was presented through the stereoscope. The task consisted of eight white rectangles, presented on a black background. This time, the mem-ory display was only presented at one side of the visual field (left or right, see Figure 4A). In IM and FM conditions, an interfering dis-play (ID) appeared for 28ms. This ID con-sisted of 4 rectangles of random orientation, masking half of the memory-display. The ID could be presented either at a congruent or incongruent location compared to the probed location. Furthermore, the ID could be pre-sented at the same or different eye compared to the memory display (Figure 4B). Similar to previous experiments, the cue and test displays were presented. In IM and FM conditions, the test display was presented after the cue had disappeared (Figure 4D, E & F), whereas in WM conditions, the cue was presented at the same time as the test display (Figure 4 C & G). Subjects had to indicate whether the rect-angle at the probed location had changed or not. No IDs were presented in the working memory condition, as earlier studies have in-dicated that WM is not susceptible to masking (Pinto et al., 2013). Here, the working mem-ory conditions served as a baseline.
A total of 11 conditions were used: memory (IM, FM) x eye (same or other eye compared to memory-display) x location (congruent or incongruent to probed location) and IM, FM and WM without an ID. Subjects were tested on two blocks of 352 trials each, after they had participated in the Experiment 2.
Data Analysis Cowan’s K was calculated
to estimate VSTM-capacity. A 2 (memory) x 2 (eye) x 2 (location) repeated measures ANOVA was conducted. Furthermore, IM, FM and WM capacity without masks were measured as a baseline.
4.2 Results
The aim of the current experiment was to in-vestigate whether IM is a cortical process us-ing a binocular maskus-ing paradigm. In this paradigm, IDs were presented during a CDT in either the same or the other eye compared to the memory display and at the congruent or incongruent location compared to the probed location. We expected that the IDs would in-terfere with the capacity of IM and FM when they were presented congruent to probed loca-tion. Furthermore, we expected that the eye to which the IDs were presented should not make a difference in capacity, since a cortically based IM should use the binocularly merged pictures that are represented in the visual cor-tex, instead of the separate pictures that are represented at a retinal level. Therefore, we expected no effect of the eye to which the ID was presented on memory capacity.
When no mask was presented, subjects could report on average 7.4 items in IM, 6.4 items in FM and 3.9 items in WM-conditions. We performed a 2 (eye (same x different)) x 2 (memory (IM x FM)) x 2 (location (congruent x incongruent)) repeated measures ANOVA to investigate whether the IDs had an effect on memory capacity (Figure 5). No main ef-fect was found for eye [F(1, 19) = 1.319, p =
.265, ÷p2= .065], as it did not matter whether the ID was presented to the same or the other eye compared to the memory display. This suggests that cross-eye interference took place, therefore IM made use of the fused images from the two eyes in the early visual cortex. Fur-thermore, there was no main effect for mem-ory, as capacity of IM compared to FM did not significantly differ [F(1, 19) = 2.384, p =
.139, ÷p2 = .111]. We did find a main effect for location [F(1, 19) = 470.4, p < .001, ÷2
p =
loca-A B C D 1 2 3 4 5
Memory Test display
G.Working Memory Trial
Respons up to 4s
Memory Cue Test display
E. Iconic Memory Trial
Respons up to 4s Interfering display (28ms)
Memory Cue
F. Fragile Memory Trial
Test display Respons up to 4s Interfering display (28ms) 500 750 1795 500 750 1795 500 750806 1306 2750 2295 2750 1750
Figure 4: Design Experiment 3 A. Memory display is presented to one eye B. Interfering displays. 1: same eye, congruent location, 2. same eye, incongruent location, 3. different eye, congruent location, 4. different eye, incongruent location, 5. no mask presented C. Working memory conditions, cue presented with test display D. Iconic memory and fragile memory conditions, cue presented before test display E. Iconic-cue condition measuring iconic memory
F. Retro-cue condition measuring fragile memory G. No-cue condition measuring working
memory.
tion of the ID was congruent to the probed lo-cation, memory capacity was hampered. The average memory capacity when IDs were pre-sented at a congruent location (2.7 items) even dropped below WM capacity (3.9 items) [t(19) = 3.0, p = .008, d = .85].
There was an interaction effect between the location of the ID and memory type that was measured [F(1, 19) = 24.992, p < .001, ÷2
p =
.568]. An ID in the congruent location had
more impact on IM compared to FM. It seems that IM is more susceptible for masking
ef-fects, whereas FM is more robust. We did not find any interactions between eye and memory type, [F(1, 19) = .096, p = .760, ÷2
p= .005] and between eye and location [F(1, 19) = .005, p =
.944, ÷p2 = .000]. This indicates that memory capacity did not differ across the eye to which the ID was presented for both location and memory type.
Finally, there was a significant interac-tion effect between memory x eye x locainterac-tion [F(1, 19) = 4.73, p = .042, ÷2
p = .199]. This
0 1 2 3 4 5 6 7 8
Items reported (Cowan's K)
Iconic memory Fragile memory
Effect of interfering displays on memory capacity
Working memory capacity
Same eye, congruent location Different eye, congruent location Same eye, incongruent location Different eye, incongruent location
No mask
Figure 5: Results Experiment 3 Memory capacity is lower when IDs are presented to the location congruent to the probed location (solid lines). No difference was found for the eye to which the ID was presented (red vs black lines).
across the levels of memory, eye and loca-tion. To further investigate this effect, post-hoc analyses were performed. Repeated mea-sures ANOVA within IM revealed that there was only a main effect of location in IM [F(1, 19) = 4.73, p = .042, ÷2
p = .199]. Nei-ther a main effect for eye [F(1, 19) = .615, p =
.442, ÷2p = .031] or an interaction effect for eye x location were found [F(1, 19) = 4.2, p =
.054, ÷2p = .181]. To summarize, these results indicate that IM is location-specific and uses a cortical representation of the memory display.
5 General discussion
In contrast to fragile memory (FM) and work-ing memory (WM), to date there is no con-sensus on the neural mechanisms underlying iconic memory (IM). Earlier research pointed towards the idea that IM is a cortical pro-cess (Sligte et al. forthcoming). However, it remained unclear what role retinal after-images play in iconic memory. This research was aimed at settling the debate on the neu-robiological mechanisms underlying IM. We
found converging evidence for the cortical brain hypothesis of IM in two adaptations of a change detection task. In both experimental paradigms, the principle of binocular percep-tion was used. Binocular perceppercep-tion is used since it is known that the brain merges visual information from both eyes at its earliest in the primary visual cortex, which can be useful in controlling for afterimages.
From the first two experiments we can conclude that IM uses a cortical represen-tation, yet it can easily be masked. The high capacity of IM is still present when the influence of a retinal afterimage is ruled out by using 3D stimuli. These stimuli only generate a cortical representation and no retinal afterimage, since they are not visible
monocularly. Although this supports our
cortical brain hypothesis, this finding was only present when a line-cue was used. To further confirm our findings, we completed a third experiment using a binocular masking paradigm. Our results support our hypothesis that IM uses a binocularly merged represen-tation. IM’s capacity deteriorated when an interfering display (ID) was presented at the probed location, regardless whether this ID was presented at the same eye or different as the memory display was presented to. This suggests that even cross-eye IDs interfere with IM capacity, which can only happen when IM uses a cortically fused representation of stimuli. Therefore, we conclude that IM is location-specific and it is created in the visual cortex.
Backward masking influences iconic memory
Although the results of our first experiment were not in line with our expectations, it can still be explained in the scope of our hypotheses. IM was completely erased across all stimulus classes when followed by a four-triangle cue, but not when followed by a line-cue. We expect these contrasting results to be a result of backward masking. Earlier studies have shown that cues similar to ours could act as a mask for the target, instead of facilitating memory retrieval of the target (Enns & Di Lollo, 1997, 2000). When such a
cue is presented within close temporal proxim-ity after the memory display, it could interfere with the memory formation of the probed item. It is found that this effect increases when set size increases and when attention has to be divided over multiple objects (Enns & Di Lollo, 1997), which is also the case in our experimental design. So unintendedly, the effect of IM was overwritten by our four-triangle cue. We can conclude that IM could be masked, in addition to providing evidence of IM as a cortical process. These results are in line with previous research indicating that IM can easily be masked by, for instance, a brief flash of light (Sligte et al., 2008). In contrast, FM appears to be more robust (Pinto et al., 2013). Also, our third experi-ment shows that both IM and FM are erased when a feature-like mask appears at the same location as the probed location, confirming the location-specific nature of both memory-types.
Limitations of our stereotest
A remarkable finding in our study is that nearly half of our subjects had to be excluded because they failed our stereotest. Most of these subjects reported that they were not able to fuse the two images together; they kept seeing the fixation-cross twice. There are some possible explanations for these inconsistencies across subjects. Firstly, for stereoscopic depth perception, binocular disparity is limited within a certain retinal area called Panum’s area. When stimuli exceed this area, it can result in double vision (Schor & Tyler, 1981). Qin, Takamatsu, and Nakashima (2006) have suggested that the fusion criterion can differ slightly among people. Possibly, our disparity was at the edge of Panum’s fusional area and therefore individual differences were found in capacity of perceiving the stimuli. Secondly, individual differences in eye-position could also have caused the differences in perception. For follow up research, it may be helpful to adjust the screens to personal settings. Lastly, the limits of Panum’s area are dependent on exposure duration and spatial resolution (Tam, Speranza, Yano, Shimono, & Ono,
2011). Further research is needed to see
whether adjustment of the resolution and duration of the stimuli will lead to better perception of our 3D stimulus class.
Contrasting results previous studies
When comparing our results to the previous study from Sligte et al. (2008), there are some major differences. First, there are differences concerning high-contrast and isoluminant stimuli. Whereas Sligte et al. (2008) find that high-contrast stimuli generate a higher mem-ory capacity because of a strong afterimage, we find no such effect. In our experiment, we chose to use subjective isoluminance instead of objective isoluminance. We found high variability between subjects in their isoluminance. As such, it is possible that our subjective isoluminant stimuli still generated a strong afterimage and thereby no differences in high-contrast versus isolumant stimuli were found. Second, Sligte et al. (2008) find in their study that IM was primarily driven by retinal afterimages. In their study IM was nearly non-existent without an afterimage, while our IM-capacity was still present even without the possibility of persistent retinal activation. Further research is needed to replicate our findings.
Iconic memory is phenomenally conscious
As greater understanding of the neural mech-anisms underlying VSTM may be helpful in understanding consciousness, it is useful to put our results in the light of consciousness. It seems that our results fit with the idea of recurrent processing (RP) as a requirement for consciousness (Lamme, 2010). This theory divides consciousness into two stages; phe-nomenal and access awareness. The first form of awareness is related to ’experience’. It is a non-cognitive form of perception, independent of attention. Access awareness is dependent on attention, reportable and often associated with working memory (Block, 1996; Lamme, 2004). The theory states that items come into awareness when not only feedforward activa-tion is present, but also recurrent processing is active. This means that when a certain area is reached by feedforward processing, recurrent
processing interacts with areas that were acti-vated earlier and therefore produces conscious
experience (Lamme, 2003). Phenomenal
awareness and IM are related to each other in different ways (Lamme, 2003; Vandenbroucke, Sligte, Fahrenfort, Ambroziak, & Lamme, 2012). Firstly, like IM, phenomenal awareness covers a rich representation of the world, as it happens at lower levels in the visual hier-archy with little competition between stimuli. Therefore, groups of recurrent interactions are possible, which enables multiple items to be stored. Secondly, phenomenal awareness and IM are thought to share the same underlying neural mechanisms (Lamme, 2003). According to our study, IM can be traced down to the visual cortex. The same goes for phenomenal awareness; stimuli would not come into aware-ness if they were merely retinal, since retinal activation involves purely feedforward activa-tion and no recurrent processing. Therefore, both IM and phenomenal awareness seem to be related to the early visual cortex. Third, previous research has ruled out that IM takes place at an unconscious level, as it was found that IM capacity benefited from visual illusions. Therefore, IM uses a higher-order integrated perceptual representation, which is phenomenological conscious (Vandenbroucke et al., 2012). Following this reasoning, it becomes clear that phenomenal awareness and IM are conceptually similar, only IM is a form of memory and thereby is present with removal of the stimulus, whereas awareness involves current experience. In conclusion, IM representations are phenomenally conscious and recurrent processing is necessary for IM to occur at this conscious level. This validates our findings that IM is found in the early visual cortex and not at a retinal level.
Iconic memory a cortical brain process
In sum, we conclude that the source of IM can be found in the early visual cortex (V1-V3). We found presence of the high capacity of IM even without presence of retinal afterimages. Thus, settling the debate with evidence supporting cortical basis of IM. Nevertheless, it is still important to confirm our results with
a replication of our study. Further knowledge on the neural base of VSTM can enhance our understandings of the neural mechanisms underlying consciousness.
References
Accornero, N., Gregori, B., Pro, S., Scappini, G., & La Riccia, M. (2008). Chromatic modulation of luminance visual evoked potential latencies in healthy subjects and patients with mild vision disorders.
Clinical Neurophysiology, 119 (7), 1683–
1688.
Adelson, E. H. (1978). Iconic storage: The role of rods. Science, 201 (4355), 544–546. Alvarez, G. A., & Cavanagh, P. (2004). The
capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological
sci-ence, 15 (2), 106–111.
Banks, W. P., & Barber, G. (1977). Color information in iconic memory.
Psycho-logical Review, 84 (6), 536.
Block, N. (1996). How can we find the neural
correlate of consciousness? Trends in
neurosciences, 19 (11), 456–459.
Coltheart, M. (1980). Iconic memory and visible persistence. Perception &
psy-chophysics, 27 (3), 183–228.
Courtney, S. M., Ungerleider, L. G., Keil, K., Haxby, J. V., et al. (1997). Transient and sustained activity in a distributed neural system for human working memory.
Na-ture, 386 (6625), 608–611.
Cowan, N. (2001). Metatheory of storage ca-pacity limits. Behavioral and brain
sci-ences, 24 (01), 154–176.
Enns, J. T., & Di Lollo, V. (1997). Object substitution: A new form of masking in unattended visual locations.
Psychologi-cal Science, 8 (2), 135–139.
Enns, J. T., & Di Lollo, V. (2000). Whatâ ès new in visual masking? Trends in
cogni-tive sciences, 4 (9), 345–352.
Haxby, J. V., Petit, L., Ungerleider, L. G., & Courtney, S. M. (2000). Distinguishing the functional roles of multiple regions in distributed neural systems for visual
working memory. Neuroimage, 11 (2), 145–156.
Knapen, T., Kanai, R., Brascamp, J., van Box-tel, J., & van Ee, R. (2007). Distance in feature space determines exclusivity in visual rivalry. Vision Research, 47 (26), 3269–3275.
Lamme, V. A. (2003). Why visual attention and awareness are different. Trends in
cognitive sciences, 7 (1), 12–18.
Lamme, V. A. (2004). Separate neural defini-tions of visual consciousness and visual attention; a case for phenomenal aware-ness. Neural networks, 17 (5), 861–872. Lamme, V. A. (2010). How neuroscience will
change our view on consciousness.
Cog-nitive Neuroscience, 1 (3), 204–220.
Landman, R., Spekreijse, H., & Lamme, V. A. (2003). Large capacity storage of inte-grated objects before change blindness.
Vision research, 43 (2), 149–164.
Luck, S. J., & Vogel, E. K. (1997). The capac-ity of visual working memory for features and conjunctions. Nature, 390 (6657), 279–281.
Luck, S. J., & Vogel, E. K. (2013). Visual working memory capacity: from psy-chophysics and neurobiology to individ-ual differences. Trends in cognitive
sci-ences, 17 (8), 391–400.
Pinto, Y., Sligte, I. G., Shapiro, K. L., & Lamme, V. A. (2013). Fragile visual short-term memory is an object-based and location-specific store. Psychonomic
bulletin & review, 20 (4), 732–739.
Postle, B. R. (2015). The cognitive neu-roscience of visual short-term memory.
Current opinion in behavioral sciences, 1 , 40–46.
Qian, N. (1997). Binocular disparity and the perception of depth. Neuron, 18 (3), 359– 368.
Qin, D., Takamatsu, M., & Nakashima, Y. (2006). Disparity limit for binocular fu-sion in fovea. Optical review, 13 (1), 34– 38.
Schor, C. M., & Tyler, C. W. (1981). Spatio-temporal properties of panum’s fusional area. Vision research, 21 (5), 683–692.
Serences, J. T., & Yantis, S. (2006). Selec-tive visual attention and perceptual co-herence. Trends in cognitive sciences,
10 (1), 38–45.
Shady, S., MacLeod, D. I., & Fisher, H. S. (2004). Adaptation from invisible flicker.
Proceedings of the National Academy of Sciences of the United States of America, 101 (14), 5170–5173.
Sligte, I. G., Scholte, H. S., & Lamme, V. A. (2008). Are there multiple visual short-term memory stores? PLOS one, 3 (2), e1699.
Sligte, I. G., Scholte, H. S., & Lamme, V. A. (2009). V4 activity predicts the strength of visual short-term memory represen-tations. The Journal of Neuroscience,
29 (23), 7432–7438.
Sligte, I. G., Vandenbroucke, A. R., Scholte, H. S., Lamme, V. A., et al. (2010). De-tailed sensory memory, sloppy working memory. Frontiers in Psychology, 1 . Sperling, G. (1960). The information
avail-able in brief visual presentations.
Psy-chological monographs: General and ap-plied, 74 (11), 1.
Tam, W. J., Speranza, F., Yano, S., Shimono, K., & Ono, H. (2011). Stereoscopic 3d-tv: visual comfort. IEEE Transactions
on Broadcasting, 57 (2), 335–346.
Todd, J. J., & Marois, R. (2004). Capacity limit of visual short-term memory in hu-man posterior parietal cortex. Nature,
428 (6984), 751–754.
Vandenbroucke, A. R., Sligte, I. G., Fahren-fort, J. J., Ambroziak, K. B., & Lamme, V. A. (2012). Non-attended represen-tations are perceptual rather than un-conscious in nature. PLoS One, 7 (11), e50042.
