Cortical and Retinal Contributions to Iconic Memory Karen Kuckelkorn
Date: 12-03-2015
Student number: 10336907 Course: Bachelor project Universiteit van Amsterdam
Docent: Ilja Sligte and Alexander Laufer Number of words: 3828
Abstract
Although images perceived in every day life are in a constant state of flux, we are able to keep a limited amount of information in visual short-term memory (VSTM). The high capacity iconic memory (IM) is a VSTM stage, where visual information is maintained for only a brief period. IM is thought to arise from retinal after-images and is therefore often considered as one of the stages of VSTM that has underlying retinal contributions. However, more recent studies provide evidence for a cortical contribution to IM. Here we investigate whether visual information processed in IM has cortical or retinal contributions. By using a change detection task and an oculus rift to create depth we measured the cortical and retinal contributions to IM. This study did not provide evidence for cortical contributions to IM. This conclusion can be drawn from the fact that we did not find the same performance in IM while measuring cortical contributions and while measuring retinal contributions to IM.
Introduction
In everyday life we perceive our surroundings in rich detail. However, those clear images seem to be in a constant state of flux, in which they disappear quickly. This visual process is related to the visual short-term memory (VSTM). It enables us to briefly maintain visual information in our memory (Sligte et al., 2010). It consists of three different stages: iconic memory (IM), fragile memory (FM) and visual working memory (VWM) (Jacob et al., 2013). IM is a short-lasting memory with a high capacity (Vandenbroucke et al., 2011). In contrast, VWM is an effortful storage, with a low capacity of approximately 4 items (Bradley & Pearson, 2012). FM embraces a longer period of time (around four seconds) and its
capacity is six items or higher (Pinto et al., 2013). Therefore, FM capacity is lower than IM capacity, but higher than VWM capacity (Sligte et al., 2008). With reference to duration it is reversed, FM embraces longer than IM, but shorter than VWM. To better understand the different VSTM stages we will consider a real world situation. Imagine going on a hiking trip and seeing a beautiful scenery of mountains. Initially you will most likely remember this scene in rich detail. This is the result of the high capacity in IM. However, because of the short lasting period of IM, which is around 1 ½ seconds, these details will disappear quickly. It could be that this is due to underlying retinal afterimages to IM. Afterwards it is only possible to remember the scene as a rough picture, since it is stored in the long-lasting VWM (Pinto et al., 2013).
In this study, we specifically focus on the contributions to IM. While studying IM researchers discovered the existence of retinal afterimages. Therefore they assumed that IM consists of prolonged retinal activation beyond stimulus duration (Sligte et al., 2008). This finding supports the idea of a retinal contribution to IM. However, some research findings rather suggest that IM might at least in part be cortical. For instance, it is found that experience, spatial aspects and attention, which are driven by cortical contributions, have different effects on the storage of IM (Graziano & Sigman, 2008; Persuh et al., 2012; Ruff et
al., 2007). Furthermore, more evidence for this view is found by investigating the underlying automatic processes to IM. IM appears to be driven by automatic processes, which are
nonretinotopic (Lin & He, 2012). In conclusion, there is no consensus on what the underlying process of IM is: retinal or cortical. Therefore, it is important to look at the different VSTM stages on a neural level.
Although some studies suggest that IM might at least in part be cortical, research to pinpoint a specific neural correlate of IM has been inconclusive (Rensink, 2014). This is partly due to the fact that differentiating between the three VSTM stages (IM, FM, VWM) has often been overlooked. However, some evidence suggests that these three processes might have different neural origin. For instance, Sligte et al. (2011) have dissociated FM from VWM by stimulating the dorsolateral prefrontal cortex (DLPFC). Further research is necessary to specify the neural origin of each specific stage. Therefore it is important to consider the functional differences of VSTM stages.
In previous studies, researchers investigated these functional differences of VSTM stages. A limitation often described in those is the lack of differentiation between all of the three VSTM stages. For instance, while studying functional differences researchers found lasting differences between IM and FM. They found that IM is overwritten by any visual information, whereas FM is overwritten by very specific visual information (Pinto et al., 2013). In this study researchers did not include VWM and therefore did not make a clear differentiation between all of the three VSTM stages. To clarify the belonging contributions to each stage it is necessary to compare all of the three stages with each other.
There are various implications of studying VSTM, particularly IM. First, it is interesting to have a look at the possibilities of improvement of VSTM considering the digitalization in today´s working environment (Sun et al., 2011). It could be interesting to consider how one can improve the perception of employees through displaying items on an effective manner on a computer screen. Second, research on improvement of IM would be a
good addition because of the increasing number of Alzheimer patients. It seemed that IM is a problem in patients with mild cognitive impairments, which have a much higher risk of developing Alzheimer disease (Lu et al., 2005).
The goal of the present study is to investigate whether cortical or retinal contributions underlie IM. We assume that IM has cortical contributions to some extent. To investigate this, we used a monocular condition to measure the retinal contributions to IM and a stereoscope condition to measure the cortical contributions to IM, while participants were tested on a change detection task to measure the capacity of all VSTM stages. Furthermore we operationalized this by using an oculus rift to create depth.
To better understand the perception of depth in the stereoscope condition we need to take into consideration that the brain utilizes the different signals from the two eyes to process (stereoscopic) depth. Neurons in the visual cortex are selective for binocular disparity,
meaning that there are different signals from the two eyes. These neurons fire when binocular disparity is present and make it possible to perceive the emerged image in 3D by combining the signals from both eyes (Poggio et al., 1988; Hubel et al., 2013). Due to the underlying cortical process through combining the image, we measure cortical contributions in the stereoscope condition. For the measurement of the retinal contributions, in the monocular condition, we used monocular vision, meaning that the eyes are used separately. In this condition it would be possible to perceive the image with one eye and combining of the images is not necessary. To make sure that we measure IM we make a comparison between all VSTM stages, the IM, FM, VWM condition.
In the monocular condition we expect participants to perform the best on the IM condition and better on the FM condition compared to the VWM condition. Referring to IM, we expect the same performance in the IM condition for the monocular condition and the stereoscope condition. This would confirm our hypotheses, which assumes that IM has some underlying cortical contributions.
Method The methods are adapted from Pinto et al. (2013). Participants
In total, 29 students from the University of Amsterdam participated in this study (20 female, 9 male; age range 19-34 years, mean age 24). All of the participants had normal or corrected-to-normal vision. Participants with a neurological disorder or a lazy eye were excluded from the study. Participants received either course credits or financial compensation for their participation. Beforehand, participants gave their written informed consent. The local ethics committee of the University of Amsterdam approved this study.
Stimuli
The trails were designed on a grey screen with a grey 440x440px background (RGB 128 128 128). A 600x600px frame, composed of black, white confetti, was placed on the grey background. A 440x440px canvas, consisting of black, white confetti, was placed on top of the grey background. Black, white confetti consisted of black, white lines (ratio 50%), which had a length of 4-10 px and a width of 1 px, were randomly placed on the canvas and on the frame. Stimuli in the memory array and in the test probe of the stereoscope condition were black, white confetti rectangles. The 51x13px confetti rectangles were cut out of another confetti canvas of the same size and pasted on a grey background, with a size of 64 by 64 px (see Fig. 1). This was pasted on the original confetti canvas and the grey background was made invisible. In the monocular condition, white rectangles were used instead of confetti rectangles and the background was grey. The memory array contained of eight of these rectangles and the test probe contained one of these rectangles that each had one of eight possible positions and one of four possible orientations. The possible orientations were horizontal, vertical, 45° to the horizontal and 135° to the vertical. For the position of the
rectangles, we used an equation to calculate the position of the rectangles in the circle. Here, the degrees were 22.5, 67.5, 112.5, 157.5, 202.5, 247.5, 292.5, 337.5. Cues were composed of four white 20x20px triangles, together forming a 128x128px square. The cue appeared in one of the eight possible locations. The fixation cross was placed in the middle of the canvas (48 font size). At the start of a new trial it was displayed in red and then it immediately changed into green.
Fig. 1 In the stereoscope condition black, white confetti were randomly placed on the canvas and on the frame. In this example a black, white confetti rectangle is placed at the top right and perceived in 3D through the oculus rift, here marked in grey for clarification purposes.
Equipment
An oculus rift (Development Kit 2) was used in all conditions and stimuli were presented on the screen in the oculus rift. The oculus rift was connected with a computer.
Trial design
A change detection task was used in all conditions. In total, there were six conditions, mono x IM, mono x FM, mono x VWM, stereo x IM, stereo x FM, stereo x VWM. In all conditions three displays were presented, a start display, a memory display and a test display. First, a start display was presented for 100ms, containing a red fixation cross. Second, a
memory display was presented for 500ms, containing eight oriented rectangles. Third, a test probe was displayed, containing one of the eight oriented rectangles, which remained on the screen until response of the participant. On 50% of the trials, the orientation of the rectangle remained the same in the third display. On the other 50% of the trials, the orientation of the rectangle changed in the third display. A cue was used for the IM condition and the FM condition. In the IM condition the cue appeared 30ms after disappearance of the second display. In the FM condition the cue appeared 1000ms after disappearance of the memory array. The cue pointed to the location where the changed or unchanged rectangle appeared in the second display. The duration of the cue was 500ms. The duration between the cue and the test probe was also 500ms. In the VWM condition no cue was displayed. In the VWM trials, the duration between the memory array and the test probe was 1000ms. Between the different displays and the cues, displays with a grey colour were presented. There were 8 blocks consisting of 96 trials each, 48 trials in the stereoscope condition and 48 trials in the
monocular condition. Thus, there were 768 trials in total. If participants answered correctly, a high sound was played, if they answered incorrectly, a low sound was played.
For the training session we used the change detection task from Vandenbroucke et al. (2011) (see Fig. 2)
Fig. 2 Trial design of the change detection task from the training session (Vandenbroucke et al., 2011). In A the trial design for the FM condition is shown. The cue appeared 1000ms after the disappearance of the memory array and is displayed for 500ms. In B the trial design for the VWM condition is shown. The cue is displayed in the test probe and disappeared after 500ms. In the change detection task of our experiment the sequence of the trial design was the same, except that we did not use a cue in the VWM condition.
Procedure
First, participants were given instructions to the task and had to sign the informed consent. Second, participants were controlled whether they can perceive 3D images through the oculus rift. Third, participants were trained on another change detection task without the oculus rift, which was not used in the original experiment. The training, previous to the test trials, was given, due to the exclusion for unmotivated and unable participants and to train participants on the change detection task. Participants were instructed that they are only allowed to participate in the current study if they were correct on 75% (later 70%) of the training trails. After the training session the participants started the change detection task of the current study. Participants wore an oculus rift and they were instructed to take a break after 96 trials. When participants started the experiment they saw on the screen that they had to press a button to start the experiment. They were instructed to press the left mouse button for change and the right mouse button for no change. Further, they were instructed to only press the change button when they are sure that they saw a change in the orientation of the rectangle. Every participant took part in all of the conditions with random sequence of trials. Dependent on the individual reaction time and duration of the break, it took around two hours to perform the training session and the experiment of this study. The average performance time on 96 trials were about eight minutes. After finishing the eight blocks of the study, participants left.
Data Analysis
For the data analysis, we used a factorial repeated measures analysis of variance (ANOVAs) to compare the mean correct percentages and the mean reaction time per condition. We used a factorial repeated measures ANOVA to compare the mean correct percentages per condition for all 8 blocks of the task and we used two factorial repeated measures to compare performance between the first and the last four blocks of the task with reference to the mean correct percentage and the mean reaction time per condition.
Results
In the training session, participants first needed to score a correct percentage of 75%. In hindsight, this criterion was too strict since half of the participants did not reach this threshold. Therefore, we adjusted the criterion to a correct percentage of 70% after half time of the running experiment. Twelve out of 29 participants were excluded from the study because they did not pass the training session. One participant was excluded from the study because he did not complete the task in one session. Our analysis is based on the data of the remaining 16 participants. Factorial repeated measures ANOVAs were used to compare the mean correct percentages and the mean reaction time per condition.
For the following two analyses, the assumption of sphericity has not been violated. We first investigated whether the type of memory used (IM, FM and VWM) influences the performance in the mono condition. Therefore, we compared the average correct percentage per type of memory in the mono condition. Here, we found that the type of memory
influences the performance (defined as correct percentage). There was a main effect of the type of memory on performance in the mono condition, F(2, 30) = 31.35, p < .0001. Contrasts revealed that performance in the IM mono condition, F(1, 15) = 40.49, p < .0001, and in the FM mono condition, F(1, 15) = 102,25, p < .0001, was significantly higher than performance in the VWM mono condition. However, there was no significant difference between IM and
FM in the mono condition, F(1, 15) = .03, p = .86, as is shown in Fig. 3. Based on these results, the first expectation, that participants would perform the best on the IM condition and better on the FM condition compared to the VWM condition, could not be confirmed.
We next investigated whether visual information processed in IM has underlying cortical or retinal contributions. Therefore, we observed the combined influence of the type of memory and the type of stimulus presentation (mono or stereo) on performance (defined as correct percentage). There was no significant interaction effect between the type of memory and the type of presentation (stereo or mono), F(2, 30) = .84, p = .441, as is shown in Fig. 3. This finding was against our expectation that performance in the IM mono condition and in the IM stereo condition would be the same. Our hypothesis that cortical contributions underlie IM could not been confirmed.
Fig. 3 Performance (defined as correct percentages) was lowest for the VWM condition, and higher for the IM and FM condition. Performance for the mono condition (green line) was for
all types of memory used (IM, FM, VWM condition) higher than performance in the stereoscope condition (blue line).
Interestingly, we observed a possible learning effect by comparing the first four blocks of the task with the last four blocks of the task with reference to the performance (correct percentage) and the reaction time. The assumption of sphericity has been violated for both analyses. Therefore, we used the Greenhouse-Geisser tests to report findings. We found an improvement in performance when comparing the last four blocks with the first four blocks, F(2, 30) = 23.32, p < .0001. Contrasts revealed that performance on the last four blocks was higher than on the first four blocks, F(2, 30) = 23.32, p < .0001 (see Fig. 4). In particular, we found that the type of presentation (mono or stereo) did not influence the performance in the first four blocks, F(1, 15) = 2.18, p = .16. However, referring to the last four blocks we did find an effect of the type of presentation on performance, F(1, 15) = 16.31, p = .001. We possibly observed that participants learn better in the mono condition (see Fig. 4). We also found an improvement in reaction time when comparing the last four blocks with the first four blocks, F(1, 15) = 27.35, p < .0001. Contrasts revealed that reaction time is higher for the first four blocks than four the last four blocks of the task F(1, 15) = 27.35, p < .0001 (see Fig. 5). These results provide evidence for a learning effect of the task.
Fig. 4 The type of presentation (stereo/mono) for the correct percentage on the type of block. Performance (defined as correct percentages) was higher for the last four blocks of the task than for the first four blocks. Performance for the mono condition (green line) was for all blocks higher than performance in the stereoscope condition (blue line).
Fig. 5 The type of presentation (stereo/mono) for the reaction time on the type of block. Reaction time was lower for the last four blocks than for the first four blocks of the task. Reaction time for the mono condition (green line) was for all blocks lower than reaction time for the stereoscope condition (blue line).
Discussion
The present study was designed to determine whether cortical or retinal contributions underlie IM. Results of the current study did not show which contributions are underlying IM. Returning to the hypothesis posed at the beginning of this study, it cannot be confirmed that IM has underlying cortical contributions. This conclusion can be drawn from the fact that we did not find the same performance for IM in the mono and stereo condition. Therefore, it seems unclear whether IM has underlying cortical contributions. Another objective of the study was to show that performance is the best for IM and better for FM compared to VWM.
The results of the study confirmed that performance in VWM is the worst compared to performance in IM and FM. However, findings did not confirm the expectation that
performance on IM is better than performance on FM. There are several possible explanations for the findings mentioned above.
Firstly, there are alternative explanations for the fact that results did not show a significant difference in capacity between FM and IM. Therefore the question came up whether we actually measured both of these VSTM stages. It may partly be explained by the fact that one could only use FM after extensive training (Matsukura & Hollingworth, 2011). Since participants in our study did not have extensive training, they possibly did not make use of FM. This suggestion may be supported by the fact that we had to adjust the criterion of the pass rate after half time of the running experiment because few participants passed the
training. Future studies should therefore use longer training sessions. However, another most likely explanation for our result findings is that we probably failed to measure IM. In the monocular condition and the stereoscope condition, we measured relative depth, which is processed in higher visual areas. However, it seemed that IM is processed in early visual areas, which we failed to measure (Parker, 2007). Therefore, we could not observe a difference between performance on IM and FM.
Secondly, there are possible explanations for the finding that we found a difference in performance in IM while measuring cortical contributions (stereo condition) and while measuring retinal contributions (mono condition) to IM. These differences can be explained in part by the complexity of the stimuli used in the stereoscope condition, which measured the cortical contributions (Eng et al., 2005; Alvarez & Cavanagh, 2004).). The rectangles in this condition, which consisted of black and white confetti, are more complex than the white rectangles used in the monocular condition. This could explain the higher performance in the monocular condition. Therefore, we cannot conclude that IM has no cortical contribution
based on these results. Instead, it is important to do further research on cortical contributions in IM while using similar stimuli in both conditions, which still make a difference between a retinal measurement and a cortical measurement of IM.
Further important contributions to our research findings are possibly the difference between the tasks used in the training and in the original experiment. There are two crucial differences between the tasks, 3D vision and the organization of objects. 3D vision through the oculus rift in the task used in our study could be confusing to the participant. This is supported by the fact that we found a learning effect referring to both the correct percentage and reaction time. Controlling for both aspects, participants performed better on the last four blocks compared to the first four blocks. Therefore we can assume that participants had to get used to the oculus rift. In future studies they should make use of a training session with an oculus rift. Furthermore, the conception differed in organization of objects because in the training task all eight rectangles were displayed compared to the original task, where only one rectangle was displayed in the test probe. Previous research already pointed out that the exact capacity depends on the organization of objects (Jiang et al., 2000). In future studies the organization of objects should be the same in the memory array and the test probe. These two crucial differences could have influenced our findings.
We can conclude that it is still unclear whether cortical contributions underlie IM. Till now we could not find cortical contributions to IM. The study should be repeated using longer training sessions, training sessions with an oculus rift, more participants and adjusted stimuli.
Reference list
Alvarez, G. A., & Cavanagh, P. (2004). The capacity of visual short-term memory is set both by visual information load and by number of objects.Psychological science, 15(2), 106-111.
Bradley, C., & Pearson, J. (2012). The sensory components of high-capacity iconic memory and visual working memory. Frontiers in psychology, 3.
Eng, H. Y., Chen, D., & Jiang, Y. (2005). Visual working memory for simple and complex visual stimuli. Psychonomic bulletin & review, 12(6), 1127-1133.
Graziano, M., & Sigman, M. (2008). The dynamics of sensory buffers: geometric, spatial, and experience-dependent shaping of iconic memory.Journal of Vision, 8(5), 9. Hubel, D. H., Wiesel, T. N., Yeagle, E. M., Lafer-Sousa, R., & Conway, B. R. (2013).
Binocular Stereoscopy in Visual Areas V-2, V-3, and V-3A of the Macaque Monkey. Cerebral Cortex, bht288.
Jacob, J., Breitmeyer, B. G., & Treviño, M. (2013). Tracking the first two seconds: three stages of visual information processing?. Psychonomic bulletin & review, 20(6), 1114-1119.
Jiang, Y., Olson, I. R., & Chun, M. M. (2000). Organization of visual short-term memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(3), 683.
Lin, Z., & He, S. (2012). Automatic frame-centered object representation and integration revealed by iconic memory, visual priming, and backward masking. Journal of vision, 12(11), 24.
Lu, Z. L., Neuse, J., Madigan, S., & Dosher, B. A. (2005). Fast decay of iconic memory in observers with mild cognitive impairments. Proceedings of the National Academy of Sciences of the United States of America, 102(5), 1797-1802.
Matsukura, M., & Hollingworth, A. (2011). Does visual short-term memory have a high- capacity stage?. Psychonomic bulletin & review, 18(6), 1098-1104.
Parker, A. J. (2007). Binocular depth perception and the cerebral cortex. Nature Reviews Neuroscience, 8(5), 379-391.
in human neuroscience, 6.
Pinto, Y., Sligte, I. G., Shapiro, K. L., & Lamme, V. A. (2013). Fragile visual short-term memory is an object-based and location-specific store. Psychonomic bulletin & review, 20(4), 732-739.
Poggio, G. F., Gonzalez, F., & Krause, F. (1988). Stereoscopic mechanisms in monkey visual cortex: binocular correlation and disparity selectivity. The Journal of
neuroscience, 8(12), 4531-4550.
Rensink, R. A. (2014). Limits to the usability of iconic memory. Frontiers in psychology, 5. Ruff, C. C., Kristjánsson, A., & Driver, J. (2007). Readout from iconic memory and selective
spatial attention involve similar neural processes. Psychological Science, 18(10), 901-909.
Saneyoshi, A., Niimi, R., Suetsugu, T., Kaminaga, T., & Yokosawa, K. (2011). Iconic memory and parietofrontal network: fMRI study using temporal
integration. Neuroreport, 22(11), 515-519.
Sligte, I. G., Scholte, H. S., & Lamme, V. A. (2008). Are there multiple visual short term memory stores?. PLOS one, 3(2), e1699.
Sligte, I. G., Vandenbroucke, A. R., Scholte, H. S., & Lamme, V. (2010). Detailed sensory memory, sloppy working memory. Frontiers in Psychology, 1, 175.
Sligte, I. G., Wokke, M. E., Tesselaar, J. P., Scholte, H. S., & Lamme, V. A. (2011). Magnetic stimulation of the dorsolateral prefrontal cortex dissociates fragile visual short-term memory from visual working memory.Neuropsychologia, 49(6), 1578-1588. Sun, H., Zimmer, H. D., & Fu, X. (2011). The influence of expertise and of physical
complexity on visual short-term memory consolidation. The Quarterly Journal of Experimental Psychology, 64(4), 707-729.
dissociate fragile visual short-term memory from visual working memory. Neuropsychologia, 49(6), 1559-1568.
