Is Iconic Memory Cortically Processed
Dewi van der Geugten
Student number: 10363041
Course: Bachelor project, Brain and Cognition
Instructor: Ilja Sligte, Alexander Laufer
Abstract: 146
Abstract
Iconic memory, part of the visual short term memory, is believed to be partially retinally processed. This experiment researched if iconic memory is partially cortically
processed. We researched this by using 3D stimuli in a change detection task that can only be cortically processed, and compared them to stimuli that are not cortically
processed. We used an oculus rift to present the 3D stimuli. We also compared the three different stages of the visual short term memory; iconic memory, fragile memory and working memory. The results show us that there were no performance
differences between iconic memory and fragile memory, and that iconic memory is not significantly equal in both the stereoscope as the monoscope condition. It is questionable if iconic memory was measured and therefore no conclusions can be
Introduction
Visual short term memory (VSTM) is responsible for retrieval of our perceptual images from memory storage. Our original perception, however, is much more
detailed than the stored memory of this perception. The functioning of VSTM, which is divided into three components, mostly takes part in the brain (Vogel & Machizawa, 2004). These three components of VSTM are iconic memory (IM), fragile memory (FM) and working memory (WM) (Jacob, Breitmeier & Treviño, 2013). Until recently, IM was defined as the brief (<1000) retinal after image of the perceptual image (Sperling, 1960). FM is defined as the high capacity memory component that is quickly present after the retinal after image has disappeared, and it can be overwritten by a new peripheral image (Sligte, Scholte & Lamme, 2008). WM is defined as the stored perceptual information that one can retain, even when the actual perceptual image is no longer in vision (Vogel, Woodman & Luck, 2001).
Especially a difference in the storing process of the different kinds of stimuli, the presented length of these stimuli, as the amount of stimuli that has been stored mark the differences between IM, FM and WM (Bradley & Pearson, 2012; Pinto, Sligte, Shapiro & Lamme, 2013; Sligte, Wokke, Tesselaar, Scholte & Lamme, 2011; Vandenbroucke, Sligte, Lamme, 2011). In this experiment, we will especially focus on the processes that are involved in the functioning of IM. Recently, experimental
evidence showed that IM might be modulated by attentional processes, contrary to the traditional view of IM being a purely retinal process (Persuh, Genzer & Melara, 2012). Since attention is a cortical process, IM might have cortical contributions. Others showed that IM consists of a mechanism that updates the object
representation, coherently with IM having cortical contributions (Lin & He, 2012). Besides, it seems that IM has a preference for certain kinds of stimuli that are presented, which contradicts a purely retinal model of IM (Kuhbander, Lichtenfeld, Pekrun, 2011) (Rensink, 2014). However, there is still no evidence to date that IM is located at a specific part in the brain (Saneyoshi, Niimi, Suetsugu, Kaminaga,
Yokosawa, 2011). This is partly due to the lack of research done in the field of IM with new research methods. However, this would be desirable, since IM takes place in a big part of daily activities. Furthermore, knowledge of IM and VSTM could be applied in practice, as for professions in which vision and storing visual signals is important, as well in court with eyewitness testimony (Lu, Neuse, Madigan, Dosher, 2005) (Tolar, Lederberg, Gokhale, Tomasello, 2008). Therefore, this experiment will research if IM is partially functioning cortically.
The research will be done with a change-detection task in which participants will use a oculus rift to look at presented 3D stimuli that only will be visible with the use of a stereoscope or oculus rift, and at stimuli that are also visible without the use of one. Because of our expectations that IM is cortically processed we chose to use 3D stimuli since these are cortically processed stimuli (Cao & Grossberg, 2005). The different components of VSTM will be tested and compared through the use of this task. Based on previous studies, we expected that participants performed best on the IM condition, second best in FM and worst on the WM condition on both the
stereoscope and the monoscope condition, due to the different capacity limits in the different components. Further, our expectations were that participants would score similar at IM in the stereoscope condition, and in the monoscope condition. This could suggest that IM is indeed partially working cortical, since one could process the 3D stimuli in the IM condition.
Method
Because of great similarities with the experiment of Pinto et al. (2013), the same construct of the method section is adopted in this paper.
Participants: In this experiment 29 participants participated, with ages between 19-34 year (M = 24, SD = 3.4). The participants received either a financial reward or course credit for their participation, or they volunteered to participate. All participants were asked about their (corrected) vision and whether they had a history with
amblyopia (lazy eyed) or strabismus (crossed eyed). All participants gave their written informed consent. The local ethics committee approved this experiment.
Materials and stimuli : The same experiment was used for the training trials as in Vandenbroucke, Sligte & Lamme (2011), see Fig. 1. The stimuli of the training trials were shown on a 17” display with a refresh rate of 60 Hz. The participants were seated 75 cm of the display.
Figure 1. Above you see one trial as was presented in the training. The background color was black with white triangles. Figure from Vandenbroucke, Sligte & Lamme (2011).
An oculus rift (type DK2) was used for the actual experiment. The background of the display was grey (RGB 128 128 128) in which a fixation cross (font 48) was displayed in the centre of the screen, which was green at the start of every new trial but changed to red during the remainder of the trial. On the display, two frames were shown of 600px by 600px each, located -300px and 300px from the centre of the screen. The frames were randomly shuffled and evenly distributed black and white pixels. At the precise centre of every frame a square was placed that was equal to the grey background colour. This grey square was 440px by 440px. Two equally sized squares were placed on the exact locations of the grey squares in the
stereoscope condition. These squares, called the confetti squares, were made out of black and white lines and were 4-10px long and 1px wide, see Fig 2. There were eight transparent squares placed of 64px by 64px in which each a rectangle was placed centrally at both the grey monoscope condition as the confetti stereoscope condition. The squares of the left frame were placed 1px to the left of the centre, and the squares of the right frame were placed 1px to the right of the centre. The
rectangles in the squares were 51px by 13px and they were randomly oriented vertical, horizontal or diagonal (0, 45, 90 or 135 degrees). The positions of the
rectangles were calculated with an equation that calculates the position of points in a circle. The degrees that were used in this calculation are as follows: 22.5, 67.5, 112.5, 157.5, 202.5, 247.5, 292.5, 337.5 degrees. The rectangles were of the same pattern as the confetti square in the stereoscope condition. The rectangles were coloured white in the monoscope condition. The cues that were used in the IM and FM condition were made out of transparent squares of 128px by 128px, and were
located at the position of the rectangle that was about to be cued. White coloured triangles of 20px by 20px were placed in the corner of the squares.
Figure 2. The stereoscope display as presented to the subjects in the stereoscope condition. The left frame is how participants see the stimuli with two eyes in the oculus rift minus the grey square that is surrounding the rectangle, this was added for this article so the rectangle would not dissapaer in the background. The right frame is how participants see the stimuli when they are only using one eye or not using the oculus rift, thus presented in 2D.
Trials: There were three conditions, iconic memory (IM), fragile memory (FM) and working memory (WM) which were all randomly displayed in both the
stereoscope condition as the monoscope condition. Each trial was made out of an alert screen, a memory display and a test display. The alert screen displayed the green fixation cross for 500ms, before switching to the memory display. The memory display consisted of the presentation of eight randomly oriented rectangles which were presented for 500ms. The test display consisted of the presentation of one rectangle on one randomly decided location, which was oriented the same or turned 90 degrees in comparison to the rectangle that was presented in that location in the memory display. The duration of the test display was dependable of whether the participant pressed a button, but could maximally take up to 4000ms. At the IM and
FM condition there was, before the test trial started, a cue presented which directed the attention of the participant to the location of the stimuli that was asked for. The cues were presented 500ms before the test display was displayed. Between the memory display and the cue of the IM condition there was 30ms, and between the cue and the test display 500ms. Between the memory display and the cue of the FM display there was a duration of 1000ms, and between the cue and the test display 500ms. The WM condition did not contain of a cue and between the memory display and the test display there was a duration of 1000ms. There were for both the
stereoscope as the monoscope condition 48 unique trials (conditions [3] X location [8] X change or no change [2]), which sums up to 96 unique trials in total. We
measured the participants accuracy in detecting changes. The possible answers are thus hit, correct rejection, miss or false alarm. Positive and negative feedback were giving during the trials as a positive or negative sound.
Procedure: At the beginning of the experiment, the participants were asked about their history concerning eye-vision. Approved participants were then instructed about the task. The instruction explicitly stated that one should only press the change button if one actually noticed a change. After the instruction, participants had an opportunity to train their memory and change-detection skills in a test that was set up independently of this experiment. This training block consisted of trials with a black background and white rectangles that lasted for about 30 minutes. No oculus rift was used during this training. After this test, we determined if the participant was skilled enough to join the experiment, or if the participants were unmotivated or not skilled enough to do a change detection task. Again, approved participants were then seated at the real experiment in which all 96 trials were shown 8 times in a random
fashion.
Data analysis: Repeated measures ANOVA was used on all the statistical data. We compared performances by the mean correct percentages and the mean reaction time per condition. We also compared the mean correct percentages and the mean reaction time per condition for the first and the second half of the 8 blocks. Differences were considered significant at a P value of 0.05 or less.
Results
Participants needed to score at least 75% correct in the training session to participate in the experiment. Due to a lack of time and a very high dropout rate in the first sessions, this score was changed to 70% correct after measuring 15 participants. Eventually, 12 out of 29 participants were excluded because they did not reach this demand. One participants data was excluded due to an error during the process. The results will be based on the data of the remaining 16 participants.
We researched if IM is partially processed retinally. We used 3D stimuli in a change detection task to measure this. Due to memory capacity, we expected that participants would perform better on IM than on FM. Repeated measures analysis shows that performance of the different conditions of memory differed from each other. Indeed, there was a significant main effect of the type of memory on correct performance, F(2,30) = 74.19, p < 0,001. The assumption of sphericity had not been violated. Participants performed better on the IM and FM conditions than in the WM condition. Indeed, contrasts revealed that performance was significantly higher in IM, F(1,15) = 120,31, p < 0.001, and FM, F(1,15) = 157,54, p < 0.001, than in WM.
Against expectations, there was no significant difference in performance between IM and FM, F(1,15) = .05, p = 0.83. Based on these results, the expectation that
participants perform better on IM than on FM, could not be confirmed.
Our second expectation was that the stimuli would be processed equally in IM on both the monoscope and stereoscope condition, which could suggest that IM is partially cortically processed. Therefore, we compared the influence of both the type of memory with the type of presentation on performance. The results show that there was no interaction effect between the type of memory and the type of presentation, F(1,30) = 0.84, p = 0.44, as is shown in Fig. 3. These results were against
expectations. Based on these results, the expectation that stimuli would be equally processed in IM on both the monoscope and stereoscope condition cannot be further looked into.
Since the experiment consisted of 8 blocks with 96 trials each, there could have been a learning effect. Therefore, the data had been split into 2 blocks; the first block consisted of the data of block 1 to 4, the second of the data of block 5 to 8. The results show that the 2 blocks differ from each other on performance, since
participants performed better on the second block. Indeed, as shown in Fig. 4a, there was a significant main effect of the type of block on the performance, F(2,30) =
23.32, p < 0.001. The assumption of sphericity had not been violated. Consistent with a possible learning effect, contrasts revealed that participants performed better at the second block than at the first block, F(1,15) = 23.32, p < 0.001. We also looked at the reaction time of the participants on both the blocks. Results show that
participants differed in reaction time in both blocks. Indeed, there was a main effect of the type of block on the reaction time of the participants, F(1,15) = 27.34, p < 0.001. The assumption of sphericity was violated and therefore greenhouse-Geisser
corrected tests are reported. Also, participants performed faster at the second block in comparison to the first blocks. Contrasts revealed that reaction time at the second block is lower than at the first block, F(1,15) = 27.34, p < 0.001, as shown in Fig.4b. These results could possibly suggest a learning effect.
B)
Figure 4. A. The means of the type of Blocks against the type of presentation for all participants. Block 1-4 are the First 4 blocks of the experiment. Block 5-8 are the last 4 blocks of the experiment. A learning effect is clearly visible, as participants scored better on the second half of the experiemtn. B. The reaction time of all participants for the First and second half of the experiment against the type of presentation. Participants took less time to respond at the second half of the experiment, than at the First half.
Discussion
The purpose of this experiment was to research if IM is cortically processed. We expected two different kinds of results. First, participants would perform better on IM, second best on FM and worst on WM. Although participants performed
significantly better on IM and FM in comparison with WM, participants performed equally well on both IM and FM. Thus, the first hypothesis was not confirmed with this experiment. Our second expectation was that participants would perform equally well on both IM in the monoscope condition as IM in the stereoscope condition. Since the data did not show a significant interaction effect between the type of memory and the type of presentation, no conclusions can be made about the performance differences of IM on the presentation type.
Since there was no difference measured between IM and FM, one could wonder if both types of memory were actually measured. Our experiment was based on the literature that suggests that the 3D stimuli, or relative depth between both eyes, is processed in higher cortical areas. Other research suggests that IM is
processed in earlier cortical regions than was expected (Parker, 2007). We therefore failed to measure IM properly, since depth was not yet processed while participants had to respond. Therefore, researching if IM is cortically processed has to be
measured in a different way. Perhaps with different kinds of stimuli or in a different time frame.
Another, more practical, flaw in our experiment could have been the duration and intensity of training the participants. Matsukura and Hollingworth (2011) found that in order to use FM in experiments, participants had to be trained extensively. In our experiment, participants were only trained for a maximum of 20 minutes, in which
both IM, FM and WM were trained. This is nowhere near an extensive training. This could have resulted in a failure to measure FM properly. In a follow up experiment, there should be given more attention to the training procedure, e.g. longer
trainingsessions and better instructions.
Our second expectation stated that the performance on IM would be the same for both the stereoscope as the monoscope condition. In a next study one should hold the complexity of the stimuli in account. It takes longer to process complex stimuli, and since the confetti stimuli of the stereoscope condition is more complex than the white stimuli in the monoscope condition, perhaps this could have also influenced the results of the study. Since we failed to measure IM, it is still unclear with what kinds of stimuli IM should be measured in a follow up stimuli, but one also has to take into account the different complexities of the stimuli.
Unexpectedly, we did find a learning effect by splitting the eight blocks in half. We found that participants did perform better on the second half than on the first blocks, and that they also took less time to respond as well. This could mean that participants needed to adjust to the oculus rift, and perhaps that the memory training beforehand was not sufficient enough. For future studies, participants should get more training before they start with the actual experiment. Also, participants
responded during the experiment that the adjustment to the oculus rift and also the different stereoscope condition took a while. In the training there was a distance of 75 cm between the screen and the participant, during the experiment the participants had the stimuli right in front of their eyes. Also, at the beginning some participants even had trouble with seeing the stimuli of the stereoscope condition. When they adjusted, after 1 or 2 blocks, they had no trouble seeing them.
In conclusion, this experiment does not find a cortical contribution to iconic memory. More research should be done to investigate the processes of iconic memory further. In follow up studies, one should invest more time in the proper
training of participants to make sure they follow the instructions well, and there needs to be thought of new ways to measure IM with stimuli that trigger lower cortical areas. There is also room to improve the adjustment from screen to oculus rift.
Literature
Bradley, C., & Pearson, J. (2012). The sensory components of high-capacity iconic memory and visual working memory. Frontiers in psychology, 3.
Cao, Y., & Grossberg, S. (2005). A laminar cortical model of stereopsis and 3D surface perception: Closure and da Vinci stereopsis. Spatial Vision, 18(5), 515-578.
Jacob, J., Breitmeyer, B. G., & Treviño, M. (2013). Tracking the first two seconds: three stages of visual information processing?. Psychonomic bulletin & review, 20(6), 1114-1119.
Kuhbandner, C., Lichtenfeld, S., & Pekrun, R. (2011). Always look on the broad side of life: Happiness increases the breadth of sensory memory. Emotion,11(4), 958. Lin, Z., & He, S. (2012). Automatic frame-centered object representation and integration revealed by iconic memory, visual priming, and backward
masking.Journal of vision, 12(11), 24.
Lu, Z. L., Neuse, J., Madigan, S., & Dosher, B. A. (2005). Fast decay of iconic memory in observers with mild cognitive impairments. Proceedings of the National Academy of Sciences of the United States of America, 102(5), 1797-1802.
Matsukura, M., & Hollingworth, A. (2011). Does visual short-term memory have a high-capacity stage? Psychonomic Bulletin & Review, 18, 1098–1104.
Parker, A. J. (2007). Binocular depth perception and the cerebral cortex. Nature Reviews Neuroscience, 8(5), 379-391.
Persuh, M., Genzer, B., & Melara, R. D. (2012). Iconic memory requires attention. Frontiers in human neuroscience, 6.
Pinto, Y., Sligte, I. G., Shapiro, K. L., & Lamme, V. A. (2013). Fragile visual short-term memory is an object-based and location-specific store. Psychonomic bulletin & review, 20(4), 732-739.
Rensink, R. A. (2014). Limits to the usability of iconic memory. Frontiers in psychology, 5.
Saneyoshi, A., Niimi, R., Suetsugu, T., Kaminaga, T., & Yokosawa, K. (2011). Iconic memory and parietofrontal network: fMRI study using temporal
integration. Neuroreport, 22(11), 515-519.
Sligte, I. G., Scholte, H. S., & Lamme, V. A. (2008). Are there multiple visual short-term memory stores?. PLOS one, 3(2), e1699.
Sligte, I. G., Wokke, M. E., Tesselaar, J. P., Scholte, H. S., & Lamme, V. A. (2011). Magnetic stimulation of the dorsolateral prefrontal cortex dissociates fragile visual short-term memory from visual working memory.Neuropsychologia, 49(6), 1578-1588.
Sperling, G. (1960). The information available in brief visual
presentations.Psychological monographs: General and applied, 74(11), 1.
Tolar, T. D., Lederberg, A. R., Gokhale, S., & Tomasello, M. (2008). The
development of the ability to recognize the meaning of iconic signs. Journal of Deaf Studies and Deaf Education, 13(2), 225-240.
Vandenbroucke, A. R., Sligte, I. G., & Lamme, V. A. (2011). Manipulations of attention dissociate fragile visual short-term memory from visual working memory. Neuropsychologia, 49(6), 1559-1568.
Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748-751.
Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions, and objects in visual working memory. Journal of Experimental Psychology: Human Perception and Performance, 27(1), 92.
