Rafał Szymanek
MSc Brain and Cognitive Sciences, 2016 University of Amsterdam
student number: 11119438 ECTS points: 32
under the supervision of Ilja Sligte, PhD
Dissociating sensory memory from working
memory
Little is currently known about the purpose and representational content of iconic memory (IM), an early sensory stage of visual short-term memory (VSTM). Here we propose that it serves as a gateway between sensory data and higher-level models of the environment, applying compression to incoming stimuli to compare them to the predicted model. Furthermore, we propose that it is a feature-based system, invariant to qualities of the object composed of those features, unlike higher stages of VSTM which are object-based. We conducted two experiments. Experiment 1 confirmed that IM is a feature-based system. Experiment 2 investigated the mechanism of compression in IM but did not yield clear results. Our findings provide new insights into the nature of iconic memory and allow to make inferences about the dissociation of sensory and working memory on a behavioural and neural level.
Introduction
Levels of visual processing
If you closed your eyes and tried to recall what you have just seen, you would find that just after doing so you have a vivid representation of the visual scene that subsides very rapidly. This is iconic memory (IM)— a visual short-term memory (VSTM) mechanism that corresponds to the earliest stages of visual processing. It is characterised by brief storage duration (less than 500ms) and high storage capacity (Sperling, 1960; Neisser, 1967). IM is classically distinguished from visual working memory (WM), a system with longer storage duration and lower capacity, typically about four objects (Cowan, 2001; Luck & Vogel, 1997).
Our understanding of visual processing has progressed significantly over the years and has benefited from the development of neuroimaging techniques. Recent studies have investigated the possibility of separate neural substrates underlying the different stages of visual processing: while WM operates on a widespread network involving prefrontal, parietal and possibly occipital regions (Linden et al., 2003; Pessoa, Gutierrez, Bandettini, & Ungerleider, 2002; Riley & Constantinidis, 2016; Sligte, van der Leij, Shapiro, & Scholte, 2014), IM seems to be mostly confined to early visual areas (Sligte et al., 2014). Recently, a third, intermediate level of visual processing has been described that stores sensory information in a fragile format over a timespan far surpassing IM (Landman, Spekreijse, & Lamme, 2003; Sligte, Scholte, & Lamme, 2008; Pinto, Sligte, Shapiro, & Lamme, 2013). It has been termed fragile visual
short-term memory or fragile memory (FM). FM can be distinguished from WM due to much higher
capacity and its available timespan far surpasses that of IM.
Different levels of VSTM can be studied in a typical change detection task. It consists of presenting a memory array containing a set of items which the observer is asked to remember. Shortly afterwards, a test array is shown and the task is to report whether it is identical to the memory array. It is possible to dissociate between different levels of VSTM by cueing one of the items at a specific point in time. Without the cue, observers can typically remember up to four items, the classical limit of WM (Cowan, 2001). However, when a cue directing attention to one of the items is introduced in between the two displays, performance increases, suggesting that even after the disappearance of the first display, it is possible to access and consciously report on the items (Lamme, 2003). This so-called retro-cue probes IM when presented shortly after the offset of the memory array and FM when presented after IM has decayed (up to 4s after offset; Sligte et al., 2008).
Using the cued change-detection paradigm, it has been possible to disentangle the three forms of VSTM: WM with a capacity of about 2-4 items, FM with a capacity that is twice that of WM, and IM with virtually unlimited capacity (Sligte et al., 2008). Furthermore, manipulating attention has been found to affect WM but not FM (Vandenbroucke, Sligte, & Lamme, 2011). A different neural substrate for these two systems has also been suggested as inducing magnetic stimulation to the prefrontal cortex reduces the capacity of WM but not FM (Sligte, Wokke, Tesselaar, Steven Scholte, & Lamme, 2011). According to a model developed at our lab (Sligte, Vandenbroucke, Scholte, & Lamme, 2010), all three forms of VSTM depend on recurrent processing (see Lamme, 2003) but involve different regions of the brain: IM operates on early visual levels (V1-V3), FM involves higher visual levels (V4/IT) and WM involves frontal and parietal regions (SPL, FEF, DLPFC).
Despite over 50 years since its discovery, there is currently little information about the nature and purpose of IM (Keysers, Xiao, Foldiak, & Perrett, 2005). In the present study we decided to investigate the representational content of IM and its interactions with the rest of the visual processing system. Our findings contribute to a coherent theory of visual processing and awareness.
Iconic memory as a predictive gateway system
We propose that iconic memory fits in the framework of predictive coding. Predictive coding is a theoretical approach to understanding the brain according to which the brain is constantly making predictions regarding incoming information and comparing them to the input received. In visual processing, this approach posits that sensory input is compared to predictions received from higher areas and the difference between sensory data and predictions (called prediction error) is sent to higher levels of processing (Rao & Ballard, 1999; Friston & Kiebel, 2009). Our proposition is that IM acts as a gateway system that dictates the content entering WM. The short duration of IM is a consequence of the constant updating of predictions based on the received prediction error which is influenced by sensory data. In this view, IM is a predictive system relating sensory inputs to a model of the visual scene generated in higher levels of visual processing.
Rensink (2014) proposed that IM may act as a link between the changing retinotopic image and the higher-level representation of an object. We believe that to achieve this, IM needs a mechanism of compressing or filtering incoming information. There is evidence that memory content can undergo compression in order to store information more efficiently by taking into account redundancies in the input (Brady, Konkle, & Alvarez, 2009). This method of processing might be useful in daily life experiences where one stimulus often signals the occurrence of another, but has proven difficult to study in the lab. Although chunking (i.e. grouping of information) in WM has been studied extensively (Cowan, 2001), compression of information in early stages of visual processing has not been tackled so far.
It has been found that IM storage is more effective when the colours used for stimuli and background highly contrast with each other (Sligte et al., 2008), suggesting that retinal after-image contributes to capacity. Indeed, light masks flashed on the entire screen can counteract this increase in performance (Sligte et al., 2008). Nonetheless, IM is not merely a product of retinal after-images since IM capacity is still higher than FM, even when the mask is present or isoluminant stimuli are used. Furthermore, retinal processing cannot by itself explain IM performance (Keysers et al., 2005). This suggests that IM is based on a representation of the stimuli in the brain.
The primary visual cortex is known to contain edge-detecting cells selectively responding to basic features of the visual field such as oriented lines (Hubel & Wiesel, 1968). Given that and the preliminary evidence that IM corresponds to activity in early visual areas, we predict that the content of IM is pre-categorical in nature and the basic unit of processing is a visual feature. WM has been found to be object-based (Olivers, Meijer, & Theeuwes, 2006; Heuer & Schubö, 2016), i.e. operating on a whole object as the basic unit of processing, and similar evidence has recently been obtained for FM (Pinto et al., 2013). Thus, confirming our prediction that IM is featured-based in nature would provide further evidence for the dissociation between IM and FM.
We hypothesize that IM is a feature-specific store, as opposed to higher-level memory systems which are object-specific. Secondly, we hypothesize that there is compression of sensory data in IM which allows to quickly retrieve information from the visual stimuli to send it to higher levels of visual processing. We
conducted two experiments to test these predictions. Experiment 1 aimed to establish if IM is feature-based and FM is object-feature-based. We used a change-detection task with cues that were either identical to the target or only shared common features. We predicted that cues sharing features with the target would have a masking effect on the target in IM but not in FM, whereas the identical cue would affect FM. The results confirmed our predictions.
Experiment 2 was conducted to evaluate different hypotheses concerning information compression in IM. We used a change-detection task in which the targets appeared inside of placeholders which looked like frames surrounding the location of the target. Firstly, we predicted that IM stores less precise representations than WM and, consequently, the presence of frames would selectively decrease IM performance due to interference. Secondly, we tested two competing hypotheses. The individual filtering hypothesis predicted that each item in IM is stored separately and filtering occurs by applying a form of compression to each item before sending it further. Therefore, presenting stimuli in a way that facilitates isolating the target from surrounding placeholders should increase performance. The competing,
summary statistics hypothesis, predicted that the whole visual scene is stored in IM in a coherent manner
and compression occurs by means of computing summary statistics of the entire scene. Therefore, performance should be higher when the stimuli are presented in a way that facilitates perceiving them as a whole. Our results confirmed the first hypothesis. As for the alternative hypotheses, the individual filtering idea did not hold ground while the results for the summary statistics hypothesis were promising but not entirely clear.
Experiment 1
Methods
23 psychology students (6 male, mean age = 20.1, SD = 2.0) from the University of Amsterdam participated in the study in exchange for course credit. They were subjected to an eye test to ensure normal or corrected to normal vision. Subjects gave written informed consent before the experiment, which was approved by the local ethics committee.
Stimuli were presented on a 23' Asus LCD monitor (type VG236HE), at a refresh rate of 120 Hz, using Presentation version 18.2 (NeuroBehavioral Systems, Inc.). Participants were seated on a leather armchair equipped with response buttons on the armrests at the distance of 57cm in front of the screen (total viewing angle 37.7° × 37.7°).
All stimuli were presented on a black background with a fixation cross in the centre. The memory and test displays comprised eight white rectangles (2.7° × 0.6°) which were placed radially around the fixation cross (6° eccentricity). The rectangles had four different orientations (0°, 45°, 90° and 135°). At most three rectangles of the same orientation were included in each display.
On each trial, the red fixation cross turned green for 1000ms to indicate the start of the trial. Then a memory display was presented for 250ms. One rectangle was cued to indicate which item was the one to report. The cue was always valid. After a retention interval of 1750-2000ms, a test display was shown and subjects were asked to indicate whether the cued item changed or not. The number of change and no-change trials in each experimental block was equal and the trials were randomised. If a no-change occurred, the new rectangle always had an orientation perpendicular to the old one. The test display was presented for 4000ms or until the participant made a response. Figure 1 shows the experimental design. Participants
were instructed to respond “change” only when they were sure of their response and “no-change” otherwise. Participants were given auditory feedback in the form of a high-pitch tone for a correct response and a low-pitch tone for an incorrect response.
The cue was shown either 10ms after offset of the memory display (IM), 1000ms after offset of the memory display (FM) or 100ms after onset of the test display (WM). The cue was presented on the screen for 500ms in each case. There were four different cue types: a set of four small triangles surrounding the item (unrelated cue), a circle (dissimilar cue), a plus sign composed of two perpendicular rectangles (similar cue) or a rectangle (identical cue). The similar cue had an orientation of 22.5° or 112.5°. The rectangle had one of four orientations: 22.5°, 67.5°, 112.5° or 157.5°. WM trials were included to estimate WM capacity, therefore only the unrelated cue was used. Figure 2 shows the different cue types used in the experiment.
Figure 1 – the design of Experiment 1. Three memory conditions were used: (top) iconic memory (IM) using an early retro-cue presented 10ms after offset of the memory display; (middle) fragile memory (FM) using a late retro-cue presented 1000ms after
offset of the memory display; and (bottom) working memory (WM) using a post-cue presented 100ms after onset of the test display. The retention time was between 1750ms and 2000ms.
Figure 2 – cues used in Experiment 1
The experiment was carried out over two separate days, each lasting two hours. On the first day, participants performed a training and one experimental session. On the second day they performed two experimental sessions. One session comprised 9 blocks of 54 trials each. There was a 1 minute break between the blocks and a 10-15 minute break between the two sessions. During the break, participants
were shown a screen displaying their performance in the previous block and the average performance in the entire session. The training comprised 5 blocks of 54 trials each (24 IM and FM trials each, and 6 WM trials). It was identical to the experimental task, but only the unrelated cue was used.
Participants were required to score at least 75% on average in the training in order to be qualified for the experiment. This was intended to exclude unmotivated and unable participants. In the breaks between the training blocks, the screen additionally displayed a message saying either “You are doing great!” or
“You need to perform better!” depending on the average score. Three participants were excluded from
the experiment because they didn’t pass the training threshold.
Results and discussion
We assessed the capacity of iconic, fragile and working memory using the capacity estimate K (Cowan, 2001), computed as: 𝐾 = (ℎ𝑖𝑡 𝑟𝑎𝑡𝑒 − 𝑐ℎ𝑎𝑛𝑐𝑒 + 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑟𝑒𝑗𝑒𝑐𝑡𝑖𝑜𝑛 − 𝑐ℎ𝑎𝑛𝑐𝑒) × 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒. Statistical analyses were performed with repeated measures ANOVAs and permutation testing (bootstrapping) over 50 000 iterations, as well as paired t-tests for specific condition pairs. The significance level was set at 0.05. Matlab R2015a, JASP 0.7.5.6 and Microsoft Excel 2013 were used to analyse the data.
A repeated-measures ANOVA using memory (IM and FM) and cue type as factors revealed a significant effect of memory (F(1,19) = 7.876, p = 0.011, η2 = 0.293) and cue type (F(3,57) = 65.992, p < 0.001, η2 = 0.776) as well as memory × cue type interaction (F(3,57) = 16.008, p < 0.001, η2 = 0.457). Mauchly's test of sphericity indicated that the assumption of sphericity had not been violated for cue type nor cue type × memory (respectively: W(5) = 0.575, p = 0.081; W(5) = 0.913, p = 0.899), therefore no correction was used.
The estimated capacity in each condition is presented inTable 1and Figure 3.
cue type
unrelated dissimilar similar identical
memory
IM 5.44 7.01 5.06 4.27
FM 5.89 6.32 5.90 4.94
WM 3.48 3.48 3.48 3.48
Figure 3 – estimated capacity of iconic (IM), fragile (FM) and working memory (WM) calculated as 𝐾 = (ℎ𝑖𝑡 𝑟𝑎𝑡𝑒 − 𝑐ℎ𝑎𝑛𝑐𝑒 +
𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑟𝑒𝑗𝑒𝑐𝑡𝑖𝑜𝑛 − 𝑐ℎ𝑎𝑛𝑐𝑒) × 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒 for each experimental condition.
Capacity in the unrelated condition was assumed to be the baseline, maximum capacity for each memory condition. We predicted no difference between the unrelated and dissimilar conditions. However, performance in the unrelated condition was lower than in the dissimilar condition in both IM and FM (T(19) = -6.241, p < 0.001, d = -1.396 and T(19) = -3.267, p = 0.004, d = -0.731, respectively). A possible explanation for this finding is that the unrelated cue could be perceived as an illusory square (or possibly diamond) similar to a Kanizsa figure (see Kanizsa, 1976 for an example). Thus, it would function as a mask, similarly to the similar and identical cues, but to a lesser degree because it did not actually occupy the same location as the target. Because of this, the performance in the dissimilar conditions was taken as the baseline for subsequent comparisons.
Firstly, we predicted a decrease in IM but not FM capacity in the similar condition. Planned comparisons using paired t-tests revealed a significant difference between dissimilar and similar conditions for both IM (T(19) = 7.830, p < 0.001, d = 1.751) and FM (T(19) = 2.392, p = 0.027, d = 0.535), the effect size was larger for IM. Bootstrapping revealed significant difference for IM (1.95, p < 0.001) but not for FM (0.42, p = 0.184). Although the results of planned comparisons did not match our predictions, if we take into account the effect sizes and the results of bootstrapping, we can conclude that IM capacity was decreased by the similar cue and that FM capacity decrease was not reliably significant and smaller than for IM.
Secondly, we predicted that the identical condition will override FM, causing a decrease in performance for both the FM and IM condition. The difference was significant for both IM (T(19) = 3.891, p < 0.001, d = 0.870) and FM (T(19) = 6.489, p < 0.001, d = 1.451), the effect size was larger for FM. Bootstrapping revealed significant difference for FM (0.95, p = 0.014) but not for IM (0.80, p = 0.097). The results are in line with our predictions.
We predicted IM would be completely erased by the similar condition which would result in a capacity drop to FM level. IM capacity in the similar condition dropped below FM capacity (T(19) = -4.978, p < 0.001, d = -1.113). IM capacity was also lower than FM in the identical condition (T(19) = -3.392, p = 0.003,
0 1 2 3 4 5 6 7 8
unrelated dissimilar similar identical
cap acity (Cowan 's K) IM FM WM
d = -0.759), but not in the dissimilar (T(19) = 5.240, p < 0.001, d = 1.172; IM capacity was higher) or unrelated (T(19) = -1.912, p = 0.071) conditions. A repeated-measures ANOVA run with only those two cue types (similar, identical) and memory (IM, FM) as factors revealed no significant effect of interaction memory × cue type (F(1,19) = 0.519, p = 0.480). This lack of specificity in IM-FM capacity difference suggests that something specific to the IM condition caused lower performance. A possible explanation for that observation is that the short interval (10ms) between the offset of the memory display and the onset of the cue used in this experiment facilitated object substitution masking (Woodman & Luck, 2003), making it difficult for participants to separate the memory array from the cue. Given that this effect was only observed for the similar and identical conditions, this explanation seems plausible, as these cues were more likely to be mistaken for the target.
We predicted FM to be completely erased by the identical condition causing a drop to WM level. FM capacity in the identical condition was significantly higher than WM capacity (T(19) = 6.632, p < 0.001, d = 1.483). We conclude that even though FM capacity is significantly decreased by identical cues, it is not erased completely.
The results obtained in Experiment 1 are generally in line with our prediction that IM is a feature-based system and FM is object-based, the latter being a confirmation of a finding by Pinto et al. (2013).
Experiment 2
Methods
22 subjects (9 male, mean age 21.1, SD = 2.4) participated in the study for course credit or monetary reward (3 participants). They were subjected to an eye test to ensure normal or corrected to normal vision. Subjects gave written informed consent before the experiment, which was approved by the local ethics committee. One participant was excluded because of sub-threshold performance on the eye test. The same equipment as in Methods was used, except participants were seated on a standard armchair and the response was provided through a keyboard.
All stimuli were presented on a white background with a fixation cross in the centre. The memory and test displays comprised eight coloured line drawn objects (3.5° × 2.5°) taken from a set of 260 objects created by Rossion and Pourtois (2004). They were placed radially around the fixation cross (11.5° eccentricity). Eight different pictures were selected randomly for each trial.
The procedure was identical to Experiment 1, except the memory display was shown for 500ms. When a change occurred, the picture was always replaced by a new one from the set so that none of the presented displays contained two identical pictures.
The cue was shown either 10ms after offset of the memory display (IM) or 100ms after onset of the test display (WM). The cue was presented on the screen for 500ms in each case and consisted of an oriented thin white line pointing from the centre to the cued item. The two experimental conditions are shown in Figure 4.
In some conditions, the items were surrounded by placeholders which were presented on the screen throughout the whole duration of the trial. These placeholders took a form of either frames that directly outlined the item (small), frames that outlined the item with a margin of about 0.45° (large), semi-frames consisting of two perpendicular lines facing the centre of the display (inner) or facing away from the centre
(outer). The semi-frames were equal in size (and margin) as the small frames. The different frames are shown in Figure 5.
Figure 4 — two experimental conditions: (top) iconic memory (IM) using an early retro-cue presented 10ms after offset of memory display; (bottom) working memory (WM) using a post-cue presented 100ms after onset of test display. The retention
time was 2000ms.
Figure 5 – frames used in Experiment 2.
The experiment was carried out on one day and consisted of a training and two experimental sessions. One session comprised 10 blocks of 40 trials each. There was a 1 minute break between the blocks and a 10-15 minute break between the two sessions. During the break, participants were shown a screen displaying their performance in the previous block and the average performance in the entire session. The training comprised 5 blocks of 54 trials each (18 trials for IM, FM and WM each). It was identical to the training in Experiment 1 with the modification of using an oriented line as the unrelated cue. The whole experiment lasted two and a half hours.
Participants were required to score at least 70% on average in the training in order to be qualified for the experiment. The different threshold compared to Experiment 1 was dictated by including more WM trials to balance the three conditions, thus making the task more difficult on average. In the breaks between the training blocks, the screen additionally displayed a message saying either “You are doing great!” or
“You need to perform better!” depending on the average score.
Two participants were excluded from analysis because their IM performance was below average by more than two standard deviations. One participant was excluded because upon debriefing they admitted to not having used the cue. The data of the remaining 17 participants was subsequently analysed.
Results and discussion
Data analysis was performed in the same way as in Experiment 1, except bootstrapping was performed over 100 000 iterations.
We hypothesised the presence of frames to have a more pronounced, detrimental effect on capacity of iconic memory than on working memory.
Participants could report on average 5.16 items in the IM condition and 2.29 in the WM condition. A repeated-measures ANOVA yielded a statistically significant main effect of memory (F(1,16) = 253.69, p < 0.001, η2 = 0.941) and frame (F(4,64) = 18.27, p < 0.001, η2 = 0.533). There was a significant effect of interaction frame × memory (F(4,64) = 3.59, p = 0.011, η2 = 0.183). Mauchly's test of sphericity indicated that the assumption of sphericity had not been violated for frame nor frame × memory (respectively: W(9) = 0.722, p = 0.862; W(9) = 0.473, p = 0.293), therefore no correction was used.
The performance and capacity in each condition are presented in Table 2.
frame
none large small inner outer
memory
IM 6.29 (87%) 4.58 (76%) 4.64 (77%) 4.88 (79%) 5.40 (81%) WM 2.65 (67%) 2.08 (62%) 2.20 (64%) 2.36 (64%) 2.21 (63%) Table 2 — estimated capacity (Cowan’s K) and performance (% correct) for each experimental condition
To evaluate whether our manipulation selectively affected iconic memory, we tested the interaction between frame and memory. We used bootstrapping over 100 000 iterations with each participant’s estimated capacity as input. Bootstrapping revealed significant differences in the average performance between the no-frame and each of the frame conditions in IM (1.72, p < 0.001; 1.66, p < 0.001; 1.41, p < 0.001; 0.89, p = 0.002 for large-, small-, inner- and outer-frame conditions, respectively). For WM, only the large-frame condition differed significantly from the no-frame condition (0.59, p = 0.0145). The small-, inner- and outer-frame conditions did not significantly differ from the no-frame condition (0.45small-, p = 0.0724; 0.29, p = 0.3466; 0.45, p = 0.1644, respectively). We conclude that frames selectively decreased IM capacity.
We then tested our competing hypotheses. The individual filtering hypothesis stated that IM capacity should be higher in the large-frame than in the small-frame condition. The summary statistics hypothesis stated that IM capacity should be higher in the outer-frame than in the inner-frame condition.
A Shapiro-Wilk test was used to test for normality on each of the IM conditions. For all four frame conditions the test yielded results which were not statistically significant (p = 0.2805, p = 0.7787, p = 0.1616, p = 0.7144 for the large-, small-, inner- and outer-frame conditions, respectively). For the no-frame condition, the result was marginally non-significant (p = 0.0506).
Planned comparison using one-directional paired t-tests revealed no significant differences in capacity between large and small frame conditions (T(16) = -0.284, p = 0.3990, d = 0.14). A trend was observed in capacity difference between inner and outer frame conditions but this difference did not reach statistical significance (T(16) = 1.684, p = 0.0557, d = 0.84).
Bootstrapping revealed no significant differences between the large-frame and small-frame conditions for IM (0.06, p = 0.8874). Similarly, no significant differences have been found between the inner-frame and outer-frame conditions (0.52, p = 0.698). However, IM capacity in the outer-frame condition was significantly higher than both the large-frame (difference: 0.82; T(19) = 3.120, p = 0.003; bootstrap: p = 0.03) and small-frame (difference: 0.76; T(19) = 3.151, p = 0.003; bootstrap: p = 0.03) conditions. There were no significant differences between the different frame conditions for WM.
Figure 6 – estimated capacity of iconic (IM) and working memory (WM) calculated as 𝐾 = (ℎ𝑖𝑡 𝑟𝑎𝑡𝑒 − 𝑐ℎ𝑎𝑛𝑐𝑒 +
𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑟𝑒𝑗𝑒𝑠𝑡𝑖𝑜𝑛 − 𝑐ℎ𝑎𝑛𝑐𝑒) × 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒 for each experimental condition.
The individual filtering hypothesis did not hold ground. Interpreting the results for the summary statistics hypothesis was less clear. Even though planned comparisons and bootstrapping did not reveal significant results, if one takes into account that the outer-frame condition was the only frame condition facilitating
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00
none large small inner outer
Cap aci ty (Co wa n 's K) frame IM WM
perceiving the items as a whole scene, then the significant differences between the outer and both the large and small conditions for IM (and no differences for WM) can be interpreted as support for the summary statistics hypothesis. The lack of difference between the inner and outer conditions could be a result of higher performance in the inner-frame condition due to a smaller number of features in the stimuli compared to both large- and small-frame conditions as the inner frame consisted of two oriented lines while the large and small frames consisted of four. Keeping in mind the results of Experiment 1 which showed that IM is feature-based, this explanation seems plausible. Further research with modifications in the task is thus required to investigate this possibility.
Furthermore, using a median split to separate low and high performers revealed a pattern of results that suggests the observed trend for inner-outer difference in IM was driven by low performing participants, whereas capacity across conditions for high performing participants was rather flat. Figure 7 shows the pattern of results for both groups. This suggests that the effect might be masked by the task being too easy for some more skilled participants. Modifying the task to make it more difficult in general or changing the procedure so that the experimental task difficulty adapts to each participant’s training performance, e.g. by increasing/decreasing the number of stimuli presented (for an example, see: Vandenbroucke et al., 2014), could make the effects more pronounced.
Figure 7 – estimated iconic memory capacity for high and low performers calculated as 𝐾 = (ℎ𝑖𝑡 𝑟𝑎𝑡𝑒 − 𝑐ℎ𝑎𝑛𝑐𝑒 +
𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑟𝑒𝑗𝑒𝑠𝑡𝑖𝑜𝑛 − 𝑐ℎ𝑎𝑛𝑐𝑒) × 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒 for each experimental condition. The groups were separated by a median split based
on average capacity across all experimental conditions.
General discussion
In Experiment 1 we used a change detection task to estimate the capacity of iconic, fragile and working memory. Different cues were employed in order to selectively override one of those visual short-term memory storage systems. As predicted, a cue sharing features with the target stimulus decreased IM
0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00
none large small inner outer
Cap
acity
(Cowan
's
K)
IM capacity of low vs. high performers
performance while affecting FM performance to a lesser degree. A cue identical to the target object, on the other hand, produced more pronounced and more reliable decrease in performance for FM than for IM. Thus, we provide new evidence that supports the notion of IM as a feature-based system and confirms the findings of Pinto et al. (2013), who argue that FM is an object-based system. These results give further credence to the hypothesis of a fragile VSTM store as a separate sensory mechanism.
Experiment 2 was conducted to evaluate different hypotheses regarding information compression in IM. By placing the stimuli in surrounding frames we were able to selectively disrupt IM performance leaving WM intact. Our results support the idea of compression occurring in early stages of visual processing. While the exact mechanisms guiding that process are not known, we conjecture that this might happen by neurons in early visual areas integrating input from multiple sources across larger receptive fields. This kind of coarse information would be sufficient to rapidly compare the incoming sensations to predictions coming from higher areas. Thus, IM could be a gateway system relating the visual input to a model of the visual scene generated at higher-level areas, propagating a prediction error and so acting as a link between the changing retinotopic image and the higher-level representation of an object, in line with propositions put forward by Rensink (2014) and with the body of research in the field of predictive coding (Friston & Kiebel, 2009; Rao & Ballard, 1999). Fine, detailed representations of the object in the visual scene would be formed more slowly, at the level of WM.
We tested whether the mechanism of information compression in IM could be better explained by individual filtering of single items or by computing summary statistics of the entire visual field. We used different kinds of frames that either facilitated or impeded processing the items individually or as a whole scene. We found that the individual filtering could not explain the obtained results as the size of the frame, which was meant to influence the ability to isolate an individual object, had no effect on performance. The summary statistics hypothesis turned out more promising. A semi-frame positioned on the outside of a display was used to facilitate processing the items as a whole scene. The performance in this condition was indeed higher than when full frames were used but no reliable difference was found when compared with a semi-frame positioned on the inside of the display. Importantly, the size and position of the frame was irrelevant for WM performance. The results, albeit not entirely clear, suggest a possibility that compression in IM takes place over the entire visual scene. In this view, coarse information about the visual field would be obtained in a form of summary statistics and compared with a higher-level model. Alternative explanations need to be considered. Given that IM is feature-based, which was confirmed by Experiment 1, one can argue that each line composing the frames is actually processed individually. In this view, one would expect decreased performance in full-frame conditions simply due to a larger number of features that require processing. This explanation on its own cannot account for the lack of difference between the inner semi-frame and the full-frames. However, with an extra assumption, namely that observers might only pay attention to a central part of the display (and, consequently, the items), one can argue that the outside parts of the frames are not processed to the same extent as the parts on the inside. Given these assumptions, one would expect a similar difference between no-frame—outer-frame and inner-frame—small-frame as in both cases, the difference between the conditions is equal. This, however, is not the case, as the difference between the no-frame and the outer-frame conditions is large (T(19) = 4.797, p < 0.001, d = 2.398) while the difference between the inner-frame and the small-frame conditions is not significant (T(19) = 0.860, p = 0.201).
One could further argue that the number of features is not the only difference between the conditions, as in all frame conditions the frames stayed on the screen across the entire duration of a trial, which might have made it more difficult to focus on the appearing stimuli. This explanation cannot be ruled out given our setup, therefore a modification in the task in which the frames only appear along with the items would be needed to ascertain this possibility. This alternative explanation could further be evaluated by designing an experiment, in which the display would be located in one quarter of the screen, chosen randomly in each trial. If the alternative explanation is true and central parts of the screen are processed in a more robust way, then we would expect higher performance for items located close to the central fixation cross than for items far from the centre.
Future research should focus on improving the task to get rid of confounding factors mentioned above. Furthermore, it would be of interest for evaluating our theoretical framework to analyse the influence of the relation between the changed and the original picture on performance across memory conditions. The set of line drawn objects used in Experiment 2 contained items that could be distinguished on many dimensions such as colour and orientation (low-level features), as well as semantic category (animal, food, tool etc.) and familiarity (high-level features). We predict that for IM, a change in low-level features produces increased performance and a change in high-level features has no effect on performance, while for WM we predict the exact opposite. Additionally, future neuroimaging studies on neural correlates of IM is needed. While we based our predictions on preliminary data linking IM with activity in early visual areas (Sligte et al., 2014) and our findings are in line with this hypothesis, additional neural evidence is required to confirm these claims.
References
Brady, T. F., Konkle, T., & Alvarez, G. A. (2009). Compression in visual working memory: Using statistical regularities to form more efficient memory representations. Journal of Experimental Psychology:
General, 138(4), 487–502. http://doi.org/10.1037/a0016797
Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage
capacity. Behavioral and Brain Sciences, 24(01), 87–114.
http://doi.org/10.1017/S0140525X01003922
Friston, K., & Kiebel, S. (2009). Predictive coding under the free-energy principle. Philosophical
Transactions of the Royal Society of London. Series B, Biological Sciences, 364(1521), 1211–21.
http://doi.org/10.1098/rstb.2008.0300
Heuer, A., & Schubö, A. (2016). Feature-based and spatial attentional selection in visual working memory.
Memory & Cognition. http://doi.org/10.3758/s13421-015-0584-5
Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex.
The Journal of Physiology, 195(1), 215–43.
http://doi.org/papers://47831562-1F78-4B52-B52E-78BF7F97A700/Paper/p352
Kanizsa, G. (1976). Subjective contours. Scientific American, 234(4), 48–52.
Keysers, C., Xiao, D.-K., Foldiak, P., & Perrett, D. I. (2005). Out of sight but not out of mind: the neurophysiology of iconic memory in the superior temporal sulcus. Cognitive Neuropsychology,
22(3), 316–32. http://doi.org/10.1080/02643290442000103
Lamme, V. A. F. (2003). Why visual attention and awareness are different. Trends in Cognitive Sciences,
7(1), 12–18.
Landman, R., Spekreijse, H., & Lamme, V. A. F. (2003). Large capacity storage of integrated objects before change blindness. Vision Research, 43(2), 149–164. http://doi.org/10.1016/S0042-6989(02)00402-9 Linden, D. E. J., Bittner, R. A., Muckli, L., Waltz, J. A., Kriegeskorte, N., Goebel, R., … Munk, M. H. J. (2003).
Cortical capacity constraints for visual working memory: Dissociation of fMRI load effects in a
fronto-parietal network. NeuroImage, 20(3), 1518–1530.
http://doi.org/10.1016/j.neuroimage.2003.07.021
Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions.
Nature, 390(6657), 279–281. http://doi.org/10.1038/36846
Neisser, U. (1967). Cognitive Psychology. Thinkingjudgement and decision makin.
http://doi.org/10.1126/science.198.4319.816
Olivers, C. N. L., Meijer, F., & Theeuwes, J. (2006). Feature-based memory-driven attentional capture: visual working memory content affects visual attention. Journal of Experimental Psychology: Human
Perception and Performance, 32(5), 1243–1265. http://doi.org/10.1037/0096-1523.32.5.1243
Pessoa, L., Gutierrez, E., Bandettini, P. a., & Ungerleider, L. G. (2002). Neural Correlates of Visual Working Memory. Neuron, 35, 975–987. http://doi.org/10.1016/S0896-6273(02)00817-6
Pinto, Y., Sligte, I. G., Shapiro, K. L., & Lamme, V. A. F. (2013). Fragile visual short-term memory is an object-based and location-specific store. Psychonomic Bulletin & Review, 20(4), 732–739. http://doi.org/10.3758/s13423-013-0393-4
Rao, R. P. N., & Ballard, D. H. (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1), 79–87. http://doi.org/10.1038/4580
Rensink, R. A. (2014). Limits to the usability of iconic memory. Frontiers in Psychology, 5(AUG), 1–9. http://doi.org/10.3389/fpsyg.2014.00971
Riley, M. R., & Constantinidis, C. (2016). Role of Prefrontal Persistent Activity in Working Memory.
Frontiers in Systems Neuroscience, 9. http://doi.org/10.3389/fnsys.2015.00181
Rossion, B., & Pourtois, G. (2004). Revisiting Snodgrass and Vanderwart’s object pictorial set: The role of surface detail in basic-level object recognition. Perception, 33(2), 217–236. http://doi.org/10.1068/p5117
Sligte, I. G., Scholte, H. S., & Lamme, V. A. F. (2008). Are There Multiple Visual Short-Term Memory Stores?
PLoS ONE, 3(2), e1699. http://doi.org/10.1371/journal.pone.0001699
Sligte, I. G., van der Leij, A. R., Shapiro, K. L., & Scholte, H. S. (2014). A link between brain structure/connectivity and visual short-term memory capacity. Journal of Vision, 14(10), 170–170. Sligte, I. G., Vandenbroucke, A. R. E., Scholte, H. S., & Lamme, V. A. F. (2010). Detailed sensory memory,
sloppy working memory. Frontiers in Psychology, (OCT). http://doi.org/10.3389/fpsyg.2010.00175 Sligte, I. G., Wokke, M. E., Tesselaar, J. P., Steven Scholte, H., & Lamme, V. A. F. (2011). Magnetic
stimulation of the dorsolateral prefrontal cortex dissociates fragile visual short-term memory from
visual working memory. Neuropsychologia, 49(6), 1578–1588.
http://doi.org/10.1016/j.neuropsychologia.2010.12.010
Sperling, G. (1960). The information available in brief visual presentations. Psychological Monographs:
General and Applied. http://doi.org/10.1037/h0093759
Vandenbroucke, A. R. E., Sligte, I. G., Barrett, A. B., Seth, A. K., Fahrenfort, J. J., & Lamme, V. A. F. (2014). Accurate metacognition for visual sensory memory representations. Psychological Science, 25(4), 861–73. http://doi.org/10.1177/0956797613516146
Vandenbroucke, A. R. E., Sligte, I. G., & Lamme, V. A. F. (2011). Manipulations of attention dissociate fragile visual short-term memory from visual working memory. Neuropsychologia, 49(6), 1559–1568. http://doi.org/10.1016/j.neuropsychologia.2010.12.044
Woodman, G. F., & Luck, S. J. (2003). Dissociations among attention, perception, and awareness during object-substitution masking. Psychological Science, 14(6), 605–611. http://doi.org/10.1046/j.0956-7976.2003.psci_1472.x
