Influence of Subjective and Objective Invisible Distractors on Visual Short-Term Memory Jamie L. van Someren
University of Amsterdam
Author Note
Jamie L. van Someren, student Brain and Cognition, Department of Psychology, University of Amsterdam. Student number: 11056223.
This research was supervised by Timo Stein, Brain and Cognition group, Department of Psychology, University of Amsterdam. Correspondence concerning this article should be addressed to Jamie L. van Someren.E-mail: jamievansomeren@gmail.com
Abstract
A crucial question in consciousness research is whether working memory (WM) can be unconscious. Previous studies claimed that it can. Briefly presented stimuli, that participants reported not to have seen, still affected their visual short-term memory (VSTM). However, these studies ignored the fact that participants can be biased to report not to have seen stimuli, that were in fact weakly conscious instead of truly unconscious. The present study used signal detection theory (SDT) to obtain an objective measure of (un)consciousness of intervening distractors in a delayed cue-target orientation discrimination task. Participants had to keep the orientation of a memory cue (a grating) in mind. In 50% of the trials a distractor (a grating) was presented during this delay. The orientation of the distractor grating was either congruent to the memory cue or differed by 40 degrees. To promote variability in consciousness of the distractors, they were presented briefly and backward masked by either a uniform black mask or a circular grating mask. At the end of each trial a test probe grating was shown. Participants had to indicate whether this probe was tilted to the left or right relative to the memory cue and whether they had seen a distractor. VSTM accuracy and d' were calculated over 480 trials. While d’ objectively indicated that the grating mask made distractors invisible while the uniform mask did not, VSTM accuracy was not affected by either of them. It remains inconclusive whether conscious and unconscious distractors can differentially affect WM.
Keywords: consciousness, working memory, visual short-term memory, distractor interference, perception awareness scale, signal detection theory
Influence of Subjective and Objective Invisible Distractors on Visual Short-Term Memory Working memory (WM) refers to a limited capacity brain system that temporarily stores and manipulates information necessary for complex cognitive tasks; WM also provides a connection between perception, long-term memory and action (Baddeley, 1992; 2003). One of the components of the broader WM system is visual short-term memory (VSTM), which is a memory system where visual information is translated and stored into a more durable representation that is able to survive interruptions, like eye blinks or eye movements (Phillips, 1974; Luck, 2007). A new line of research examines whether processing into a durable representation needs to be conscious. Consciousness has different meanings and can relate to state of wakening, for example awake or asleep, or to conscious content. This study uses the latter description of consciousness, in which some of the available information can be accessed consciously (and is reportable), while other information that is processed simultaneously remains unconscious (Dehaene & Changeux, 2011). It is debated whether consciousness is also necessary for processing in VSTM. Traditionally, WM and consciousness are considered to be closely related: “Without memory no conscious sensation, without memory no consciousness” (Richet, 1886). Baddeley (2003) suggests that WM input and output are always accompanied by consciousness. Baars and Franklin (2003) claim that rehearsal, visuospatial operations, recall, and report of WM are also conscious. Baars’ (1998) global workspace theory (GWT) provides an alternative view on consciousness and states that consciousness is needed to retrieve unconscious networks that perform explicit functions (Baars & Franklin, 2003; Dehaene & Naccache, 2001). According to a different model, by Sligte and colleagues (2008), recurrent interactions between neurons in visual areas might result in consciousness: WM makes the information consciously accessible (Sligte, Scholte & Lamme, 2008; Lamme, 2006). All these views suggest a close relationship between WM and consciousness, but does WM really need to be conscious to modulate information processing?
Previously, it was even unclear if it was possible for unconscious information to be maintained over a period of time. However, Reber et al. (2012) observed a one-minute maintenance for unconscious information, using masked word pairs. Another study showed an even longer effect of 47 minutes (Gaillard et al., 2007). It has been an important line of research to examine if unconscious information can be operated by WM. One study of implicit learning effects showed it was possible for subjects to use their WM, without them being aware of using these operations (Hassin, 2013). Soto, Mantyla and Silvanto (2011) reported that subjectively invisible stimuli can be localized above chance by WM, even with a delay. And Silvanto and Soto (2012) showed that WM does not necessarily need to be conscious. They explored memory effects, and their paradigm included backward masking of a distractor, to render it invisible. Subjects had to keep the orientation of a grating in mind, and during the delay a backward masked distractor was sometimes present. After delay a memory probe was given, and participants had to decide whether the grating was tilted left or right compared to the first grating. To establish whether the distractor was invisible, a subjective measure was used. After each memory probe, subjects had to decide whether they saw the distractor, maybe saw it, or did not see it, which they indicated on the perception awareness scale (PAS). The PAS included the following scores: 1 = did not see it; 2 = maybe saw something; 3 = saw the stimulus but not its orientation; 4 = saw the stimulus and its orientation (Overgaard et al., 2010; Sandberg et al., 2010). Stimuli were regarded invisible if subjects assigned a PAS score of 1.
Subjective measures, like the PAS, have some disadvantages. It is possible that subjects claim they did not see the distractor, while they actually saw part of it or all of it. This decision is influenced by the criteria they set (Kunimoto, Miller & Pashler, 2001). For example: one subject may count a flickering as seeing something, while someone else may need to see a clear distractor before indicating to have seen something. These decisional criteria lead to the
problem that subjective measures do not exclude the possibility that stimuli in fact were weakly conscious (Snodgrass, Bernat & Shevrin, 2004).
To avoid bias due to the influence of decision criteria, and to ensure that distractors are invisible, objective measures are needed. These are not dependent on a decision criterion, but on a measure that assesses if someone performed correctly or incorrectly. One way of quantifying objective measures is provided by the signal detection theory (SDT) (Green & Swets, 1966). This theory uses accuracy to measure the ability of subjects to distinguish between a distractor being present or absent. For example, this can be assessed by comparing the hit rate and false alarm rate and calculating the sensitivity index (d’) from these two (Kunimoto, Miller & Pashler, 2001). In case of the PAS, a ‘hit’ would be defined as the distractor being present and responding PAS > 1; a ‘miss’ would be assigned when the distractor was present and a PAS = 1 response; a ‘false’ alarm would be assigned with the distractor being absent and a PAS > 1 response, and a correct rejection would be identified as the distractor being absent and a PAS = 1 response (Overgaard et al., 2010; Sandberg et al., 2010). In conclusion, bias related to decision criteria is eliminated by the SDT.
The present study compares interference from distractors rendered invisible according to both definitions: subjective and objective. This could provide new insights in the question whether WM needs to be conscious. This study uses two different ways of backward masking. The subjective measure tests the influence of a distractor on VSTM accuracy with a mask that is weak: it might be possible for the subject to see the distractor, replicating Silvanto and Soto’s study (2012). The objective measure tests the influence of a distractor on VSTM accuracy with a stronger mask, to ensure that the subject cannot see the distractor. This assigns a score that is independent of judgment. Under the unconscious WM model, interference is expected on both the subjective and objective level. Under the weakly conscious model, effects are only expected on the subjective level.
Method Participants
A total of 32 participants were enrolled in this study (25 females, mean age 21 years). They were recruited through lab.uva.nl and were all first year Psychology students at the University of Amsterdam. The study was approved by the Ethics Board of the Department of Psychology. Participants provided informed consent and received 1.5 Psychology Research Credit as an incentive.
Stimuli and experimental procedure
A monitor with display resolution of 1920 x 1080 pixels and refresh rate of 60 Hz was used to present stimuli. Stimuli and task were controlled by Matlab. Each trial started with a black fixation point that was presented in the middle of the screen (500 ms), followed by a blank screen (500 ms). Next, the memory cue, a Gabor grating (166 pixels in width and height, 4 cycles, 10% contrast), was presented for 300 ms, followed by a mask (83 ms). The grating contrast could be tilted 10, 20, 30, 40 or 50 degrees to the left or right from vertical. Participants were told to keep the orientation of the memory cue in mind. After another blank delay of 1500 ms, a distractor was presented for 17 ms on 66% of the trials. The distractor was another Gabor grating, and either had the same orientation as the memory cue (congruent) or had the opposite orientation to the memory cue (incongruent). The distractor was followed randomly by either a uniform mask (50% of trials) or a circular grating mask (50% of trials), presented for 83 ms. A blank delay (1500 ms) and fixation point (500 ms) preceded the memory probe (300 ms). The memory probe was a grating tilted either 10 degrees to the left or right, compared to the memory cue. Following a blank delay of 100 ms, participants had to indicate whether it was tilted left or right by pressing one of two buttons during an unlimited time period. In addition, participants had to indicate on the PAS whether they had seen the visual distractor, using the following
scores; 1 = did not see it; 2 = maybe saw something; 3 = saw the stimulus but not its orientation; 4 = saw the stimulus and its orientation (Overgaard et al., 2010; Sandberg et al., 2010).
Participants were shown a click-through version before starting the experiment and first completed 5 practice trials. The experiment consisted of 480 trials, and participants had to take a break of at least 20 seconds each time they completed 80 trials. Accuracy was measured with the memory probe test; cue orientation and congruency varied randomly for each trial. Figure 1 illustrates the timeline of one trial.
Figure 1. Timeline of an experimental trial. Examples are with distractor. Upper: circular grating mask. Lower: uniform mask.
Results
Three out of 32 participants were excluded due to chance level performance in the VSTM task; a total of 29 participants (22 females, mean age 21 years) were included. The criterion for chance level performance was set at a score lower than 60% (three standard deviations below the average of all observations) at the mean of one of the three variables: no distractor, congruent distractor, or incongruent distractor.
Distractor visibility
First of all, a manipulation check for distractor visibility was conducted, independently of orientation. The proportion of responses for distractor visibility for the uniform mask are plotted in Figure 2A. To measure perceptual sensitivity for the distractor, we used SDT (Green & Swets, 1966). As mentioned in the introduction, d prime (d’) can be calculated using the hit and false alarm rate (see Table 1 for definition). The higher d’, the more signal can be detected and conscious access to the stimulus is more likely; if d’ significantly differs from chance, participants are sensitive to the distractor.
The mean d’ for the uniform mask was 0.49 (SD = 0.69). An one sample t-test showed a significant difference from zero, t(28) = 3.839, p = .001, which means participants generally had more hits than false alarms. This indicates that the participants were on average able to distinguish the presence from absence of the distractor, showing that the distractor was not rendered objectively invisible.
Distractor visibility for the circular grating mask is plotted in Figure 2B. The mean d’ was -0.018 (SD = 0.35). An one sample t-test revealed that the d’ did not differ significantly from zero, t(28) = -0.282, p = .78, which means that participants generally did not differ in their hit and false alarm rate. This indicates that participants were on average not able to distinguish the presence from absence of the distractor, showing the mask objectively rendered the distractor highly invisible.
Comparing the d’ of the two different masks with a paired t-test, showed that the scores were significantly different, t(28) = 3.914, p = .001. This indicates that the perceptual sensitivity for the two masks was significantly different. Thus, the uniform mask is less effective in masking the distractor compared to the circular grating mask, which means that the manipulation was effective.
Table 1
Definition of hits and misses in the experiment, according to signal detection theory Distractor Present Distractor Absent
PAS > 1 Hit False Alarm
PAS = 1 Miss Correct Rejection
Note. The signal detection theory is from Green & Swets (1966). Participants had to indicate whether they saw the distractor on the perception awareness scale (PAS; Adapted from Overgaard et al., 2010; Sandberg et al., 2010). A score of PAS > 1 means participants responded with either ‘maybe saw something’, ‘saw the distractor but not its orientation’ or ‘saw the distractor and its orientation’. A score of PAS = 1 means the participants responded with ‘did not see it’.
Impact of visibility and distractor orientation on VSTM accuracy
Previously mentioned results showed that the uniform mask did not render the distractor objectively invisible, but the circular grating mask did. Thus, the uniform mask condition reflects visibility of the distractor, and the circular grating mask condition reflects invisibility.
This finding leaded to the main analysis, in which the effect of distractor orientation and visibility on VSTM accuracy for the separate masks was assessed. A repeated-measures ANOVA test was performed, with mask (uniform or circular grating) and orientation difference
in comparison to the orientation cue (no distractor, 0 or 40 degrees, i.e. congruent or incongruent) as main factors. Analysis showed that there was no effect of mask (i.e. visibility), F(1, 27) = 0.049 p = .83, or distractor orientation, F(2, 26) = 1.746, p = .18, and no interaction effect, F(2, 26) = 0.091, p = .91. (See Figure 3 for a plot of distractor interference for the different masks). Thus, both orientation and visibility did not affect VSTM accuracy.
To examine if the significant results of Silvanto and Soto (2012) and Bona et al. (2013) on the distractor could be replicated, a separate repeated-measures ANOVA factor was carried out for the uniform mask. Opposite to the findings of the previous mentioned studies, there was no effect of the distractor, F(2, 26) = 0.555, p = .58.
Finally, we assessed if the separate analysis on subjectively visible and invisible trials conducted by Silvanto and Soto (2012) and Bona et al. (2013), could be replicated. In the first and second t-test respectively, one and 13 participants were excluded, because they did not have enough trials in which they had judged the distractor to be visible. The criterion was set on at least 5 trials. Separate tests – with only the uniform mask trials - were carried out. A paired t-test revealed that on trials where participants indicated not seeing the distractor (PAS = 1), the distractor orientation had no effect, t(27) = 0.015, p = .99. On trials where participants indicated that the distractor was visible (PAS > 1), a paired t-test showed no significant effect either, t(15) = 1.502, p = .15 In conclusion, the effects of visibility and distractor orientation on VSTM accuracy found by Silvanto and Soto (2012) and Bona et al. (2013) were not replicated in this study.
A
B
Figure 1. (A) Distractor visibility in uniform mask model, presented by the proportion of visibility responses for the masked distractor. On the left for the trials in which the distractor was present and on the right in which it was absent. Error bars indicate ± 1 SEM. (B) Distractor visibility in circular grating mask model, presented by the proportion of visibility responses for the masked distractor. On the left for the trials in which the distractor was present and on the right for the trials in which it was absent. Error bars indicate ± 1 SEM.
Figure 3. The effect of distractor orientation on VSTM accuracy. Congruent = distractor orientation differs 0 degrees from memory cue, incongruent = distractor orientation differs 40 degrees from memory cue. Error bars indicate ± 1 SEM. Uniform = uniform mask. Circular grating = circular grating mask.
Discussion
The aim of this study was to investigate whether WM has to be conscious or can operate unconsciously. In order to do so, the present study looked at interference effects on VSTM from distractors rendered invisible on both a subjective and objective level. Previous studies had used subjective measures and concluded that a distractor was unconscious when a person indicated they did not see it (PAS = 1). These studies only analyzed the trials for which participants indicated they had not seen the distractor. Schmidt (2015), however, pointed out that this not-seen-judgment-only procedure assumes a two-state concept of consciousness: absent or present. Schmidt (2015) argued that this assumption is incompatible with the insights gained from SDT in modern psychophysics. In SDT it is well established that a response is determined both by
perceptual sensitivity and decisional criteria. SDT proposes to utilize all responses, false or true, and positive or negative, to calculate the sensitivity index (d') which separates perceptual sensitivity from decisional criteria. When the PAS is used, decisional criteria are indeed at play and it is possible that distractors reported not been seen, actually were weakly conscious (Snodgrass, Bernat & Shevrin, 2004). To examine if the same effects could be found if the distractor was truly unconscious, d’ was added as an objective measure. It was predicted that under the unconscious WM model, interference on both the subjective and objective level would be found. Under the weakly conscious model, it was expected to only find effects on the subjective level.
First of all, a manipulation check confirmed that the perceptual sensitivity for the two masks differed. In the weaker, uniform, mask condition, the d' of participants was on average significantly higher than zero, indicating that they could discriminate whether the distractor was present or absent. In the stronger, grating, mask condition, their d' on average did not differ significantly from zero, indicating that participants could not discriminate whether the distractor was present or absent. The main analysis showed there was no effect of distractor visibility or distractor orientation on VSTM accuracy on both the subjective and the objective level. Separate tests on the subjective level also did not show any effects of distractor visibility or orientation on VSTM. In short, neither of the predictions were met: both on the subjective and objective level there was no interference effect from distractors on VSTM. Therefore, it is concluded that VSTM information could not be processed unconsciously or weakly conscious.
There are several explanations why predicted effects were not found. First, the theoretical framework may have been incorrect, as it is based on studies that only assessed subjective measures. This theoretical framework may only apply if possibly weakly conscious trials are included as if they were unconscious. Instead of only using the not-seen-judgments (PAS =1), the SDT method this study used may have been more proper to use in all trials.
Second, subtle differences in protocols could explain the fact that this study was unable to replicate the findings of Silvanto and Soto (2012) and Bona et al. (2013). Remarkably, participants in the Bona et al. (2013) experiment reported they did not see anything (PAS =1) in about 30% of the distractor present trials, while this percentage was much higher (70%) in our study. Although this study tried to have the exact same experimental setup for the uniform mask condition, differences may have occurred in technical setup. It is possible that the monitor luminance differed, which could have made the distinction between the Gabor grating and the white background less clear.
In addition, instructions in the current study may have been different from the instructions participants in other studies received. Even though this study tried to address the importance of attention to both the distractor as well as the orientation of the memory cue, it is possible that different researchers emphasized one or the other. If more attention is paid to the memory cue orientation than the distractor, the distractor might be missed more often. Also, some researchers may have unintentionally mentioned that it is easy to see the distractor or, alternatively, that it is difficult to see the distractor, which could have influenced participants’ decision criteria.
Finally, the overall attention for the test could have explained our results as the experiment was lengthy (1.5 hours) and attention may not continuously have been optimal because of test fatigue. DeHaene (2014) pointed out that weakly conscious or unconscious stimuli are not processed in a passive bottom-up way, unaffected by attention or instructions. Attention influences the extent to which unconscious stimuli are processed, and they equally benefit from focused attention as conscious stimuli do.
Some possible limitations of the current study need to be discussed. First, the current study had a moderate number of participants. However, the number of participants included here was expected to be more than sufficient, since Silvanto and Soto (2012) reported a very
large effect size of d = 1.61 for the difference between congruent and incongruent distractors. If indeed an effect size of this order of magnitude would be realistic, even only six
participants would provide sufficient statistical power (1 - ß = 0.88) at a significance threshold of p = .05. Although the effect found in the current study was in the expected direction, its size was small (d = 0.18) compared to what was reported by Silvanto and Soto (2012). It is likely that the true effect was strongly overestimated by these authors. Our sample size of 29 provided sufficient statistical power (1 - ß = 0.80) for a medium effect size (d = 0.54). Yielding significance for effects as small as d = 0.18 may not be realistic for laboratory studies, since it would require 225 participants in order to have a power of 1- ß = 0.80.
A second limitation was, as described earlier, that multiple researchers were involved in data collection which may have influenced participants’ interpretation of the test or their decision criteria.
Future research should try to take the mentioned limitations into account. First, attention to the task should ideally both be optimized and measured during the task. A possible way to reach optimal attention is to give participants sufficient breaks. These breaks should be frequent and long lasting, in order for participants to keep focused while performing the task. A way of measuring participants’ attention could be eye tracking. If participants have their eyes at or move their eyes to the location of the distractor at the time it is presented, chances are high that they attended to it. A study by Hoffman and Subramaniam (1995) showed that it is impossible to move the eyes to one location, while attending to a different location. Attention to the unconscious stimuli will promote further processing in the brain (DeHaene, 2014).
Second, this study used only two different masks: the uniform black mask and the circular grating mask. It was shown that the circular grating mask was better in rendering distractors objectively invisible than the uniform black mask. However, was the circular grating
mask maybe stronger than strictly necessary or desired? Possibly, stronger masks than strictly necessary impede processing of otherwise effective subconscious distractors. Future studies could systematically vary the mask strength and select the strength that is just sufficient to render the distractor objectively invisible. Ideally, this calibration should be done for each individual separately. Using the minimally required mask may make it easier to reveal possibly vulnerable effects of unconscious distractors.
In summary, these results demonstrate the use of signal detection theory to better define whether a mask renders a distractor objectively invisible. However, they remain inconclusive about whether the unconscious and weakly conscious distractors differ with respect to their effect on working memory.
References
Atkinson, R. C., & Shiffrin, R. M. (1971). The control of short-term memory. Scientific
American, 225(2), 82-91.
Baars, B. J. (1998). Metaphors of consciousness and attention in the brain. Trends in
neurosciences,21(2),58-62.Baddeley,A. (1992). Working memory. Science, 255(5044), 556-559.
Baars, B. J., & Franklin, S. (2003). How conscious experience and working memory interact. Trends in Cognitive Sciences, 7(4), 166-172.
Baddeley, A. (2003). Working memory: looking back and looking forward. Nature Reviews
Neuroscience, 4(10), 829.
Bergström, F., & Eriksson, J. (2015). The conjunction of non-consciously perceived object identity and spatial position can be retained during a visual short-term memory task.
Frontiers in Psychology, 6, 1470.
Bona, S., Cattaneo, Z., Vecchi, T., Soto, D., & Silvanto, J. (2013). Metacognition of visual short-term memory: dissociation between objective and subjective components of VSTM. Frontiers in Psychology, 4, 62.
Dehaene, S. (2014). Consciousness and the brain: Deciphering how the brain codes our thoughts. Penguin.
Dehaene, S., & Changeux, J. P. (2011). Experimental and theoretical approaches to conscious processing. Neuron, 70(2), 200-227.Luck, S. J. (2007), Scholarpedia, 2(6):3328. Dehaene, S., & Naccache, L. (2001). Towards a cognitive neuroscience of consciousness:
Basic evidence and a workspace framework. Cognition, 79(1-2), 1-37.
Gaillard, R., Cohen, L., Adam, C., Clemenceau, S., Hasboun, D., Baulac, M.,... & Naccache, L. (2007). Subliminal words durably affect neuronal activity. NeuroReport, 18(15), 1527-1531.
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.
Hassin, R. R. (2013). Yes it can: On the functional abilities of the human unconscious.
Perspectives on Psychological Science, 8(2), 195-207.
Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Perception & psychophysics, 57(6), 787-795.
Kunimoto, C., Miller, J., & Pashler, H. (2001). Confidence and accuracy of near-threshold discrimination responses. Consciousness and Cognition, 10(3), 294-340.
Lamme, V. A. (2006). Towards a true neural stance on consciousness. Trends in Cognitive
Sciences, 10(11), 494-501.
Overgaard, M., Timmermans, B., Sandberg, K., & Cleeremans, A. (2010). Optimizing subjective measures of consciousness. Consciousness and Cognition, 19(2), 682-684. Phillips, W. A. (1974). On the distinction between sensory storage and short-term visual
memory. Perception & Psychophysics, 16(2), 283-290.
Reber, T. P., Luechinger, R., Boesiger, P., & Henke, K. (2012). Unconscious relational inference recruits the hippocampus. Journal of Neuroscience, 32(18), 6138-6148. Richet, C. (1886). Les origines & Les modalités de la mémoire: Essai de psychologie
générale. Revue Philosophique de la France et de l'Étranger, 21, 561-590. Sandberg, K., Timmermans, B., Overgaard, M., & Cleeremans, A. (2010). Measuring
consciousness: is one measure better than the other? Consciousness and Cognition, 19(4), 1069-1078.
Schmidt, T. (2015). Invisible stimuli, implicit thresholds: Why invisibility judgments cannot be interpreted in isolation. Advances in Cognitive Psychology, 11(2), 31.
Silvanto, J., & Soto, D. (2012). Causal evidence for subliminal percept-to-memory interference in early visual cortex. Neuroimage, 59(1), 840-845.
Sligte, I. G., Scholte, H. S., & Lamme, V. A. (2008). Are there multiple visual short-term memory stores? PLOS One, 3(2), e1699.
Snodgrass, M., Bernat, E., & Shevrin, H. (2004). Unconscious perception: A model-based approach to method and evidence. Perception & Psychophysics, 66(5), 846-867.
Soto, D., Mäntylä, T., & Silvanto, J. (2011). Working memory without consciousness. Current Biology, 21(22), R912-R913.
