Line orientation in peripheral vision : an attempt to decode illusory representations in the visual cortex

(1)

Master thesis

Line Orientation in Peripheral Vision:

An Attempt to Decode Illusory Representations

in the Visual Cortex.

J.P.R.A. Schuurmans

Student number: 10892532

Supervisors: Marte Otten and Yaïr Pinto

Date: October 2017

Programme group Brain and Cognition

(2)

1 Preface

Before you lies my master thesis, 32 pages representing what kept me busy the last couple of months. During this period of time, I've learned a great deal, not only about analysing and writing but also about recruiting participants and operating an MRI machine.

While reading, you will perhaps notice that my discussion does not have a traditional setup. In addition to possible improvements of the study, it contains a scenario. This choice was made because the MRI dataset likely contained too much noise, making the results

inadequate to reflect on the literature discussed in the introduction. On the recommendation of one of my supervisors, Yaïr Pinto, I wrote about a scenario in which the results would have been compatible. This way I could still practice in writing a scientificly relevant discussion.

I received a lot of help during these last couple of months. I would like to take this opportunity to thank all the people who have helped me in this last stage of my master of psychology.

First, I want to thank my supervisors Marte Otten and Yaïr Pinto for providing me with the opportunity to do this research. By giving me a lot of responsibility, which was challenging at times, I’ve learned a great deal. Second, Hidde Pielage for being my study-buddy a.k.a. research-partner in crime. Conducting this research together was a nice last chapter of years studying together. Djimo van Berlo for critically reading every sketch I wrote, most of the time resulting in interesting discussions. Astrid Nielsen and Milan van Berlo for helping me out with my English writing. Lukas Snoek for all his time he spent helping with the analysis. The Spinoza Centre staff for teaching me how to operate an MRI machine. And lastly, my friends and family who spent hours with me in the lab, looking at stripes and dots changing in the periphery.

Jolien Schuurmans Utrecht, 25th of October 2017

(3)

2 Index Abstract ... 3 Introduction ... 4 Methods ... 11 Participants ... 11 Stimuli ... 11 Procedure ... 12 Data acquisition ... 14

fMRI data pre-processing ... 14

Mapping and region of interest ... 15

First level and Multi Voxel Pattern Analysis ... 15

Results ... 16 Behavioural results ... 16 Decoding results ... 16 Explorative analyses ... 17 Discussion ... 19 Improvements ... 19 Alternative scenario ... 21 Conclusion ... 23 Literature ... 24 Appendix 1 ... 29 Appendix 2 ... 30

(4)

3 Abstract

When you look up from this thesis, you see a detailed and colourful world. However, your eye only records details and colour in the fovea, a very small central area of the retina, with a field of view in the size of a thumbnail at arm's length. The question arises, how can we inhibit a rich visual world, when our eyes do not provide sufficient information? One idea is that we do not. We simply misinterpret what we experience. Another option is that our knowledge and expectations change visual processing at the earliest levels. In the current research, the recently discovered Uniformity Illusion was used, in attempt to find an answer to the question if an active computation is necessary for filling in of lacking peripheral visual information, or if the brain simply ignores the lack of visual information in a more passive manner. fMRI was used and a MVPA was executed to acquire the activity patterns in the visual cortex when the orientation based illusion manifests itself. However, the classifier algorithm was not successful in decoding lines with different orientations above chance rate. This means the brain activity resulting from Uniformity Illusion could not be decoded in this study. A possible explanation for this outcome is that there was too much noise in the dataset, or poor choices were made during the analysis. If the classifier algorithm would have been successful and also able to decode the illusory trials above chance level, the theory about filling-in as a passive visual process should be rejected and filling-in of missing peripheral information would presumably be an active feedback process. Providing a contribution to the understanding of conscious (visual) experiences.

(5)

4 Line Orientation in Peripheral Vision: An Attempt to Decode Illusory Representations

in the Visual Cortex.

When you look around, you experience a rich, detailed and colourful world. It presents itself as a picture, sharp in focus and detailed (Anderson, Mullen, & Hess, 1991; Block, 2014). An example of this experience is represented by a drawing by Ernst Mach in 1988 (Noë & Thompson, 2002). The "view from the left eye" is from the centre out to the periphery sharp in focus and uniformly detailed (Figure 1). His drawing represents his (monocular) visual field. Even though our experience seems a one-to-one map of the visual world, this is physically not possible. Our eye only records details and colour in the fovea (Anderson et al., 1991), a central area of the retina with a visual view in the size of a thumbnail at arm's length (Wandell, Dumoulin & Brewer, 2007). How can we experience a rich visual world, when our eyes do not provide any such information? The brain seems to draw conclusions about the peripheral areas of our field of vision with very limited input (Anderson et al., 1991). It appears this distorted information is filled-in (Pessoa, Thompson, & Noë, 1998).

(6)

5 There are different theories about the underlying neural processes of filling in of visual information (De Weerd, 2006). One possibility is that there are no active neuronal processes involved to explain this phenomenon. Our visual system simply ignores the lack of visual input and the filling in is a passive result of this (Dennett, 1992). This theory proposes that visual processes take place via a bottom-up process in the brain (Appendix 1). This means that visual information enters the eye, and then enters the cortex via the retina and the LGN. After this, it is processed in a hierarchical manner. According to Dennett (1992), it is not necessary for the visual system to literally “fill in” visual information. He describes that the visual representations that do not occur physically, are not "analogue" adjusted in the brain. Filling-in missing information is in these terms symbolic (De Weerd, 2006). Despite the name, no actual “filling-in” occurs. There is no medium in the brain representing visual information in a way pixels represent visual information in a computer. Thus the retinotopic organisation of neurons is not identical to the experience of visual information. The

characteristics of visual stimuli are processed in the early areas of the brain and only in the higher cognitive areas do these visual stimuli receive their meaning (Todorovic, 1987; Van den Brink & Keemink, 1976). Dennett (1992) illustrates this by use of the neon colour

spreading illusion (Van Tuijl, 1975). This is an optical illusion in which vividly coloured lines on a white background give rise to the experience of the background being lightly coloured (with red lines, the background seems to have a slightly pink glow). In the brain, there is a circuit specialised in processing the shapes and location of the objects we see (Livingstone & Hubel, 1988). Another circuit is specialised in processing colour. This results in one "colour-label" for a surface, and not pixel-for-pixel, for this circuit cannot process shape and location of objects efficiently. This "label" of colour is associated with the simultaneous processed shape and location, which is processed by the shape-circuit. Thus with the neon colour spreading illusion, there is merely a "mislabelling" of visual information. There is no specific

(7)

6 "product" of the visual system in the brain. This means that the representation of the visual world does not exist in the early visual areas, but is a cognitive process that takes place in ‘high’ areas of the brain (Van den Brink & Keemink, 1976).

Different studies provide support for the concept that representations of visual stimuli do not take place in early visual areas. For example, Logotheis and Schall (1989) found in a binocular rivalry study with monkeys, that there was no correlation between the conscious representation of differently orientated gradings and brain activity in early visual areas.

Moreover, it appears that cells in early visual areas flicker back and forth when presented with quick altering (isoluminant) colour stimuli. Still just one colour is perceived (Gur &

Snodderly, 1997). Furthermore, results by He, Cavanagh and Intriligator (1996) show that with humans, even though cells in the V1 carry visual information about orientation, the representation of these stimuli can fade. The results of these studies suggest that the

retinotopic organisation of neurons in early visual areas is not identical to the experience of visual information. The activity in these areas is stimulus-driven (bottom-up) and remain unchanged. In short, according to the theory about filling-in as a passive process, labelling of visual surfaces occurs in higher areas of the cortex. Visual information is processed in a bottom-up manner and its representation will not occur in early visual areas. Filling-in is a symbolic term for rich and detailed representations when there is distorted visual information.

However, different studies have found evidence for a more active process for filling in visual information. This theory proposes that filling-in is both a bottom-up and top-down process (Komatsu, 2006). When visual information enters the eyes, its first processed in a bottom-up and subsequently in a hierarchical manner (Appendix 1). When visual information from peripheral areas is missing, this is adjusted for via top-down processes in the visual processing stream. This means that there is feedback from late areas to early visual areas (Kok & De Lange, 2014). These top-down signals are determined by internally generated

(8)

7 predictions about the nature of the periphery (Clark, 2013). The signals are based on the strongest information available. Distorted bottom-up information is overwritten by predicted top-down information (Komatsu, 2006). This implies the existence of an interpolation process in the early visual areas, whereby the activation in these areas is isomorphic to the subjective representation of the visual stimuli (De Weerd, 2006). In other words, the activation patterns in early visual areas are a reflection of the subjective representation (Sasaki & Watanabe, 2004).

Several studies have found evidence that early visual areas (e.g. V1 and V2) are indeed involved in the filling in of visual information (Ramachandran & Gregory, 1991; Sasaki & Watanabe, 2004; Meng, Remus & Tong, 2005; Komatsu, 2006). Komatsu (2006) reports that V1 and V2 are activated when subjects are presented with several illusions, such as, neon colour spreading (Grossberg & Mingolla, 1985), the Craik–O’Brien–Cornsweet illusion (Davey, Maddess & Srinivasan, 1998) and phantom illusions (Kitaoka, Gyoba & Sakurai, 2006). Kok and de Lange (2014) have demonstrated that when perceiving Kanisza figures, V1 displays stimulation and inhibition processes. This entails that for the neurons in the V1 with a field of view that falls within the illusory figure, the neural activity is stronger compared to the neurons that do not have a field of view that falls within this figure.

Furthermore, the neurons in the V1 that react to local illusion elements show less activity. The studies into these illusions indicate that early visual areas are involved in the filling in, which indicates an active reconstruction process. Moreover, Ramachandran and Gregory (1991) have found evidence of an active process during the filling in of peripheral areas of the visual field. This study discovered differences in the timing of the filling in depending on the stimulus. When filling in a small visual area, an artificially induced scotoma, first the colour and second the texture were filled in. During a short moment, the representation of the filled-in field did not cofilled-incide with the filled-information that was projected onto the retfilled-ina. If this had

(9)

8 been a passive bottom-up process, the representation of the filled-in area should have been the same as the representation of the surrounding input. However, this was not the case.

Therefore, this finding yields support for an active reconstruction of the peripheral visual information. In short, according to this theory, filling in missing visual information is an active process that takes place in early visual areas.

Now, the fundamental question is, if an active computation is necessary for filling in of visual information, or if the brain simply ignores the lack of visual information in a more passive manner. Different studies have found evidence for representations not being present in early visual areas, but in other studies the opposite is found. A way to investigate the process behind rich visual peripheral experiences is by inducing a controlled visual experience, in which the physical input differs from the experience, and measure brain activity in early visual areas during this process. Visual illusions provide this opportunity (De Weerd, 2006). However, the illusions that are described in the previous paragraphs involved the filling in of a figure, or show only the filling in of a small area in the visual field. The present study uses the recently discovered Uniformity Illusion (Otten, Pinto, Paffen, Seth & Kanai, 2016). This is an illusion that distorts peripheral vision, even though the physical stimulus does not

change (Figure 2, for more examples, visit: http://www.uniformillusion.com). The Uniformity Illusion differs from the previously described illusions because it covers the entire peripheral area. The illusion occurs with a broad spectrum of stimuli (such as colour, luminance, shape and orientation), which suggests that there is a general mechanism for visual processing in play when this illusion occurs (Otten et al., 2016). This way, the Uniformity Illusion provides the opportunity to investigate the experience of rich peripheral vision with a lacking

(10)

9

Figure 2. Example of a luminance based Uniformity Illusion

Using Functional Magnetic Resonance Imaging (fMRI), the activation pattern of the illusory percept can be extracted from the data at specific locations in the brain, which can be used to exclude one of the two hypothesis about filling-in. For this study, participants were shown the orientation based Uniformity Illusion (Figure 3a). fMRI was used to measure the blood-oxygen-level-dependent (BOLD) signal in the brain during the experiment. A multi voxel pattern analysis (MVPA) was done to compute the activation pattern in early visual areas for lines with different orientations in peripheral areas of the visual field. First, a classifier algorithm was trained for the differently orientated lines. Later this could be cross-validated on illusory displays. If filling in of peripheral visual information involves a passive, bottom-up process, it is expected that no activation pattern of the illusory representation will be present early visual areas. After prolonged fixation on the illusory displays, the initial activation pattern will not change, or will no longer be present, due to extinction because of adaptation. This means that the classifier algorithm will not be successful in decoding the illusory displays when the illusion manifests itself. On the other hand, there is the active process, in which both bottom-up and top-down processes are playing a role. In the case of

(11)

10 this active process, it is expected that the visual representation will be isomorphic to a neural activation in early visual areas. In this case, the activation pattern of the representation should already be visible in these areas when the illusion happens. Thus the activation pattern from the peripheral illusory representation shows a similar pattern as the matching representation due to actual peripheral physical input (Figure 3b). The classifier algorithm will be successful in decoding the illusory displays when the illusion manifests itself.

(a)

(b)

Figure 3. (a) Orientation based Uniformity Illusion as used in the current study. (b) Displays

(12)

11 Methods

Participants

All 33 participants (20 female, 13 male; with an average age of 22.2 years (range 17-28 years)) had normal or corrected-to-normal vision. Prior to participating, the participants were screened on the possibility of metal in their bodies and other risk factors precluding participation in MRI studies. All participants were fully informed about the nature of the experiment and a written informed consent was obtained before experimentation. Participants received credits or € 23.75 for their participation in each scanning session. The experiment was approved by the Ethical Committee of the Psychology Department of the University of Amsterdam.

Stimuli

To construct the different displays (Figure 3 and Figure 4) the screen (79.7 × 49.1 cm, VA = 64.63° × 42.58°) was divided into a grid with square cells (1.59 × 1.59 cm, VA = 1.45°

× 1.45°). The central and peripheral lines, located in the middle of their cells (0.53 cm, VA = 0.48°), were drawn at the centre of the screen in circular formation within two concentric circles. The central lines were drawn inside the inner circle (7.97 cm, VA = 7.24°) and the peripheral lines were drawn between this inner circle and the outer circle (26.57 cm, VA = 23.82°). The orientation of the lines depended on the trial type (Appendix 2). The light grey lines (RGB = (150, 150, 150)) were presented on a dark grey background (RGB = (100, 100, 100)). This colour difference was minimal in order to prevent a strong afterimage.

(13)

12 Procedure

Each participant took part in two separate sessions. Each session consisted of one mapper block and two training blocks. The experiment included a mapper block in order to indicate the regions of interest (ROI) during the viewing of lines with different orientations. The mapper block contained 96 trials (Appendix 2, Table 1). The training blocks were for the classifier algorithm to train on different trials. Both training blocks consisted of 80 trials (Appendix 2, Table 2).

The participants were presented with stimuli on a 32" BOLD screen from Cambridge Research Systems (1920 × 1080 with a 120 Hz refresh rate) and viewed the screen at a distance of 63 cm through a mirror attached to the head coil. During the experiment,

participants had to fixate on a yellow fixation dot (RGB = (250, 250, 0)) in the centre of the screen (Figure 4). In a third of all of the trails (with exception of the illusion trails), a small dot appeared at a random location in the periphery. Participants had to indicate whether the small dot (Figure 4a), or the illusion appeared (Figure 4b). This was to make sure participants kept their attention on peripheral areas of their visual field. The responses were registered after each trial by use of MRI compatible button boxes (Current Designs Fiber Optic Response Pads). For example, during the mapper block and the first training block, participants had to respond with their right index finger if a dot appeared or if they experienced the illusion and if not with their right middle finger. During the last training block, participants had to use the opposite fingers to respond: the index finger when there was no dot or illusion and the middle finger when there was a dot or they experienced the illusion. This way motor movement of the finger would be filtered out, and the classifier algorithm would not pick it up for the pattern analysis. Which finger the participants had to use was counterbalanced for each session and for all of the participants.

(14)

13 (a)

(b)

Figure 4. Examples of trials used in the study. (a) Trial with the central patch empty,

peripheral tilted lines of 45° and a dot appearing. (b) Illusion trial with the tilted lines in the central patch of 15° and peripheral tilted lines between 20° and 40°.

(15)

14 Data acquisition

The neuroimaging data were acquired at the Spinoza Centre for Neuroimaging on a Philips 3 Tesla MRI Scanner (Philips Ingenia CX system) using a standard head coil. A T1-weighted anatomical image (220 transverse slices, TE = 3.7 ms, TR = 8.2 ms, flip angle = 8°, 1 × 1 × 1 mm voxels, FOV (ap, rl, fh) = 240 × 188 × 220) was recorded for each participant. Functional MRI was recorded using gradient-echo, echo-planner pulse sequence (37

transverse slices, thickness = 3 mm, TR = 2000 ms, TE = 28 ms, flip angle = 76°, 3.00 × 3.08

× 3.00 mm voxels, FOV (ap, rl, fh) = 240 × 240 × 122, matrix size = 80 × 78).

fMRI data pre-processing

Data were processed using an fMRI prep pre-processing model

(https://github.com/poldracklab/fmriprep). For the first level analysis FSL was used (Woolrich, et al., 2009). Distortion correction was applied to correct for the effects of

magnetic field inhomogeneity by using a field map top-up (b0). To correct for effects of small head movements between runs, translation and rotation of the participants, the six parameters were estimated by use of a rigid body spatial transformation model. Slice time correction was not carried out because of relatively short repetition times (TR = 2 sec) (Poldrack, Mumford & Nichols, 2011). Also, no functional data were smoothed in the space domain. Because of the activation of relatively small structures, spatial smoothing frequently causes information loss (Misaki, Luh, Bandettini, 2013). Temporal filtering was carried out using a 100 seconds high pass filter. Spatial normalization was conducted for each participant by the registration of the image into an MNI152 space (3 × 3 × 3 mm voxels).

(16)

15 Mapping and region of interest

To prevent the classifier algorithm from training on information of lines in the central patch, a region of interest (ROI) was estimated for each participant. Using FSL feat, the z-statistics of the mapper block, with contrast peripheral lines minus lines in the centre, were estimated. Only the voxels significantly active for seeing lines in peripheral areas were used for the classifier algorithm to train. In order to exclude one of the two hypotheses, it was necessary to train the classifier algorithm on just occipital areas. The standard

Harvard-Oxford cortical structural occipital pole mask was set to 3 × 3 × 3 mm voxel space and used to mask the occipital area of the brain.

First level and Multi Voxel Pattern Analysis

Using FSL feat, the z-statistics of all trials were estimated, with a contrast of peripheral lines minus lines in the centre. For every trial, one regressor was obtained for analysis, accumulating to a total of 160 regressors per session. Pattern classification was carried out for each session for all participants individually. Support Vector Machine (SVM) with a three-way classification was used. The classifier algorithm was trained and tested on the classification of three different displays: peripheral lines with an orientation of 15°, 45° and random between 20° and 40° (20-40°). To prevent mistakes, Pipeline from Sklearn was used for the analysis, using a standard scalar. For cross-validating the data with

StratifiedKFold, a total of 10 folds was used. This gave all the accuracy values per participant per session. To test the reliability of the found results, a one-sample t-test was conducted on the mean accuracy values and a chance rate of 33%.

With a successful classifier, a cross-validation with the illusory trials could check whether the activation in occipital areas of the brain is similar to subjective representation of visual stimuli and thus if filling-in is predominantly an active or passive process in the brain.

(17)

16 Results

Behavioural results

From the total amount of 33 participants, two did not finish their second session. Also, due to incorrect settings, two sessions were unsuited for the analysis, leaving 62 sessions to analyse. With a mean accuracy of 95.5%, participants were very accurate in reporting the dot. This implies that they were focused on the task. On average, participants indicated that they experienced the illusion with a central patch with lines of 15° more often (39%) than with a central patch of 45° (27%). An independent samples t-test showed that this difference was significant t(1951)= 5.31, p < .001.

Decoding results

For each session of each participant, the accuracy percentage of the testing data was obtained by use of SVM (Figure 5). A one-sample t-test was performed to test whether the classifier algorithm performed significantly greater than chance rate. The chance rate was 33% because the test data could be classified into three possible categories. However, the classifier algorithm could not classify above chance level t(32) = -1.26, p = 0.11. Classifying line orientation from illusory displays turned out to be impossible because the classifier algorithm could not perform above chance.

(18)

17

Figure 5. Mean accuracy values for decoding peripheral orientation from all participants,

compared to the chance rate (33%).

Explorative analyses

A number of extra tests were conducted to check whether the classifier algorithm performed at chance level because of artefacts in the data. First, the classifier algorithm was tested in order to determine whether it was able to decode between stimuli in the centre and stimuli in the periphery. For this analysis the ROI mask was removed. This way, any activity induced by the centre of the screen would show. After obtaining the accuracy values for each session of all participants using SVM (Figure 6), the decoding turned out to be successful in decoding between central and peripheral lines (M = .95). To test whether this result was significant, a one-sample t-test was conducted on the mean accuracy and chance rate (50%). The statistics show that the decoding of peripheral versus central was indeed significantly higher than chance t(32) = 81.70, p < 0.01. These results suggest that the data did not contain a significant number of artefacts and the classifier algorithm used was indeed functional.

0 0,2 0,4 0,6 0,8 1 Ac cura cy Participants

(19)

18

Figure 6. Mean accuracy values for decoding central and peripheral stimuli from all

participants, compared to the chance rate (50%).

To test if the data contained too much noise, the classifier algorithm was trained to classify orientation of lines, just in the centre of the visual field. The used trials contained central lines with an orientation of 15° and central lines with an orientation of 45°. Again SVM was used for training and testing on data from both sessions of all participants. After obtaining all the accuracy values (Figure 7), a one-sample t-test showed that the null-hypotheses, stating that orientation in the centre is not decodable above chance rate (50%), cannot be maintained t(32) = 2.33, p = 0.01. This means the classifier algorithm was able to decode orientation above chance rate, with an average accuracy of M = .533.

Figure 7. Mean accuracy values for decoding central and peripheral orientation from all

participants, compared to the chance rate (50%). 0 0,2 0,4 0,6 0,8 1 Ac cura cy Participants 0 0,2 0,4 0,6 0,8 1 Ac cura cy Participants

(20)

19 Discussion

The Uniformity Illusion was used in an attempt to examine if filling-in is an active or passive process. A classifier algorithm was trained to decode between differently orientated lines in peripheral visual areas, which could be used to cross-validate on illusory displays. However, in the current study, the classifier algorithm was not able to successfully

discriminate between differently orientated lines in peripheral visual areas. First, some discussion issues regarding the dataset and improvements of this current study will be illuminated. Second, a scenario in which the classifier algorithm would have been successful will be discussed.

Improvements

Several studies have shown that decoding orientation in the visual cortex using MVPA is possible (Hayes & Rees, 2005; Kamitani & Tong, 2005; Hsieh & Tse, 2010). For example, Kamitani and Tong (2005) trained linear SVMs to recognise striped patterns. Although the activity of orientation in early visual areas is at subvoxel level, in this instance the classifier was able to decode between different orientated lines. Combining the signals from across multiple voxels in V1 and V2 gave enough information for the classifier algorithm to do so. A study by Hayes and Rees (2005) confirmed these results. Even visual masking was used, so the participants were not consciously aware of the orientation of lines. Still, their classifier algorithm was able to correctly decode between different orientations within the V1. These results show that decoding orientation by use of MVPA classifier algorithms should be possible in these early visual areas. Still, in the current study, these results were not found. To exclude possible errors or artefacts in the used data set, location (peripheral versus central lines) was decoded using the same methods, which had a success rate of 95%. This means that the data contained enough information to of use. To check if orientation was decodable at all

(21)

20 in the current dataset, the orientation of lines in the centre was also decoded. This gave a significant result above chance level. The result is more compatible with literature describing the successful decoding of orientation using MVPA (Hayes & Rees, 2005; Kamitani & Tong, 2005; Hsieh & Tse, 2010). A possible explanation for this finding is the relatively large representation of foveal compared to peripheral vision in retinotopic maps of the visual cortex (Dow, Snyder, Vautin & Bauer, 1981; Wandell et al., 2007). In this case, activation patterns for central stimuli would be more clear than patterns for peripheral stimuli. However, a

chance rate of 53% is still rather towards the low end. Meaning the dataset probably contained too much noise for it to decode above chance level.

Perhaps poor choices were made during the analysis. Different regressors could have resulted in analysing more specific orientation specified voxels (Poldrack, Mumford & Nichols, 2011). Instead of using the z-statistics of lines in the periphery versus lines in the centre of the screen, the regressor could have been established by calculating the z-statistics of peripheral tilted lines with an orientation of 15° versus 40° and the same for 45° versus 20-14°. Possibly these different contrasts would have established regressors more sensitive for different orientation in the periphery. Moreover, choosing a whole brain univariate analysis over MVPA could have shown a more clear location (Poldrack, Mumford & Nichols, 2011) for voxels more active for different orientations, instead of an activation pattern in just the occipital cortex. This way the illusory displays could have been analysed a statement could be made if filling-in is predominantly an active or passive process.

(22)

21 Alternative scenario

Should the classifier algorithm would been successful in decoding between different orientations in peripheral areas and after cross-validating, the illusory trials would also be classified above chance level. This would give interesting implications for the current literature about filling-in, but also about perception in general. Finding the activation pattern of illusory displays in the visual cortex would suggest that 'mislabelling' in high cognitive areas is not a sufficient theory to describe the process behind filling-in. In order for the

activity to change in the visual cortex, without changing the sensory-driven information, there must be an active process (Komatsu, 2006). With an interaction between sensory-driven and knowledge-driven information, activation in early visual areas could could be adjusted

without a change of physical stimuli. The brain does this based on our knowledge of the world (Clark, 2013). Using this prior knowledge, internally generated predictions provide feedback to early visual areas (Kok & De Lange, 2014). Our knowledge and expectations change visual processing at the earliest levels. These activation patterns in early visual areas seem to

correlate with (visual) perceptual experiences.

Visual perception seems to be partly based on sensory-driven information and partly on predictions. Possibly conscious experience as a whole is subject to this principle of feedback from higher levels (Panichello, Cheung & Bar, 2013). However, is this feedback a necessary condition for conscious experiences? Different researchers propose feedback is crucial for experiences (Roelfsema, Lamme & Spekreijse, 1998; Dehaene & Changeux, 2011; Meyer, 2012). For example, Roelfsema et al. (1998) measured orientation selective V1 cells in monkeys. These monkeys were shown a background with specifically orientated stripes or the same stimulus as a figure on a differently orientated background. Initially, the activity of the cells was the same for both displays. But after 100 ms, the cells showed a larger response to the figure, compared to just the background. If the monkey did not see the figure, there was

(23)

22 no difference between the two displays. This implies that the activity was mediated by

feedback from higher areas when the monkey was aware of the stimuli. When disrupting the top-down process, representations of stimuli seemed to be impaired (Supèr, Spekreijse & Lamme, 2001; Self, Kooijmans, Supèr, Lamme & Roelfsema, 2012). This was demonstrated in two studies using a similar experiment as that of Roelfsema et al. (1998). With anaesthesia, masked stimuli, TMS over the V1 or lesions in higher areas, the difference of activity in cells between figure and background were absent (Supèr, Spekreijse & Lamme, 2001; Self,

Kooijmans, Supèr, Lamme & Roelfsema, 2012). Similar results were also found in human participants (Ro, Breitmeyer, Burton, Singhal, & Lane, 2003; Fahrenfort, Scholte & Lamme, 2007; Dux, Visser, Goodhew, & Lipp, 2010). This implies that if feedback from high level to early visual areas would be absent, visual stimuli would not reach consciousness. Predictions from high-level areas providing feedback to early visual areas seem to be a necessary step for visual conscious representations.

If there is indeed feedback based on prior knowledge, priming could influence these predictions. This means that when exposed to priming, visual representations can be altered. Future research could explore whether these predictions could be reinforced by the effect of priming on visual perception. In order find out more about the content of conscious (visual) experiences and the effects of priming on this matter, the uniformity illusion could contribute a great deal, as it manipulates visual perception. It works for a wide range of stimuli and for a large surface of the visual field (Otten et al., 2016). This could provide good opportunities to investigate how the brain constructs conscious visual experiences.

(24)

23 Conclusion

In conclusion, there are various possible explanations why decoding orientation of peripheral vision using MVPA was not possible. It is possible noise in the data generated activation patterns not sufficient for decoding orientation or poor choices were made during the analysis. A solution for the current dataset is a whole brain univariate analysis for a better localization of illusory brain activity. This way, a statement could be made if filling-in is predominantly an active or passive process.

If the classifier algorithm would have been successful in decoding orientation and cross-validating on illusory trials, the theory about filling-in as a passive visual process should be rejected. The filling-in of missing peripheral information would be an active feedback process, based on prior knowledge of the world. This way, even though our eyes do not provide us with the information, we still experience a rich visual world, explaining why Ernst Mach in 1988 thought the view from his left eye was sharp in focus and uniformly detailed. Thus, assuming these results would indeed have been found, when you look up from this thesis and look around, you will perceive a small piece of the world in detail and sharp in focus. The rest is merely a product of your own predictions.

(25)

24 Literature

Anderson, S. J., Mullen, K. T., & Hess, R. F. (1991). Human peripheral spatial resolution for

achromatic and chromatic stimuli: Limits imposed by optical and retinal

factors. The Journal of Physiology, 442, 47–64.

Block, N. (2014). Rich conscious perception outside focal attention. Trends in Cognitive

Sciences, 18, 445-447.

Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of

cognitive science. Behavioral and Brain Sciences, 36, 181-204.

Davey, M. P., Maddess, T., & Srinivasan, M. V. (1998). The spatiotemporal properties of the

Craik–O’Brien–Cornsweet effect are consistent with ‘filling-in’. Vision Res.

38, 2037–2046

De Weerd, P. (2006). Perceptual filling-in: More than the eye can see. Progress in brain

research, 154, 227-245.

Dehaene, S., & Changeux, J. P. (2011). Experimental and theoretical approaches to conscious

processing. Neuron, 70, 200-227.

Dennett, D. C. (1992). Filling in versus finding out: A ubiquitous confusion in cognitive

science. In: Pick, Van den Broek, Knill (eds) Cognition, Conception, and

Methodological Issues. American Psychological Association.

Dow, B. M., Snyder, A. Z., Vautin, R. G., & Bauer, R. (1981). Magnification factor and

receptive field size in foveal striate cortex of the monkey. Experimental brain

research, 44, 213-228.

Dux, P. E., Visser, T. A. W., Goodhew, S. C., & Lipp, O. V. (2010). Delayed reentrant

processing impairs visual awareness: an object substitution masking study.

(26)

25

Fahrenfort, J. J., Scholte, H. S., & Lamme, V. A. F. (2007). Masking disrupts reentrant

processing in human visual cortex. J. Cogn. Neurosci. 30, 7414–7422.

Grossberg, S., & Mingolla, E. (1985). Neural dynamics of form perception: Boundary

completion, illusory figures, and neon color spreading. Psychological

Review, 92, 173–211.

Guillery, R. W., & Sherman, S. M. (2002). Thalamic relay functions and their role in corticocortical communication: generalizations from the visual system.

Neuron, 33, 163-175.

Gur, M., & Snodderly, D. M. (1997). A dissociation between brain activity and perception: Chromatically active cortical neurons signal chromatic activity that is not perceived. Vision Research, 37, 377-382.

Haynes, J. D., & Rees, G. (2005). Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat Neurosci, 8, 686–691.

He, S., Cavanagh, P., & Intriligator, J. (1996). Attentional resolution and the locus of visual awareness. Nature, 384, 334-337.

Hsieh, P. J., & Tse, P. U. (2010). Brain‐reading of perceived colors reveals a feature mixing mechanism underlying perceptual filling‐in in cortical area V1. Human brain

mapping, 31, 1395-1407.

Kaas, J. H., Krubitzer, L. A., Chino, Y. M., Langston, A. L., Polley, E. H., & Blair, N. (1990). Reorganization of retinotopic cortical maps in adult mammals after lesions of the retina. Science, 248, 229.

Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nat Neurosci. 8, 679–685.

(27)

26

Kitaoka, A., Gyoba, J., & Sakurai, K. (2006). The visual phantom illusion: A perceptual

product of surface completion depending on brightness and contrast. Progress

in Brain Research, 154, 247–262.

Kok, P., & de Lange, F. P. (2014). Shape perception simultaneously up-and downregulates

neural activity in the primary visual cortex. Current Biology, 24, 1531-1535.

Komatsu, H. (2006). The neural mechanisms of perceptual filling-in. Nature reviews

neuroscience, 7, 220-231.

Livingstone, M., & Hubel, D. (1988). Segregation of form, color, movement and depth:

Anatomy, physiology and perception. Nature, 240, 740-749.

Logothetis, N., & Schall, J. (1989). Neuronal correlates of subjective visual perception.

Science, 245, 761-763.

Meng, M., Remus, D. A., & Tong, F. (2005). Filling-in of visual phantoms in the human brain. Nature neuroscience, 8, 1248-1254.

Meyer, K. (2012). Another remembered present. Science, 335, 415–416.

Misaki, M.., Luh, W. M., & Bandettini, P. A. (2013). The effect of spatial smoothing on fMRI decoding of columnar-level organization with linear support vector

machine. Journal of neuroscience methods, 212, 355-361.

Noë, A., & Thompson, E. (2002). Vision and mind: Selected readings in the philosophy of perception. MIT Press.

Otten, M., Pinto, Y., Paffen, C. L., Seth, A. K., & Kanai, R. (2016). The uniformity illusion: central stimuli can determine peripheral perception. Psychological science, 28, 56-68.

Panichello, M. F., Cheung, O. S., & Bar, M. (2013). Predictive feedback and conscious visual experience. Frontiers in psychology, 3, 620.

(28)

27 Pessoa, L., Thompson, E., & Noë, A. (1998). Filling-in is for finding out. Behavioral and

brain sciences, 21, 781-796.

Poldrack, R. A., Mumford, J. A., & Nichols, T. E. (2011). Handbook of functional MRI data

analysis. Cambridge University Press.

Ramachandran, V. S., & Gregory, R. L. (1991). Perceptual filling in of artificially induced

scotomas in human vision. Nature, 350, 699–702.

Ro, T., Breitmeyer, B., Burton, P., Singhal, N. S., & Lane, D. (2003). Feedback contributions

to visual awareness in human occipital cortex. Current Biology, 13, 1038–

1041.

Roelfsema, P. R., Lamme, V. A. F., & Spekreijse, H. (1998) Object-based attention in the primary visual cortex of the macaque monkey. Nature 395, 376–381.

Sasaki, Y., & Watanabe, T. (2004). The primary visual cortex fills in color. Proceedings of

the National Academy of Sciences, 101, 18251-18256.

Self, M. W., Kooijmans, R. N., Supèr, H., Lamme, V. A., & Roelfsema, P. R. (2012).

Different glutamate receptors convey feedforward and recurrent processing in macaque V1. Proceedings of the National Academy of Sciences, 109, 11031-11036.

Supèr, H., Spekreijse, H., & Lamme, V. A. (2001). Two distinct modes of sensory processing observed in monkey primary visual cortex (V1). Nature neuroscience, 4, 304-310.

Todorovic, D. (1987). The Craik-O’Brien-Cornsweet effect: new varieties and their

theoretical implications. Percept. Psychophys., 42, 545–560.

Van den Brink, G., & Keemink, C. J. (1976). Luminance gradients and edge effects. Vision

(29)

28 Van Tuijl, H. F. J. M. (1975). A new visual illusion: neonlike color spreading and

complementary color induction between subjective contours. Acta

Psychologica, 39, 441-45.

Wandell, B. A., Dumoulin, S. O., & Brewer, A. A. (2007). Visual field maps in human cortex. Neuron, 56, 366-383.

Woolrich, M. W., Jbabdi, S., Patenaude, B., Chappell, M., Makni, S., Behrens, T., & Smith, S. M. (2009). Bayesian analysis of neuroimaging data in FSL. Neuroimage, 45, S173-S186.

(30)

29 Appendix 1

Hierarchical processing of visual information

Visual information is processed in a hierarchically organised visual system (Figure 8). Starting with the retina being stimulated by light. This light is converted to brain signals (action potentials), which are transmitted via the thalamus (LGN) to the occipital lobe in the cortex (Guillery & Sherman, 2002). The primary visual cortex and other ‘early’ visual areas (V1, V2, etc.) contain a layer of neurons with small receptive fields organised in a retinotopic manner (Kaas et al., 1990). These areas represent simple image properties, as orientation and contrast (Hubel & Wiesel, 1962). Sequentially, later visual areas process information from a larger region of the visual field. These areas represent more complex information about shape and meaning of objects. However, this process does not explain the richness and details of our daily visual experiences. What exactly happens in this stream of visual processes causing us to experience a fully detailed visual representation, despite limited visual input?

Figure 8. Processing path of visual information

(31)

30 Appendix 2

Overview of trial types in different blocks used in the study Table 1

Display Types of the Mapper Block with a Total of 96 Trials

Display number

Number of trials without dot

Number of trials with dot

Trial Type

3) 9 3 central patch empty,

peripheral tilted lines of 20-40°

peripheral tilted lines of 15°

6) 9 3 central tilted lines of 15°,

peripheral patch empty

8) 9 3 central tilted lines of 20-40°,

(32)

31 Table 2

Display Types of Training Blocks with a Total of 80 Trials

Display number Number of trials without dot Number of trials with dot

Trial Type

1) 8 0 illusion trial, central patch of 15°,

2) 8 0 illusion trial, central patch of 45°,