The effects of statistical structure on (irrelevant) visual search in natural scenes

The effects of statistical structure on

(irrelevant) visual search in natural

scenes

A research providing insights into its underlying

mechanisms

Renée G.M. Olislagers

11670525

Supervised by Lynn Sörensen en Steven Scholte

University of Amsterdam, Department of Brain & Cognition

1

The effects of statistical structure on (irrelevant) visual search in natural scenes

A research providing insights into its underlying mechanisms

Renée G.M. Olislagers

Abstract

Human beings are remarkably fast with detecting objects in natural scenes. Natural scenes have a statistical structure in both the relative and absolute location of objects. Visual processing is adapted to this statistical structure, which increases the efficiency of visual search: object detection and perceptual processing is facilitated when objects appear in a statistically likely location. It remains unknown whether the statistical structure also affects the simultaneous search for an object with no positional regularity, and which is not relevant for the task. Here, the visual search for this object is called the irrelevant search. If true, this would be evidential for the underlying mechanism to be a spatial attentional bias based on statistical knowledge. In this thesis, a research is proposed in detailed elaboration to address this research question. A computer task shortly presenting images of street scenes was designed. Participants were cued with a target category (i.e. bicycle), defining the target for the relevant search. The images sometimes contained a Gabor patch, a stripy pattern with no positional regularity. The response time for reporting the detection of a Gabor patch was used as a measurement for irrelevant search. To analyse whether its detection was affected by a spatial bias, the statistical correlation was determined between a spatial representation of the response times according to Gabor patch location in the image and the spatial probability map for the location of the target object, and objects in street scenes in general. If this correlation indeed demonstrates a spatial bias in the detection of the Gabor patch, it suggests a spatial attentional bias to be present, induced by the task demands of the relevant search. However, if the results do not demonstrate the presence of a spatial bias in irrelevant search, this would suggest other mechanisms to be more plausible to underly the spatial bias in relevant search. Thus, the research elaborated in this thesis sheds light on the mechanisms underlying the effects of statistical structure of natural scenes on visual search. It therefore contributes to a better understanding of human vision in natural scenes.

Keywords: visual search, natural scenes, statistical scene structure, Gabor patch

Introduction

A broad spectrum of everyday life tasks relies on accurately detecting objects in our visual environment. Therefore, object search has been a topic of interest in the field of cognitive neuroscience. In artificial displays, when detecting a target within distractors sharing one or more visual features, the search effectiveness is highly decreased with the number of distractors is the display (Treisman & Gelade, 1980). However, in natural scenes humans are remarkably fast at detecting behaviourally relevant objects in the presence of a large number of distractors (Peelen & Kastner, 2014). This is true in spite of the similarity in visual features between target and distractor objects, like colour, size, and orientation. Thus, object search in natural scenes appears to be highly efficient, compared to artificial displays. The difference in search efficiency might be elicited by basal differences between natural scenes and artificial displays. Natural scenes differ from artificial displays in some important aspects: Peelen and Kastner (2014) stated that, in natural scenes, there is a regularity in the arrangement of distractors. Furthermore, they put forward that the effective search space for an object is reduced by the context of a natural scene. Hence, natural scenes have a statistical structure in both the relative and absolute location of objects. The relative location of an object is its location relative to other objects. The absolute location of an object can be defined as its location in visual space if one would fixate on the centre of a scene, or rather, its retinotopic coordinates (Kaiser, Quek, Cichy, & Peelen, 2019). For example, a carpet appears mostly in the lower parts of the visual field, whether a lamp appears in the upper parts of the visual fields (Russel et al., 2008).

2

Research has shown that statistical structure facilitates object detection. Violations of this statistical structure in natural scenes appears to decrease accuracy and speed of detection of objects (Biederman, Mezzanotte, & Rabinowitz, 1982). Also, perceptual performance is enhanced when objects appear in a statistically likely location (De Haas et al., 2016; Quek & Finkbeiner, 2014). Furthermore, the distribution of fixations is shifted from being more centrally biased toward the bottom of a scene for urban scenes, or toward the top of a scene for scenes of interiors (Parkurst & Niebur, 2003). Neider and Zelinsky (2006) showed object detection to be faster and with fewer eye movements when targets appeared on an expected location based on the contextual information in the scenes, compared to when targets appeared on random locations in the scene.

Moreover, statistical structure affects perceptual processing on a neuronal level. Continuous flash suppression breaks sooner when objects appear in typical absolute locations, suggesting this facilitates basic perceptual processing (Kaiser & Cichy, 2018a). Furthermore, when objects appeared in a statistically likely location, response patterns from fMRI data were decoded more successfully compared to objects appearing in a statistically unlikely location. This finding suggests that statistical structure enhances encoding efficiency of objects (Kaiser & Cichy, 2018b).

The mechanism underlying the facilitation of visual search in natural scenes by statistical structure could be caused by the statistical structure of natural scenes posing an attentional bias. The context and implicit memory of a scene is long known to guide spatial attention and eye movements, and thereby serve as a mechanism for effective indexing in a scene (Chun & Yuhong, 1998; Chun & Nakayama, 2000; Olivia, Torralba, Castelhano, & Henderson, 2003). Peelen and Kastner (2014) describe this attentional bias as a ‘where template’, defined as an attentional template for high probability locations for target presence based on scene context and episodic memory. A more tangible description is the attentional bias to be a spatial priority map, defined as the attentional priority in a topographic representation of the environment (Theeuwes, 2019). The spatial priority map is weighted by the spatial target probability, which is the probability at any location in the scene for the searched object (the target) being present. On a neuronal level, the spatial priority map of attention is defined by Zelinsky and Bisley (2015) as a neural representation of a topographic space. Here, different levels of activity in neurons representing the topographic space encode the relative priority of the corresponding locations. This priority is unaffected by bottom-up or top-down factors of attention. Thus, spatial priority maps are attentional priority maps/ attentional biases encoded in neural representations of the topographic space. Relative priority of locations is encoded with different levels of activity, weighted by the spatial target probability and thus statistical structure of a scene. Via this mechanism, statistical structure in natural scenes could facilitate object search.

Altogether, it appears that knowledge of the statistical structure of a scene, and thus statistical knowledge of object locations, facilitates search. The mechanism underlying this facilitation is suggested to be an attentional bias posed by the statistical structure of natural scenes, in the form of priority maps. Here, the facilitated search is the search for a target object, further called relevant visual search. However, it remains unknown whether statistical knowledge of scenes also affects irrelevant visual search. Irrelevant visual search here is defined as the search for an object irrelevant for the task, and for which the statistical structure does not apply and is thus irrelevant. Therefore, the research question proposed in this study is whether the statistical knowledge of both context and target location affects not only relevant search (target detection), but also irrelevant search in natural scenes. This is hypothesized to be true, for the attentional bias encoded in priority maps would facilitate the processing of objects in high target probability locations in the scene, and therefore also the processing of an irrelevant object located in these high probability locations. Hence, this hypothesis is directional, such that the statistical knowledge is positively affecting irrelevant visual search (facilitation), when the irrelevant object is located in a high probability target location. If the null-hypothesis is rejected, this would suggest statistical knowledge of target location to indeed pose an attentional bias functioning as a top-down attentional template. Thus, detection of irrelevant targets would be facilitated by statistical knowledge of the location of the relevant target. This effect would be independent of bottom-up effects due to the presence of a relevant target.

3

To address this question a computer task was designed in which participants were shortly presented with images of a street scenes. During the experiment, participants received an cue for a target category. This cue defines the (relevant) target objects, and tasked them with the relevant search. There are four different categories, being bicycle, car, stop sign, and fire hydrant. Every participant is cued with only one target category. To research irrelevant search, a Gabor patch was used as irrelevant target. A Gabor patch is a stripy pattern or irregularity (Figure 1B), which was randomly placed in the image, in 75% of the trials. The Gabor patch was chosen for this purpose because we can assume that no participant has visual experience with this stimulus, and thus has no implicit expectation of its location induced by statistical knowledge. After the picture was presented, participants had to report whether they detected a Gabor patch, or not. The statistical knowledge of the location of the target object is expected to induce a target bias. Moreover, a context bias is expected to be induced by the fact that all images present street scenes. Therefore, it is hypothesized that if Gabor patches appear in a high probability location for the target object, the response time for reporting its detection is decreased. To test if this is true, for every participant, a spatial representation of the response times for Gabor patch detection report, hereafter named response time grid, is created according to Gabor patch location. Furthermore, a spatial probability map of the individual target categories, and objects in street scenes (the context) in general, is created based on a data set of street scene images. These spatial probability maps represent the target bias and context bias induced by statistical knowledge of natural scenes, respectively. Subsequently, a repeated measures correlation (Bakdash & Marusich, 2017) will be performed on the response time data with the target bias and the response time data with the context bias. Furthermore, the same analysis is executed for a centre bias, for this represents an attentional bias induced by looking at the fixation cross. Namely, the fixation distribution is centrally biased when experiments use a pre-trial fixation cross (Parkurst & Niebur, 2003). In line with the hypothesis, the target bias and context bias are both expected to significantly correlate to the response time grids. Furthermore, the correlation of response time with the centre bias is expected to approach significance, because of its similarities with the context bias. In this thesis, the methodology of the proposed study is described in detail, and regarding choices are explained. A pilot study was done to finetune experimental settings and perform a small data analysis, which is described throughout the methods section. In the pilot study, participants were not cued with a target category. Consequently, they were only tasked with detecting the Gabor patch. Results of the pilot study are incorporated and factored in methodological choices. Furthermore, different outcome scenarios of the study are scrutinized with both a methodological and theoretical point of view. Methodological considerations and alternatives are discussed. From a theoretical point of view, the possible outcome scenarios are explained and implications are elaborated on. Lastly, suggestions for further research are given, aiming to further distinguish possible underlying mechanisms for the facilitation of object search in natural scenes by statistical knowledge.

Methodology

Participants

Data collection will be performed using human subjects. Subjects will mostly be psychology and psychobiology students from the University of Amsterdam. Recruitment will be done by advertising the study on lab.uva.nl and with posters on campus. The offered reward will be either 1.5 psychology research credit or 15 euros. Inclusion criteria are normal or corrected vision and no history of psychiatric or neurological disorders. Reason for this was a probable impaired ability to detect mainly irrelevant targets, but also relevant targets, and influences on concentration and engagement in the task, respectively. Impaired vision and psychiatric or neurological disorders are therefore seen as factors that could influence the outcome of the study. Moreover, participants have to be between the age of 16 and 39. This range is chosen to avoid the effect of age on performance in visual search tasks (Potter, Grealy, Elliott, & Andrés, 2012).

To estimate the appropriate sample size, the possibilities for performing a power analysis were explored. However, in this study data analysis is done by performing a repeated measures correlation (see statistical analysis section). As yet, there is no published power analysis method for this data analysis

4

method. Bakdash and Marusich (2017) advise to use the entries for a Pearson correlation in G*Power (Faul, Erdfelder, Lang, & Buchner, 2007), and adjust the degrees of freedom. However, this does not suffice for a priori analysis, for the degrees of freedom depend on the sample size (Bakdash & Marusich, 2017). Moreover, selecting an appropriate sample size is complicated for repeated measures designs, for calculations are often based on oversimplified assumptions, giving false confidence on the chosen sample size (Guo, Logan, Glueck, & Muller, 2013). Therefore, I chose to estimate the sample size based on previous research. Sample size is primarily affected by study design, method of sampling and outcome measure (Chander, 2017). Preferably, the sample size estimation is thus based on studies with a similar study design. Specifically, studies measuring response times for detection of objects in shortly presented natural stimuli, are preferred. Sample sizes of 69 and 84 participants were used in a study measuring response time for object detection in 150 ms presented drawings of natural scenes, for the purpose of researching the effect of object location violations (Biederman et al., 1982). The study is similar to our study in terms of stimulus presentation time and study design. Furthermore, a study analysing response time for reporting the location (left or right) of a cued target object in a 67 ms presented image, followed by a mask, had a sample size of 66 participants (Reeder & Peelen, 2013). Similarities between this and our study are mainly in study design and method of sampling. Another study, testing only 12 participants, matched methodologically with our study in the sense that it researched the effects of object location in scenes on detection abilities (Neider & Zelinsky, 2006). However, the main focus of this study was eye-movements. Consequently stimuli-offset was when participants reported detection, making response time in this study is more a measurement of how long it takes to find the target through eye-saccades. Therefore, additionally considering their sample size being in another range, the two previously described studies are considered most comparable to our study on relevant aspects of study design. Accordingly, sample size estimations are based on these studies. The sample sizes are approximately in the range of 65 to 68 participants. Hence, a sample size of 75 participants is estimated to be appropriate for this study.

In the pilot study, 10 participants executed the experiment. The experiment varied in the first five participants, to finetune the experimental settings. The data obtained from the last five participants was used in the data analysis.

Experimental design

This study will have a within subjects design. All subjects are cued with one target category throughout the experiment. There are four different categories: bicycle, stop sign, car, and fire hydrant. These categories are selected because their spatial probability maps differ from each other in where in the image their high probability locations for object presence mainly are. Moreover, all four spatial probability maps differed in this sense from the spatial probability map of objects in street scenes in general as well. This is visible in Figure 4A and 4B. Except for the cued target category being different, the experiment will be the same across participants. Therefore, the resulting data will have the same format.

A design where every participant would be cued with all four categories throughout the experiment has been taken into consideration. However, this design was rejected due to the limited amount of trials per participant. During piloting, 1000 trials was determined to be the highest feasible amount of trials in a given session. Only trials containing a Gabor patch are used for the response time analysis. Since the amount of trials where a Gabor patch is present is set at 75%, only 750 trials are available for data analysis. Furthermore, trials with a category cue have to contain a target object in 50% of trials, to prevent implicit learning and thereby prompting a bias to not look for the target object anymore. These trials are also not included in the analysis. The why of this is explained later under ‘data analysis’. Moreover, only trials where the Gabor patch was detected correctly are included in the data analysis, which is aimed to be around 75% of the trials containing a Gabor patch. Altogether, this results in about 420 trials per category, and thus four response time grids based on about 105 trials each. For the purpose of obtaining evenly distributed response times, the 500x500 pixels (px) pictures are divided in a 7x7 grid. Thus, when evenly distributed, 105 trials would maximally result in 2-3 trials defining the response time represented in every box (105 trials divided by 49 boxes in the grid). Hence, due to the limited amount of trials per participant, a design where every participant is cued with all categories is not

5

considered possible. Therefore, in the current study design participants are cued with only one target category.

Allocation of participants to the different target categories will happen pseudorandomly, using block randomization. This allocation can be double blinded. However, knowing the cued category will have no influence on the results, because varying the category is not a manipulative condition, but simply a version of the experiment.

Apparatus

During the experiment participants will be placed in a chin-holder to secure a constant distance to the screen between trials and between subjects. This distance is set at 76 cm from chin to screen-centre. The experiment will be presented on a screen with a screen resolution of 1024 x 1280 pixels.

The picture presentation time will be set at 150 ms. Therefore, it is important the participant is fixating at the onset of the trial. To make sure of this, every trial will start after 500 ms of fixation. Fixation will be monitored using an Eyetracker (Eyelink). The fixation is allowed to deviate 50 pixels from the centre of the screen (picture size:500x500 px). Calibration accuracy has to be rated as ‘good’ for the experiment to proceed. However, if not possible, ‘fair’ was also accepted.

Stimuli

The images of natural scenes used in this experiment were from the 2017 training set from the MS COCO database (Lin et al., 2014). The images in this data base have pixel by pixel stuff annotations, meaning for every pixel it is annotated by what objects and suchlike they are occupied. To obtain street scene images with similar context only images annotated with ‘road’ were selected (Caesar, Uijlings, & Ferrari, 2018). After selection, images were resized to a 500 x 500 pixel format.

During the experiment, there will be 4 different kinds of stimuli, as 75% of stimuli will contain a Gabor patch, and 50% of stimuli will contain a target object from the cued category.

In this experiment, Gabor patches are used as irrelevant targets (See Figure 1B). A Gabor patch is a sinusoidal grating with a Gaussian envelope (Fredericksen, Bex, & Verstraten, 1997). Gabor patch detection is a newly introduced measurement for irrelevant search in natural scenes. Therefore, the pilot experiment was, among other things, used to optimize the settings for the features of the Gabor patch for Gabor patch detection to be not too hard and not too easy. This resulted in the following settings for the Gabor patch: The spatial frequency of the Gabor patch was ranged between 0.1 and 0.25, and randomly sampled within this range. Similarly, the orientation of the Gabor patch was randomly sampled between a range of -90°– 90°. The spatial frequency and the orientation of the Gabor patch were ranged to disrupt participants from developing an expectation of its visual appearance. Participants are preferred to develop a rather categorical idea of Gabor patches, robust to variations. The size of the Gabor patches was set at 50 pixels and the standard deviation of the gaussian envelope was set at 10 pixels.

For the Gabor patches to be evenly sampled across the images, the images are divided in a 7x7 grid. Gabor patch placement is then done pseudorandomly along the 7x7 grid. For a schematic representation of the grid and Gabor patch placement see Figure 1A . The Gabor patch was never placed in the centre of the image, for the fixation cross could interfere with its detection.

6

A B

C D

Figure 1. This figure clarifies the way stimuli are created and what they look like. A: The figure gives a schematic representation of the 7x7 grid used to subdivide the images. The squares are numbered in bold. The images are resized to 500 x 500 pixels. The numbers at the bottom of the grid show the starting points of the squares in pixels. The numbers above and on the left side of the grid show the x- and y- coordinates of the centre of the Gabor patch in pixels. To illustrate, the coordinates of the centre of a Gabor patch placed in the first square (1, 1) are shown. In the central square (4, 4), the fixation cross is shown. B: Gabor patch. C+D: Examples of stimuli containing a Gabor patch.

Trials

Every trial starts with a fixation cross. After 500 ms of fixation, the picture is presented for 150 ms. The reason for this is to make sure the participant is focused on the centre of the screen when the picture is presented. After every trial, participants are asked if a Gabor patch was present. Gabor patch detection was reported by pressing the m- and z-key on a QWERTY keyboard. Here, pressing the m-key meant ‘yes, I detected a Gabor patch’, and pressing the z-key meant ‘no, I did not detect a Gabor patch’. It is important the participant stays engaged with the task of detecting a target object from the cued category as well. Therefore, after every trials they will be asked if the target was present as well. This does not apply for the pilot experiment, for participants were not cued or tasked with detecting the target here.

Procedure

All participants will be asked to read the information brochure and sign the informed consent beforehand. The experiment starts with 40 practice trials. In the first 21 trials, the picture presentation time is set at 300 ms, whereafter it decreases every trial until 200 ms is reached, for the participant to

7

adjust gradually. Participants will be asked to respond as fast and accurately as possible. The main experiment consists of a 1000 trials, subdivided in blocks of 50 trials. This was determined to be a feasible amount based on pilot participants’ reports on the length of the experiment, and blocks. The target category is cued at the start of the experiment, and at the start of every block. Breaks are inserted after every block, where participants receive feedback on their general accuracy percentage of both target object and Gabor patch detection separately and mean response time for Gabor patch detection report in the last block. Feedback after every block is given because pilot participants reported this to motivate them to improve in the next block. Also, the percentage of trials where there was a false positive, a report of detection of a Gabor patch in the absence of it, is shown. This is done to prevent participants from strategically reporting they detected a Gabor patch when they were uncertain, since 75% of the trials contained a Gabor patch. The experiment proceeds when pressing a random key. Conducting the experiment takes up one hour in total. Therefore, after 500 trials a big break is inserted, where participants can stand up and walk around. For a task flow of one block see Figure 2A.

The pilot experiment followed a similar procedure, however without a category cue, and with only feedback concerning Gabor patch detection. For a task flow of one block of the pilot experiment see Figure 2B.

A B

Figure 2. Task flow of one block. The part of the task flow indicated with braces is repeated 50 times in each block. A: task flow of the main experiment. B: task flow of the pilot experiment.

Data analysis

There are two outcome variables in this study, accuracy and response time. Both outcome variables have been considered to be the focus of this study. However, in this study design, the data analysis focusses on the response time as outcome variable, for it is, contrary to accuracy, a gradual measurement of search. Data analysis is conducted in Google Colaboratory (2018).

Pre-processing

Only trials where participants correctly detected the Gabor patch are included in the analysis of the response time (RT) data. Moreover, only trials in which there was no target object from the cued category present are included in the analysis, for the presence of a target object draws attention. Therefore, the response time and accuracy of reporting detection of the Gabor patch could be affected by the location of the target object. For instance, when the Gabor patch is close to the target object, the response time for its detection could be decreased and vice versa. Outliers are defined by the ±3 SD rule (Lachaud & Ranaud, 2011), and are removed from the dataset. If 10% of trials or more would have to be excluded due to outliers, the participant will be excluded as a whole (Endo & Takeda, 2004). These

8

outlier criteria were applied to the pilot data, and resulted in the exclusion of 9.8 trials on average. None of the participants had to be excluded as a consequence of more than 10% of outliers. Outlier removal resulted in the response time distributions to approach a more normal distribution. Moreover, this applies to the response time grids. Based on the effects of the proposed outlier criteria on the pilot data, the criteria were considered to be suited for the data of the main experiment.

Pre-processing of the response time data from the pilot experiment will be done according to similar criteria. Here, only trials where participants correctly detected the Gabor patch are included in the analysis of the RT data. Outlier and subject exclusion criteria remain the same.

Response time grids

For every participant a 7x7 response time grid was obtained. For every individual square, we compute the median response time of all trials with the Gabor patch located within that square. The reason for calculating the median response time instead of the mean is that response time is usually not normally distributed but always right skewed. In the pilot study, because of the small sample size, the response time grids were smoothed by applying a Gaussian filter. To do so, the missing value in the centred square was first replaced by zero (Gabor patches were never located in the centred square because of the fixation cross). Constant padding was done with the mean of the 48 medians. For an example of a response time grid and the corresponding response time distribution see Figure 3A and 3B, respectively. For the response time grids of all pilot participants see Appendix Figure 1.

A B

Figure 3. The purpose of this figure is to illustrate the transformation of the response time distribution to a response time grid, by showing an example of one pilot participant. The response time grids of all participants can be found in Appendix Figure 1. A: Example of a 7x7 colour-coded grid of one pilot participant presenting the median response time according to Gabor patch location. B:The response time distribution (histogram) of the pre-processed response time data of the same participant as corresponding to the response time grid in A. The dashed line indicates the median.

Spatial probability maps

The spatial probability maps are created based on images from the same database as used for the stimuli. Again, images annotated with ‘road’ were selected to obtain images of street scenes. Thereafter, for every individual object category (bicycle, car, motorcycle, bus, truck, fire hydrant, stop sign, parking meter and bench), images were selected on containing only one instance from the category. The resulting 111529 images were then standardized for the purpose of similar spatial dimensions, by resizing all images to 224x224 pixels. Here, the rule of thumb was for the randomly selected portion of the picture to contain at least 80% of the original target object. The spatial probability maps where obtained by calculating for every pixel of the standardized picture format how often a given pixel was occupied by a target object. Then, a normalization is applied so that the sum of all pixels is equal to one. This results in the value in each pixel to represent the probability of that pixel being ‘occupied’ by an object (See Figure 4A and 4B). The spatial probability maps are then transformed to a 7x7 grid, for them to be spatially comparable to the response time grids. To do so, the 224 x 224 pixel images are divided in 49

9

squares of 32 x 32 pixels, and the sum of all 32 pixels per square is calculated to attain the proportional value for the ‘occupation’ of the squares (See Figure 4C and 4D).

To obtain the spatial target probability maps for the four target categories cued in the experiment the following amount of images was used: stop sign: 510, fire hydrant: 560, car: 3500, bicycle: 769. Because the amount of images meeting the criteria differs between categories, the spatial target probabilities are corrected for the amount of images used to produce them.

To obtain the spatial probability map for a street context in general, images of 9 different object categories were used. Image selection resulted in the following amount of images per category (bicycle: 769, car: 3500, motorcycle: 1107, bus: 2088, truck: 2428, fire hydrant: 560, stop sign: 510, parking meter: 210, bench: 357).

A B

Street scene

C D

Street scene

Figure 4. The purpose of this figure is to illustrate the idea of the spatial probability maps (A+B) and to visualized the spatial biases used in the statistical analysis A: Spatial target probability map of street scenes in general, so for all object categories. This map represents the context bias. B: Spatial target probability maps of the individual cued target categories. These maps represent the target biases. The spatial probability maps in A and B are not divided in a 7x7 grid, but the proportions are calculated per pixel. C and D present the spatial target probability maps divided into a 7x7 grid. C shows the context bias and D shows the target biases of the cued target categories, as used in the statistical analysis.

Centre bias

The centre bias was based on what part of the picture is processed in foveal and perifoveal vision, which is 6° of visual angle (Engbert, Longrin, & Kliegl, 2002; Tatler, 2007). This was calculated to be 199.93 pixels horizontally, and 286.22 pixels vertically, based on the screen resolutions and dimensions of the screen used to present the experiment. It was considered to confine the centre bias to only foveal vision. However, the resulting centre bias only varied in the centre of the image, in the square where we have no datapoint, for no Gabor patches were placed on the fixation cross. The centre bias was created using a 2-dimensional gaussian function with σx= 199.93 and σy=286.22, corresponding to the perifoveal

10

reach. Both x and y ranged from 0 to 500 (dimensions of the images in pixels), and x0 and y0 where set

at the centre (250 pixels), corresponding to the location of the fixation cross in the presented images. The resulting matrix was normalized to represent proportional values (sum equal to 1), and transformed to a 7x7 grid to be statistically comparable to the response time grids.

A B

Figure 5. The purpose of this figure is to illustrate the idea of the centre bias (A) and to visualize the centre bias as used in the statistical analysis (B). A: Centre bias with same dimensions as images (500x500), represented proportionally. B: Centre bias transformed to a 7x7 grid, by taking the sum of all proportions in each square.

Statistical analysis

The next step in the data analysis is to compute how well the obtained response time grids correspond to the spatial probability maps. To do so, a repeated measures correlation test is used (rmcorr). This statistical method determines the within-individual association, in this case between the median normalized response time and the proportional probability of a target being there (biases) for each square in the 7x7 grid (datapoints) (Bakdash & Marusich, 2017). It thus adjusts for inter individual variability. A rmcorr is executed for comparision between RT and the target bias, the context bias, and the centre bias. The rm_corr and plot_rm_corr function from the python package pingouin are used (Vallat, 2018). In the case that the assumption of normality is not met for either the biases or the response time grids, the rm corr will be transformed in a non-parametric test by ranking all values first. The assumption of normality is checked using the Shapiro test of normality. Only the centre bias (W=0.964, p=0.147) and the target bias of the target category ‘car’ (W=0.962, p=0.121) are normally distributed. All other biases are not normally distributed (context: W=0.933, p=0.009; bicycle: W=0.912, p=0.002; fire hydrant: W=0.943, p=0.022; stop sign: W=0.921, p=0.003). Thus, solely the centre bias can be parametrically tested, provided that all response time grids are normally distributed. In the pilot experiment, the median response times of 3 of the 5 participants were normally distributed (W=0.953, p=0.052; W=0.964, p=0.148; W=0.978, p=0.516). The median response times of two participants were not normally distributed (W=0.889, p=0.0003; W=0.950, p=0.041). The relatively low amount of participants with a normally distributed response time grid indicates a high possibility of non-parametric testing in the main experiment as well. Therefore, all testing is expected to be non-parametric.

In the pilot experiment, the response time data was only compared to the context bias and the centre bias, for the target category cue was not present. Since not all response time distributions were normally distributed, both the median response times and the biases were ranked before performing the rmcorr. The rmcorr on the ranked response time data with the ranked context bias resulted in a rm correlation value (r) of 0.016934 (df=234, p=0.796, 95% CI [-0.11, 0.14]). For the ranked centre bias, a rm correlation value of 0. 062505 was found (df=234, p=0.33904, 95% CI [-0.07, 0.19]). See Figure 6 for a visualisation of both rm correlations. The power was respectively calculated to be 0.058 and 0.156. For a visualisation of all response time distributions and their medians, see Appendix Figure 2.

These results show no significant correlation between the biases with the response times, and thus suggest no predictive abilities of the biases for the response time grids. However, only five participant were measured in this pilot study, which is observable in the low power of the currently performed rm

11

correlations. This is likely the cause of the low correlations and high p-values found in the pilot study. With an appropriate sample size, this will likely not be an issue. For now, the results of the pilot study just show the statistical analysis to be suited for the type of data obtained in this study.

A B

Figure 6. This figure provides a visualisation of the results of the rmcorr on the pilot data. A: rmcorr plot for the ranked median of the response time for every Gabor patch location with the probability values in the spatial probability map of the ranked context bias (df=234, p=0.796, 95% CI [-0.11, 0.14]). B: rmcorr plot for the ranked median of the response time for every Gabor patch location with the probability values in ranked centre bias (df=234, p=0.33904, 95% CI [-0.07, 0.19]).

Scenarios

To recap, natural scenes have a statistical structure in object location which affects visual search on both a behavioural and neuronal level. The underlying mechanism is thought to be an attentional spatial bias based on this statistical knowledge of both context and the location of task specific targets. The research question of the proposed research is whether this statistical knowledge also affects the search for an object which is irrelevant for the task. If true, this would be of evidential value for an attentional bias to be the underlying mechanism, as the spatial bias would then also affect irrelevant visual search. A study was proposed to address this research question, wherein participants are tasked with detecting a target from a cued object category in images of street scenes. We assumed this to induce both a context and a target bias. Moreover, participants were tasked with detecting Gabor patches in these scenes, functioning as the irrelevant target. Consequently, we expect response time of Gabor patch detection to be affected by the spatial attentional bias induced searching for the target object. The response time for Gabor patch detection report was placed into a 7x7 grid according to Gabor patch location. Ultimately, the response time grid was compared to the spatial biases performing a rm correlation.

In this section different outcome scenarios for the described study will be scrutinized. First, methodological considerations and alternatives, and their effects will be discussed. Thereafter, theoretical explanations of outcome scenarios of the experiment will be discussed, and placed into a bigger picture.

Methodological criticism and control experiment

If no significant statistical correlation is found for the response time data with the target- and/or the context bias, there are several potential methodological explanations.

The first explanation is the experimental design not being capable of inducing the target bias. In the proposed experimental design, participants are tasked with two searches, being the Gabor patch detection and the target detection. Tasking participants with detecting objects from the cued category is meant to induce an attentional bias based on the statistical knowledge of target location in the scene, called the target bias. However, participants could, either implicitly or explicitly, rate the importance of the detection of the Gabor patch above the detection of the target, and thereby focus on the first task. There are several cues for the importance of Gabor patch detection in the experiment, like the first question after every trial being if they detected the Gabor patch. The presence of the target is always

12

questioned secondly. Alternating this sequence is not possible, for this would reduce the validity of the measured response time. Moreover, answering the question on target detection first could reduce report performance on the second question (Averbach & Sperling, 1961). Furthermore, feedback after every block only incorporates the response time for report of Gabor patch detection, which also implicitly states the importance of detecting the Gabor patch over the detection of the target. Incorporating feedback on the response time of the target detection report as well has been considered, however rejected, for as described before, the response time for answering a second question is not reliable. Taken together, the experimental design could imply a lesser importance of detecting the target. As a result, no attentional bias for the target category (the target bias) would be induced, and thus no significant statistical correlation the response time data and this bias can be found.

Secondly, it should be taken into account that if a significant statistical correlation is found for the response time data with the target biases, and especially the context bias, this could be due to its similarity with the centre bias. A centre bias, that is, a fixation bias, can be induced by fixating on the centre of the screen before stimulus onset (Parkurst & Niebur, 2003). If this bias is relatively strong, the centre bias will explain the response time data and therefore result in a high statistical correlation with the response time data. If there is indeed a strong centre bias demonstrated, more attention has to be paid to what extent the resulting statistical correlation for the context and target bias is due to its similarity with the centre bias. Hence, conclusions on the statistical correlation results for the context and target bias have to be nuanced by taking into account the statistical correlation result of the centre bias. Lastly, there is a possibility for the experimental design to not be powerful enough to demonstrate any bias. If true, no significant correlation results from the main experiment. These results could then be incorrectly interpreted, while they just show the experimental approach to be incapable of demonstrating spatial biases in visual search. Therefore, I suggest a control experiment, to test whether the current method is capable of showing the biases to be present. This experiment would act as a positive control, and therefore focusses only on target detection. The experimental design would be as similar as possible to the main experiment, however it would leave out the Gabor patches. In short, participants would be presented with images of natural scenes, and are asked after every image if they detected an object from the cued target category. Their response times would be organized according to target location, and similarly compared to a target-, context-, and centre bias using a rmcorr. If results are significant, this would show the experimental design capable of demonstrating the presence of biases in relevant search. Therefore, this design is suited to demonstrate these biases for irrelevant search as well.

However, this control experiment is not yet incorporated in the methods section, because it is rather complicated to be properly designed. Firstly, there is the problem of defining the location of the target object. In the main experiment, the size of the Gabor patch remains constant and is artificially placed in the centre of a square of a 7x7 grid. Consequently, it always occupies only one square. However, objects in natural scenes vary in size, and will therefore differ in the amount of pixels they ‘occupy’ in the image. This raises the problem of where to ‘place’ the response time for detection report of the object, when making the response time grid. Furthermore, target objects have a bias in their location in natural scenes, and are thus often placed in statistically likely locations. However, in this experiment images with even sampling of the target objects along the image are required, meaning objects appear equally as much in every part of the image. Thus, it would require a careful selection of images.

A solution for these problems would be to artificially place a target object along a 7x7 grid, small enough to occupy exactly one square. However, this raises another problem, namely the relative size of the target object. When a target object with a constant size is randomly placed in natural images, the relative size and location of the object varies depending on both the location in the image and the angle in which the picture was taken. The relative location of objects is known to affect visual processing (Bar, 2004; Kaiser, Stein, & Peelen, 2014) and could therefore be of influence on object detection. Thus, artificially placing a target object in the scene could have undesired effect on the results, and is therefore considered unsatisfactory.

Altogether, there are several methodological difficulties that have to be considered and solved. However, if these difficulties are addressed, the proposed control experiment could function as a strong positive control for the experimental design.

13

Theoretical explanations and implications

Besides the methodological view on different outcome scenarios of this study, it is important to place the scenarios in a theoretical framework. Here I discuss the theoretical implications of different outcome scenarios in the case the findings are not the result of methodological flaws.

In the hypothesized scenario, both the target bias and the context bias show a significant statistical correlation with the response time data. On an experimental level this means participants are faster at reporting a Gabor patch when it is placed in a location where it is statistically likely for objects in general, or specifically the target object to be placed, given the context of the image (street scene). On a more general level this suggests statistical knowledge imposes an expectation of the location of a relevant target, which in its turn affects the search for an irrelevant target. More specifically, it suggests an attentional bias to be imposed by the statistical knowledge of object locations in the given context. As described in the introduction, this would be encoded as an attentional priority map, a neural representation of the topographic space with the levels of activity of the concerning neurons encoding the relative attentional priority of represented locations. Placed in a practical context, this would imply the demands for one task to affect a simultaneously executed task. That is, if statistical knowledge of both context in general and specific targets is relevant for one task, however not for a simultaneously executed task, the resulting spatial bias will influence processing for both tasks. The topographic attentional template is then driven by task demands, and applied on visual information processing in general, independent of whether it is simultaneously used for a task in which this statistical knowledge is irrelevant.

However, the attentional bias is not necessarily applied on visual information processing in general. Participants are aware of the fact that the Gabor patch can appear at any location in the image, and thus have no statistical expectation for its placement. Therefore, it is given that their statistical knowledge of the location of objects in contexts, in this case street scenes, is of no value for the Gabor patch search. Hence, if no significant statistical correlation could be found, this could suggest participants to be adaptive to the usefulness of statistical priors.

A second outcome scenario for the analysis of this study would be the context bias to show a significant statistical correlation with the response time data, however not the target bias. On an experimental level this means participants are faster with reporting Gabor patch detection when the Gabor patch appears in high probability locations for objects in general in this specific context (street scenes). However, report of Gabor patch detection has no clear correlation with high probability locations for a specific object category, when participants are simultaneously tasked with searching for these objects. This indicates the statistically based attentional spatial bias to be induced, not driven by task demands, but implicitly and automatically, just by scene context. Thus, when performing a search in a street scene, this attentional template automatically guides attention to locations where objects likely appear. For example, the road, a high probability location, is often more centred and in lower part of the picture, while the sky, a low probability location, is often more in the upper part of the picture. In contrast, a target bias is primarily relevant in the specific task of searching the target, and is therefore not induced as an attentional bias.

An explanation for the biases not to correlate with the response time data for the Gabor patch detection can be found in the nature of priority maps. Previously, we treated these exclusively as attentional biases based on statistical knowledge of object location. However, priority maps of topographic location might not be solely based on this statistical knowledge. Contrarily, important locations in space are feature-based prioritized instead of statistically-feature-based prioritized. Zelinsky and Bisley (2015) describe this in their review as follows: feature importance is weighted in the topographic space. Processing of locations in the topographic space is then prioritized in a competitive manner, based on feature presence. Supporting research comes from Reeder and Peelen (2013). In this study, they cued participants to detect either people or cars in natural scenes. In some trials, instead of an image of a natural scene, participants were presented with silhouettes of either cars or people. This stimulus was directly followed by a dot. Faster detection of the dot was interpreted as the stimulus in this location matching the active search template, and thereby capturing attention. Silhouettes of the cued search category were shown to capture attention, irrespective of orientation and especially strongly on the location of category-diagnostic object parts. This research shows attentional priority of topographic locations to be guided in a feature-based

14

manner. If priority maps are feature-based, and the category cue activates a certain search template, Gabor patches will not match this search template, and therefore not capture attention. Thus, if true, the response time of Gabor patch detection report will not correlate with the spatial biases.

Another explanation involves the way in which spatial biases are encoded in the neural system. In this thesis, I argue the spatial biases to be encoded as attentional biases. In contrast, they could also be encoded in the neural tuning of receptive fields higher in the visual stream, towards relevant areas of the visual field. In this case, the spatial bias is not induced by task demands, but is present by default. This would explain the absence of spatial biases in irrelevant visual search. The previously referred to review by Kaiser et al. (2019) gives a clear overview of evidential research for this statement. The in my opinion most important discussed researches are described below. Evidence for the statement mainly comes from population receptive field (pRF) analysis, a method for estimating the receptive fields of neuron populations, described by Dumoulin and Wandel (2008). Le, Witthoft, Ben-Shachar, and Wandell (2017) researched the reading region selective to word forms in the ventral occipital-temporal cortex using this method. They showed central vision and the horizontal median to be the preeminent portion of the visual field to evoke a response in this area, which is the portion considered to contain relevant information for reading. Furthermore, similar research from Silson, Groen, Kravitz and Baker (2016) show the occipital face area (OFA) to have a more fovea concentrated bias, while the occipital place area (OPA) shows a more peripheral bias. Moreover, they show the lateral occipital cortex (LO), the area containing object selective regions, to have a centred lower visual field bias. Altogether, these studies suggest neural tuning of higher visual areas to relevant or high probability locations of the receptive field in regard to their function.

To further research on whether the statistically based spatial bias observed in previous research (Biederman et al., 1982; De Haas et al., 2016; Quek & Finkbeiner, 2014) is (partially) explainable by this neural tuning, using the pRF method on the more specific object selective regions of the LO is to be explored. To do so, I propose roughly the following study: the pRF of regions responsive to specific object categories like vehicles, persons, or traffic signs to be determined. Furthermore, spatial probability maps are created for these specific object categories based on large-scale databases, similarly to the current study’s approach. To test whether these spatial biases in neural tuning are indeed based on statistical likelihood of object locations, the similarity of the spatial probability maps to the pRFs of the specific object categories is analysed. If a clear similarity is demonstrated, this is of evidential value for the theory that statistically based spatial biases are indeed encoded by neural tuning of receptive field higher in the visual stream.

To summarize, there are methodological, as well as theoretical explanations to appoint regarding the different outcome scenarios for the proposed research. The method could fail to induce the spatial biases by implicitly prioritizing Gabor patch detection. Furthermore, the presence of a centre bias could partially explain the potentially demonstrated spatial biases. From a theoretical point of view, a significant statistical correlation between the spatial biases and the response time data would be of evidential value for an attentional bias to be the underlying mechanism. The spatial bias could either be driven by task demands and apply to visual processing in general, or apply to only the demanding task. In contrary, the spatial bias could not be driven by task demands, but induced implicitly and automatically, just by scene context. Moreover, rather than an attentional bias, the spatial biases could be encoded in the neural tuning of receptive fields higher in the visual stream towards areas of the visual field, relevant to the concerning region. Follow-up research could further explore the contribution of this underlying mechanism, by comparing pRFs of category selective regions in the lateral occipital cortex to the concerning spatial target biases.

To come to a conclusion, the field of human vision research currently knows the statistical structure of object locations in natural scenes to affect visual search on both a behavioural and neural processing level. The spatial biases are here argued to be encoded as an attentional bias. The outcome of the currently designed research could either be evidential for this theory, or it could indicate a different mechanism to be more plausible. Thus, the research elaborated in this thesis sheds light on the mechanisms underlying the effects of statistical structure. It therefore contributes to a better understanding of human vision in natural scenes.

15

Appendix

Figure 1 . The figure shows the response time grids for all five pilot participants, to provide a complete illustration of the collected data.

16

Figure 2. The figure shows the response time distributions (normed histograms) of all five pilot participants plotted in one figure. The dashed lines indicate median for every participant. The purpose of this figure is to visualize the median response times and response time distributions and how they relate to each other.

17

Literature

Averbach, E., and Sperling, G. (1961). Short-term storage of information in vision. In Information

Theory, ed. C. Cherry, 196–211. London: Butterworth.

Bakdash, J.Z.,& Marusich, L.R. (2017). Repeated Measures Correlation. Front. Psychol., 8, 456.doi: 10.3389/fpsyg.2017.00456

Bar, M. (2004). Visual Objects in Context. Nature reviews Neuroscience, 5(8), 617-29. doi:10.1038/nrn1476.

Biederman, I., Mezzanotte, R.J., Rabinowitz, J.C. (1982). Scene Perception" Detecting and Judging Objects Undergoing Relational Violations. Cognitive psychology, 14, 143-177.

Caesar, H., Uijlings, J., Ferrari, V. (2018). COCO-Stuff: Thing and Stuff Classes in Context. The

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1209-1218.

Chander, N.G. (2017). Sample size estimation. J Indian Prosthodont Soc., 17(c), 217-218. doi:10.4103/jips.jips_169_17

Chun, M.M., Nakayama, K. (2000). On the Functional Role of Implicit Visual Memory for the Adaptive Deployment of Attention Across Scenes. Visual cognition, 7(1/2/3), 65-81.

Chun, M.M, Yuhong, J. (1998). Contextual Cueing: Implicit Learning and Memory of Visual Context Guides Spatial Attention. Cognitive psychology, 36, 28-71.

De Haas, B., et al. (2016). Perception and processing of faces in the human brain is tuned to typical feature locations. J. Neurosci., 36, 9289-9302.

Dumoulin, S.O., Wandel, B.A. (2008). Population receptive field estimates in human visual cortex.

NeuroImage, 39(2), 647-660. doi:https://doi.org/10.1016/j.neuroimage.2007.09.034

Endo, N., & Takeda, Y. (2004). Selective learning of spatial configuration and object identity in visual search. Perception & Psychophysics, 66(2), 293-302.

Engbert, R., Longrin, A., Kliegl, R. (2002). A dynamical model of saccade generation in reading based on spatially distributed lexical processing. Vision Research, 42(5), 621-636. doi:10.1016/s0042-6989(01)00301-7

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.

Fredericksen, R.E., Bex, P.J., Verstraten, F.A.J. (1997). How big is a Gabor patch, and why should we care?. Opt. Soc. Am. A, 14(1)

Colaboratory: Frequently Asked Questions, (2018, June), [online] Available: https://research.google.com/colaboratory/faq.html.

Guo, Y., Logan, H.L., Glueck, D.H., Muller, K.E. (2013). Selecting a sample size for studies with repeated measures. BCM Medical Research Methodology, 13, 100. doi:https://doi.org/10.1186/1471-2288-13-100

Kaiser, D., Cichy, R.M. (2018a). Typical visual-field locations facilitate access to awareness for everyday objects. Cognition., 180: 118-122

18

Kaiser, D., Cichy, R.M. (2018b). Tyical visual-field locations enhance processing in object-selective channels of human occipital cortex. J Neurophysiol, 120(2), 848-853. doi:10.1152/jn.00229.2018 Kaiser, D., Quek, G. L., Cichy, R. M., & Peelen, M. V. (2019). Object vision in a structured world.

Trends in cognitive sciences.

Kaiser, D., Stein, T., & Peelen, M. V. (2014). Object grouping based on real-world regularities facilitates perception by reducing competitive interactions in visual cortex. PNAS Proceedings of the

National Academy of Sciences of the United States of America, 111(30), 11217–

11222. doi:https://doi.org/10.1073/pnas.1400559111

Lachuad, C.M., Reanaud, O. (2011). A tutorial for analyzing human reaction times: How to filter data, manage missing values, and choose a statistical model. Applied Psycholinguistics, 32, 389-416. doi:10.1017/S0142716410000457

Le, R., Witthoft, N., Ben-Shachar, M., Wandell, B. (2017). The field of view available to the ventral occipito-temporal reading circuitry. Journal of vision, 17(4), 6. doi:https://doi.org/10.1167/17.4.6 Lin, T.Y. et al. (2014) Microsoft COCO: Common Objects in Context. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8693. Springer, Cham. doi: https://doi.org/10.1007/978-3-319-10602-1_48

Neider, M.B., Zelinsky, G.J. (2006). Scene context guides eye movements during visual search. Vision

Research, 46(5), 614-621. doi: https://doi.org/10.1016/j.visres.2005.08.025

Olivia, A., Torralba, A., Castelhano, M.S., Henderson, J.M. (2003). Top-down control of visual attention in object detection. Proceedings 2003 International Conference on Image Processing (Cat.

No.03CH37429) doi:10.1109/ICIP.2003.1246946

Parkhurst, D.J., Niebur, E. (2003). Scene Content Selected by Active Vision. Spatial vision, 16(2), 125-54. doi:10.1163/15685680360511645

Peelen, M. V., & Kastner, S. (2014). Attention in the real world: toward understanding its neural basis.

Trends in cognitive sciences, 18(5), 242-250.

Potter, L. M., Grealy, M. A., Elliott, M. A., & Andrés, P. (2012). Aging and performance on an everyday-based visual search task. Acta psychologica, 140(3), 208–217. doi:

https://doi.org/10.1016/j.actpsy.2012.05.001

Quek, G.L., Finkbeiner, M. (2014). Face-sex categorization is better above fixation than below: evidence from the reach-to-touch paradigm. Cogn. Affect. Behav. Neurosci., 14, 1407-1419. Reeder, R.R., Peelen, M.V. (2013). The contents of the search template for category-level search in natural scenes. Journal of Vision, 13(3), 1-13. doi:https://doi.org/10.1167/13.3.13

Russel, B.C., et al. (2008). LabelMe: a database and web-based tool for image annotation. Int. J.

Comput. Vis., 77, 157-173.

Silson, E.H., Groen, I.I.A., Kravitz, D.J., Baker, C.I. (2016). Evaluating the correspondence between face-, scene-, and object-selectivity and retinotopic organization within lateral occipitotemporal cortex.

19

Tatler, B.W. (2007). The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions. Journal of Vision, 14(7), 4.

doi:https://doi.org/10.1167/7.14.4

Theeuwes, J. (2019). Goal-driven, stimulus-driven, and history-driven selection. Curr Opin Psychol.,

29, 97-101. doi: 10.1016/j.copsyc.2018.12.024

Vallat, R. (2018). Source code for pingouin.correlation. [Online] Available: https://pingouin-stats.org/_modules/pingouin/correlation.html#

Zelinsky, G.J., Bisley, J.W. (2015). The what, where, and why of priority maps and their interactions with visual working memory. Annals of the New York Academy of Sciences, 1339 (1), 154-164. doi: 10.1111/nyas.12606