Neural Mechanisms underlying the Development of Optimal Depth Cue
Combination in childhood
Bauke van der Velde, BSc.
ABSTRACT - The brain continuously receives information from the world through all sensory modalities. By integrating these cues in an optimal fashion, adults minimize variance to make more successful decisions. In children, however, optimal cue integration develops surprisingly late. This study explored two questions: At what age do children start integrating motion parallax and binocular disparity depth cues? And do these behavioural changes coincide with changes in the neural mechanisms underlying optimal cue integration? A behavioural experiment in which children were asked to determine which of two planes was further away, showed that only children of 11 years and older have started integrating motion parallax and binocular disparity depth cues in an optimal fashion. No evidence was found that suggests differences in neural mechanisms for cue integrators compared to non-cue-integrators. MVPA data of the fMRI experiment presented here thus supports the hypothesis that behaviour and neural mechanisms of optimal cue integration are dissociated during development. The delayed maturation of optimal cue integration might thus not be caused by young children lacking the tools to optimally integrate cues but by an inability to access this information efficiently at younger ages.
Table of Contents
Introduction 3
Methods & Materials 10
Results 15
Discussion 22
Summary 26
Acknowledgements 27
Introduction
The brain continuously receives information from the world through all sensory modalities. To react to the environment effectively, this information needs to be integrated into one robustly unified percept. Besides simplifying information, integration of multiple sources has one vital benefit: limiting the variance of the final percept to increase its reliability (Ernst & Banks, 2002).
Whenever the adult human nervous system is employed to estimate a physical property, this estimation is corrupted by noise (e.g. through internal neural noise, Eqn. 1). Some sensory cues are more susceptible to noise than others. By weighting all available cues according to their individual reliability, humans can improve the precision of their estimation beyond that of perceptual judgments based on single cues alone. Assuming that noise of each cue is independent and Gaussian, the estimate that maximizes the reduction of variability is the Maximum Likelihood Estimate (MLE); in which all single cues are weighted according to their reliability (Eqn. 2). The MLE is equal to the inverse of their variance (Eqn. 3; Cochran, 1937) 𝑆! = 𝑓 𝑆 (Eqn. 1) 𝑆 = !𝑤!𝑆! With !𝑤! = 1 (Eqn. 2) 𝑤! = ! !!! !! !! !!!…!…! (Eqn. 3)
In 2002, Ernst and Banks found that when subjects judged the size of a bar based on visual and haptic cues combined, performance improved markedly compared to judgments based on either single cue. Furthermore, when noise was artificially added to a single cue, subjects weighted this cue less heavily in the combined cue estimate, in accordance with the MLE model (Ernst & Banks, 2002). This result has been replicated on numerous occasions
either with multisensory cues (Alais & Burr, 2004; Tsakiris Haggard, 2005) and cues from within the same modality (Hillis, Watt, Landy & Banks, 2004; Knill & Saunders, 2003). Oruç and colleagues later found that even in the case of single cues with correlated noise the MLE still markedly improves the estimation compared to single cues (Oruç, Maloney & Landy, 2003).
Hillis and colleagues (2002) noted an important difference between within and across modality cue integration. Subjects were asked to judge the steepness of a slant based on two cues either from within a single modality (texture and disparity) or between two modalities (vision and touch). Cues were presented in either a congruent (both cues gave identical information) or incongruent fashion (the information of both cues was conflicting). Only within modality did this incongruency lead to a loss of performance. Hillis and colleagues termed this effect mandatory fusion. Within a sense the separate cues are fused in such a way that it is impossible to dissect the individual cues from the unified percept, making it impossible to ignore one faulty cue. While effect of mandatory fusion is also present in multisensory cue integration, it is a lot less pronounced (Hillis, Ernst, Banks & Landy, 2002).
More recent studies have begun to reveal that the ability to make optimal use of multiple sensory cues does not develop until late childhood (Gori Sandini, Martinoli & Burr, 2008; Nardini, Jones, Bedford & Braddick, 2008; Nardini, Bedford & Mareschal, 2010; Barutchu, Crewther & Crewther, 2009). Gori and colleagues explored multisensory integration in children during visuo-haptic size discrimination. Children had to decide which of two blocks was smallest, while size information was conveyed through touch or vision alone or by both simultaneously. Where adult performance improved when multiple cues were available, performance of children under the age of 8 did not. Their performance was very similar to their performance for the best single cue, implying that children base their decision on the most reliable available cue alone. These findings suggest that optimal visuo-haptic cue integration does not fully develop until late childhood (Gori et al., 2008)
This has been replicated several times across a range of tasks and modalities (Barutchu et al., 2009; Nardini et al., 2008; Nardini et al., 2010). Barutchu and colleagues used a detection task of audio-visual stimuli and found that integration of visual and auditory cues did not fully mature until 10 to 11 years old (Barutchu et al., 2009). Nardini and colleagues compared child and adult performance on a navigational task where self-motion (egocentric) and visual landmark (allocentric) cues were used as sources of location information. They found adults’
localisation accuracy improved when two sources of information were available, whereas children younger than 9 years old performed as well in the combined cue condition as in their best single cue condition (Nardini et al., 2008). To follow up on this, Nardini and colleagues conducted a study using two cues from within one modality. They reasoned that optimal cue integration might develop earlier for within sense cues compared to between sense cues. They showed that when judging slants based on stereo and texture cues, improvements in slant discrimination with both cues available only started meeting the MLE-model prediction between the ages of 10 to 12 years. Also, mandatory fusion that characterizes within modality cue integration was absent in 6-yearold children (Nardini et al., 2010).
Why the clearly important ability to benefit from redundant information develops so late is currently unclear. Broadly, there are two possibilities: (1) children do not possess the tools necessary to integrate optimally at a younger age or (2) children do possess the tools, but do not use them in an adult-like manner. Gori and colleagues argued in favour of the first possibility (Gori et al., 2008). They argued that senses are constantly changing during development. For cues to be optimally integrated, the reliability of one cue compared to a different cue needs to be identified. With ever-changing reliabilities of sensory information, this calibration process might prove difficult. This would imply that only through learning children eventually get the tools needed to integrate cues optimally. In response to this however, Ernst argued in 2008 that it is unlikely that this calibration process can explain sub-optimal cue integration in children, since adults are also constantly calibrating their senses without any loss of cue integration performance. For example, when hitting a nail with a hammer instead of your own hand adults immediately recalibrate the range of their arms. Therefore, adults are still able to hit the nail with high precision (Ernst, 2008).
Nardini and colleagues underlined a different possibility. They showed that reaction times in young children did improve when more than one information cue was available, even
adults. This raises the possibility that, while children have in fact the tools ready for optimal cue integration, they opt to choose for different strategies in a cue combination problem, e.g., prioritizing speed over accuracy (Nardini et al., 2010). It might also be possible that children possess the tools, but lack of optimal cue integration in younger children is the result of attention problems. However, Van der Velde and colleagues showed that the suboptimality of cue integration was repeatable over sessions, making attention problems an unlikely cause (van der Velde, Dekker, Aisbitt & Nardini, 2013).
One way to distinguish between these conflicting hypotheses is by investigating the neurological mechanisms underlying cue integration in the developing child. If children do not possess the tools needed for optimal cue integration, neural mechanisms are likely to be different from the mature visual system. However, if the mechanisms prove to be functioning in a similar way, it might be that children have the tools available, but rather optimize their choices in a different way. Comparing the fundamental neural mechanisms between adults and children, both integrators and non-integrators, might therefore be vital for a further understanding. This study will investigate this in children for the first time.
Recently, Ban and colleagues investigated the neural mechanisms of adult cue integration. In their task, subjects looked at squares depicted either far way (into the screen) or nearby (popping out of the screen). The perception of depth was created through motion parallax or stereovision alone or through both cues combined. Activity patterns of subjects looking at these particular stimuli were analysed using support vector machine learning. Ban and colleagues tested how well one could differentiate between activity patterns for squares that were depicted far away and squares that were depicted nearby. They argued that areas important for integrating congruent depth cues would perform better - would show more distinct patterns of activation for far or near objects - in the congruent condition compared to either of the single cues alone. The single cues used in this experiment, however, were not truly single cues. For example, the motion parallax only cue was depicted on a flat screen. The zero disparity of the screen could therefore interfere with the motion parallax cue, deteriorating performance in single cue conditions. Ban and colleagues argued that therefore the performance of the congruent condition not only needed to exceed either of the single cues alone, but more so, needed to surpass the quadratic summation of both single cues (for a more detailed explanation, see Figure 1). They also added an incongruent condition to ensure that brain areas were only sensitive for congruent depth cue integration and not merely sensitive to
having more sources of random information available. The one area that showed more distinct activity patterns in the congruent cue condition compared to either the quadratic summation of the single cues or the incongruent condition was V3B, providing evidence that this area might be vital for mature depth cue integration (Ban, Preston, Meeson & Welchman, 2012).
Figure 1. In-depth explanation of why the sensitivity in the congruent condition should be higher than the quadratic summation of both single cues (from Ban et al., 2012). (a) To judge the distance of
a ballet dancer, both motion and disparity cues can be used. In order to create a unified percept and minimize variance both cues will be fused (purple Gaussian). (b) To differentiate between two blobs which are defined by a bivariate Gaussian (green and purple) four detectors can be used: two single cue detectors (disparity and motion) that will use only one dimension of the square depicted, an independent detector which keeps the two blobs separated by using the optimal separation line (the gray diagonal line), which is equal to the quadratic
summation of both single cues (motion2 + disparity2 = optimal separation line2) or the fusion detector (top
right) which completely integrates the two signals. (c) A single cue case where the optimal separation is now a vertical line (because this is still a cue conflict situation, with one cue depicting depth and the other depicting zero depth). Because of this the fusion mechanism is compromised, since complete fusion is impossible. (d) An incongruent cues case. The optimal separation line is once again the diagonal. The independent performance will therefore be equal to the one in b. The performance of the fusion detector has two possibilities. A strict fusion detector will be unable to fuse the signals and will therefore be completely insensitive. A robust fusion detector will revert to the performance of one of the single cues. (e) Outcome if signals are independent. Both in the congruent and the incongruent condition performance will be equal to the quadratic summation of both single cues. (f) Outcome if signals are fused. In this case, the congruent condition performance will be equally high as in e. Both single cues, which are not truly single cues, are now lower however, because incongruent signals are fused. Therefore congruent performance will be higher than the quadratic summation of both single
Over the last decade more and more studies researched the neural mechanisms behind optimal cue integration. However, there are no known examples of studies researching the development of cue integration. The aim of this study is to investigate the neural mechanisms driving the development of motion parallax and binocular disparity depth cue integration. The reason behind choosing motion parallax and binocular disparity depth cues was twofold. Firstly, the study done by Ban and colleagues supplied perfect material for comparison between adult and child cue integration. A second reason for choosing these cues, is that motion parallax and disparity depth cues have been found to be strongly related in how they can be computed from the environment as well as in the underlying neural pathways (Rogers & Graham; Rogers & Collett, 1989; Bradshaw & Rogers, 1996). This strong relation between these two depth cues could result in a quicker optimization of the integration of these particular cues compared to the previously researched texture and disparity depth cues. However, if these cues would integrate optimally around the same age as for example texture and disparity depth cues, it may imply a common developmental mechanism for visual cue integration. To discriminate between these two possible developmental trajectories, we tested when adult-like integration of motion parallax and binocular depth cues arises.
The work described here encompasses two experiments. In a large behavioural study, children between the ages of 6 to 13 years judged which of two planes was further away (i.e. had more depth). In replication of the study reported by Ban et al. (2012), the depth of these stimuli was created either through motion parallax or binocular disparity alone or through a congruent or incongruent combination of the two. Sensitivity to depth differences was measured for each of these four conditions. Based on previous developmental work, it was expected that younger children’s sensitivity would not improve when both depth cues were present. Older children’s sensitivity however, will probably improve, but only when the two available cues are congruent with each other. Moreover, the sensitivity in the congruent condition was expected to be higher than the quadratic summation of the sensitivities for both single cues, since the single cues used in this study cannot be truly single cues.
The second study investigated neural mechanisms of the development of cue integration, which tried to understand whether children’s behaviour and neural mechanisms of cue integration are dissociated. A behavioural experiment determined whether the child combined cues optimally and an fMRI experiment determined whether there were differences in neural mechanisms between optimal and non-optimal cue integrators. The fMRI experiment was a replication of the study reported by Ban and colleagues (2012).
Consequently, a child’s neural mechanisms for cue integration were deemed adult-like if MVPA accuracies in V3B were higher in the congruent condition than in the incongruent condition and that accuracy in the congruent condition also surpassed the quadratic summation of the accuracies of both single cue conditions. Also, if there are changes in neural mechanisms during the development of cue integration, it will be interesting to determine whether the change from child to adult-like cue integration will develop gradually over time or abruptly. The first possibility would imply a level of learning, since the neural mechanisms show signs of becoming more and more optimized. The second possibility might imply a change in strategy, which abruptly changes both the neural mechanisms and the behaviour of cue integration.
NOTE
As of this writing, the data for the fMRI experiment is still being gathered and analysed. The remainder of this report will therefore focus mainly on the behavioural experiment. Pilot fMRI data of five adults will also be shortly discussed and lastly, the fMRI data of two randomly picked children will be presented.
Methods & Materials
ParticipantsThe study was divided into two parts: a behavioural and an fMRI study. The behavioural study was first extensively piloted, during which 44 adult participants (age range 22-40) were tested to ensure an optimal task design, which was then used to test 27 adult participants and 74 children. The children were divided into three age groups: 6-8 years old, 8-11 years old, and 11 years and older. The children were tested at a local London school. All participants gave written consent and, in the case of child participants, so did their guardians.
The fMRI study was first piloted on 5 adult participants. Later, child participants were recruited through adverts in local newspapers. During this currently on-going study at least 30 children will be scanned in the age group 9-13 years old.
Materials
The behavioural task was presented using a 1920x1080 27” Samsung monitor. Depth perception was created using random dot stereograms (RDSs) and red-cyan glasses. The RDSs consisted of a background rectangle (20° by 16°) with zero depth and an inner square
Table 1. Summary of the subjects participating in the behavioral experiment
(11° by 11°) with varying depth. Depth through binocular disparity was created by superimposing a cyan and a red RDS. The inner square of the red RDS was slightly displaced to the right and the inner square of the cyan RDS was slightly displaced to the left. These images are fused by the brain while looking through red-cyan glasses creating the perception of depth. Larger horizontal separations create a larger perceived depth. Depth through motion parallax was depicted by moving both the inner and the outer parts horizontally following a sinusoidal velocity with a period of one second. The motion amplitude of the outer rectangle was fixed at .5°, while the amplitude of the motion of the inner square was varied in six equal levels from .25° (furthest) to .5° (nearest). The relative motion of the inner square compared to the outer rectangle gave rise to the perception of depth. A grey background with some randomly placed black and white squares surrounded the RDSs (for example of stimulus with background, see figure 2).
Design and Procedure
Two separate studies were done for this study: a behavioural study in which participants only participated in the behavioural experiment and an fMRI study in which participants both participated in the behavioural and the fMRI experiment. The behavioural experiment was
Figure 2. Example of stimulus shown during behavioral and MRI task. Depicted here is
an example of the RDS shown during the behavioral and MRI task. Both the outer rectangle and the inner square move independently from each other (depicted by the blue and orange arrows).
divided into four blocks, in which depth was created differently: through binocular disparity only, through motion parallax only, through both cues congruently or through both cues incongruently. Each trial consisted of two consecutively shown RDSs. One, a reference RDS at standard depth (8 arcmin) and the other, a target RDS at varying depths (3, 5, 7, 9, 11 or 13 arcmin, see figure 3 for example of trial). Subjects were asked to determine which of the two shown RDSs was furthest away. Each of the six depth levels was presented 15 times per block. So in total 360 trials were presented. At the end of a block, participants received a score based on their performance on the easiest trials. This score was then converted into coins, which children could exchange for small presents at the end of the experiment. Each block started with a practice round to familiarise participants with the task.
The fMRI experiment consisted of two parts. First, children participated in the behavioural experiment described above to determine the maturation of their cue integration. This was followed by two MRI sessions: a retinotopic mapping session and an experimental MRI session. During the experimental MRI session participants were scanned while looking at RDSs similar to the ones described above. However, the depth was this time fixed at 6 arcmin throughout the experiment and the inner square was now being depicted either in front or behind the screen. Depth was created in the four ways described above: through disparity only, motion parallax only, both cues congruent or both cue incongruent. The fMRI experiment followed a blocked design. During each of 6 runs each condition was randomly repeated during 3 separate blocks (there were 12 blocks in total). During a block, each RDS
Figure 3. Example of a trial in the behavioral experiment. Each trial consists of two
RDSs being shown for one second. After both RDSs have been shown, the subject is asked to decide which of the two RDSs had more depth.
was shown 8 times consecutively for 1 second with an inter stimulus interval of 1 second. A fixation cross was present throughout the experiment and subjects were asked to perform a vernier fixation task, in which they decided whether a vertical stripe was presented on the right or the left side of the fixation cross (see figure). Each run started and ended with a 16-seconds-long null block, during which only the fixation cross was present. Performance on the vernier task was monitored to ensure both fixation and fusion of the red-cyan RDSs.
Statistics
The number of times the reference was chosen over the target during the experiment was recorded for all disparity levels. The resulting psychometric function was analysed using MATLAB (2012b, TheMathWorks), and psignifit 2.5.6 (see http://bootstrap-software.org/psignifit) was used to fit a cumulative Gaussian to the data. Psignifit 2.5.6 implements the maximum-likelihood method described by Wichmann & Hill (2001). Goodness of fit (gof) and bias were determined. A gof of at least 65 % and a bias not larger than two arcmin were set as minimum requirements. A total of 13 and children and 6 adults were excluded because they did not meet these criteria, 58 children and 17 adults remained. The ability of a participant to detect depth differences is a function of the steepness of the slope of the psychometric function. A steeper slope indicates that the subject has a higher sensitivity to detect differences between depths.
The quadratic summation (quadsum) of the two single cues was calculated and T-tests were performed to compare the means of the quadsum and the congruent condition.
𝑆!!+ 𝑆!!
Another t-test was done comparing the slopes of the congruent and incongruent conditions. This was done across all age groups. Individual integration coefficients were calculated by subtracting the individual quadsum (the most optimal summation of the two single cues) from the individual congruent slopes.
Figure 4. Fixation cross with vernier task. During some trials a vertical line appeared
inside the fixation cross. The participants were asked to decide whether this line was on the left or the right side of the fixation cross. This task ensured not only that participants were fixating, but also that the two images presented on both retinae were fused in a correct manner.
𝑆!− 𝑆!!+ 𝑆!!
This integration coefficient was then correlated with age in the children to test for age-related increases in cue integration.
Imaging
Blood-oxygen-level-dependent (BOLD) fMRI data was acquired using a 1.5 Tesla Siemens Avanto scanner. During the experimental cue integration runs, 22 2.3 mm coronal slices were acquired using echo planar imaging (matrix size: 96 x 96, voxel size: 2.3x2.3x2.3, repetition time (TR): 2 ms), recording 208 volumes. Two runs for the retinotopic mapping were acquired with an echo-planar sequence (matrix size: 64 x 64, voxel size: 3.2 x 3.2 x 3.2 mm, TR, 2.5, 30 coronal slices), recording 128 volumes. Each run consisted of six cycles and was 230s long.
For each participant retinotopic regions were defined using a standard retinotopic mapping procedure (Sereno, Dale, Reppas, Kwong, Belliveau, et al., 1995). Area V3B was defined accordingly to the procedure described in the study of Ban and colleagues as the area anterior to V3A and superior and anterior to V3, inferior to V7 and posterior to MT, containing a superior/anterior upper and inferior/posterior lower field bordering V3A.
Volumetric segmentation and cortical reconstruction was done with the Freesurfer image analysis suite (http://surfer.nmr.mgh.harvard.edu/).
Multi-voxel pattern analysis
Data was aligned and slice-time corrected. Linear trends were removed and the data was registered to a hi-res MPRAGE structural image using AFNI (Cox, 1996) and an in-house adaptation of Freesurfer tools. The time series was convolved with the hemodynamic response function including regressors of interest and non-interest (motion parameters). Within each ROI, voxels were sorted based on their response to all conditions combined, compared to the fixation baseline (from the two fixation only trials). This resulted in a selection of maximally 300 voxels per ROI. The time course of each individual voxel was normalized for each run to minimize effects of baseline variations across runs. The time courses were then shifted 4 seconds to account for the hemodynamic response lag. A support vector machine (svm) was used (libsvm, Chang & Lin, 2011) for classification, using a six fold leave one out cross validation. In this validation method, in which each fold of data from a new combination of five scans is used for training and the last remaining scan for testing. In other words, the svm
was trained on the activity for the trials of 5 of 6 runs. Then, its knowledge was tested on the sixth run. Accuracies were calculated based on how well the svm could predict whether the depicted depth in a run was far away or nearby, based purely on activity patterns. Areas involved in the detection of depth will probably show more similar activity patterns and therefore show higher accuracies. Prediction accuracies were converted to units of discrimination using the following formula:
𝑑! = 2 ∗ 𝑒𝑟𝑓𝑖𝑛𝑣(2𝑝 − 1)
Where erfinv is the inverse of the error function and p is the proportion of correct predictions. Further, statistical analysis was performed using SPSS (ref).
Results
BehaviourFor every adult the amount of reference RDS (of 8 arcmin depth) chosen over the target RDS (of varying depth) was counted for each category separately. This data was then psychometrically fitted. The average slope for each category is shown in figure 5B. The ability to detect differences in depth from motion parallax only proved very difficult, which was in accordance with the study of Ban and colleagues (2012). Surprisingly, adults showed no higher sensitivity in the congruent condition compared to either the quadratic summation of the performance on the single cues (t(18)=0.159,p=n.s.) or the performance in the incongruent condition (t(18)=0.983, p=n.s.). This was in contradiction with previous work done by Ban and colleagues (2012).
Figure 6 shows the average fits for the child data, broken down into age groups. The results for the two younger groups appear to be similar. The older group however, shows a
1
Slopes Old Disparity Parallax Congruent Incongruent
1
Slopes Middle Disparity Parallax Congruent Incongruent 0 0.1 0.2 0.3 0.4 0.5 0.6 1
Slopes Young Disparity Parallax Congruent Incongruent
1
Slopes Old Disparity Parallax Congruent Incongruent
1
Slopes Middle Disparity Parallax Congruent Incongruent
1
Slopes Young Disparity Parallax Congruent Incongruent
6-8 yr 8-11 yr 11+ yr
Slopes (sensitivity) per category for children and adults
M ean slope * + n.s. n.s. n.s. n.s. 1
Slopes Adults Disparity Parallax Congruent Incongruent 0 0.1 0.2 0.3 0.4 0.5 0.6 Adults n.s. n.s. Disparity Parallax Congruent Incongruent Quadsum
Figure 5. Slopes of the psychometric function per category for children and adults.
(a) The average slopes of the psychometric function for the different categories separated by age group are shown. Only in the oldest age group a clear improvement has been found in the congruent condition compared to the quadratic summation of the single cues (p < 0.05). While not significant, a clear trend can be seen for the congruent condition being higher than the incongruent condition in the oldest age group (p=0.07). (b) The average
a) b) M ea n s lo pe
steeper slope in the congruent cue condition. Subsequently, the slopes of these fits analysed. Condition significantly influenced performance (F3,1.419= 102.424, p<0.001) and a significant
interaction effect between age and condition was found (F6,0.5 = 3.593, p<0.01) showing that
as children grow older, the performance in the different conditions changes. This finding is depicted differently in figure 5A, which shows the average slopes for all conditions for the different age groups. Children in the two younger groups showed no improvement in ability to detect depth differences in the congruent conditions compared to either the quadratic summation (6-8 year olds, t(13)=1.211,p=n.s.; 8-11 year olds, t(21)=-2.312, p=n.s.) or the performance in the incongruent condition (6-8 year olds, t(13)=1.426, p=n.s.; 8-11 year olds, t(21)=-2.312, p=n.s.). A change however can be seen in the 11+ year olds. They showed a significantly increased sensitivity in the congruent condition compared to the quadratic summation (t(19)=2.345, p<0.05). The sensitivity in the congruent condition was not significantly higher than the sensitivity in the incongruent condition. A strong trend, however, was present (t(19)=1.918, p=0.07). This implies an effect of age on the amount of benefit from availability of more than one congruent cue. No effects of block order or gender have been found (F7,0.14 = 0.814, p=n.s. and F3,0.18=1.187, p=n.s.).
To further analyse this maturation of cue integration, an integration index was calculated for each subject individually by subtracting the sensitivity in the congruent
Figure 6. Percentage reference RDS chosen over target for all age groups. The
average fits for all the different age groups for each condition separately are shown. The y-values represent the percentage of trials chosen for the reference trial, which has a standard disparity of 8. No clear biases can be seen. The most striking feature is the steeper slope of the congruent condition in the oldest age group.
2 4 6 8 10 12 14 0 20 40 60 80 100 2 4 6 8 10 12 14 0 20 40 60 80 100 2 4 6 8 10 12 14 0 20 40 60 80 100 2 4 6 8 10 12 14 0 20 40 60 80 100 4 6 8 10 12 14 4 6 8 10 12 14 4 6 8 10 12 14
Depth of target (in arcmin) Depth of target (in arcmin) Depth of target (in arcmin) Average fit for all categories seperated by age group
6-8 yrs 8-11 yrs 11+ yrs
Pe rce nt ag e re fe re nce ch ose n ove r ta rg et 2 4 6 8 10 12 14 0 20 40 60 80 100 DisparityParallax Congruent Incongruent Pe rc en ta ge r ef er en ce c ho se n ov er ta rg et
condition by the quadratic summation of both single cues. This integration index was then correlated with age. This however, yielded no significant results (ρ = 0.2, p =n.s.).
To interpret this result, a scatter plot was made (figure 7A), depicting the integration index versus the age of the participant. The resulting plot shows a u-shaped trend caused by a larger variance in performance of the children between the ages 6 to 8, little to no
6 7 8 9 10 11 12 −0.4 −0.2 0 0.2 0.4 0.6 CongruentSpecificVsAge Age (years) IntegrationValue 6 7 8 9 10 11 12 −0.4 −0.2 0 0.2 0.4 0.6 IntegrationVsAge Age (years) IntegrationValue
Age$versus$Integra-on$Index$$
Age versus Integration Index
Congruent over Incongruent versus age
Sl op e co ng ru en t – in co ng ru en t Sl op e co ng ru en t – q ua dsu m
Age$
vers
us$
Inte
gra-on$
Inde
x$$
Age$
vers
us$
Inte
gra-on$
Inde
x$$
6 7 8 9 10 11 12 −0.4 −0.2 0 0.2 0.4 0.6 CongruentSpecificVsAge Age (years) IntegrationValue 6 7 8 9 10 11 12 −0.4 −0.2 0 0.2 0.4 0.6 IntegrationVsAge Age (years) IntegrationValueAge$versus$Integra-on$Index$$
Age versus Integration Index
Congruent over Incongruent versus age
Sl op e co ng ru en t – in co ng ru en t Sl op e co ng ru en t – q ua dsu m
Age$
vers
us$
Inte
gra-on$
Inde
x$$
Age$
vers
us$
Inte
gra-on$
Inde
x$$
Figure 7. Scatter plots. (a) The graph shows a scatter plot from the integration index
(Congruent slope minus quadsum) versus age. A clear u-shaped trend can be seen implying better cue integration both in the younger and older age groups. (b) A scatter plot shows the correlation of how much more sensitive children were in the congruent condition compared to the incongruent condition compared with age. Just as in (a) the graph shows a clear u-shaped trend. (c) & (d) scatter plots showing data from (a) & (b), but specifically for the ages 10 and up. Both the integration index (ρ = 0.615, p<0.001) and congruent – incongruent index (ρ = 0.390, p <0.05) correlated strongly with age.
a) b) d) 10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5
IntegrationVsAge
Age (years) IntegrationValue 10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5CongruentSpecificVsAge
Age (years) IntegrationValueAge versus Integration Index Congruent over Incongruent versus age
10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5
IntegrationVsAge
Age (years) IntegrationValue 10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5CongruentSpecificVsAge
Age (years) IntegrationValueAge versus Integration Index Congruent over Incongruent versus age
Sl op e co ng ru en t – q ua dsu m Sl op e co ng ru en t – in co ng ru en t 10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5
IntegrationVsAge
Age (years) IntegrationValue 10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5CongruentSpecificVsAge
Age (years) IntegrationValueAge versus Integration Index Congruent over Incongruent versus age
10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5
IntegrationVsAge
Age (years) IntegrationValue 10 11 12 −0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5CongruentSpecificVsAge
Age (years) IntegrationValueAge versus Integration Index Congruent over Incongruent versus age
Sl op e co ng ru en t – q ua dsu m Sl op e co ng ru en t – in co ng ru en t c) 6 7 8 9 10 11 12 −0.4 −0.2 0 0.2 0.4 0.6 CongruentSpecificVsAge Age (years) IntegrationValue 6 7 8 9 10 11 12 −0.4 −0.2 0 0.2 0.4 0.6 IntegrationVsAge Age (years) IntegrationValue
Age$versus$Integra-on$Index$$
Age versus Integration Index
Congruent over Incongruent versus age
Sl op e co ng ru en t – in co ng ru en t Sl op e co ng ru en t – q ua dsu m
Age$
vers
us$
Inte
gra-on$
Inde
x$$
Age$
vers
us$
Inte
gra-on$
Inde
x$$
performance boost in the congruent condition for the children between the ages 8 to 11 and a clear benefit of cue integration in the 11+ group. A scatterplot zoomed in specifically on the oldest children showed a strong correlation between age and integration index (ρ = 0.615, p<0.001; figure 7C). This also shows that the benefits of cue integration seem to gradually increase as children grow older. Figure 7B depicts the relation between congruency-specific benefits of viewing two cues versus age (𝑆!− 𝑆!), which shows a strikingly similar pattern to integration index scatterplot, albeit less extreme. A zoomed in scatter plot was made which showed a correlation between age and congruent specific benefits in the older age groups (ρ = 0.390, p <0.05; figure 7D). Lastly, integration index and congruent specific benefits correlated strongly in the older group of children? (ρ=0.684, p<0.0001), possibly implying similar underlying processes.
fMRI Adult Data
The following results are taken from the functional MRI experiment. It is important to note that at the moment of writing, child data is still being gathered. Presented here is therefore pilot adult data and data from two children. The subject pool is therefore very limited and statistical analysis
will therefore be inaccurate.
Nevertheless, statistical tests are included to demonstrate in which way the data can be analysed. While any conclusions based on these
results can only be highly
preliminary, it is possible to determine whether the preliminary
data (so far) is in concurrence with previous work.
FMRI responses were measured in 10 regions of interest (see figure 8). MVPA was used to determine which areas enable machine learned classifying. Disparity and motion
V1 V1 V2 V2 VP VP V4 V4 LOLO MT MT V7 V7 V3B V3B V3A V3A V3 V3 V2 V2
Figure 8. Drawn brain areas for the one child. The
drawn brain areas are shown superimposed on the map of t-values for all conditions combined.
Sl op e c on gr ue nt – qu ad su m Sl op e c on gr ue nt – inc ong ru ent Sl op e c on gr ue nt – inc ong ru ent
parallax defined depth were reliably decodable, but no interaction effect between area and condition was found (F9,110.45= 1.203, p=0.312).
Figure 9 displays the d’ sensitivity index calculated for all regions of interest. None of the areas yielded significantly higher sensitivity for the congruent condition compared to the quadratic summation of the single cues. Areas V3A (t(4)=1.516, p=n.s.), MT (t(4)=0.627, p=n.s.) and V2 (t(4)=1.840, p=n.s.), however, showed the expected cue integration pattern. V3B, an area previously implied with the cue integration of disparity and motion parallax did not show the correct pattern of sensitivities, showing higher sensitivities for the quadratic summation of both single cues compared to the congruent cue (t(4)=-1.067, p=n.s.). Figure 10 compares accuracies for congruent versus incongruent cues. No significant differences were found. MT (t(4)=1.639, p=n.s.) and V3A (t(4)=0.548, p=n.s.) once do show patterns similar to what was expected. V3B (t(4)=-0.893, p=n.s.) did not show the expected pattern, with higher sensitivities for in the incongruent condition.
Figure 9. Sensitivities for conditions for all analyzed brain areas.
Depicted are the sensitivities (in d’) of all brain areas for the two single cue conditions and the congruent combined cue condition. The quadratic summation of both single cues is depicted as a red bar. No significant differences have been found.
Se ns it iv it y ( in d -‐pr im e) LO MT V1 V2 V3 V3A V3B V4 V7 VP 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 Disparity Parallax Congruent Quad Sum
Child example data
From the data set, two children were randomly picked for this results section. One younger child aged 10 (JF) and one older child aged 12 (MJ). Figure 10 shows the
psychometric functions of both children. MJ shows an increase in sensitivity for detecting depth differences in the congruent combined cue condition compared to the single cue and conflicting cue conditions. This increase in sensitivity is absent in JF’s psychometric functions. This effect can also be seen in figure 11, where the slopes of each psychometric function of each category are depicted for both children. MJ shows an increase in the congruent condition larger than the quadratic summation of both single cues. JF does not.
MRI MVPA accuracies for both children are shown in figure 12. Extra attention was paid to area V3B, an area previously implied to play an important role in optimal cue integration. MJ shows an effect of cue integration in area V3B. In the congruent condition, the SVM was more successful to differentiate between far and near trials based on the activity patterns in this particular ROI. MJ therefore, shows both a behavioural and a neuronal effect of optimal cue integration. In JF however, an effect of congruent cue combination was also present in V3B. She therefore shows no behavioural effect of optimal cue integration, while
LO MT V1 V2 V3 V3A V3B V4 V7 VP 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 Congruent Incongruent
Figure 10. Sensitivities for congruent and incongruent conditions for all analyzed brain areas. Depicted are the sensitivities (in d’) of all brain
areas for the two combined cue conditions (congruent and incongruent). No significant differences have been found although both MT and V3A show encouraging patterns. Se ns it iv it y ( in d -‐pr im e)
Figure 12. Bar plots of the slopes of the psychometric functions of the two children. The two plots here depict the slopes of the psychometric
function for an older (MJ) and a younger (JF) child. MJ clearly shows some benefits from cue combination, while these benefits seem to be absent in JF’s plot. MJ JF Sl op es o f t he p sy ch om et ri c f un ct io n
Figure 11. Psychometric functions of the two children MJ and JF. The two plots
here depict the psychometric functions for an older (MJ) and a younger (JF) child. MJ clearly shows some benefits from cue combination, while these benefits seem to be absent in JF’s plot. MJ JF Pe rc en ta ge r ef er en ce c ho se n o ve r ta rg et
. MJ JF Me an s lo pe MJ JF A cc ur ac ie s ( in pe rc en ta ge ) A cc ur ac ie s ( in p ec en ta ge )
Figure 13. MVPA accuracies of the two children. Depicted here are the
MRI accuracies for both children. MJ shows a cue integration effect in V3B in accordance with his behavioral data. JF however, does not show a behavioral cue integration effect, while her there is a cue integration effect visible in V3B.
Discussion
The aim of this study was to investigate the development of cue integration of motion parallax and binocular disparity depth cues in children. More specifically, we were interested in when children start integrating these cues optimally and what the neural mechanism underlying this development is. Previously, it was thought that children needed experience in order to integrate cues optimally. Recent research, however, showed that children start integrating cues of different senses around the same time as cues from within a sense, which made the experience theory unlikely. We therefore posed the two separate hypotheses. Firstly, from the behavioural point of view, optimal cue integration of motion parallax – disparity depth cues develop at the same age as stereo-texture depth cues. Secondly, neurologically, the mechanisms coding for depth would be similar in cue integrators and non-cue integrators. Both these hypotheses underline the possibility that children have all the tools needed for optimal cue integration, but that the brain chooses to perform sub-optimally in cue integration conditions. The preliminary data presented in this report show some evidence for both these hypotheses.
Behavioural adult data
Adult cue integration data did not replicate previous adult cue integration findings. Participants’ performance did not improve significantly in the congruent cue condition. This is especially noteworthy, since a cue integration effect has been found in the oldest children. One possible explanation for this can be found in the overall better performance of adults. Adults were often almost faultless in their best single cue condition, making it very difficult to improve even the slightest in the combined cue condition. This could also explain the dissimilarity in results between our study and the one done by Ban and colleagues (2012).
While the task used in this study was very, they made use of an adaptive staircase paradigm. This paradigm is likely to be frustrating for younger children since the task gradually becomes more difficult and can lead to severe lapses in attention, preventing the staircase from converging.In order to make the task more suitable for younger children, we therefore opted for the use of a constant stimuli design. The levels deemed to appropriate for children based on piloting, may however, have been too easy for adults, resulting in ceiling effects. I.e., adults depth discriminations for single cues was already so good that they could inly gain minimally from adding an additional cue.
Behavioural child data
The psychophysical data for children were in line with our expectations that young children do not integrate optimally. Performance of children in the youngest age groups (6-10 years old) did not show any effect of cue integration. However, older children’s (11+) performance improved significantly in the congruent cue condition compared to the quadratic summation of both single cues. This improvement was specifically found for congruent stimuli and could not be found for incongruent stimuli. Until now, it was unknown whether this was also the case in the integration of motion parallax and disparity depth cues and at what age the turning point for this specific type of cue integration lies. The results of this study give reason to believe that optimal cue integration of motion parallax and disparity depth cues does not fully mature until 11 years old, similar to the maturation age of the integration of disparity and texture depth cues (Nardini, 2010).
It seems therefore, that children start to optimally integrate motion parallax and disparity cues, which are strongly correlated in the brain and the world at a similar time in life than the less strongly associated cues of texture and disparity. This would make the explanation that children have not calibrated their cues yet, and are therefore unable to integrate them in an optimal fashion, unlikely. If some cues were more frequently presented together in the real world, the calibration process of these particular cues should be quicker than cues presented together less frequently in the real world. Therefore optimal integration for these cues should arise earlier in life. Also, previous studies have found that optimality in multisensory cue integration might actually arise earlier in life than optimality in within sensory cue integration (Gori et al., 2008; Nardini et al., 2008), which is difficult to rhyme with the hypothesis that the tuning of cues causes delayed optimal cue integration as well.
One thing to note, however, is that the change from immature to mature cue integration is not abrupt, but gradually over the course of two year (see figure 7C). This in turn would be more difficult to explain with the change in strategy hypothesis and probably implies a sort of ambivalent strategy state. Also, while the results of the behavioural experiment did not find a difference in the age of onset of the optimal cue integration of similar and dissimilar depth cues, it might be that there are in fact very slight differences in onset, which are detectable only in larger sample sizes. Future research could compare the performance of the same group of children on both texture and disparity and motion parallax and disparity cue integration paradigms, to explore correlations between the ability to integrate motion parallax and disparity and disparity and texture cues in an optimal fashion. A second point of interest in our data is the found u-shape when comparing the benefits of cue combination with age. Some young children’s performance does improve when more cues are available. While counterintuitive, this phenomenon has also been found by two similar studies which showed a multicue facilitation for younger and older age groups, but not for children between the ages 9 and 11 (Nardini et al., 2010; Barutchu et al., 2009).
This effect, however, could be caused by concentration problems in the younger age group. Due to the difficulty of perceiving depth from motion parallax solely, a drop of concentration in the other single cue condition, binocular disparity, could lead to both single cue performances being low. This would make a positive cue integration effect easier to achieve. This is underlined by the larger variance in performance of younger children. An additional explanation for this pattern of results, is a selectivity bias for well-performing younger children, which may have arisen because this task proved difficult to complete for the youngest children.
fMRI pilot data
The adult pilot data did not show the expected optimal cue integration pattern in V3B, which was found in the study of Ban and colleagues (2012). The most likely cause of this is that this data was gathered from only five participants. Noteworthy is the pattern in V3A which, while not statistically significant, was the pattern to be expected if the area influences optimal cue integration. This area has previously been implied to be vital for disparity defined depth (Adams & Zeki, 2001; Tsao, Vanduffel, Sasaki, Fize, Knutsen et al., 2003) and might be involved in motion defined depth as well (Paradis, Cornilleau-Peres, Droulez, Van de
Moortele, Lobel et al., 2000; Vanduffel, Fize, Peuskens, Denys, Sunaert, et al., 2002). The most important fact taken from the pilot is that it seems that our setup should be working correctly and will therefore present no problems for the rest of the study.
fMRI child data
The data from two child participants is shown in this report as well. One older, cue integrating participant and one younger non-cue-integrating participant. MVPA accuracies of both participants are strikingly similar in areas previously implied with depth cue integration (V3B) or in depth created by either single cue (V3A and MT). This is consistent with the hypothesis that behaviour and neural mechanisms of cue integration are somehow dissociated. This could point to the possibility that children do have tools available for optimal cue integration, but opt for a different strategy when more than one source of information is available. It is important to note that due to the limited subject pool of this particular data, conclusions from this are very much preliminary. Analysis of the remaining child subjects is needed for a more conclusive remark regarding the dissociation of brain and behaviour during optimal cue integration.
Summary
This study aimed to shed light on both the neural and behavioural changes during the maturation of motion parallax and binocular disparity depth cue integration. We used a behavioural task, where both children and adults were asked to decide which of two RDSs was depicted further away. Only children age 11 and over were better able to detect depth differences when both motion parallax and disparity depth cues were combined. The similar onset point of cue integration of motion parallax and disparity depth cues compared to texture and disparity depth cue implies a similar underlying mechanism driving the maturation of within sense cue integration. Two example children took part in a follow-up fMRI study. For both, the depth of a plane (in front/behind a screen) could be decoded with more accuracy from activity in area V3B, if plane depth was defined by congruent motion parallax and disparity cues than when defined on either cue alone or the cues in conflict. However, only the older child showed optimal cue integration behaviourally. FMRI data presented in this study thus was consistent with the hypothesis that behaviour and neural changes of cue integration are dissociated, and that children do not show optimal cue integration behaviourally because they fail to access this information. The MRI data shown here, however, is only taken from two child participants and future research is needed to understand the exact link between neural mechanisms and behaviour during optimal cue integration.
Acknowledgements
This report could not have been completed without the incredible help from the following people.
Dr. Tessa Dekker
As the lead researcher my main support. Thank you for being patient with me and willing to teach me. You are the best!
Dr. Hiroshi Ban
For lending us your experiment and insights throughout the study Dr. Marko Nardini
For teaching me everything I need to know concerning the development of cue integration
Prof. Marty Sereno
For incredible support in the gathering and analysis of the MRI data Dr. Karin Petrini
For being your happy self, while giving loads of help during the analysis of the behavioural data
Dr. Pete Jones
For giving your insights on the programming and the optimizing of the task Rachel Fahy, BSc
For helping me test almost 100 children
Sarah Kalwarowsky, Sara Garcia and Eliza Burton for helping with making phone calls, answering questions or simply for being there.
References
Adams, D. L., & Zeki, S. (2001). Functional organization of macaque V3 for stereoscopic depth. Journal of Neurophysiology, 86(5), 2195-2203.
Alais, D., & Burr, D. (2004). The Ventriloquist Effect Results from Near-Optimal Bimodal Integration. Current biology : CB, 14(3), 6–6. doi:10.1016/j.cub.2004.01.029
Ban, H. H., Preston, T. J. T., Meeson, A. A., & Welchman, A. E. A. (2012). The integration of motion and disparity cues to depth in dorsal visual cortex. Nature Neuroscience, 15(4), 636–643. doi:10.1038/nn.3046
Barutchu, A., Crewther, D. P., & Crewther, S. G. (2009). The race that precedes coactivation: development of multisensory facilitation in children. Developmental science, 12(3), 464-473. Bradshaw, M. F., & Rogers, B. J. (1996). The interaction of binocular disparity and motion parallax in the computation of depth. Vision Research, 36(21), 3457-3468.
Cox, R. W. (1996). AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Computers and Biomedical research, 29(3), 162-173.
Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27, 2011
Cochran, W. G. (1937). Problems arising in the analysis of a series of similar experiments. Supplement to the Journal of the Royal Statistical ….
Ernst, M. O. (2008). Multisensory integration: a late bloomer. Current biology : CB, 18(12), R519–21. doi:10.1016/j.cub.2008.05.002
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429–433. doi:10.1038/415429a
Gori, M., Sandini, G., Martinoli, C., & Burr, D. (2010). Poor haptic orientation discrimination in nonsighted children may reflect disruption of cross-sensory calibration. Current biology : CB, 20(3), 223–225. doi:10.1016/j.cub.2009.11.069
Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002). Combining sensory information: mandatory fusion within, but not between, senses. Science (New York, N.Y.), 298(5598), 1627–1630. doi:10.1126/science.1075396
Hillis, J. M., Watt, S. J., Landy, M. S., & Banks, M. S. (2004). Slant from texture and disparity cues: optimal cue combination. Journal of Vision, 4(12), 967–992. doi:10.1167/4.1.1 Knill, D. C., & Saunders, J. A. (2003). Do humans optimally integrate stereo and texture information for judgments of surface slant?. Vision research, 43(24), 2539-2558.
Nardini, M., Bedford, R., & Mareschal, D. (2010). Fusion of visual cues is not mandatory in children. Proceedings of the National Academy of Sciences of the United States of America,
107(39), 17041–17046.
doi:10.1073/pnas.1001699107/-/DCSupplemental/pnas.201001699SI.pdf
Nardini, M., Jones, P., Bedford, R., & Braddick, O. (2008). ScienceDirect.com - Current Biology - Development of Cue Integration in Human Navigation. Current Biology.
Oruç, I., Maloney, L. T., & Landy, M. S. (2003). Weighted linear cue combination with possibly correlated error. Vision research, 43(23), 2451–2468.
Paradis, A. L., Cornilleau-Peres, V., Droulez, J., Van De Moortele, P. F., Lobel, E., Berthoz, A., ... & Poline, J. B. (2000). Visual perception of motion and 3-D structure from motion: an fMRI study. Cerebral Cortex, 10(8), 772-783.
Rogers, B. J., & Collett, T. S. (1989). The appearance of surfaces specified by motion parallax and binocular disparity. The Quarterly Journal of Experimental Psychology, 41(4), 697-717. Rogers, B., & Graham, M. (1982). Similarities between motion parallax and stereopsis in human depth perception. Vision research, 22(2), 261-270.
Sereno, M. I., Dale, A. M., Reppas, J. B., Kwong, K. K., Belliveau, J. W., Brady, T. J., ... & Tootell, R. B. (1995). Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science, 268(5212), 889-893.
Tsakiris, M., & Haggard, P. (2005). The rubber hand illusion revisited: visuotactile integration and self-attribution. Journal of Experimental Psychology: Human Perception and Performance, 31(1), 80.
Tsao, D. Y., Vanduffel, W., Sasaki, Y., Fize, D., Knutsen, T. A., Mandeville, J. B., ... & Tootell, R. B. (2003). Stereopsis activates V3A and caudal intraparietal areas in macaques and humans. Neuron, 39(3), 555-568.
van der Velde, B., Dekker, T., Aisbitt, G., & Nardini, M. (2013). Individual Differences in the Development of Depth Cue Combination. Journal of Vision,13(9), 1186-1186.
Vanduffel, W., Fize, D., Peuskens, H., Denys, K., Sunaert, S., Todd, J. T., & Orban, G. A. (2002). Extracting 3D from motion: differences in human and monkey intraparietal cortex. Science, 298(5592), 413-415.
