Integration of information across the senses

(1)

Integration of information across the senses

David Burr

University of Florence CNR Pisa

University of Western Australia

(2)

Sensory fusion

The key to robust perception is the

combination and integration of multiple sources of sensory information.

Because no information-processing system is powerful enough to ‘perceive and act’

accurately under all conditions.

Ernst & Bülthoff

TICS 2004

(3)

Today’s talk

• A bit of background

• The “ventriloquist effect” (David Alais)

• The “ventriloquist effect” during saccades (Paola Binda)

• Temporal aspects of audio-visual binding

• Visuo-acoustic integration during biological motion (Roberto Arrighi)

• Development of visuo-haptic integration (Monica Gori)

• Abnormal face coding with autism (Liz Pellicano)

• A visual sense of number (John Ross)

(4)

Bayesian inference and sensory fusion

Clark JJ, Yuille AL (1990) Data

fusion for sensory information

processing. Kluwer Academic.

(5)

“Bayesian” combination of multi-sensory information

/

2

1

_A

A

W

A

^  ^ 

w S w S

S ˆ ^VA ^ ^A ˆ ^A ^ ^V ˆ ^V

Weights proportional to precision, the reciprocal of variance

 1



i

w

i

/ 2

1 _V

V

W V    

(6)

-10 -5 0 5 10

P ro ba bi lit y

Space

Visual Auditory

 =8

 =2

Maximum likelihood estimation (MLE)

combination of information

(7)

-10 -5 0 5 10

P ro ba bi lit y

Space

Bimodal Visual Auditory

 =8

 =2

MLE combination of information

S w S

S ˆ

^VA ^

w

^A ^A^ ^V ^V

(8)

S w

S

S ˆ

^VA

^ w

^A ^A

^

^V ^V

-10 -5 0 5 10

Probability

Space

Bimodal Visual Auditory

=8

=2

Bayesian combination of information



² ² ²

1 1

1

A V

VA

p

VA

^ ^ ^

-10 -5 0 5 10

Probability

Space

=4

=2.8

(9)

Combining haptic and visual information

Ernst & Banks Nature (2002)

(10)

The “ ventriloquist effect”

Explanations :

• Magic

• Performers “project

their voices” to puppet.

• Vision “captures” sound:

inherent dominance of vision.

• “Bayesian” combination

of information.

(11)

The “ ventriloquist effect”

(12)

Visual stimuli: 4 blobs

(13)

Visual stimuli: 32 blobs

(14)

Visual stimuli: 64 blobs

(15)

Spatial localization

-20 -10 0 10 20

0.0 0.2 0.4 0.6 0.8 1.0

P ro po rt io n rig ht

Displacement right (degs)

(16)

-20 -10 0 10 20 0.0

0.2 0.4 0.6 0.8 1.0

Proportion right

Displacement right (degs)

P



Spatial localization



² ²



exp ^x 

p  

(17)

Spatial localization

-20 -10 0 10 20

0.0 0.2 0.4 0.6 0.8 1.0

Proportion right

Displacement right (degs)

σ

 ^ _ ^ ^ ^ ^ _ ^

 x x dx

p exp ( )

² ₂



/

2

1 

 

(18)

Auditory and visual localization

-20 -10 0 10 20

0.0 0.5 1.0

P ro po rt io n rig ht

Displacement right (degs) 4

^o

64

^o

32

^o

Click

(19)

2

Conflict presentation

Probe: no conflict

Which interval was to the left?

0

variable

(20)

The ventriloquist effect:

For small blobs, vision dominates.

Alais & Burr

Current Biology, 2004

(21)

The ventriloquist effect:

For blurred blobs, audition can dominate

Alais & Burr

PSE PSE

(22)

Cross-modal conflict

0.0 0.5 1.0

-20 -10 0 10 20

0.0 0.5 1.0 0.0 0.5 1.0

4^oblob

=5^o

-5^o ^=0

Position of Probe (degs)

Proportion "left"

^

^

64^o blob 32^o blob

(23)

MLE Predictions of cross-modal PSEs from uni-modal variances.

-5 0 5

4ô 32ô 64ô

LM

P oi nt o f S ub je ct iv e E qu al ity ( de gs )

Audio-visual Conflict - (degs)

DA SD

i i

V V A

VA A

w

S w S

S w

1

ˆ







Alais & Burr

(24)

The ventriloquist effect:

Perceived position well predicted by the 1-cue auditory and visual thresholds

Measured conflict dependency ()

P re di ct ed c on fli ct d e pe nd en cy

Alais & Burr

(25)

Multi-modal thresholds are lower:

signature of fusion

Visual Auditory Two Cue Predicted 0.0

0.5 1.0 1.5

Noramlised Threshold

Alais & Burr

0 5 10

0 5

Measured 2-cue thresholds

Predicted 2-cue thresholds

Slope = 1.0



² ² ²

1 1

1

A V

VA





(26)

Interim conclusion

• Like visual-tactile judgments, audio-visual spatial localization – the “ventriloquist

effect” – are well explained by the

maximum-likelihood model of sensory

fusion.

(27)

Using audio-visual judgments to study visual localization during saccades

Binda et al.

J. Neuroscience, 2007

(28)

-200 -100 0 100 200 -40

-20 0 20 40

Delay from saccadic onset

A p pa re nt p os iti o n (d eg s)

Eye trace

Compression of space during saccades

Ross, Morrone & Burr Nature 1997

(29)

-40 -20 0 20 40 -40

-20 0 20 40

Targ Fix

peri-saccadic fixation

Real position (deg)

MCM JR

A pp ar en t p os iti on ( de g)

-40 -20 0 20 40

Spatial Compression during saccades

Ross, Morrone & Burr Nature 1997

(30)

Morrone, Ross & Burr J Neuroscience 1997, Kaiser & Lappe, Neuron, 2004

Compression is one-dimensional, in

the direction of the saccade

(31)

Combination of visual and auditory information during saccades

-16° +16°

150 °

Binda, Bruno, Burr, Morrone J. Neuroscience, 2007

(32)

-50 -40 -30 -20 -10 0 10 0.0

0.5 1.0

Vision

Pr op or tio n "ri gh t"

Position of the perisaccadic stimulus (°)

Visual localization: biased and less reliable during

saccades

ACTUAL LOCATION

-16° +16°

(33)

Auditory localization during saccades:

veridical (but inprecise)

-50 -40 -30 -20 -10 0 10

0.0 0.5

1.0 Audition Vision

Pr op or tio n "ri gh t"

(34)

-50 -40 -30 -20 -10 0 10 0.0

0.5

1.0 Audition Vision Bimodal

Pr op or tio n "ri gh t"

Audio-visual localization: a compromise

(35)

Average mislocalization errors

0 -5 -10 -15

*

M is lo ca liz at io n ( de g )

Pred A+V

A V

w S w S

S ˆ

^VA

^

^A

ˆ

^A

^

^V

ˆ

^V

1 _i 2

w i  

σ is threshold

(36)

Average thresholds

0 1 2

3 *

*

N or m al iz ed T hr es ho ld s

Pred A+V

A V



² ² ²

1 1

1

A V

VA





(37)

-100 -50 0 50 10

0

-100 -50 0 50

10 0

-100 -50 0 50 0

-10

0 -10

Threshold (deg) Threshold (deg)

Time ( ms )

Observed Predicted

Bias (deg) Bias (deg)

Bimodal Audition Vision

Dynamics

Binda, Bruno, Burr, Morrone, J. Neuroscience, 2007

w S w S

S ˆ

^VA

^

^A

ˆ

^A

^

^V

ˆ

^V

Multisensory weights are adjusted rapidly, on-line

(38)

External position given by sum of retinal signal and Corollary discharge

-20 0 20

0.0 0.5

-20 0 20

0.0 0.5

P ro ba bi lit y

Fixation Saccades

Retinal eccentricity External space

0.0 0.5 1.0

∑

Eye-position signal

Retinal signal

Alex Pouget

(39)

Corolary discharge from superior colliculus

Sommers & Wurtz, Nature 2006

(40)

Retinal eccentricity R es p o n se t o t h e tr an si e n t s ti m u lu s

The spatial properties of shifting RFs are predicted by corollary discharge from the SC–MD–FEF pathway

PRE POST

Sommers & Wurtz, Nature 2006

(41)

-60 -30 0 30 0.0

0.1

-60 -30 0 30 0.0

0.1

-60 -30 0 30 0.0

0.1

Space ( deg )

eye-pos 1 eye-pos 2

MLE(eye-pos)

A ct iv ity

Modeling visual localization

Corollary discharge

Tim e

(42)

-60 -30 0 30 0.0

0.1

-60 -30 0 30 0.0

0.1

-60 -30 0 30 0.0

0.1

Space ( deg )

eye-pos 1 eye-pos 2 MLE(eye-pos)

Activity

-100 0 100

10 0 -10 -20 -30

Time ( ms )

R et in al S pa ce ( de g )

-100 0 100

-30 -20 -10 0

Time ( ms )

R et in al S pa ce ( de g )

-50 0 50

10 0 0 -10

Time ( ms )

Threshold (deg)

Bias (deg)

+

Modeling visual localization

Visual localization Corollary discharge

Retinal input

Position

σ

(43)

Dynamics of shifting RF in LIP

Simulated Dynamics of the Eye- position

signal

Kusonoki & Goldberg, J Neurophys (2003)

(44)

“Inverse ventriloquist effect”:

“What you see is what you hear”

Shams, Kamitani & Shimojo Nature (2000)

(45)

“Temporal ventriloquism”: Can the dynamics of the mislocalization be

altered by a sound?

time (ms)

0 400

-400

The perceived timing of a visual event can be “captured” by an auditory stimulus

Paola Binda PhD project

(46)

-60 -40 -20 0 20 40 60 -60

-40 -20 0 20 40 60

P er ce iv ed ti m in g (m s)

Audio-visual conflict (ms)

Saccade Fixation

Auditory dominance

Visual

dominance

Δ

(47)

-100 -50 0 50 100 0

5 10 0 5 10 0 5 10

Time (ms)

SILENCE

SIMULTANEOUS 50 ms BEFORE 50 ms AFTER

AG

A pp ar en t p os iti on ( de g)

PB SS

time

(48)

Interim summary

• Audio-visual stimuli show reduced perisaccadic mislocalization

• Updating of the visual signal weights is dynamic.

• The maximum-likelihood approach can lead to important insights of how visual continuity is maintained.

• Multi-sensory fusion occurs after the corollary

discharge affect visual spatial mislocalization

(49)

Temporal alignment of sight and sound

What are the mechanisms for

judging simultaneity?

(50)

Temporal alignment of sight and sound

Does perceived simultaneity take into account the

relatively low speed of sound?

(51)

Subjective audiovisual alignment scales with perceived auditory depth.

Alais & Carlile PNAS 2005

(52)

Subjective audiovisual alignment scales with perceived auditory depth.

Alais & Carlile PNAS 2005

(53)

Subjective audiovisual alignment scales

with perceived auditory depth.

(54)

Subjective audiovisual alignment scales with

perceived auditory depth

(55)

Subjective audiovisual alignment scales with

perceived auditory depth

0 10 20 30 40 50

0 50 100 150

A ud ito ry la g (m s)

Auditory distance (m)

Travel time for sound:2.9 ms/m (434 m/s)

Slope 3.2 ms/m

(56)

Judging perceptual synchrony

Arrighi, Alais & Burr, JoV 2006

(57)

Over a wide range of physical asynchronies, visuo-auditory

events seem synchronous

-200 -100 0 100 200 300 0.0

0.5 1.0

Sound delay (ms)

P (in p ha se )

Arrighi, Alais & Burr, JoV 2006

(58)

Temporal increment

discrimination thresholds

AUDITORY

VISUAL

BIMODAL (+) BIMODAL (-)

 Δ

Banks, Burr & Morrone, IMRF, 2006

(59)

Dipper

functions in all conditions

-400 -200 0 200 400 0

30 60 90 0 20 40 60 0 10 20 30 40 50 60

GC

Base Interval (ms)

Bimodal E

In te rv al D is cr im in at io n T he sh ol d (m s)

C Visual

0 20

0 5 10

A Acoustic

(60)

-400 -200 0 200 400 0

25 50 75 100

D is cr im in at io n T h re sh ol d ( m s)

Base Interval (ms)

Bimodal

Visual

Auditory

Average results

(61)

Effect of filtering on spatial discrimination

0 2σ 4σ

(62)

0 200 400 600 800 1000 0

1

0 500

0 1 2

0 500

0 10 100 1000

1 10 100

1000 Auditory

Visual Bimodal

Activity (arbitrary units)

Time (ms)

F

Response (arbitraty units)

Time (ms)

A B C D

E

Transducer Function 100 ms

50 ms

Interval discrimination threshold (ms)

Base Interval (ms)

0 ms

(63)

-200 -100 0 100 200 0

25 50 75

D is cr im in at io n T h re sh ol d ( m s)

Base Interval (ms)

Category discrimination

Simultaneous

(64)

Conclusions

• The data suggest that audio-visual signals pass through a filter-like process to

determine what stimuli should be fused together.

• The purpose of the filtering maybe to make a category judgment about

simultaneity to help decide whether to fuse or to segregate. Thresholds reflect the

difficulties of within category judgments.

(65)

Can integration be better than MLE?

Audio-visual integration of biological motion

Burr & Arrighi, in preparation

(66)

Summation of visual and auditory motion

Same direction

Opposite direction

Alais & Burr (2004)

Brain Res Cogn Brain Res

(67)

Summation of visual and auditory motion

   

 

   



 



right

left

Visual threshold Auditory threshold

Linear summation Baysian prediction

left

Alais & Burr (2004)

Brain Res Cogn Brain Res

Signature of compulsory fusion

(Hillis et al, Science 2002)

(68)

Scuola di danza di Signa (Firenze)

ORIGINAL THRESHOLDED

(69)

Examples of stimuli

Tap sequence embedded in

noise Only noise

(70)

2 10 100 300 50

75 100

P e rc en t C o rr e ct

Number of noise items

Vision

Chance level

Visual sensitivity

(71)

2 10 100 300 50

75 100

P e rc en t C or re ct

Number of noise items

Vision Sound

Chance level

Auditory and visual sensitivity

(72)

Visual and auditory

normalization to threshold

0.1 1 6

50 75 100

P er ce nt C o rr ec t

Normalized noise

Chance level

(73)

Stimuli near discrimination threshold

In synch Out of synch

ECVP Arezzo 2007

(74)

Bimodal discrimination

0.1 1 6

50 75 100

P er ce nt C or re ct

Normalized noise

Chance level

in synch

out of

synch

(75)

Various models of summation

0.0 0.5 1.0

U ni m od al v is ua l t hr es ho ld s

Unimodal acoustic thresholds Synch

Desynch

Independent Probability summation

MLE fusion



² ² ²

1 1

1

A V

VA





(76)

Audio-visual integration

For a natural stimulus auditory and visual information can be integrated to improve detectability, by more than predicted by the Bayesian maximum likelihood model

Psychophysics

can be fun

(77)

Development of multi-sensory integration?

•Do children have to learn to integrate?

•When does this ability develop?

•Does “touch educate vision”

(Bishop Berkeley)?

Monica Gori, grad student IIT, Genoa

(78)

Virtual reality presentation of haptic and visual stimuli

Ernst & Banks Nature (2002)

(79)

Non-virtual reality

(80)

(81)

SetUp Used

Blurring the non-virtual reality

(82)

Adult uni-modal thresholds

-6 -3 0 3 6 0.0

0.5 1.0

Haptic No blur Med blur Heavy blur

P ro p or tio n (" ta lle r" )

Probe size (mm)

MG

Gori et al. Curr. Biol 2008

(83)

PSEs with visuo-haptic conflict:

adults

-4 -2 0 2 4 -4

-2 0 2 4

No Blur Med Blur Max Blur

P S E ( m m )

Conflict (mm)

Gori et al. Curr. Biol 2008 Replicate Ernst and Banks

w S w S

S ˆ

^VA

^

^A

ˆ

^A

^

^V

ˆ

^V

1 _i 2

w i  

(84)

MLE predicts PSEs at all visuo- haptic conflicts: adults

0 2 4 6

Predicted PSEs (mm)

M ea su re d P S E s (m m )

(85)

Improvements in thresholds - adults

Visual Haptic Dual MLE

2 3 4 5

T hr e sh ol d (m m )

Gori et al. Curr. Biol 2008



² ² ²

1 1

1

A V

VA





(86)

Uni-modal size discriminations:

children

-6 -3 0 3 6

0.0 0.5

1.0

^{10 Years}

-6 -3 0 3 6

0.0 0.5

1.0 5 Years

Relative probe size (mm)

Gori et al. Curr. Biol 2008

(87)

5 6 8 Adult 0

2 4 6

T H R E S H O LD S ( M M )

AGE

Haptic Visual

Uni-modal size discriminations

Gori et al. Curr. Biol 2008

(88)

Visuo-haptic conflict - children

-6 -3 0 3 6 0.0

0.5 1.0

P ro p o rt io n “ ta ll er ”

Relative probe size (mm)

10 year-old 5 year-old

-6 -3 0 3 6 0.0

0.5 1.0

 = 0 mm

 = 3 mm

 = -3 mm

Gori et al. Curr. Biol 2008

(89)

-4 -2 0 2 4 -4

-2 0 2 4

Predicted PSE (mm)

M ea su re d P S E ( m m )

Visuo-haptic conflict:

5-year-olds

Gori et al. Curr. Biol 2008

(90)

Visuo-haptic conflict:

development

-4 -2 0 2 4 -4

-2 0 2 4

-4 -2 0 2 4 -4 -2 0 2 4 -4 -2 0 2 4 -4 -2 0 2 4

5 YOs 6 YOs 8 YOs Adults

Predicted PSE (mm)

M ea su re d P S E ( m m )

10 YOs

Gori et al. Curr. Biol 2008

(91)

Size discrimination thresholds: no improvement in 5-year-olds

Gori et al. Curr. Biol 2008

Visual Haptic Dual MLE

2 5 10

T hr es ho ld ( m m )

(92)

3 10 Adult 1

3 10

Blur

T hr es ho ld s (m m )

Haptic Vision MLE

Cross Modal

Development of thresholds

Age (years)

Gori et al. Curr. Biol 2008

(93)

Development of haptic weight

3 10 Adult

0.0 0.5 1.0

0.5

1.0 H ap tic W ei gh t

Age (years)

PSEs

Thresholds

0.0 V is ua l W ei gh t

Gori et al. Curr. Biol 2008

(94)

Interim conclusions

• Touch dominates size discriminations in young children, even though it is less

precise than vision.

• Does this confirm Bishop Berkeley’s idea that “Touch educates vision”?

Gori et al. Curr. Biol 2008

(95)

Size is not given directly by

vision, but depends on distance

(96)

Orientation selectivity is a

primary property of area V1

(97)

Will orientation discrimination be

haptically dominated in children?

(98)

Uni-modal orientation discrimination

0.0 0.5

1.0 ^{8 Years}

-30 -15 0 15 30 0.0

0.5

1.0 5 Years

Probe orientation (degs)

P ro po rt io n “s te ep e r”

Gori et al. Curr. Biol 2008

(99)

Visuo-haptic conflict: orientation

-12 -6 0 6 12 0.0

0.5 1.0

-12 -6 0 6 12 0.0

0.5 1.0

Relative probe orientation (deg)







10 year-old 5 year-old

P ro po rt io n “ st e ep er ”

Gori et al. Curr. Biol 2008

(100)

Visuo-haptic conflict - orientation

-6 -3 0 3 6 -6

-3 0 3 6

-6 -3 0 3 6 -6

-3 0 3 6

-6 -3 0 3 6 -6

-3 0 3 6

-6 -3 0 3 6

Predicted PSE (deg)

M ea su re d P S E ( d eg )

_{5 YOs} _{6 YOs} _{8 YOs} _Adults

Gori et al. Curr. Biol 2008

(101)

-4 -2 0 2 4 -4

-2 0 2 4

-4 -2 0 2 4 -4 -2 0 2 4 -4 -2 0 2 4 -4 -2 0 2 4

Visuo-haptic conflict

-6 -3 0 3 6 -6

-3 0 3 6

-6 -3 0 3 6 -6

-3 0 3 6

-6 -3 0 3 6 -6

-3 0 3 6

-6 -3 0 3 6

Predicted PSE (mm or deg)

M ea su re d P S E ( m m o r de g)

^{5 YOs} ^{6 YOs} ^{8 YOs} ^Adults^{10 YOs}

Size

Orientation

Gori et al. Curr. Biol 2008

(102)

Development of thresholds

3 10 Adult

2 5 10 30

Thresholds (deg)

3 10 Adult

1 3 10

Blur

Thresholds (mm)

Haptic Vision MLE

Cross Modal

H

Age (years)

Size Orientation

Gori et al. Curr. Biol 2008

(103)

Development of weights

3 10 Adult

0.0 0.5

1.0 _PSEs

Thresholds

3 10 Adult

0.0 0.5 1.0

Age (years)

H ap tic W ei gh t V is ua l W ei gh t

1.0

0.0

Size Orientation

Gori et al. Curr. Biol 2008

(104)

Concluding remarks

• During development, the senses require continuous calibration to compensate for physical growth (eye size and separation etc.).

• Perhaps the more robust sense is used to calibrate the others, even if it is less

precise.

• Using one sense to calibrate the other

precludes integration.

(105)

Abnormal Adaptive Face-Coding in

Children with Autism Spectrum Disorder

Pellicano, Jeffery, Burr, & Rhodes, Curr

Biol, 2007

(106)

Adaptation to faces

(107)

Adaptation to faces

Prof J. Mollon, FRS Dr Rowan Williams

Archbishop of Canterbury

?

(108)

Measuring the aftereffect

PSE = stimulus strength at p=0.5

aftereffect = difference in PSE

(109)

pre-adaptation data

Pellicano, Jeffery, Burr, & Rhodes, Curr

Biol, 2007

(110)

FIAE: sample individual data

Adapt antiDan Adapt antiJim Pellicano, Jeffery, Burr, & Rhodes, Curr Biol, 2007

(111)

Size of the aftereffect

*

t(27) = 2.11, p < * 0.05

Cohen’s d = .81

Pellicano, Jeffery, Burr, & Rhodes, Curr

Biol, 2007

(112)

AE correlates with score on Social Communication Questionnaire (SCQ)

r = – 0.60,

p <

0.05

(113)

Thoughts

• Combined with other evidence, our findings suggest that adaptive coding mechanisms are atypical in ASD.

• Does this occur just for faces, or for other

aftereffects too? High-level and low-level?

(114)

A visual sense of number

David Burr & John Ross

Burr & Ross

Current Biology 2008

The University

of Florence

(115)

Numerosity Estimation

Rather than counting them, economist William Jevons estimated numbers of beans thrown into a dish, and made errors when there were more

than 4 beans. Errors in estimate varied with bean number:

Weber’s law. William Stanley Jevons

(116)

Weber’s law for numerosity

Ross, Perception, 2003

25% Weber

fraction explains the subitizing

limit of 4 25

.

 0

 

N

W N

(117)

The natural scale for numbers: log or linear?

Dehaene et al. Science 2008

(118)

Dehaene et al. Science 2008

(119)

Dehaene et al. Science 2008

(120)

B

Time

N u m b e r o f ite m s ( lo g s c a l e )

0 2 5 5 0 7 5 1 0 0

Normalized response (%)

0 2 5 5 0 7 5 1 0 0

1 2 3 4 5

A C D

Fixation 500 ms

Sample 800 ms

Delay 1000 ms

1200 msTest Match

1200 msTest Match 1200 msTest

Non-Match P=0.25

P=0.25 P=0.50

0 500 1000 1500 2000

0 10 20 30 40

50 1

2 3 4 5

Time

Spike rate (Hz)

0 500 1000 1500 2000

0 10

20 1

2 3 4 5

Spike rate (Hz)

“Number-neurons” in monkey pre-frontal and parietal cortex

Selectivity follows a log scale

(121)

Nieder & Merten

J Neuroscience 2007

Number neurons cover a

large range

(122)

Human brain imaging

CS

IPS

Right hemisphere Left hemisphere

left angular gyrus (AG)

bilateral posterior superior parietal lobe (PSPL)

bilateral horizontal segment of intraparietal sulcus (HIPS)

Top view

A

L

C B

(123)

LIP neurons respond in graded fashion to total number in RF

Roitman, Brannon &Platt

PLoS 2007

(124)

Could numerosity be a visual attribute?

If so it should be subject to

adaptation .

(125)

Adaptation demo

(126)

Adaptation demo

(127)

Where did the other dots go?

(We’ll come back to that)

(128)

Adaptation: 45 sec + 8 sec top-up

(129)

Test stimulus (500 ms)

(130)

0.5 sec pause

(131)

Probe stimulus (500 ms)

(132)

Psychometric functions with adaptation

10 100 400

0.0 0.2 0.4 0.6 0.8 1.0

P (g re at er )

Matched dot number

Control Adapt to 400 dots

Probe

(133)

Adaptation vs dot number

3 10 100 300

0.5 1.0 1.5 2.0 2.5 3.0

10 100

0.5 1.0 1.5 2.0 2.5 3.0

P ro po rt io n in cr ea se

Probe dot number

DB JR

Adapt to 400 dots

(134)

Adaptation:

magnitude estimation

0 20 40 60

Estimate

Dot Number

JR

0 20 40 60 80

E st im at e

Dot Number

DB

Adapt 7

No adapt

Adapt 120

(135)

Numerosity or texture?

(136)

Size of rectangular elements:

paired comparisons

10 100

0.0 0.5 1.0

P (m or e)

Dot number

Small-small Big-big

Big-small Small-big

JB

(137)

Adaptation does not depend on element orientation

Control Parallel Orthog

0 20 40 60

M at ch ed d ot n um be r

^DB^ED

PB

Adapt

(138)

Effect of the test contrast

2 10 100

1 10

P ro po rt io n in cr ea se

Contrast of test (%)

Unadapted Threshold

Adapted threshold

(139)

Effect of adaptor contrast

10 100

0.3 1

6 DB

PB

P ro po rt io n in cr es e

Adaptor contrast (%)

No adaptation baselines detection

thresholds

(140)

Numerosity or texture

Neither PSE nor Weber fractions depend on:

• Size or shape of elements

• Orientation of elements

• Fourier sprectra of stimuli

• Contrast, or contrast sign

• Chromaticity

(141)

Colour-contingency after-effect

(142)

Colour-contingency after-effect

(143)

Colour-contingency after-effect

0.0 0.5 1.0

P ro po rt io n "m or e b lu e"

Ratio (Blue/Total)

88% Yellow

88% Blue

No Adapt

(144)

10 100 0.3

1 2 DB

PB

P ro po rt io n in cr ea se

Adapt Dot Number

Baselines

Effect of number of adaptor dots

(145)

Neural mechanisms?

LIP VIP

(146)

Neural

mechanisms?

LIP VIP

10 100

0.1 1 10

Predicted proportion increase

Adapt dot number Adapt 50

VIP LIP

(147)

10 100 0.3

1 2 DB

PB

P ro po rt io n in cr ea se

Adapt Dot Number

Baselines

Data

(148)

Interim conclusions

• The capacity to estimate number is built into vision.

• Numerosity is a primary visual

attribute: a dozen ripe cherries look twelvish , just as they look reddish.

• Like other visual attributes,

numerosity obeys Weber’s Law, is

subject to spatially local adaptation and contingency aftereffects.

Burr & Ross Curr Biol 2008 see also Butterworth Curr. Biol. 2008

(149)

Integration of information across the senses