Index of /SISTA/sdesouza

(1)

Optimisation and Robustness of Cellular

Neural Networks

Palestra para o DCA/UFRN base de sistemas inteligentes

Samuel Xavier de Souza

Katholieke Universiteit Leuven Departement Elektrotechniek (ESAT)

SCD - SISTA

(2)

Outline

1 Introduction

2 Robust CNN Tuning and Learning

3 _{Coupled Simulated Annealing}

4 _{Applying CSA to CNN Optimisation}

5 _{Conclusions and Challenges for Future Work}

(3)

Outline

1 _Introduction

The potential of Cellular Neural Networks

How does a CNN work? Goals of this thesis

2 _{Robust CNN Tuning and Learning}

3 Coupled Simulated Annealing

4 Applying CSA to CNN Optimisation

(4)

CNN as an alternative to classical digital computation

Cells replication

→ →

Local connections_→Large number of cells_→High parallelism

Local connections & analog signals_→Low area & Low power High parallelism_→ supercomputer power_→order of Tera ops.

[Chua and Yang, IEEE TCAS, 1988]

(5)

CNN as an alternative to classical digital computation

Cells replication

→ →

Local connections_→Large number of cells_→High parallelism Local connections & analog signals_→Low area & Low power

High parallelism_→supercomputer power_→order of Tera ops.

(6)

CNN as an alternative to classical digital computation

Cells replication

→ →

Local connections_→Large number of cells_→High parallelism Local connections & analog signals_→Low area & Low power High parallelism_→ supercomputer power_→order of Tera ops.

[Chua and Yang, IEEE TCAS, 1988]

(7)

CNN are very suitable for image processing

Evolution of CNN chips New Q-Eye CNN chip

176_×144 cells

>10,000 fps <100mW

Early vision processing Redundant image data

→relevant information Wide application range:

Visual applications demanding high throughput speed

(8)

A wide range of applications

Ultra-high speed event/fault monitoring in production lines

Smart security cameras

Automotive applications such as

smart airbag deployment blind spot detection navigation

collision detection

3D Echocardiogram

Vision for industrial/domestic robots Object tracking

Eye tracking and gaze estimation Visual game/computer interfaces Handsfree wheelchair driving, etc.

(9)

A wide range of applications

Ultra-high speed event/fault monitoring in production lines Smart security cameras

collision detection

3D Echocardiogram

(10)

A wide range of applications

collision detection

3D Echocardiogram

(11)

A wide range of applications

collision detection

3D Echocardiogram

(12)

A wide range of applications

collision detection

3D Echocardiogram

Vision for industrial/domestic robots

Object tracking

(13)

A wide range of applications

collision detection

3D Echocardiogram

(14)

A wide range of applications

collision detection

3D Echocardiogram

Eye tracking and gaze estimation

Visual game/computer interfaces Handsfree wheelchair driving, etc.

(15)

A wide range of applications

collision detection

3D Echocardiogram

Eye tracking and gaze estimation Visual game/computer interfaces

(16)

A wide range of applications

collision detection

3D Echocardiogram

(17)

Outline

1 _Introduction

The potential of Cellular Neural Networks

How does a CNN work?

Goals of this thesis

(18)

A template controls space-invariant CNNs

The equation of a cell

dxi,j dt = −xi,j+ X |k,l|6r ak,lyi−k,j−l+ X |k,l|6r bk,lui−k,j−l+z, with yi,j(t) =f(xi,j)

Atemplatespecifies acoupling law

19 template values(9+9+1)

Controls the entire network

CNN computer Template Digital computer Instruction CNN template values A= 2 4 a−1,−1 a−1,0 a−1,1 a0,−1 a0,0 a0,1 a_1,−1 a1,0 a1,1 3 5 ,B= 2 4 b−1,−1 b−1,0 b−1,1 b0,−1 b0,0 b0,1 b1,−1 b1,0 b1,1 3 5 ,z 8 / 54

(19)

A template controls space-invariant CNNs

CNN computer Template Digital computer Instruction CNN template values A= 2 4 a−1,−1 a−1,0 a−1,1 a0,−1 a0,0 a0,1 a_1,−1 a1,0 a1,1 3 5 ,B= 2 4 b−1,−1 b−1,0 b−1,1 b0,−1 b0,0 b0,1 b1,−1 b1,0 b1,1 3 5 ,z

(20)

A template controls space-invariant CNNs

(21)

A template controls space-invariant CNNs

CNN computer Template Digital computer Instruction CNN template values A= 2 4 a−1,−1 a−1,0 a−1,1 a0,−1 a0,0 a0,1 a_1,−1 a1,0 a1,1 3 5 ,B= 2 4 b−1,−1 b−1,0 b−1,1 b0,−1 b0,0 b0,1 b1,−1 b1,0 b1,1 3 5 ,z

(22)

A template controls space-invariant CNNs

(23)

A programmable CNN is ideal for image processing

Templates provide an ideal framework for Single-Instruction Multiple-Data computation Image processing operations are mostly SIMD

Most of these tasks have a corresponding CNN template CNN-UM allows execution of a sequence of templates

[Roska and Chua, IEEE TCAS, 1993]

One-to-one cell-pixel mapping Advantages

high parallelism much faster less silicon area very low power Theonlydrawback isrobustness

(24)

CNN chips do not behave as ideal CNNs

Example (d) (b) (a) (c) Causes of errors Parameter spread Electrical noise

Imperfect loading from off-chip to on-chip memory Temperature variation

Chip-independent approaches

Design of robust templates Template decomposition Stochastic template optimisation

(25)

Outline

1 _Introduction

The potential of Cellular Neural Networks How does a CNN work?

Goals of this thesis

(26)

Methodologies for robustness in CNN chips

Goals

Robust tuning of CNN chips Learning of complex behaviour in CNNs

Optimisation techniques for robust CNN tuning and learning Integrate the methodologies Develop proof-of-concept applications

(27)

Outline

1 _Introduction

Chip-specific Methods

Learning complex dynamics Experiments and Results

(28)

Optimising with chip measurements works better

Chip-specific

[F ¨oldesy et al., IEEE TCAS-I,1999]

Optimise-decompose-optimise approach

Uses chip measurements to calculate error Searches the minimum cost via gradient search

Non-optimal templates are decomposed

Disadvantages

Needs explicit calculation of error gradient Only optimises B and z: uncoupled templates May cause unnecessary template

decomposition

The use of chip-measurements does improve robustness

(29)

Training set is generated using an ideal CNN simulator

[Xavier-de-Souza et al., 2003]

Generating a training set triplet Training set triplet: input, initial state, and desired images Original template

p_init

Ideal CNN simulator

A good starting point

Original template pinit is a good initial approximation

(30)

Presenting our tuning setup

[Xavier-de-Souza et al., IEEE TCAS-I, 2004]

Chip-specific CNN tuning setup Optimisation core

A global optimisation method Generates and accepts, or not, a new template p Optimisation problem min p g(p), where g(p) = 1 √ k v u u t k X i=1 (y_id₋yi(∞))2 16 / 54

(31)

Major advantages, a minor drawback

Advantages of this tuning setup

No restrictions related to template type Propagating templates can be tuned too

Poor local solutions are avoided due to global optimisation No premature template decomposition

The drawback:global optimisation is slower

What makes it worth?

Number of unknowns619 compensates this drawback Original template is a good starting point

(32)

Optimising for robustness

Typical template chip error Chip-specific robustness

Gross tuning optimum

chip-robust optimum

Max. value Min value pmin pinit pmax

pmin pmax

pinit e

upper bound Theoretical

lower bound _Theoretical

Optimisation problem min p 1 r r X j=1 g(p+ej) 18 / 54

(33)

Outline

1 _Introduction

2 _{Robust CNN Tuning and Learning} Chip-specific Methods

Learning complex dynamics

Experiments and Results

(34)

Learning is more difficult than tuning

No know good starting point

Desired output is obtained either manually or via other methodologies Potentially, the entire template parameter space needs to be searched

Helpful strategies

The search space can be reduced with simple initial assumptions, e.g. symmetry, zero values

These assumptions can be relaxed in subsequent optimisation runs

(35)

Learning is more difficult than tuning

No know good starting point

Desired output is obtained either manually or via other methodologies Potentially, the entire template parameter space needs to be searched

Helpful strategies

The search space can be reduced with simple initial assumptions, e.g. symmetry, zero values

These assumptions can be relaxed in subsequent optimisation runs

(36)

Learning dynamics requires more desired outputs

Desired output is a sequence of images Example

Example

[Xavier-de-Souza et al., JCTA, 2006]

NewActive Wave Computingoperations Example s p ir a l a u to w a v e 21 / 54

(37)

Learning dynamics requires more desired outputs

Desired output is a sequence of images Example

Example

[Xavier-de-Souza et al., JCTA, 2006]

NewActive Wave Computingoperations Example s p ir a l a u to w a v e

(38)

Outline

1 _Introduction

2 _{Robust CNN Tuning and Learning} Chip-specific Methods

Learning complex dynamics

Experiments and Results

(39)

Tuning results for ACE4k chip (64

_×

64 cells)

Edge detection

Average

Half toning

(40)

Tuning results for ACE16k chip (128

_×

128 cells)

input image initial state ideal CNN output orig. tem. output final tem. output

Average

Threshold

Sobel

(41)

On-chip learning results for ACE4k (64

_×

64 cells)

Travelling wave

Combustion wave

Pyramiding up

(42)

Simulation learning of a spiral autowave (16

_×

16 cells)

D e s ir e d D e s ir e d 1 s t la y e r L e a rn e d G e n e ra lis e d 1 s t la y e r D e s ir e d D e s ir e d 2 n d la y e r L e a rn e d G e n e ra lis e d 2 n d la y e r 26 / 54

(43)

Outline

1 _Introduction

Simulated annealing and cooperative behaviour

General principles of Coupled SA Variance control of acceptances Experiments and results

(44)

SA is based on two stochastic processes

Temperature Tk Temperature Tac

k Current Cost E(x) Current Solution x Program flow Data flow No Yes No Stopping Criterion Equilibrium Criterion Acceptance x ← y if E(y) ≤ E(x), or Generation yi= f (xi, ε) Temperature Scheduling Tk+1= U(Tk, k) Tac k+1= V (T ac k, k) Yes END Initialization x= x0 Tk= T0 Tac k= T0ac k= 1 assess E(x) ∀i = 1, · · · , D; ε ← gk(Tk)

if r < A(x → y), r ← U[0, 1]

Simulating physical annealing

Physical process:

heat, hold and cool slowly down Mathematical modelling:

Material states _→ Problem solutions Energy of a state _→ Cost of a solution

Temperature _→ Control parameter Convergence proof exists fort_{→ ∞}

Many speed up versions exist: quality of solution_↔speed With distributed versions: quality_↔speed_↔resources

(45)

Why a new Simulated Annealing (SA) method?

Example

Adaptive SA: less suitable for CNN tuning

Too many run-modify-rerun loops Problem representing template solutions in Genetic Algorithms

The coupling idea

Coupled Local Minimisers couples local gradient searching processes

(46)

Cooperative behaviour in SA

Parallel SA versions

Concurrent processes move independently Best solution is exchanged among process

Ensemble SA

No exchange of best solutions

Entire ensemble moves according to average cost

[Xavier-de-Souza et al., 2006a]

The new Coupled Simulated Annealing class of methods

Coupledacceptance probabilities for SA Efficient parallel/distributed implementation Less sensitive to initialisation parameters

(47)

Outline

1 _Introduction

Simulated annealing and cooperative behaviour

General principles of Coupled SA

Variance control of acceptances Experiments and results

(48)

Comparing SA and Coupled SA

Properties of the CSA class of methods

.. . .. . AΘ y1 y2 yi ym x2 Ω SA CSA x y A xi x1 Θ xm E E E E f γ 0 ≤ A(x → y) ≤ 1,

∀ x, y ∈ Ω. Θ ⊂ Ω, and γ = f [E(x1), E(x2), · · · , E(xm)] .

0 ≤ AΘ(γ, xi→ yi) ≤ 1, ∀ x ∈ Θ, y ∈ Ω, with Ω AΘ AΘ AΘ 32 / 54

(49)

Three CSA instance methods

Method 1 Multi-state SA AΘ(γ,xi →yi) = exp −E(yi) Tac k exp −E(y_i) T_kac + exp −E(x_i) T_kac

(50)

Three CSA instance methods

Method 1 Multi-state SA AΘ(γ,xi →yi) = exp −E(yi) Tac k exp −E(yi) Tac k + X xj∈Θ exp −E(xj) Tac k

Generalises classical SA acceptance for_{Θ ≡}x1

(51)

Three CSA instance methods

Method 1 Multi-state SA AΘ(γ,xi →yi) = exp −E(yi) T_kac exp −E(yi) T_kac + X xj∈Θ exp −E(xj) T_kac | {z } γ

(52)

Three CSA instance methods

Method 2 Blind acceptance AΘ(γ,xi →yi) = 1 − exp −E(xi) T_kac X xj∈Θ exp −E(xj) T_kac

Acceptance of yi is independent of its energy when

E(x_i) <E(y_i)

(53)

Three CSA instance methods

Method 2 Blind acceptance AΘ(γ,xi →yi) = 1 − exp −E(xi) Tac k X xj∈Θ exp −E(xj) Tac k | {z } γ

Acceptance of yi is independent of its energy when

(54)

Three CSA instance methods

Method 3 Coupled SA-Modified AΘ(γ,xi →yi) = exp E(xi) Tac k X xj∈Θ exp _E (xj) Tac k

Also perform blind acceptance

Strategy: probability ofanycurrent stateleave

to a new state sums to 1

(55)

Three CSA instance methods

Method 3 Coupled SA-Modified AΘ(γ,xi →yi) = exp E(xi) T_kac X xj∈Θ exp _E_(x j) T_kac | {z } γ

(56)

Three CSA instance methods

Method 3 Coupled SA-Modified AΘ(γ,xi →yi) = exp   E(x_i)₋max z∈Θ(E(z)) T_kac   X xj∈Θ exp   E(xj)−max z∈Θ(E(z)) T_kac   | {z } γ

to a new state sums to 1

(57)

Outline

1 _Introduction

Simulated annealing and cooperative behaviour General principles of Coupled SA

Variance control of acceptances

Experiments and results

(58)

Adapting the schedule of the acceptance temperature

Acceptance variance in CSA-M

06 σ2(AΘ) 6

m₋1

m2

Simple way to control the variance ifσ2(AΘ) < σ2_D, T_kac=T_kac₋₁(1_{− α)} ifσ2(AΘ) > σ2_D, T_kac=T_kac₋₁(1+ α) Reduces sensitivity to initial parameters Adaptive schedule for T_kac Experimentally: Best values σ_D2 _≈ 99%of m_m−21 α _≈ 0.05 37 / 54

(59)

Outline

1 _Introduction

Simulated annealing and cooperative behaviour General principles of Coupled SA

Variance control of acceptances

Experiments and results

(60)

Presenting the test problems

Test problems group 1

Unimodal and simple multi-modal problems

6 multidimensional and multi-modal problems Many local minima

6 rotated problems from group 2 Difficult for multiple 1-D searches

Example −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 0 1000 2000 3000 4000 Example −500 0 500 −500 0 500 −1000 −500 0 500 1000 39 / 54

(61)

Presenting the test problems

Example −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 0 1000 2000 3000 4000 Example −500 0 500 −500 0 500 −1000 −500 0 500 1000

(62)

Presenting the test problems

Example −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 0 1000 2000 3000 4000 Example −500 0 500 −500 0 500 −1000 −500 0 500 1000 39 / 54

(63)

Comparison experiments

Multi-start SA versus CSA-M with variance control, D=30

sa csa 0 2 4 6 8 10 12 14 16 18 20 f3, D=30 sa csa 0 10 20 30 40 50 60 70 80 f4, D=30 sa csa 0 10 20 30 40 50 f5, D=30 sa csa 0 20 40 60 80 100 120 140 160 180 f6, D=30 sa csa 0 50 100 150 f7, D=30 sa csa 0 10 20 30 40 50 60 70 80 90 f10, D=30 sa csa 50 60 70 80 90 100 110 120 130 f12, D=30 sa csa 30 40 50 60 70 80 f13, D=30

Comparable performancewithoutvariance control When testing sensitivity against initialisation parameters Clear CSA-Msuperiority with variance control

(64)

Other experiments

Effect of variance control

5 10 15 20 25 30 10−5 10−4 10−3 10−2 10−1 100 101 102 Cost Number of optimizers CSAM (a) CSAM (b) CSAM w/ Var. Control (c)

Variance control approximates best average run

Scaling with dimension

4 6 8 10 12 14 16 18 20 101 102 103 104 105 Number of Optimizers

Number of Cost Function Evaluations per Optimizer

D=20 D=16 D=12

D=8

D=4

Final cost improves consistently with m Scales with higher dimensions

(65)

Outline

1 _Introduction

General approach to CNN optimisation

A systematic approach Applications

(66)

CSA for tuning and learning

Cell model dxi,j dt = −ˆg(xi,j,ΦSig) +z(ΦTem) + X |k,l|6r ak,l(ΦTem)xi−k,j−l + X |k,l|6r

bk,j(ΦTem)ui−k,j−l(ΦOpt,ΦSig)

Hardware parameters

Φ = [ΦSig,ΦTem,ΦOpt]

xi,j

ˆ

g(xi,j)

gmax(ΦSig)

gmin(ΦSig)

The optimisation problem within the unified framework min A,B,z,Φ,∆t1,···,∆tT 1 NSNRNT NS X s=1 NR X n=1 NT X k=1 X i,j (x_id ,j;s,k,n−xi,j(A,B,z,Φ,tk)) 2 43 / 54

(67)

Outline

1 _Introduction

General approach to CNN optimisation

A systematic approach

Applications

(68)

Unified CNN optimisation framework

Block diagram _{[Hillier et al., 2006]}

CNN optimisation cases Chip-specific template tuning Learning of fixed-point templates Learning of spatio-temporal dynamics Modular, replicable to many CSA processes CSA reduces

run-modify-rerun loops

(69)

Guidelines for defining a training set

Example

Include all essential input-output mapping functionalities

Consider also bias and fixed state images

Balance dark and light pixels Embed many functionalities in a single TS instance

Consider exchanging input and initial state

Ensure continuity of dynamics

CNNOPT: Matlab toolbox

(70)

Outline

1 _Introduction

General approach to CNN optimisation A systematic approach

Applications

(71)

Real-time object tracking

Tracking the object

Mi Ai Oi Fi None None None i = 0 i = 1 i = 2 i = 3 [Xavier-de-Souza et al., 2004]

Building blockfor other applications e.g. computer/game interfaces, eye tracking, face tracking, gaze estimation, target tracking, multiple object tracking.

More than300 frames/sec. in 2nd _{generation CNN chips}

(72)

Handsfree wheelchair driving

Shifting chip window

Face tracking no yes no no yes yes current frame all objects object positions Infer unknown Initialization lost?

Isolate objects in Acquire new frame if necessary Resize objects object lost? Shift tracking window resize object? Adaptive resizing 3-D Testing environment driving action driving action Bi-I Matlab Computer Video 1 Video 2 [Xavier-de-Souza et al., 2006b] 49 / 54

(73)

Outline

1 _Introduction

Conclusions

(74)

Conclusions

(75)

(76)

Conclusions

(77)

(78)

Conclusions

(79)

(80)

Conclusions

(81)

(82)

Conclusions

(83)

(84)

Conclusions

(85)

(86)

Conclusions

(87)

Outline

1 _Introduction

5 _{Conclusions and Challenges for Future Work} Conclusions

(88)

”To be continued...”

New template decomposition techniques Multi-layer emulation in single-layer CNN chips General tuning of hardware parameters

On-chip tuning of CNN algorithms Self-tuning of CNN chips

(89)

Promoters

Johan Suykens, Joos Vandewalle

Advisory committee

Andr ´e Barb ´e, Marc Van Hulle, Jan Van Impe

Co-authors

Desir é Boll é, D ániel Hillier,

Michiel Van Dyck, M ¨us¸tak E. Yalc¸ın

Prof. T ´amas Roska

for the opportunity to visit his lab in Hungary and for the honor to have him in my Ph.D. jury

SISTA colleagues