Implementation and evaluation of two prediction techniques for the Lorenz time series

(1)

Implementation and Evaluation of

Two Prediction Techniques

for the Lorenz Time Series

G r a n t E H u d d l e s t o n e

A s s i g n m e n t p r e s e n t e d i n p a r t i a l f u l f i l m e n t o f t h e r e q u i r e m e n t s f o r t h e

d e g r e e M a s t e r o f S c i e n c e a t t h e D e p a r t m e n t o f A p p l i e d M a t h e m a t i c s

o f t h e U n i v e r s i t y o f S t e l l e n b o s c h , S o u t h A f r i c a

(2)

Declaration:

I, the undersigned,

hereby declare that the work contained in this assignment

is my own original work,

and that I have not previously in its entirety, or in part, submitted

it at any university

for a degree.

Signatm~

.

(3)

A b s t r a c t

T h i s t h e s i s i m p l e m e n t s a n d e v a l u a t e s t w o p r e d i c t i o n t e c h n i q u e s u s e d t o f o r e c a s t d e t e r m i n i s t i c c h a o t i c t i m e s e r i e s . F o r a l a r g e n u m b e r o f s u c h t e c h n i q u e s , t h e r e c o n s t r u c t i o n o f t h e p h a s e s p a c e a t t r a c t o r a s s o c i a t e d w i t h t h e t i m e s e r i e s i s r e q u i r e d . E m b e d d i n g i s p r e s e n t e d a s t h e m e a n s o f r e c o n s t r u c t i n g t h e a t t r a c t o r f r o m l i m i t e d d a t a . M e t h o d s f o r o b t a i n i n g t h e m i n i m a l e m b e d d i n g d i m e n s i o n a n d o p t i m a l t i m e d e l a y f r o m t h e f a l s e n e i g h b o u r h e u r i s t i c a n d a v e r a g e m u t u a l i n f o r m a t i o n m e t h o d a r e d i s c u s s e d . T h e f i r s t p r e d i c t i o n a l g o r i t h m t h a t i s d i s c u s s e d i s b a s e d o n w o r k b y S a u e r , w h i c h i n c l u d e s t h e i m p l e -m e n t a t i o n o f t h e s i n g u l a r v a l u e d e c o m p o s i t i o n o n d a t a o b t a i n e d f r o m t h e e m b e d d i n g o f t h e t i m e s e r i e s b e i n g p r e d i c t e d . T h e s e c o n d p r e d i c t i o n a l g o r i t h m i s b a s e d o n n e u r a l n e t w o r k s . A s p e c i f i c a r c h i t e c t u r e , s u i t e d t o t h e p r e d i c t i o n o f d e t e r m i n i s t i c c h a o t i c t i m e s e r i e s , n a m e l y t h e t i m e d e p e n d e n t n e u r a l n e t w o r k a r c h i t e c t u r e i s d i s c u s s e d a n d i m p l e m e n t e d . A d a p t a t i o n s t o t h e b a c k p r o p a g a t i o n t r a i n i n g a l g o r i t h m f o r u s e w i t h t h e t i m e d e p e n d e n t n e u r a l n e t w o r k s a r e a l s o p r e s e n t e d . B o t h a l g o r i t h m s a r e e v a l u a t e d b y m e a n s o f p r e d i c t i o n s m a d e f o r t h e w e l l - k n o w n L o r e n z t i m e s e r i e s . D i f f e r e n t e m b e d d i n g a n d a l g o r i t h m - s p e c i f i c p a r a m e t e r s a r e u s e d t o o b t a i n p r e d i c t e d t i m e s e r i e s . A c t u a l v a l u e s c o r r e s p o n d i n g t o t h e p r e d i c t i o n s a r e o b t a i n e d f r o m L o r e n z t i m e s e r i e s , w h i c h a i d i n e v a l u a t i n g t h e p r e d i c t i o n a c c u r a c i e s . T h e p r e d i c t e d t i m e s e r i e s a r e e v a l u a t e d i n t e r m s o f t w o c r i t e r i a , p r e d i c t i o n a c c u r a c y a n d q u a l i t a t i v e b e h a v i o u r a l a c c u r a c y . B e h a v i o u r a l a c c u r a c y r e f e r s t o t h e a b i l i t y o f t h e a l g o r i t h m t o s i m u l a t e q u a l i t a t i v e f e a t u r e s o f t h e t i m e s e r i e s b e i n g p r e d i c t e d . I t i s s h o w n t h a t f o r b o t h a l g o r i t h m s t h e c h o i c e o f t h e e m b e d d i n g d i m e n s i o n g r e a t e r t h a n t h e m i n i m u m e m b e d d i n g d i m e n s i o n , o b t a i n e d f r o m t h e f a l s e n e i g h b o u r h e u r i s t i c , p r o d u c e s g r e a t e r p r e d i c t i o n a c c u r a c y . F o r t h e n e u r a l n e t w o r k a l g o r i t h m , v a l u e s o f t h e e m b e d d i n g d i m e n s i o n g r e a t e r t h a n t h e m i n i m u m e m -b e d d i n g d i m e n s i o n s a t i s f y t h e b e h a v i o u r a l c r i t e r i o n a d e q u a t e l y , a s e x p e c t e d . S a u e r ' s a l g o r i t h m h a s t h e g r e a t e s t b e h a v i o u r a l a c c u r a c y f o r e m b e d d i n g d i m e n s i o n s s m a l l e r t h a n t h e m i n i m a l e m b e d d i n g d i m e n s i o n . I n t e r m s o f t h e t i m e d e l a y , i t i s s h o w n t h a t b o t h a l g o r i t h m s h a v e t h e g r e a t e s t p r e d i c t i o n a c c u r a c y f o r v a l u e s o f t h e t i m e d e l a y i n a s m a l l i n t e r v a l a r o u n d t h e o p t i m a l t i m e d e l a y . T h e n e u r a l n e t w o r k a l g o r i t h m i s s h o w n t o h a v e t h e g r e a t e s t b e h a v i o u r a l a c c u r a c y f o r t i m e d e l a y c l o s e t o t h e o p t i m a l t i m e d e l a y a n d S a u e r ' s a l g o r i t h m h a s t h e b e s t b e h a v i o u r a l a c c u r a c y f o r s m a l l v a l u e s o f t h e t i m e d e l a y . M a t l a b c o d e i s p r e s e n t e d f o r b o t h a l g o r i t h m s .

(4)

Opsomming

In hierdie

tesis word twee voorspellings-tegnieke

geskik vir voorspelling

van determ inistiese

chaotiese

tydreekse

ge"im plem enteer en geevalueer.

Vir sulke tegnieke word die rekonstruksie

van die aantrekker

in

fase-ruim te

geassosieer m et die tydreeks

gewoonlik vereis.

Inbedm etodes

word aangebied

as 'n m anier om die aantrekker

te rekonstrueer

uit beperkte

data.

M etodes

om die m inim um

inbed-dim ensie

te bereken uit gem iddelde

wedersydse

inligting

sowel as die optim ale

tydsvertraging

te bereken uit vals-buurpunt-heuristiek,

word bespreek.

Die eerste voorspellingsalgoritm e

wat bespreek

word is gebaseer op 'n tegniek

van Sauer.

Hierdie

algo-ritm e m aak gebruik van die im plem entering

van singulierwaarde-ontbinding

van die ingebedde

tydreeks

wat voorspel word.

Die tweede

voorspellingsalgoritm e

is gebaseer

op neurale

netwerke.

'n Spesifieke netwerkargitektuur

geskik vir determ inistiese

chaotiese

tydreekse,

naam lik

die tydafhanklike

neurale

netwerk

argitektuur

word bespreek

en ge"im plem enteer.

'n M odifikasie van die terugprapagerende

leer-algoritm e

vir gebruik

m et die tydafhanklike

neurale

netwerk word ook aangebied.

Albei algoritm es

word geevalueer deur voorspellings

te m aak vir die bekende

Lorenz tydreeks.

Verskeie

inbed param eters

en ander algoritm e-spesifieke

param eters

word gebruik

om die voorspelling

te m aak.

Die werklike waardes

vanuit die Lorentz tydreeks

word gebruik om die voorspellings

te evalueer en om

voorspellingsakkuraatheid

te bepaal.

Die voorspelde tydreekse

word geevalueer op grand van twee kriteria,

naam lik voorspellingsakkuraatheid,

en kwalitatiewe

gedragsakkuraatheid.

Gedragsakkuraatheid

verwys na die verm oe van die algoritm e

om

die kwalitatiewe

eienskappe

van die tydreeks

korrek te sim uleer.

Daar word aangetoon

dat vir beide algoritm es die keuse van inbed-dim ensie

grater as die m inim um

inbed-dim ensie soos bereken uit die vals-buurpunt-heuristiek,

grater akkuraatheid

gee. Vir die neurale

netwerk-algoritm e

gee 'n inbed-dim ensie

grater

as die m inim um

inbed-dim ensie

ook betel' gedragsakkuraatheid

so os verwag. Vir Sauer se algoritm e,

egter, word betel' gedragsakkuraatheid

gevind vir 'n inbed-dim ensie

kleiner as die m inim ale

inbed-dim ensie.

In term e van tydsvertraging

word dit aangetoon

dat vir beide algoritm es

die grootste

voorspellingsakku-raatheid

verkry word by tydvertragings

in 'n interval rondom

die optim ale

tydsvetraging.

Daar

word ook aangetoon

dat

die neurale

netwerk-algoritm e

die beste

gedragsakkuraatheid

gee vir

tydsvertragings

naby

aan

die optim ale

tydsvertraging,

terwyl

Sauer

se algoritm e

betel'

gedragsakku-raatheid

gee by kleineI' waardes van die tydsvertraging.

(5)

1 Introduction

2.2 Tim e D elay

2

Concepts in embedding theory

3.3 S election of the N earest N eighbours

3 Prediction with the Sauer algorithm

3.4 C enter of M ass of the N earest N eighbours

1.1

1.2

1.3

1.4

1.5

2.1

2.3

3.1

3.2

3.5

3.6

3.7

B ackground P roblem D escription .

Lorenz system , general inform ation and equations

N otation conventions

C hapter P review . . .

P hase space reconstruction

E m bedding D im ension.

P rem ise of S auer's A lgorithm ..

R econstruction of the A ttractor .

S ingular V alue D ecom position and P rojections

R egression of the Linear M odel

C alculation of the Target P oint

1

2

5

7

8

9

11

15

16

17

18

20

4 Prediction with Time Delayed Neural Networks

4.1 A n Introduction to N eural N etw orks ...

23

(6)

II C O N T E N T S

4 .2 S tru c tu re s in N e u ra l N e tw o rk A rc h ite c tu re . . . .. 2 5

4 .3 T im e D e p e n d e n t N e u ra l N e tw o rk s . . . .. 2 9

4 .4 T ra in in g A lg o rith m fo r T im e -D e la y N e u ra l N e tw o rk s . . . .. 3 2

5 Evaluation

of the algorithms

37

5 .1 E v a lu a tio n o f th e T D N N a lg o rith m . . . .. 3 8

5 .1 .1 P e rfo rm a n c e C rite ria 3 9

5 .1 .2 E x p e rim e n ta l re s u lts o b ta in e d fo r th e T D N N a lg o rith m . . . .. 4 3

5 .1 .3 A n a ly s is o f re s u lts fo r th e T D N N a lg o rith m . . . .. 4 4

5 .2 E v a lu a tio n o f th e S a u e r a lg o rith m . . . .. 4 6

5 .2 .1 P e rfo rm a n c e C rite ria

. . . ..

47

5 .2 .2 E x p e rim e n ta l re s u lts o b ta in e d fo r th e S a u e r a lg o rith m . . . .. 4 7

5 .2 .3 A n a ly s is o f re s u lts fo r th e S a u e r a lg o rith m . . . .. 4 8

6 Conclusion

51 A

Time series scaling functions

53 B

Sauer algorithm

Matlab code

55

B .1 M a in p ro g ra m . . . .. 5 5 B .2 E m b e d d in g th e tim e s e rie s . . . .. 5 6 B .3 F in d in g n e a re s t n e ig h b o u rs . . . .. 5 6 B .4 C a lc u la tin g th e c e n tre o f m a s s o f th e n e a re s t n e ig h b o u rs . . . .. 5 6 . . . .. 5 7 B .5 S in g u la r v a lu e d e c o m p o s itio n a n d th e s p a n n in g m a trix B .6 P ro je c tin g th e d is p la c e m e n t m a trix B .7 P e rfo rm in g th e lin e a r re g re s s io n . B .8 E s tim a tin g th e ta rg e t p o in t ... . . .. 5 7 . . .. 5 7 5 7

C

TDNN

algorithm

Matlab code

59

C .1 T D N N tra in in g c o d e . . .. 5 9

(7)

CJ

n

0 '<

w

~

'<

I'D

~

riG"

::r-

...•

::J ;:;: iii"

N"

OJ

...•

0"

::J

-.

C ::J n

...•

0"

::J

(8)

(9)

List of Figures

1.1 P hase space plot of the L orenz attractor obtained from the solution of the three L orenz equations. 3

1.2 P lot of the solution of 1J

(t)

for the L orenz system s of equations. . . .. 4

2.2 A m ap that is not one-to-one.

2.3 A n em bedding F of the sm ooth m ap A .

2.1

2.4

2.5

2.6

A one-to-one m ap that is not continuously differentiable. i.e. not an im m ersion.

D ifferent choices of tim e delays on a L orenz tim e series ..

A verage m utual inform ation versus tim e delay for the L orenz tim e series ..

A n illustration of false and true nearest neighbours. . .

9 9 9

10

1 1

12

3.1 Illustration of points on the attractor relevant to the S auer algorithm . . . .. 19

4.4 T he effect of the bias on activation function output.

4 .1

4.2

4.3

N eural netw ork architecture.

T he structure of a single neuron.

P opular activation functions. . .

24

25

27

28

4.5

Left:

F eedforw ard connections for a T D N N .

Right:

F eedback connections for a T D N N . . .. 30

4.6 F eedback and F eedforw ard connection detail.

4.7

Left:

F eedforw ard connections for R N N .

Right:

F eedback connections for R N N ..

4.8

D ecision tree for adapted back-propagation neural netw ork training. . . .

31

33

5.1 P lot illustrating a predicted tim e series versus the associated real tim e series obtained from a

prediction by the

_TDNN

algorithm . . . .. 38

5.2 C haracteristic points for the L orenz tim e series. . . .. 40

(10)

V I

5 .3 C h a ra c te ris tic p o in ts o n th e L o re n z a ttra c to r. ..

LIST

O F

FIGURES

41

5 .1 0 P lo t o f a q u a lita tiv e ly c o rre c t s im u la tio n b y th e S a u e r a lg o rith m . 5 .4 P lo t o f th e C -s p a c e p o in ts fo r re a l tim e s e rie s fro m tw o v ie w p o in ts .

5 .1 3 P lo t o f a p re d ic tio n b y th e S a u e r a lg o rith m th a t d is p la y s d iv e rg e n t b e h a v io u r. . 5 .1 1 P lo t o f a p re d ic tio n b y th e S a u e r a lg o rith m th a t is tra p p e d o n o n e w in g .

5 .1 2 P lo t o f a p re d ic tio n b y th e S a u e r a lg o rith m th a t d is p la y s o s c illa to ry b e h a v io u r ..

41

4 2

43

4 3 4 3 4 3

48

4 8

48

P lo t o f a p re d ic tio n th a t firs t c a p tu re s th e d y n a m ic s b u t is tra p p e d .

P lo t o f a p re d ic tio n th a t d is p la y s o s c illa to ry b e h a v io u r. .. . .. P lo t o f a p re d ic tio n th a t d e c a y s to w a rd s a fa ls e fix e d p o in t. P lo t o f a q u a lita tiv e ly c o rre c t s im u la tio n .

P lo t o f th e firs t d e riv a tiv e o f th e L o re n z tim e s e rie s v e rs u s th e L o re n z tim e s e rie s . 5 .5

5.6

5.7

5 .8

(11)

List of Tables

2 .1 P e rc e n ta g e o f fa ls e n e a re s t n e ig h b o u rs a t d iffe re n t e m b e d d in g d im e n s io n s fo r th e L o re n z tim e s e rie s . 1 3

5.1

5 .2 T e s tin g p a ra m e te rs fo r T D N N a lg o rith m .

A v e ra g e M S E v a lu e s fo r T D N N p re d ic tio n s u s in g a c c u ra c y c rite rio n .

38

44

5 .3 A v e ra g e e u c lid e a n d is ta n c e b y T D N N p re d ic tio n fo r th e b e h a v io u r c rite rio n u s in g th e fx -s p a c e

d is c rim in a tio n . . . .. 4 4

5 .4 A v e ra g e e u c lid e a n d is ta n c e b y T D N N p re d ic tio n fo r th e b e h a v io u r c rite rio n u s in g th e (-s p a c e

d is c rim in a tio n . . . .. 4 5

5 .5 A v e ra g e M S E v a lu e s fo r T D N N p re d ic tio n s u s in g a c c u ra c y c rite rio n w ith h id d e n c o n te x t la y e rs . 4 5

5 .6 A v e ra g e e u c lid e a n d is ta n c e b y T D N N p re d ic tio n w ith h id d e n c o n te x t la y e rs fo r th e b e h a v io u r

c rite rio n u s in g th e (-s p a c e d is c rim in a tio n . . . .. 4 5

5 .7 A v e ra g e e u c lid e a n d is ta n c e b y T D N N p re d ic tio n w ith h id d e n c o n te x t la y e rs fo r th e b e h a v io u r

c rite rio n u s in g th e fx -s p a c e d is c rim in a tio n . . . .. 4 5

5 .9 A v e ra g e M S E v a lu e s fo r S a u e r a lg o rith m p re d ic tio n s u s in g th e b e h a v io u r c rite rio n .

5 .1 0 A v e ra g e M S E v a lu e s fo r S a u e r a lg o rith m p re d ic tio n s u s in g th e a c c u ra c y c rite rio n .

5.8

T e s tin g p a ra m e te rs fo r S a u e r a lg o rith m . .

47

49

. . . 4 9 5 .1 1 A v e ra g e M S E v a lu e s fo r d im e n s io n v e rs u s d e la y . . . .. 4 9 5 .1 2 A v e ra g e M S E v a lu e s fo r d im e n s io n v e rs u s s p a n n in g d im e n s io n . m a x im u m v a lu e s fo r a ro w .) . VB (V a lu e s in ita lic s in d ic a te

49

(12)

(13)

List of Symbols

T : T h e L o r e n z t i m e s e r i e s u s e d f o r a l g o r i t h m e v a l u a t i o n . N : T h e l e n g t h o f t h e t i m e s e r i e s T . X t : A d i s c r e t e v a r i a b l e d e s c r i b i n g t h e s t a t e o f t h e s y s t e m a t t i m e

t.

m:

T h e e m b e d d i n g d i m e n s i o n . l : T h e t i m e d e l a y . s : T h e s p a n n i n g d i m e n s i o n .

b

t: T h e p r e d i c t i o n p o i n t o n t h e r e c o n s t r u c t e d L o r e n z a t t r a c t o r . l l i : T h e i - t h n e a r e s t n e i g h b o u r t o t h e p r e d i c t i o n p o i n t o n t h e L o r e n z a t t r a c t o r . ei: T h e t a r g e t p o i n t f o r t h e i - t h n e a r e s t n e i g h b o u r . c : T h e c e n t r e o f m a s s o f t h e n e a r e s t n e i g h b o u r s .

d(t):

A t i m e d e l a y e d c o o r d i n a t e s e t a t t i m e

t.

k : R e g r e s s i o n c o e f f i c i e n t s . h ~ : T h e s u m m i n g s u b - u n i t v a l u e f o r n o d e i i n l a y e r

n.

l ~ : T h e v a l u e f o r n o d e i i n l a y e r

n.

wD:

T h e w e i g h t c o n n e c t i n g t h e j - t h n o d e i n l a y e r

n -

1 w i t h t h e i - t h n o d e i n l a y e r

n.

'! '( p ) : I n p u t p a t t e r n p f o r a n e u r a l n e t w o r k . Y ( p ) : O u t p u t p a t t e r n p f o r a n e u r a l n e t w o r k .

!\.:

T h e n u m b e r o f l a y e r s i n a n e u r a l n e t w o r k . H : T h e n u m b e r o f c o n t e x t l a y e r s i n a n e u r a l n e t w o r k .

J :

T h e n u m b e r o f s e t s o f i n p u t p a t t e r n s , '! '( p ) . a n d a s s o c i a t e d o u t p u t p a t t e r n s , Y ( p ) , f o r a n e u r a l n e t w o r k . MT l : T h e n u m b e r o f n o d e s i n l a y e r

n.

I X

(14)

Chapter 1 Introduction

1.1 Background

W h e n a d isc u ssio n tu rn s to p re d ic tio n , a p e rso n w ill in v a ria b ly a sk w h e th e r p re d ic tin g th e fin a n c ia l m a rk e ts

is p o ssib le . T h is q u e stio n h ig h lig h ts th e c o m m o n d e sire to d iv in e th e fu tu re a n d h a s b e e n w ith h u m a n k in d

fo r m ille n n ia , a t first sim p ly w o n d e rin g w h a t th e w e a th e r m ig h t h o ld fo r c ro p s to th e m o d e rn d a y d e sire to

fo re c a st th e m a rk e t-d riv e n e c o n o m ie s o f th e w o rld .

In e x p la in in g th e p ro c e ss o f p re d ic tio n to a p e rso n , th e d istin c tio n b e tw e e n d iffe re n t fo rm s o f c h a o tic b e h a v io u r

h a s to b e h ig h lig h te d . P u re ly sto c h a stic b e h a v io u r is w h a t p e o p le im m e d ia te ly th in k a b o u t w h e n th e y h e a r

c h a o s. T h e te rm c h a o s is h o w e v e r a lso c o n sid e re d fo r sy ste m s th a t illu stra te irre g u la r b e h a v io u r b u t a t th e

sa m e tim e a re n o t c o m p le te ly ra n d o m .

T h is fo rm o f c h a o s is d e te rm in istic [1 ] [1 2 ] a n d h a s tw o k e y fe a tu re s; th e sy ste m is se n sitiv e to in itia l c o n d itio n s

a n d th e sy ste m , o v e r tim e , se ttle s d o w n to a fix e d re g io n o f th e sp a c e d e fin e d b y its v a ria b le s.

S e n sitiv ity to in itia l c o n d itio n s m e a n s th a t a v e ry sm a ll c h a n g e in th e in itia l, o r sta rtin g , c o n d itio n s o f a

sy ste m c a n le a d to v e ry d iffe re n t b e h a v io u r in a sh o rt p e rio d o f tim e . F o r th e p u rp o se s o f th is th e sis, o n ly a

d e te rm in istic c h a o tic sy ste m th a t p o sse sse s a n a ttra c to r, b e in g a lo c a liz e d a re a in th e v a ria b le sp a c e fo r th e

sy ste m to w h ic h th e so lu tio n w ill c o n v e rg e fro m in itia l c o n d itio n s, w ill b e c o n sid e re d .

A c c u ra te ly p re d ic tin g th e fu tu re fo r a sto c h a stic sy ste m m ig h t to a ll in te n ts a n d p u rp o se s b e im p o ssib le . T h is

h o w e v e r is n o t th e c a se fo r a sy ste m e x h ib itin g d e te rm in istic c h a o tic b e h a v io u r w h e re m e th o d s m a y b e so u g h t

to m a k e p re d ic tio n s [1 1 ] [2 7 ] a b o u t th e fu tu re b e h a v io u r o f th e sy ste m .

A s w ith a ll a n a ly tic a l m e th o d s th e d a ta u n d e r in v e stig a tio n , in th is se n se th e tim e se rie s c o n ta in in g o b se rv a tio n s

o f th e c h a o tic sy ste m , h a s to b e p ro c e sse d in to a fo rm th a t is c o n v e n ie n t fo r a n a ly sis.

A la rg e n u m b e r o f p re d ic tio n a lg o rith m s [3 ] [4 ] [8 ] [1 8 ] [2 1 ] re ly o n th e re c o n stru c tio n o f th e a ttra c to r fro m

th e o b se rv e d m e a su re m e n ts. D u e to th e p o te n tia lly la rg e n u m b e r o f v a ria b le s a t p la y in a c h a o tic sy ste m ,

(15)

2

CHAPTER

_{1. INTRODUCTION}

r e c o n s tr u c tio n o f th e a ttr a c to r is u s u a lly n o t p o s s ib le a n d a m e th o d is s o u g h t to a p p r o x im a te th e a ttr a c to r .

S a u e r [ 2 0 ] ju s tif ie d th e u s e o f e m b e d d in g a tim e s e r ie s in r e c o n s tr u c tin g a n a ttr a c to r . S u c c e s s f u l e m b e d d in g

r e q u ir e s th e c h o ic e o f a m in im a l e m b e d d in g d im e n s io n [ 1 4 ] a n d a n o p tim a l tim e d e la y [ 9 ] . M e th o d s e x is t to

d e te r m in e th e s e o p tim a l p a r a m e te r s , a n d th e s e m e th o d s w ill b e d is c u s s e d in c h a p te r 2 .

1.2 Problem Description

T h is th e s is im p le m e n ts a n d e v a lu a te s th e p e r f o r m a n c e o f tw o p r e d ic tio n a lg o r ith m s w h e n a p p lie d to th e L o r e n z

tim e s e r ie s ( s e c tio n 1 .3 ) .

T h e f ir s t a lg o r ith m is b a s e d o n w o r k b y S a u e r [ 1 8 ] a n d u tiliz e s lo c a l lin e a r m o d e ls , in w h ic h th e s in g u la r v a lu e

d e c o m p o s itio n p la y s a p iv o ta l r o le , to p e r f o r m th e p r e d ic tio n s .

T h e s e c o n d a lg o r ith m e m p lo y s a n e u r a l n e tw o r k a p p r o a c h , w h ic h , b e c a u s e o f th e n o n - lin e a r n a tu r e o f th e

m e th o d h a s a p p e a r e d s u ita b le f o r th e a n a ly s is a n d p r e d ic tio n o f c h a o tic d a ta . Q u o tin g S w in g le r [ 2 5 ] :"

applications

which require pattern

recognition

or simulation

of a physical system

too complex to model with rules are

perfectly

suited to neural computing

techniques."

V a r io u s p r e d ic tio n a lg o r ith m s [ 2 ] [ 3 ] [ 4 ] [ 7 ] [ 1 7 ] b a s e d o n

n e u r a l n e tw o r k s h a v e b e e n f o r m u la te d .

T h e p e r f o r m a n c e o f th e s e tw o a lg o r ith m s is e v a lu a te d o n th e b a s is o f ( 1 ) p r e d ic tio n a c c u r a c y a n d ( 2 ) w h e th e r

th e p r e d ic te d tim e s e r ie s c a p tu r e s th e d y n a m ic s o f th e tim e s e r ie s u n d e r in v e s tig a tio n .

T h e f ir s t c r ite r io n o f p r e d ic tio n a c c u r a c y a s a m e a s u r e o f p e r f o r m a n c e is th e p r im e p u r p o s e o f th e a lg o r ith m .

H o w e v e r , th e s e c o n d c r ite r io n a s k s a m o r e s tr in g e n t q u e s tio n : if th e a lg o r ith m p r e d ic ts a n u m b e r o f tim e

s e r ie s v a lu e s in to th e f u tu r e , d o th o s e v a lu e s im ita te th e b e h a v io u r o f th e tim e s e r ie s b e in g p r e d ic te d ?

T h e q u e s tio n th e n a r is e s a b o u t w h ic h c o m b in a tio n s o f e m b e d d in g d im e n s io n a n d tim e la g a r e th e b e s t f o r

p r e d ic tio n a c c u r a c y a n d f o r c a p tu r in g o f th e d y n a m ic s . F u r th e r m o r e , a r e th e tw o c r ite r ia s a tis f ie d b y th e

s a m e c o m b in a tio n o f te s t p a r a m e te r s ? F in a lly , a r e th e b e s t c o m b in a tio n s in d e p e n d e n t o f th e c h o ic e o f th e

a lg o r ith m ?

1.3 Lorenz

s y s t e m ,

general information and equations

T h e L o r e n z tim e s e r ie s [ 1 6 ] [ 2 3 ] is a s ta p le s o u r c e o f d a ta f o r th e e v a lu a tio n o f p r e d ic tio n a lg o r ith m s .

T h e L o r e n z e q u a tio n s d e s c r ib e a s im p lif ie d m o d e l o f f lu id m o tio n d r iv e n b y c o n v e c tio n in th e a tm o s p h e r e b y

in c o r p o r a tin g v a r ia b le s f o r th e r a te o f c o n v e c tio n o v e r tu r n in g ( x ) , th e h o r iz o n ta l te m p e r a tu r e v a r ia tio n ( 1 } ) ,

(16)

1.3. LORENZ SYSTEM,

GENERAL

INFORMATION

AND EQUATIONS

3

50

Phase space plot of the Lorenz attractor

y(t) N -20 -15 -10 -5 x (t) o 5 1 0 1 5 20 F ig u r e 1 .1 : P h a s e s p a c e p lo t o f th e L o r e n z a ttr a c to r o b ta in e d f r o m th e s o lu tio n o f th e th r e e L o r e n z e q u a tio n s . d x u ( y - x ) ,

=

( 1 .1 ) d t d y -

=

p x - y - x z , ( 1 .2 ) d t d z

=

- l3 z

+

x y . ( 1 .3 ) d t

T h e P r a n d tl n u m b e r ,

u,

is p r o p o r tio n a l to r a tio o f th e f lu id v is c o s ity o f a s u b s ta n c e to its th e r m a l c o n d u c tiv ity .

T h e R a y le ig h n u m b e r , p , is p r o p o r tio n a l to d if f e r e n c e in te m p e r a tu r e b e tw e e n th e to p a n d b o tto m o f th e

s y s te m a n d 1 3 is th e r a tio o f th e w id th to th e h e ig h t o f th e R a y le ig h - B e n a r d c e ll e n c a s in g th e s y s te m . T h e s e

p a r a m e te r s a r e u s u a lly s e t to th e f o llo w in g v a lu e s ;

u

=

10,

p

=

2 8 a n d 1 3

=

l

T h e L o r e n z e q u a tio n s h a v e c e r ta in im p o r ta n t f e a tu r e s :

• T h e e q u a tio n s a r e a u to n o m o u s , i.e . th e r ig h t- h a n d s id e s a r e n o t f u n c tio n s o f tim e .

• T h e e q u a tio n s a r e n o n - lin e a r b e c a u s e o f th e x z a n d x y te r m s .

• T h e s o lu tio n s a r e b o u n d e d .

• T h e r e a r e n o u n s ta b le p e r io d ic o r b its o r u n s ta b le s ta tio n a r y p o in ts .

E v e n th o u g h th e e q u a tio n s c o n ta in n o r a n d o m , n o is y o r s to c h a s tic te r m s th e r e is a s u r p r is in g le v e l o f u n p r e

-d ic ta b ility in th e s o lu tio n , a s is e v id e n t f r o m f ig u r e

1.1.

(17)

4

CHAPTER

1 .

INTRODUCTION

• T h e in fin ite n u m b e r o f d a ta p o in ts th a t c a n re a d ily b e g e n e ra te d c o m p u ta tio n a lly fro m th e e q u a tio n s .

• T h e s o lu tio n o f th e L o re n z e q u a tio n s d e s c rib e s a s tra n g e a ttra c to r in p h a s e s p a c e .

T h e s o lu tio n fo r e q u a tio n s (1 .1 ). (1 .2 ) a n d (1 .3 ) w e re o b ta in e d fro m th e u tilis a tio n o f a n e x p lic it R u n g e -K u tta

fo rm u la . In th is w o rk th e o d e 4 5 fu n c tio n o f M a tla b w a s u s e d .

S o lu tio n s c a n b e fo u n d fo r x . 1 ) a n d z b u t th is th e s is w ill o n ly u tilis e th e tim e s e rie s o b ta in e d fro m th e s o lu tio n

o f

1 ).

L e t X i =1 ) (( i. - l) h ) . w h e re h is th e tim e s te p fo r th e s o lu tio n o b ta in e d fro m th e fu n c tio n s o lv e r a n d fo r th is p a p e r is s e t a t 0 .0 1 7 0 s . T h e re fo re , th e tim e s e rie s u s e d fo r a lg o rith m e v a lu a tio n in th is th e s is h a s th e fo rm ,

T

a n d is g ra p h ic a lly re p re s e n te d in fig u re 1 .2 . { X l,X l, ... ,Xn } , P lo t o f L o re n z tim e s e rie s (1 .4 ) 0 .9 0 .8 . " ~ ~ 0 .4 0 .3 0 .2 0 .1 0 2 4 6 8 1 \ 1 0 1 2 1 4 1 6 -F ig u re 1 .2 : P lo t o f th e s o lu tio n o f 1 )( t

1

fo r th e L o re n z s y s te m s o f e q u a tio n s .

(18)

1.4. NOTATION

CONVENTIONS

1.4 Notation

conventions

5 V e c t o r s a r e p r i n t e d i n b o l d f a c e a n d d e n o t e a c o l u m n v e c t o r . T h e t r a n s p o s e o f v e c t o r s w i l l b e i n d i c a t e d b y t h e s u p e r s c r i p t T . T h e E u c l i d e a n n o r m o f a v e c t o r a

=

[a

1 ,

a2, ... , an]T

w i l l b e r e p r e s e n t e d b y

II

a

II

w i t h

1.5 Chapter Preview

IIall=

Ja:a

=

JL;-

af.

( 1 . 5 )

C h a p t e r 2 e x p l a i n s t h e f u n d a m e n t a l i d e a s b e h i n d e m b e d d i n g t h e o r y a n d e l a b o r a t e s o n t h e p r o c e s s e s b e h i n d c a l c u l a t i n g t h e o p t i m a l e m b e d d i n g d i m e n s i o n a n d t i m e d e l a y f o r a g i v e n t i m e s e r i e s . H a v i n g l a i d t h e g r o u n d w o r k f o r p r e d i c t i o n a l g o r i t h m s t o c o m e , c h a p t e r 3 e x p l a i n s S a u e r 's a l g o r i t h m . C h a p t e r 4 b r i e f l y i n t r o d u c e s t h e r e a d e r t o m a i n c o n c e p t s i n n e u r a l n e t w o r k a r c h i t e c t u r e a s w e l l a s e x p l a i n i n g t h e t i m e d e p e n d e n t a r c h i t e c t u r e u t i l i z e d i n t h e a l g o r i t h m . T h e i m p l e m e n t a t i o n o f t h e a l g o r i t h m s i s e x p l a i n e d i n c h a p t e r 5 a n d r e s u l t s a n d a n a l y s i s o f b o t h a l g o r i t h m s a r e p r e s e n t e d .

(19)

(20)

Chapter 2 Concepts

In

• embedding theory

A d y n a m ic a l s y s te m is a s e t o f e q u a tio n s th a t g iv e s th e tim e e v o lu tio n o f th e s y s te m 's s ta te fro m in itia l

c o n d itio n s .

T h e d y n a m ic a l s y s te m m a y e ith e r b e d e s c rib e d b y c o n tin u o u s v a ria b le s , w h e re a c o m m o n fo rm o f th e d y n a m ic a l

s y s te m c a n b e d e s c rib e d b y firs t-o rd e r a u to n o m o u s o rd in a ry d iffe re n tia l e q u a tio n s s u c h a s th o s e fo r th e L o re n z

e q u a tio n s (1 .1 , 1 .2 , 1 .3 ). T h e s o lu tio n fo r th e d iffe re n tia l e q u a tio n s d e s c rib e s a c o n tin u o u s e v o lu tio n o f th e

s ta te o f th e s y s te m fro m in itia l c o n d itio n s .

S im ila rly , fo r d is c re te v a ria b le s , th e d y n a m ic a l s y s te m m a y b e d e s c rib e d b y a m a p o f th e fo rm ,

X n + l

=

F (x n l.

fo r w h ic h th e s o lu tio n d e s c rib e s a d is c re te e v o lu tio n o f th e s ta te o f th e s y s te m fro m in itia l c o n d itio n s . (2 .1 )

E a c h m u lti-d im e n s io n a l p o in t o b ta in e d fro m th e s o lu tio n o f th e d y n a m ic a l s y s te m d e s c rib in g th e s ta te o f th e

s y s te m is re p re s e n te d b y a c o rre s p o n d in g p o in t in p h a s e s p a c e w h o s e c o o rd in a te s a re th e v a ria b le s o f th e s ta te .

T h e re fo re a n y p o in t in p h a s e s p a c e c o m p le te ly d e s c rib e s th e s ta te o f th e s y s te m a n d c o n v e rs e ly a n y s ta te o f

th e s y s te m h a s a c o rre s p o n d in g p o in t in p h a s e s p a c e . T h e re la tio n s h ip (in te rc o n n e c tio n ) o f p o in ts in p h a s e

s p a c e d e s c rib in g th e tim e e v o lu tio n o f th e s y s te m is te rm e d a tra je c to ry .

If, fo r s o m e in itia l c o n d itio n s th e tra je c to rie s c o n v e rg e to s o m e b o u n d e d s u b s e t o f p h a s e s p a c e , th is s e t is

re fe rre d to a s a n a ttra c to r. T ra n s ie n t d a ta a re v a lu e s o b ta in e d fo r th e s y s te m th a t h a v e n o t y e t c o n v e rg e d

o n to th e a ttra c to r. D u e to fa c t th a t tra n s ie n t b e h a v io u r d o e s n o t c a p tu re th e s ta b le d y n a m ic s o f th e s y s te m ,

its e ffe c t o n p re d ic tio n a c c u ra c y is a n im p o rta n t c o n s id e ra tio n a n d th u s fo r th e re m a in d e r o f th is th e s is it is

a s s u m e d th a t a ll tra n s ie n t b e h a v io u r is o m itte d fro m th e re le v a n t d a ta s e ts .

If th e v a ria b le s o b s e rv e d fro m a s y s te m d o n o t fu lly d e s c rib e a p h a s e s p a c e p o in t th e n s o m e m e th o d is re q u ire d

to re c o n s tru c t p h a s e s p a c e fro m th e in c o m p le te s e t o f v a ria b le s .

(21)

8 CHAPTER

_2.

CONCEPTS

IN EMBEDDING

THEORY

2 .1

P h a s e s p a c e re c o n s tru c tio n

R eco n stru ctio n o f th e o rig in al p h ase sp ace fro m o b serv ed d ata is u su ally n o t p o ssib le. M eth o d s are th u s so u g h t

w h ich allo w th e reco n stru ctio n o f th e attracto r in p h ase sp ace fro m th e av ailab le d ata (fo r earlier m eth o d s

see [1 0 ] an d [1 9 ]). T h e req u irem en t fo r reco n stru ctio n is th at lin earisatio n o f th e d y n am ics at an y p o in t in

th e p h ase sp ace is p reserv ed b y th e reco n stru ctio n p ro cess. T h is is d u e to th e n eed to ex p lo it d eterm in ism in

th e d ata an d th e p ro v isio n th at th e tim e ev o lu tio n o f a trajecto ry in reco n stru cted p h ase sp ace o n ly d ep en d s

o n its cu rren t p o sitio n .

G iv en an o b serv ed d iscrete tim e series, T , a d elay reco n stru ctio n , w ith d im en sio n m an d tim e d elay 1 , is as

fo llo w s,

b (m -l )l+ n

=

[X (m -l )l+ n ,'" , X 2 l+ n , X l+ n ,

x

n ] T .

(2 .2 )

E m b ed d in g th eo ry states th at fo r id eal n o ise-free d ata th ere ex ists a m in im al v alu e fo r th e d im en sio n , m , su ch

th at th e reco n stru cted v ecto rs

b

are eq u iv alen t to p h ase sp ace v ecto rs. E q u iv alen ce, in th is sen se, refers to th e fact th at th e m ap is a o n e-to -o n e im m ersio n (to b e ex p lain ed b elo w ).

A co n stru ctio n is o n ly co n sid ered an em b ed d in g if th e reco n stru cted m ap d o es n o t co llap se p o in ts o r tan g en t

d irectio n s, in o th er w o rd s, th at th e m ap p in g is o n e-to -o n e an d th at d ifferen tial in fo rm atio n is p reserv ed .

W h en th e state o f th e d eterm in istic d y n am ical sy stem , an d th e fu tu re ev o lu tio n , is co m p letely sp ecified b y a

sin g le p o in t in p h ase sp ace th en th e m ap p in g is co n sid ered o n e-to -o n e. If at a g iv en state x th e v alu e f(x )

is o b serv ed in th e reco n stru cted sp ace an d th at after a fix ed tim e in terv al an o th er ev en t fo llo w s, th en if

f

is o n e-to -o n e each ap p earan ce o f f(x ) w ill b e fo llo w ed b y th e sam e ev en t.

If a sm o o th m ap , F o n

A,

is a co n tin u o u sly d ifferen tiab le m an ifo ld an d th e d eriv ativ e m ap D F (x ) is o n e-to -o n e fo r ev ery p o in t x o n

A

th en th e m ap is an im m ersio n . U n d er an im m ersio n n o d ifferen tial stru ctu re is lo st in

g o in g fro m

A

to

F(A).

T h erefo re a m ap , F , is an em b ed d in g if an d o n ly if it is a o n e-to -o n e im m ersio n . F ig u res 2 .1 , 2 .2 an d 2 .3

illu strate d ifferen t v alid an d in v alid em b ed d in g m ap s.

In o rd er to em b ed a scalar tim e series w e n eed to co n stru ct m in d ep en d en t v ariab les, th e n u m b er req u ired

to u n iq u ely ch aracterize th e sy stem . T h e v ariab le set (v ecto r) w ill th u s ex ist in so m e m -d im en sio n al sp ace.

T h erefo re, to reco n stru ct th e attracto r in th e n ew p h ase sp ace w e n eed to fin d th e em b ed d in g d im en sio n , m ,

an d an o p tim al tim e d elay , w h ich w ill allo w th e b est ch o ice o f v alu es fro m th e tim e series so th at th e v ariab les

(22)

2.2. TIME DELAY

₉

F

F ig u re 2 .1 : A o n e-to -o n e m ap th at is n o t co n

-tin u o u sly d ifferen tiab le, i.e. n o t an im m ersio n .

F ig u re 2 .2 : A m ap th at is n o t o n e-to -o n e.

F

F (A )

F ig u re 2 .3 : A n em b ed d in g F o f th e sm o o th m ap A .

2.2 _{Time Delay}

T he reconstruction of phase space is equivalent to the original space of variables and is a m ethod for providing

independent coordinates com posed of the present observation Xt and earlier observations of the system . T he

tim e delayed coordinate set at tim e

t,

w ith tim e delay 1, appears as follow s

d(t)

[X t, X t~ l, X t-2 l, ... ,] . (2.3)

If

the tim e delay is chosen too sm all then successive delayed coordinates w ill not be independent enough

and not enough tim e w ould have passed for the system to have explored the state space sufficiently. A s

an exam ple, consider a highly oversam pled data set w here given a sm all tim e delay successive tim e delayed

m easurem ents w ould be alm ost identical and provide no inform ation about the system .

C onversely, if the tim e delay w ere too large, tw o successive m easurem ents in equation 2.3 w ould be nearly

random w ith respect to each other.

(23)

1 0 xx -Time delay =2 Time delay =9 Time delay = 30

CHAPTER

2 .

CONCEPTS

IN EMBEDDING

THEORY

'"

z x X l

'"

+ x -X l o

'"

xZ X l F i g u r e 2 . 4 : D i f f e r e n t c h o i c e s o f t i m e d e l a y s o n a L o r e n z t i m e s e r i e s . o f 2 v e r y l i t t l e i n f o r m a t i o n a b o u t t h e e v o l u t i o n o f t h e s y s t e m i s o b t a i n e d a n d t h a t f o r a t i m e d e l a y o f 3 0 t h e p o i n t s h a v e a n e a r r a n d o m d i s t r i b u t i o n a n d l i k e w i s e g i v e n o i n f o r m a t i o n o n t h e s t a t e o f t h e s y s t e m . F o r a t i m e d e l a y o f 9 i t c a n b e f e l t b y v i s u a l i n s p e c t i o n t h a t t h e d y n a m i c s o f t h e s y s t e m i s b e i n g c a p t u r e d . T h i s v a l u e f o r t h e t i m e d e l a y w i l l b e s h o w n t o b e t h e t h e o r e t i c a l c o r r e c t c h o i c e f o r t h e L o r e n z t i m e s e r i e s . F i g u r e 2 .4 a l s o s h o w s t h e L o r e n z a t t r a c t o r i n 2 d i m e n s i o n s f o r t h e d i f f e r e n t t i m e d e l a y s . I t c a n o n c e a g a i n b e s e e n t h a t i n c o r r e c t c h o i c e s o f t h e t i m e d e l a y l e a d t o b a d r e c o n s t r u c t i o n s o f t h e a t t r a c t o r . A v a l u e f o r t h e t i m e d e l a y i s t h u s s o u g h t , w h i c h w i l l e n s u r e t h a t s u c c e s s i v e t i m e d e l a y e d m e a s u r e m e n t s , a r e n o t c o m p l e t e l y i n d e p e n d e n t i n a s t a t i s t i c a l s e n s e . A m e t h o d i s s o u g h t t h a t q u a n t i f i e s t h e d e p e n d e n c e o f

d(t -

1 ) o n v a l u e s o f

d(t)

i n a t i m e s e r i e s . M u t u a l i n f o r m a t i o n i s d e f i n e d b y 1 ( 1 )

=

-.L

P i j ( 1 ) I n P i j ( 1 ) i j P i P j , w h e r e P i i s t h e p r o b a b i l i t y t o f i n d a n o b s e r v a b l e i n t h e i .- t h i n t e r v a l o f a t i m e s e r i e s a n d P i j i s t h e p r o b a b i l i t y t h a t a n o b s e r v a t i o n f a l l s i n t o t h e i .- t h i n t e r v a l a n d a f t e r a t i m e 1 i s o b s e r v e d i n t h e j - t h i n t e r v a l . F r a s e r [ 9 ] d e m o n s t r a t e s t h a t t h e f i r s t m i n i m u m o f t h e m u t u a l i n f o r m a t i o n o f a t i m e s e r i e s c o r r e s p o n d s t o t h e b e s t c h o i c e o f t i m e d e l a y . I n f i g u r e 2 .5 t h e a v e r a g e m u t u a l i n f o r m a t i o n f o r t h e L o r e n z t i m e s e r i e s i s s h o w n w i t h t h e f i r s t m i n i m a i n d i c a t i n g

(24)

2 .3 . E M B E D D I N G D I M E N S I O N

Average mutual information for the Lorenz time series

2 . 5 2

11

c o

'iii

E

.E

. S ~ 1 . 5 ::> E O J O l co Q ; >

«

0 . 5 IS> 2 IS> 3 (S) 4 (S) 5 l" l s 6 7 Time delay 8 8 9 A 1 0 e 1 1 8 1 2 F i g u r e 2 . 5 : A v e r a g e m u t u a l i n f o r m a t i o n v e r s u s t i m e d e l a y f o r t h e L o r e n z t i m e s e r i e s . t h e o p t i m u m t i m e d e l a y a t 9 . E x p e r i m e n t a l r e s u l t s i n c h a p t e r 5 i l l u s t r a t e h o w t h e c h o i c e o f t i m e d e l a y a f f e c t s p r e d i c t i o n a c c u r a c y .

2 . 3

E m b e d d i n g

D i m e n s i o n

A n a t t r a c t o r i s c o n s i d e r e d u n f o l d e d , a n d t h u s a n e m b e d d i n g , i f a l l n e i g h b o u r i n g p o i n t s i n t h e a t t r a c t o r a r e t r u e n e i g h b o u r s . K e n n e l [ 1 4 ] d e f i n e s a f a l s e n e a r e s t n e i g h b o u r a s f o l l o w s : " A fa ls e n e ig h b o u r is a p o in t in th e d a ta s e t th a t is a n e ig h b o u r s o le ly b e c a u s e

we

a r e v ie w in g th e o r b it ( th e a ttr a c to r ) in to o s m a ll a n e m b e d d in g s p a c e ." T h e r e f o r e a l l n e i g h b o u r i n g p o i n t s , i n a n i n c o m p l e t e u n f o l d e d a t t r a c t o r , w i l l n o t b e s o s o l e l y d u e t o t h e d y n a m i c s o f t h e s y s t e m b u t a l s o b e c a u s e t h e g e o m e t r i c s t r u c t u r e o f t h e a t t r a c t o r w a s p r o j e c t e d d o w n t o s m a l l e r s p a c e t h a n r e q u i r e d b y t h e a t t r a c t o r . I t i s o b v i o u s t h a t p r e d i c t i o n s m a d e w i t h i n f o r m a t i o n f r o m f a l s e n e i g h b o u r s w o u l d d e l i v e r u n r e l i a b l e r e s u l t s . T h e m e t h o d o f f a l s e n e a r e s t n e i g h b o u r s , i n t r o d u c e d b y K e n n e l [ 1 4 ] ' i s u s e d t o c a l c u l a t e a m i n i m u m e m b e d d i n g d i m e n s i o n f o r t h e a t t r a c t o r s o a s t o a v o i d p r o j e c t i n g p o i n t s i n t o n e i g h b o u r h o o d s o f t h e a t t r a c t o r t o w h i c h t h e y d o n o t b e l o n g . T h i s m e a n s t h a t f o r d i m e n s i o n s m l e s s t h a n t h e m i n i m u m e m b e d d i n g d i m e n s i o n m a t h e t o p o l o g i c a l s t r u c t u r e o f t h e a t t r a c t o r i s n o t p r e s e r v e d a n d t h e r e c o n s t r u c t i o n i s n o t a o n e - t o - o n e i m a g e . T h e r e f o r e , o n c e t h e n u m b e r o f f a l s e n e a r e s t n e i g h b o u r s i s z e r o t h e m i n i m u m e m b e d d i n g d i m e n s i o n h a s b e e n

(25)

12 CHAPTER

_2.

CONCEPTS

IN EMBEDDING

THEORY

re a c h e d a n d th e a ttra c to r h a s b e e n u n fo ld e d in p h a s e s p a c e . F ig u re 2 .6 illu s tra te s th e e ffe c t o H a ls e n e ig h b o u rs .

False and true nearest neighbours on the Lorenz attractor

0 -.

+

+'

><

X t F ig u r e 2 .6 : A n illu s tr a tio n o f f a ls e a n d tr u e n e a r e s t n e ig h b o u r s . F o r th e p o in t P th e n e a re s t n e ig h b o u r is th e fa ls e n e ig h b o u r F n (

t*)

a s c a n b e s e e n b y th e d iv e rg e n c e o f th e fa ls e n e ig h b o u r's tra je c to ry . T h e re a l n e ig h b o u r is p o in t

Nn(t)

w h o s e tra je c to ry is s im ila r to th a t o f p o in t P .

T h e s q u a re o f E u c lid e a n d is ta n c e b e tw e e n a p o in t

b

n a n d a n e a re s t n e ig h b o u r

b

r in d -d im e n s io n a l s p a c e is [R d (n , r)]2 d-l

L.

(X n -k l - X r_ k l)2 ,

k=O

w h e re 1 is th e c h o s e n tim e d e la y . If th e e m b e d d in g d im e n s io n is in c re a s e d b y o n e to d

+

1 , th e n [R d

+

1 (n , r)]2 - [R d (n , r)]2 _{[X n -d l} _- _{X r-d tl} 2 _.

A c c o rd in g to K e n n e l [1 4 ]' a firs t o f tw o c rite ria fo r a p o in t to b e d e fin e d a s a fa ls e n e ig h b o u r is

IX n d l - X r-d ll

>

R

to 1, R d (n ,r)

w h e re Rto l is a c h o s e n to le ra n c e .

A s e c o n d c rite rio n is re la te d to th e ra d iu s o f th e a ttra c to r. If th e s u p p lie d tim e s e rie s is s m a ll a p ro b le m

m ig h t a ris e b e c a u s e a s th e e m b e d d in g d im e n s io n s in c re a s e s , s o th e p o in ts in th e a ttra c to r s p re a d o u t to fill th e p h a s e s p a c e . T h u s in s m a ll d a ta s e ts n e a re s t n e ig h b o u rs m ig h t n o t b e c lo s e , i.e . th e E u c lid e a n d is ta n c e

(26)

2.3. EMBEDDING

DIMENSION

D i m e n s i o n 1 2 3 4 P e r c e n t a g e o f f a l s e n e a r e s t n e i g h b o u r s 4 2 .6 1 .6 0 0

13

T a b l e 2 . 1 : P e r c e n t a g e o f f a l s e n e a r e s t n e i g h b o u r s a t d i f f e r e n t e m b e d d i n g d i m e n s i o n s f o r t h e L o r e n z t i m e s e r i e s . b e t w e e n n e i g h b o u r i n g p o i n t s i s o f t h e s a m e o r d e r a s t h a t o f t h e 'r a d i u s ' o f t h e a t t r a c t o r , R A, w h e r e

[R

A ] 2

1 N (

1 N

)2

N';

xn-

N,;

xn R A i s t h e n t h e r o o t m e a n s q u a r e v a l u e o f t h e d a t a a b o u t i t s m e a n . T h e r e f o r e , i f a n e i g h b o u r i n g p o i n t m e e t s t h e c r i t e r i o n i n e q u a t i o n 2 .4 i t i s d e e m e d f a l s e , w h e r e A t o l i s a c h o s e n t o l e r a n c e . Rd

+

1 _{( n , r )}

>

_A t o l .

RA

( 2 .4 ) A n i m p o r t a n t c o n s i d e r a t i o n n e e d s t o b e t a k e n i n t o a c c o u n t i f t h e d a t a i s o v e r s a m p l e d . T h e n , w h e n e x a m i n i n g a n e a r e s t n e i g h b o u r t h e p o i n t c o u l d e x i s t o n t h e s a m e t r a j e c t o r y a n d j u s t b e d i s p l a c e d b y t h e t i m e e q u i v a l e n t o f t h e s a m p l i n g r a t e . T h i s p o i n t i s a l w a y s a t r u e n e i g h b o u r a n d m i g h t i n f l u e n c e t h e f a l s e n e i g h b o u r s t a t i s t i c . A s i m p l e s o l u t i o n i s t o d o w n s a m p l e b u t t h i s d i s c a r d s d a t a a n d i s u s u a l l y u n a c c e p t a b l e . A n o t h e r s o l u t i o n i s t o d i s c a r d p o i n t s t h a t h a v e a t i m e i n d e x w i t h i n a b o u n d f r o m t h e i n d e x o f t h e p o i n t u n d e r i n v e s t i g a t i o n . A m o r e e l e g a n t s o l u t i o n i s t h a t o f f a l s e s t r a n d s , a s i n t r o d u c e d b y K e n n e l [ 1 3 ] . T h e c o n c e p t i s s i m i l a r t o t h a t o f t h e f a l s e n e a r e s t n e i g h b o u r a p p r o a c h e x c e p t t h a t a g r o u p o f n e a r e s t n e i g h b o u r i n g p o i n t s o n a t r a j e c t o r y , o r s t r a n d , a r e c h o s e n . I f i n i n c r e a s i n g t h e d i m e n s i o n a n y p o i n t o n a s t r a n d i s f a l s e , t h e e n t i r e s t r a n d i s d e e m e d f a l s e . T h e m e t h o d p r o c e e d s i n a s i m i l a r f a s h i o n t o t h a t o f t h e f a l s e n e a r e s t n e i g h b o u r s m e t h o d . E x p e r i m e n t a l r e s u l t s i n c h a p t e r 5 i l l u s t r a t e h o w t h e c h o i c e o f a n e m b e d d i n g d i m e n s i o n s a f f e c t s t h e p r e d i c t i o n a c c u r a c y .

(27)

(28)

Chapter 3 Prediction with the Sauer algorithm

T o m a k e a s e r ie s o f p r e d ic tio n s f o r a tim e s e r ie s , a m o d e l o f th e g iv e n tim e s e r ie s h a s to b e c o n s tr u c te d .

T h e m e th o d u s e d f o r p r e d ic tio n th e n e m p lo y s a c o n tin u a tio n o f th e m o d e l s o th a t f u tu r e v a lu e s o f th e tim e

s e r ie s m a y b e c a lc u la te d b y m e a n s o f s o m e f o r m o f e x tr a p o la tio n . T h e r e f o r e , p r e d ic tio n m o d e ls illu s tr a te h o w

th e tim e s e r ie s d a ta m u s t b e v ie w e d o r in te r p r e te d a n d p r e d ic tio n m e th o d s p r o c e s s e s th e d a ta to m a k e th e

p r e d ic tio n .

M o d e ls m a y b e c la s s if ie d a s e ith e r lo c a l o r g lo b a l m o d e ls . G lo b a l m o d e ls m a k e p r e d ic tio n s o f f u tu r e v a lu e s o f

th e tim e s e r ie s p o s s ib le b y u s in g a ll k n o w n p o in ts in th e tim e s e r ie s . N e u r a l n e tw o r k s ( c h a p te r 4 ) a r e g lo b a l

m o d e ls a s th e y a r e tr a in e d o n a ll a v a ila b le tim e s e r ie s d a ta to m a k e p r e d ic tio n s . L o c a l m o d e ls in tu r n o n ly

u tiliz e a p o r tio n o f th e a ttr a c to r in a n e ig h b o u r h o o d a r o u n d th e d a ta p o in t f o r w h ic h th e p r e d ic tio n is b e in g

m a d e . T h e a lg o r ith m d e v is e d b y S a u e r [ 1 8 ] a n d s tu d ie d in th is c h a p te r is a n e x a m p le o f a lo c a l m o d e l.

T h e s im p le s t m o d e ls u s e d f o r p r e d ic tio n o f a tim e s e r ie s a r e b a s e d o n lin e a r te c h n iq u e s . L in e a r m o d e ls o n ly

p r e d ic t p e r io d ic o s c illa tin g , e x p o n e n tia l s o lu tio n s o r c o m b in a tio n s th e r e o f .

I n tu r n , n o n - lin e a r p r e d ic tio n te c h n iq u e s a tte m p t to a p p r o x im a te m o r e o f th e d y n a m ic s o f a s y s te m a n d in

s o d o in g a p p r o x im a te b o th th e lin e a r a n d n o n - lin e a r b e h a v io u r o f th e tim e s e r ie s . D u e to th e f le x ib ility o f

n o n - lin e a r m o d e ls o v e r f ittin g o n d a ta c a n b e c o m e a r e a l c o n s id e r a tio n b e c a u s e th e m o d e l f its e q u a lly w e ll o n

tr u e tim e s e r ie s f e a tu r e s a n d o n n o is e in th e tim e s e r ie s .

(29)

16 CHAPTER

_3.

PREDICTION

WITH

THE SAUER

ALGORITHM

3 .1

P re m is e o f S a u e r's A lg o rith m

T h e S a u e r a lg o rith m to b e d isc u sse d in th is c h a p te r h a s its o rig in s in a v e ry sim p le b u t p o o r p re d ic tio n

te c h n iq u e u tiliz in g e m b e d d e d a ttra c to rs th a t w ill b e te rm e d th e c e n te r o f m a ss (C O M ) p re d ic tio n te c h n iq u e .

N o te th a t fo r th is c h a p te r a tra je c to ry re fe rs to a n o rd e re d se t o f se q u e n tia l p o in ts in a lo c a liz e d a re a o f

th e a ttra c to r a ro u n d th e p re d ic tio n p o in t. It th e re fo re m a k e s se n se to re fe r to d iffe re n t tra je c to rie s o n o n e

a ttra c to r. T h is is in c o n tra st to th e p re v io u sly d e fin e d v ie w o f a sin g le tra je c to ry b e in g d e fin e d fo r th e e n tire

a ttra c to r.

In sh o rt, th e C O M te c h n iq u e fin d s a n u m b e r o f n e a re st n e ig h b o u rs a ro u n d th e la st p o in t o n th e a ttra c to r,

sh ifts th e n e a re st n e ig h b o u rs o n e p o in t fo rw a rd o n th e re sp e c tiv e tra je c to rie s a n d p re d ic ts th e n e x t p o in t o n

th e a ttra c to r a s b e in g th e c e n te r o f m a ss o f th e se n e ig h b o u rs. T h is te c h n iq u e is v e ry se n sitiv e to p re d ic tio n

e rro rs a n d m u ltiste p p re d ic tio n s q u ic k ly fa il.

S a u e r ta k e s th is sim p le a p p ro a c h a n d im p ro v e s o n it b y in tro d u c in g th e c o n c e p t o f p rin c ip a l c o m p o n e n t

a n a ly sis th ro u g h th e u se o f sin g u la r v a lu e d e c o m p o sitio n (S V D ).

G iv e n a n e m b e d d e d tim e se rie s (e m b e d d in g d im e n sio n

m),

th e n e ig h b o u rin g p o in ts to th e fin a l, o r p re d ic tio n , p o in t o n th e a ttra c to r a re fo u n d . T h e S V D o f th e m a trix fo rm e d b y th e v e c to rs d e sc rib in g th e p o sitio n o f

n e ig h b o u rin g p o in ts re la tiv e to th e c e n tre o f m a ss o f th e se n e ig h b o u rin g p o in ts is o b ta in e d .

A se t o f o rth o g o n a l v e c to rs sp a n n in g th e ro w sp a c e o f a m a trix is o b ta in e d fro m th e S V D o f th e m a trix .

B y c h o o sin g th e first s o rth o g o n a l v e c to rs, a su b sp a c e is fo u n d w h ic h is th e b e st s-d im e n sio n a l lin e a r sp a c e

(s ::;

m )

th ro u g h th e p o in ts in a le a st-sq u a re s se n se .

P ro je c tio n s o f th e m a trix o n to th e c o lu m n sp a c e p ro v id e d b y th e S V D a re c a lc u la te d . L in e a r m o d e ls a re b u ilt

w h ic h m a p th e p ro je c tio n s to k n o w n fu tu re tim e se rie s v a lu e s o f th e n e ig h b o u rin g p o in ts. A sim ila r m a p p in g

is th e n a p p lie d to th e p re d ic tio n p o in t to p re d ic t th e n e x t v a lu e in th e tim e se rie s.

3 .2

R e c o n s tru c tio n

o f th e A ttra c to r

T h e re c o n stru c tio n sta rts w ith c h o o sin g a d im e n sio n ,

m,

a n d tim e d e la y , 1 . If th e tim e se rie s h a s th e sa m e fo rm a s p re v io u sly d e fin e d in c h a p te r 2 th e n th e e m b e d d in g m a trix , B , a p p e a rs a s fo llo w s,

B

X ( m - l) l+ l X ( m - l ) l+ 2 X l+ l X l+ 2 X l X2 b ( m - l) l+ l b ( m - lll+ 2 (3 .1 ) X t X t- ( m - 2 ) l X t- ( m - l) l

b

Tt

It c a n b e n o te d fro m e q u a tio n 3 .1 th a t th e tim e se rie s is e m b e d d e d in 're v e rse ' a c ro ss th e ro w s o f th e m a trix .

T h is is u se fu l in sim p lify in g so m e in te rp re ta tio n a sp e c ts a n d d o e s n o t c h a n g e th e fu n c tio n in g o f th e a lg o rith m

(30)

3.3. SELECTION

O F

THE NEAREST

NEIGHBOURS

₁₇

T h e p re d ic tio n p o in t,

h

t , in te rm s o f th e e m b e d d e d tim e s e rie s is th e la s t ro w , o r c o o rd in a te p o in t, in B . T h e

ta rg e t p o in t is h t + 1 a n d is to c o n ta in th e firs t p re d ic te d v a lu e .

T a rg e t v a lu e s , W i = X ( m - l ) l + ( i + 1), a re d e fin e d a s th e s c a la r v a lu e s s u c h th a t h T m - l ) l + i m a p s to W i . It is th u s im m e d ia te ly a p p a re n t th a t th e p re d ic tio n p o in t, h t, m a p s to th e ta rg e t v a lu e W t = X t +1 w h ic h is n o t a n e m b e d d e d v a lu e in (3 .1 ).

3 .3

S e l e c t i o n o f t h e

N e a r e s t

N e i g h b o u r s

T h e firs t ta s k is lo c a te th e 'V n e a re s t n e ig h b o u rs to th e e m b e d d e d p re d ic tio n p o in t,

h

t , fro m th e tra in in g

s e t (d a ta p o in ts g iv e n a t th e s ta rt o f th e p re d ic tio n ). In e x te n d e d p re d ic tio n ru n s o f m o re th a n o n e p o in t it

is im p o rta n t to d iffe re n tia te b e tw e e n th e tra in in g s e t a n d p re d ic tio n s e t. T h e tra in in g s e t is th e e m b e d d in g

m a trix w ith w h ic h th e in itia l p re d ic tio n ru n b e g a n . N e ig h b o u rs a re o n ly c h o s e n fro m th e tra in in g s e t s o th a t

e rro rs in p re d ic tio n d o n o t in flu e n c e th e p re d ic tio n a c c u ra c y o f la te r p o in ts .

T h e E u c lid e a n d is ta n c e fro m h t to a n y p o in t h i in th e a ttra c to r is c a lc u la te d a s fo llo w s

. 6 t i = l l h t - h i

II.

T h e v e c to rs h i c o rre s p o n d in g to th e s m a lle s t . 6 t i 'S a re c a n d id a te s fo r n e a re s t n e ig h b o u rs . H o w e v e r, o n ly o n e p o in t o n e a c h tra je c to ry s h o u ld b e ta k e n a s a n e a re s t n e ig h b o u r to e n s u re th a t e n o u g h d iv e rs e in fo rm a tio n

c a n b e fo u n d a b o u t th e b e h a v io u r o f th e tra je c to rie s in th e a p p lic a b le re g io n o f th e a ttra c to r. T h e re fo re th e 'V

c lo s e s t n e ig h b o u rs o n s e p a ra te tra je c to rie s a re c h o s e n a s th e 'V n e a re s t n e ig h b o u rs , a n d a re re la b e lle d ,

nj,

fo r

j

=

1 , ... ,'V. T h e m a trix N is c o n s tru c te d w ith th e 'Vv e c to rs

nT

a s ro w s a n d th e ir a s s o c ia te d ta rg e t v a lu e s a re re la b e lle d C j. T h e ta rg e t v e c to r is d e fin e d a s

N=

e =

nT

nI

T noy C l

Cz

C o y

(3.2)

( 3 .3 )

A m e th o d o f e n s u rin g a c h o ic e o f 'V n e ig h b o u rs o n 'V d iffe re n t tra je c to rie s is : if a n y c a n d id a te n e ig h b o u r, h i

h a s a n in d e x w h ic h d iffe rs a c h o s e n

b

o r le s s fro m a p re v io u s ly s e le c te d n e ig h b o u r's in d e x th e n th e c a n d id a te

(31)

1 8

CHAPTER

_3.

PREDICTION

WITH

THE SAUER ALGORITHM

A s tr a ig h tf o r w a r d w a y to p r e d ic t w o u ld b e to f in d th e b e s t r e g r e s s io n o f e j o n l l j a n d to u s e it to p r e

-d ic t W t f r o m

b

t . H o w e v e r , th is m e th o d is s u s c e p tib le to n o is e in th e d a ta . I t is th e r e f o r e m o r e a p p r o p r ia te

to d r a w th e m o r e d o m in a n t tr e n d s f r o m th e m a tr ix N a n d u s e o n ly th a t f o r th e r e g r e s s io n . T h e u s e o f th e

S V D th u s b e c o m e s p r u d e n t a n d th e f o llo w in g s e c tio n s d e s c r ib e th e p r o c e s s in v o lv e d .

3.4 Center of Mass of the Nearest Neighbours

H a v in g o b ta in e d th e "V n e a r e s t n e ig h b o u r s to

b

t th e c e n te r o f m a s s o f th e s e n e ig h b o u r s h a s to b e f o u n d . T h e n o r m a l m e th o d f o r c a lc u la tin g th e c e n te r o f m a s s v e c to r , c o f a n u m b e r o f p o in ts is a s f o llo w s , "V . L l l j j = l C=--"V i = 1 , ... ,m . ( 3 .4 )

A w e ig h te d c e n tr e o f m a s s is u s e d in th is th e s is , b e c a u s e o f s lig h tly im p r o v e d p r e d ic tio n a c c u r a c y , w ith ,

"V . L W j l l j j = l

c=-

"V . L W j j = l i = 1 , ... ,m , ( 3 .5 ) w h e r e Wj is th e w e ig h t c o n s tr u c te d in s u c h a w a y th a t th e w e ig h tin g in c r e a s e s w ith a d e c r e a s e in th e d is ta n c e

o f a n e a r e s t n e ig h b o u r to th e p r e d ic tio n p o in t. T h e c h o s e n w e ig h tin g e x p r e s s io n , Wj, is a s f o llo w s ,

W j

=

( 1 _ ~ (

II

l l j - b t

II)

2) 3

2

II

ll" V - bt

II

A m a tr ix C is c o n s tr u c te d w ith "V r o w s o f c a s f o llo w s , c

T

c

T

C=

c

T

3.5 Singular Value Decomposition

and Projections

( 3 .6 )

H a v in g o b ta in e d th e m a tr ix C th e s in g u la r v a lu e d e c o m p o s itio n , S V D [ 2 4 ] ' o f th e r e s u ltin g d is p la c e m e n t

m a tr ix ,

D

=

N - C c a n n o w b e c a lc u la te d . T h e d is p la c e m e n t m a tr ix ,

D,

d e s c r ib e s p o s itio n v e c to r s f o r e a c h n e ig h b o u r r e la tiv e to th e c e n tr e o f m a s s . B y c a lc u la tin g th e S V D f o r th e m a tr ix , v e c to r s a r e o b ta in e d th a t

s p a n th e r o w s p a c e o f th e m a tr ix .

S in g u la r v a lu e d e c o m p o s itio n a llo w s th e f a c to r is a tio n o f th e "V x m m a tr ix ,

D,

a s f o llo w s ,

(32)

3.5. SINGULAR

VALUE DECOMPOSITION

AND PROJECTIONS

o

P r e d ic tio n

P o in t

• N e ig h b o u r s

E 9 C O M

o f N e ig h b o u r s

• o th e r p o in ts

F ig u r e 3 .1 : I llu s tr a tio n o f p o in ts o n th e a ttr a c to r r e le v a n t to th e S a u e r a lg o r ith m .

s o th a t

r

0"1 liL VT ~ [

u, ... u,

]~m

[ v , ... v

m

r

0

1 9 w h e re U ( v X 'V ) a n d V ( m x m ) a re b o th o rth o g o n a l m a tric e s a n d

L

( 'V X m ) is d ia g o n a l. T h e v e c to rs , V 1 , ... , Vm, a re o rth o n o rm a l b a s e s fo r th e ro w s p a c e o f 0 a n d U 1 , ... ,u " a re lik e w is e o rth o n o rm a l b a s e s fo r th e c o lu m n s p a c e o f D . H a v in g p re v io u s ly c h o s e n a s p a n n in g d im e n s io n s , s o th a t s :::;

m,

th e firs t s c o lu m n s in V a re s e le c te d . T h e s e le c te d v e c to rs a re k n o w n a s th e d o m in a n t rig h t s in g u la r v e c to rs a n d s p a n th e lin e a r s p a c e , RS . RS is a s u b s p a c e in th e p h a s e s p a c e Rm. T h e s p a n n in g v e c to rs , V 1 , V 2 ," " Vs , a re p a c k e d in to th e ( m x s ) m a trix S a s fo llo w s ,

(3.7)

S

= [

V1 Vs ] , ( 3 .8 )

(33)

2 0

CHAPTER

3. PREDICTION

WITH

THE SAUER ALGORITHM

T h e s p a n n in g v e c to r s , S , th u s d e s c r ib e a lo w - d im e n s io n a l lin e a r s p a c e p a s s in g th r o u g h th e c e n te r o f m a s s , c , o f th e n e ig h b o u r s . T h e a d v a n ta g e o f u s in g th e S V D b e c o m e s a p p a r e n t b e c a u s e th e d y n a m ic in f o r m a tio n o f th e d is p la c e m e n t v e c to r s a r e c o m p r e s s e d in to th e f ir s t f e w d o m in a n t v e c to r s w h e r e a s th e n o n - d o m in a n t v e c to r s m a in ly s to r e le s s im p o r ta n t in f o r m a tio n s u c h a s n o is e . T h e r e f o r e , b y p r o je c tin g th e d is p la c e m e n t v e c to r s o n to th e s u b s p a c e RS _{th e} _{v e c to r s} _{a r e} _{f o r c e d} _{o n to} _{th e} s u b s p a c e p a s s in g th r o u g h th e c e n tr e o f m a s s w h ils t d e c r e a s in g th e e f f e c t o f n o is e in th e d a ta . T h e p r o je c tio n , P , is c a lc u la te d a s f o llo w s p

T

=

eDT,

w h e r e

e

is th e p r o je c tio n m a tr ix .

e

=

S ( S T S ) - ' S T .

3.6 Regression of the Linear Model

I n th e f o llo w in g s e c tio n it is a s s u m e d th a t th e d is c u s s io n r e v o lv e s a r o u n d a a ttr a c to r f o r w h ic h th e r o w c o u n t

o f th e e m b e d d e d m a tr ix is f a r g r e a te r th a n th e d im e n s io n

m.

O n c e th e c o lu m n s o f 0 h a v e b e e n p r o je c te d d o w n to th e s u b s p a c e , RS

, th e lin e a r r e g r e s s io n m o d e l c a n b e

f o u n d w h ic h b e s t f its th e p r o je c te d v e c to r s .

L e t

Q

b e th e p r o je c tio n m a tr ix P a u g m e n te d w ith a c o lu m n o f o n e s , v iz .

Q

= [P u ] w ith u = [ 1 ,1 , ... , l] T . B y a u g m e n tin g th e p r o je c tio n m a tr ix w ith o n e s th e r e g r e s s io n c o e f f ic ie n ts a s s o c ia te d w ith th a t c o lu m n p la y s

th e r o le o f a c o n s ta n t. I t is th u s r e q u ir e d to s o lv e th e r e g r e s s io n c o e f f ic ie n ts , k , in th e f o llo w in g e q u a tio n , Q k e . E x p a n d in g ( 3 .9 ) g iv e s , r P ','

P',l

...

P ',m 1

k,

Pl,'

Pl,l

...

Pl,m

1

kl

P " , ,

P",l

...

P " ,m 1

k

m

+,

( 3 .9 )

C,

Cl ( 3 .1 0 ) C"

3.7 Calculation of the Target Point

H a v in g o b ta in e d a lin e a r e q u a tio n f o r th e p r e d ic tio n p o in t b t th e ta r g e t p o in t b t + ' c a n b e c a lc u la te d .

(34)

3.7. CALCULATION

O F

THE TARGET

POINT

₂₁

T h e so lu tio n fo r

k

le ts

e ~

Qk

su c h th a t

IIQk -

ell

is a s sm a ll a s p o ssib le . T h e re fo re th e e x p re ssio n q

Tk

a p p ro x im a te s W t =b t+ l (').

T h e c o m p le te e x p re ssio n fo r th e p re d ic te d e m b e d d e d p o in t b t+ l is

b t+ l

=

[q T k b t-l+ l(l) b t-1 + 1 (1 )

b

t-1

+

1

(m - 1) ] .

(3 .1 1 )

N o te th a t th e

1

th ro u g h m -th e n trie s in b t+ l (3 .1 1 ) a re p re v io u sly e m b e d d e d p o in ts in B .

T h e ta rg e t p o in t th e n b e c o m e s th e p re d ic tio n p o in t in th e n e x t p re d ic tio n a n d th e a lg o rith m p ro c e e d s in th e

(35)

(36)

Chapter

4 Prediction with Time Delayed

Neural Networks

4.1 An Introduction to Neural Networks

A n e u ra l n e tw o rk [5 ] [2 2 ] [2 5 ] [2 8 ] is a s y s te m o f d a ta s to ra g e p o in ts , c a lle d n o d e s . T h e s e n o d e s a re s e t u p

in s u c h a w a y th a t th e v a lu e s in s o m e o f th e n o d e s d e te rm in e th e v a lu e s in o th e r n o d e s .

If

th e v a lu e in o n e

n o d e d e p e n d s o n th e v a lu e in a n o th e r n o d e th e n th e s e n o d e s a re s a id to b e

connected.

A w e ig h t is a s s o c ia te d

w ith e a c h c o n n e c tio n th a t d e te rm in e s to w h ic h d e g re e th e v a lu e in th e n e x t n o d e is in flu e n c e d b y th e v a lu e

o f th e p re v io u s n o d e .

A n e u ra l n e tw o rk a lw a y s h a s a s e t o f in p u t n o d e s a n d a s e t o f o u tp u t n o d e s . It a ls o h a s s o m e in te rm e d ia te

n o d e s c a lle d

hidden

_{n o d e s .} _{T h e s im p le s t} _{d e s c rip tio n} _{o f th e fu n c tio n in g} _{o f a n e u ra l} _{n e tw o rk} _{(N N ) is th a t} _{it is}

a n o n lin e a r o p e ra to r th a t m a p s a v e c to r, d im e n s io n

n,

in th e in p u t s p a c e o n to a v e c to r, d im e n s io n

m,

in th e o u tp u t s p a c e , w h e re th e re a re

n

in p u t n o d e s a n d

m

o u tp u t n o d e s .

W h e th e r th e N N is u s e d fo r c la s s ific a tio n o r fo r c o m p u ta tio n a l p u rp o s e s , th e u s e r o f th e N N s h o u ld h a v e a

te s t s p a c e o f in p u t v e c to rs w ith th e ir a s s o c ia te d d e s ire d o u tp u t v e c to rs . T h e te s t s p a c e , o r tra in in g s e t, is

th e n u s e d to tra in th e n e tw o rk in th e fo llo w in g w a y : fo r e a c h in p u t a p p lie d to th e n e tw o rk th e w e ig h ts a re

a d ju s te d u n til th e e rro r b e tw e e n th e a c tu a l o u tp u t a n d th e c o rre c t o u tp u t is m in im is e d . O n c e a s a tis fa c to rily

lo w e rro r is o b ta in e d th e n e tw o rk is c o n s id e re d

trained

a n d th e n e tw o rk c a n th e n b e u s e d to o u tp u t v a lu e s fo r

in p u t v e c to rs w h ic h h a v e n o t p re v io u s ly b e e n p re s e n te d to th e m o d e l.

T h e fu n c tio n in g o f a n e u ra l n e tw o rk is a s fo llo w s . A n in p u t v e c to r in tro d u c e d to a n e tw o rk is s to re d in th e

in p u t la y e r, (s e e fig u re 4 .1 ). T h e n o d e s o f th e in p u t la y e r a re c o n n e c te d to e v e ry n o d e in th e n e x t la y e r, th e

firs t h id d e n la y e r. If a ll p o s s ib le c o n n e c tio n s a re m a d e b e tw e e n th e n o d e s in tw o a d ja c e n t la y e rs th e n th e

la y e rs a re c o n s id e re d fu lly c o n n e c te d (in g ra p h th e o ry th is c o n fig u ra tio n is re fe rre d to a s a c o m p le te b ip a rtite