The derivation of the optimal Karhunen-Loève coordinate functions

(1)

Tilburg University

The derivation of the optimal Karhunen-Loève coordinate functions

Stobberingh, R.

Publication date:

1971

Document Version

Publisher's PDF, also known as Version of record

Link to publication in Tilburg University Research Portal

Citation for published version (APA):

Stobberingh, R. (1971). The derivation of the optimal Karhunen-Loève coordinate functions. (EIT Research

Memorandum). Stichting Economisch Instituut Tilburg.

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners

and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.

• You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal

Take down policy

If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately

and investigate your claim.

(2)

~

7626 '

is~~ -1T

24

24 ! lj ..d~i;~:-~:...ii- : ~i~b~.ltii.: -U

~?e~.~mming

Ri~.d~'.

..T~ f:..~,

~

n:,`. ~'::C'~Lt~.~-~~:1;

xuaL~~-h~:~a1~

TIL~?ï3RC3

R. Stobberingh

The derivation of the optimal

Karhunen-Loève

coordinate functions

IInl~lIIIIIIIIVIIIIIIIIIIluiu~lN~~II~II

Research memorandum

TILBURG INSTITUTE OF ECONOMICS

(3)

(4)

4 li.t:,.;

_É

Sil~ . l~S

:

;. Introducticn.

The Karhunen - Loève expansíon, over the interval [a,b], of a random proces f(t) is an expansion in the form

f(t) - ~ xi Y'i (t) i

where the set of orthonormal functions {Y'.(t)} is chosen i

such that the coefficients x. are uncorrelated. i

These functions {Y'.(t)}_i are specífied by the solution of a homogeneous íntegral equation.

b

~ Kf(t,T)~i(T)dT - ai~i(t)

a ~ t ~ b.

(])

where the kernel Kf(t,T) is the covariance of the expanded random proces, Y'i(t) is an eígenvalue solution and ai íts associated eigenvalue.

The magnitude x.,_i or more conveniently the squared magnitude

~xi~2 of the component along the ith axis `Yi(t) of the

coor-dinate system {Y'.(t)}

_i

in the expansion of

the function f(t)

may be considered as a good measure of the usefulness or im-portance of the ith coordinate axis 4'. in representing the

i given function.

Those coordinate axes along which f(t) has small components

can be ignored without altering f(t) appreciably.

Since we have f(t) with relative frequency p(a) in the

semble, the average importance of the axis Y'.(t) in the

en-i

semble is given by

(a)

(a) 2

Pi - ~ p

Ixi

I

(5)

From the point of vieuw of information compression it is de-sirable that the p's be concentrated on a few axes instead of being widely spread over many axes.

To formulate this idea mathematically it would be convenient to introduce the entropyfunction in terms of the p's,

S({`~i})

- - ~

pi log

pi

.

i

The desirable choice of the coordinate system {Y'.(t)} will then 1

be characterized as one that minimizes S.

It is to be proved that a coordinate system ís optimal in this sense if and only if it is a Karhunen - Loève coordinate sys-t em.

(6)

2. Canonical expansions of random functions over a discrete series of poínts.

A canonical expansion of a random function f(t) is defined as any representation in the form,

f(t) - mf(t) t ~ xiY'i(t),

i

where: mf(t) is the mathematical expectation of f(t), xi are uncorrelated random variables, and Y'i(t) non-random, coordinate functions.

(2)

Expressing the random function f(t) by the canonical expan-sion (2), we can calculate its covariance function,

Kf(t,t') - ~ Di4'i(t)~i(t')~

i

(3)

where D._i are the variances of the x.. i

Putting t- t' we ge[ the variance of the random function f(t):

Kf(t.t) - ~ Di ~`~i(t)~Z i

(4)

The canonical

expansion

(3) of the covariance

function of

f(t) follows from the canonical expansion (2) for the random

(7)

Let x. be uncorrelated random variables, wíth zero expected i

value and variances D., i

E[ x.] - 0; E[ x. x.] - 0

i i ~

2]

Instead of (2), we now consider

fo(t) - f(t) - mf(t) - ~ xi~i(t) i

From (6) we get:

E[fo(t)x.] -~ Y'i(t) E[xi xj ]- DjWj(t), J

i

(6)

and the required, unknown coordinate functions `Yj(t) can be

found:

4'j (t) - ~ E [ fo(t) xj D.

J

This formula shows us that fqr the random function fo(t) to be expressible in terms of the canonical expansion (6), it is necessary to take the coefficients x. all correlated with the

i random function fo(t).

There fore we take a linear combination of values of fo(t) corresponding to different values of the argument t,

(i ~ j) (5)

(8)

where the aih are as yet quite arbitrary coefficients Substituting this expression in (7), we get

~(t)-D ~ a~hE [fo(t)fo(th)]-~ ~ a.hKf(t,th). (9)

~ D. ~

h ~ h

From (8}, the variances and covariances of the random varia-bles x. can be calculated,

i

E I xix~] -~ aiha~k E[ fo(th)fo(tk)] -h,k - ~ aiha.kKf(th,tk). J h,k Di - E Í~xi~Z] - L _{aihaikKf(th,tk).} h,k (10)

According to (S), the coefficients aih have to satisfy the conditions

L aihajkKf(th'tk) - 0 if i~ j (11)

h,k

Using (9) another expression can be obtained for the covarian-ces of x. ,_i

E [xix~ ] - ~ aiha.kKf(th,tk)

J h,k

-~ aih ~ a~kKf(th,tk) - D~ L _aih~j(th),

(9)

and cosequently, because of (10)

L aih ~j (th) - dij h

(12)

The number of equations (11) is always smaller than the num-ber of coefficients aih; there fore these coeffícients can be chosen in an infinite number of ways.

We can determine the uncorrelated random variables x. in i succession from (8) in the following manner.

First, let the xl be determíned, prescribing arbitrarily all the coefficients alh. Then let the coefficients a2h be deter-mined in such a way that the x2 is uncorrelated with xl; this condition is the only one to be satisfied by the coefficients aZh.

There fore all the coeffícients, with the exception of one, can be prescribed arbitrarily.

As a general rule the coefficients anh have to be determined ín such a way that xn is uncorrelated with x1,x2,...xn-1. This allows all the coefficients anh to be specified

arbitra-rily, except (n-1) of them. For instance we can put:

a. - 1 and a. - 0 if h~ i

ii ih

(13)

Then, by substituting i-1 ín,formulae (8), (9) and ( 10), we

get:

xl - fo(tl); D1 - E~~x1~21 - Kf(tl.tl)

~1(t)- D Kf(t,tl). (14)

(10)

For j ~ i(i-2,3,4,...) the conditions (12) become

i-1

L aih~jCth)}~j(ti) - 0 (j-1,2,3,...,(i-])). (]5) h-1

The sequence of operations for each given i will thus have to be performed as follows:

First one finds the coefficients

ail,ai2,...ai ~i-1 by solving the set of equations (]S);

then one determines x., D. and 4'.(t) from;

i i i i-I xi - ~ aihfo(th) t fo(ti) h-1 i-1 Di - _{E ~~xi~21} - _L _{aihaiQKf(th'tQ)tKfCti,ti)} h,Q-1 i- 1 } L ~aihKf(th'ti) } aihKf(ti.th) ~ h-1 i-1 i-1

~i(t)~D L aihKf(t.th)-D L aihKf(t.th)}Kf(t.ti))

1h-1 1 h-1

Remarks.

l. If we define the functions ~Y.(t)_i by formula (7), then for any given number of terms in the series (6) and for the given selection of randomcoefficients x.,_i this gives the best approximation to the random function fo(t) in the sen-se that among all possible series (6) having the same

num-ber of terms, this minimizes the mathematical expectation of the square of the modulus of the residual term for any t. Let {~.(t)} be a set of arbitrary functions;

i

n

0

set: f ( t) -~ xi~i(t) t Rn(t), then it is to be

(11)

rived that the square of the modulus of the residual term,

n

E I IRn(t)I2~ - E( Ifo(t) - ~ xi~i(t)~Z) i- 1 n

- E I ~fa(t)~~l -

~

~i~`~i(t)~~

i-1 n 2 t~ D.

_i

~~.(t)-

_i

4'.(t)

_~

~ i-1

From this we can easily see that for a given n, and for

~

all t, the expected value E(IR (t)I~( will be a minimum if: n

~P. (t) - 'U. (t)

_i

_~

(i-1, 2, 3,... n)

2. If the coefficients xi and the coordinate functions '~i(t) are determined from formulae (8) and (9), then the expan-sion of the random function fo(t) in the form (6) repre-sents this function exactly at all points t- th.

3. By selecting the linear combinations of the random func-0

tion f(t) at a discrete series of points as the coeffi-cients x. of the canonical expansion, we get an accurate

i

canonical expansion of the random function over this dis-crete series.

The canonical expansion will be a finite sum if the num-ber of points th is finite and it will be a convergent series in the mean square if the points th form an infi-nite set.

Since the points th can be chosen arbitrarily close to one another, (6) can be made arbitrarily accurate for

(12)

3. Canonical expansions of a random function in a prescribed domain of variation of the argument.

Now we take for the coefficients x._i of the canonical expan-sion of the random function fo(t), the integrals

b .1

a

a.(t) fo(t) dt

i

(16)

The functions a.(t)_i have to meet only one restriction, follo-wing from the condition that the variables x. must be

uncorre-i lated.

To find the equation which has to be satisfied by the func-tions a.(t),_i we calculate the covariance of the random va-riables x._i x.,_~ determined by (16):

E [ x. x. ~

-i ~

,b b

ai(t) aj(t')Kf(t,t')dt'-0 (i~j)(17)

Putting i~j we get the formula for the variance of x. i

b b

Di - E [Ixil2) - ( ~ ai(t)ai(t')Kf(t,t')dt dt'(18) a~ a

To determine _{the coordinate functions 4'.(t) we substitute} i

(16) into formula (7):

`~i(t)-~ E [ fo(t)xi~ -~ E [ fo(t)

(13)

Substituting this formula in (17), we get: ,b a a ai(t)a.(t')Kf(t,t')dt dt' -J

b

(

a~(t')

[

(

ai(t)Kf(t,t')dt ) dt b - D. I a.(t') ~.(t') dt' (20) i I ~ i aJ

Formulae (17), (18), and (20) yield the following relation-ships between the functions a~(t) and ~i(t):

b

a.(t) Y'i(t) dt - c~i~ J

These relationships, necessary and sufficíent for the random variables to be uncorrelated are called the bi-orthogonality conditions of the functions a.(t) and Y'.(t).

J 1

Bearing in mind the symmetry of the covariance functíon, we get from (17) the following identity

(14)

and from a comparison of (22) with (20), ,b D. J aJ

b

ai(t)

Y'~ (t)dt

- Di

a~ (t)

4'i(t)dt

(23)

a

Hence it can be seen that if the function a.(t) is orthogo-i

nal to the function 4'.(t), then the function a.(t) is

ortho-J J

gonal to ~Y. (t). i

Any set of pairs of functions a.(t) and `Y.(t) which

satis-i ~

fies equations (19) and the conditions of bi-orthogonality (21), gives in prínciple a canonical expansion of the ran-dom function fo(t).

Let us take an arbitrary system of functions f~(t), f2(t), f3(t),... and put:

a~(t)

- f~(t).

The variance D~ and the coordinate function `Y~(t) can be determined according to formulae(18) and (19).

Moreover, we write now

a2(t) - c21 a~(t) t f2(t).

(15)

a b b b a2(t)'i'1 (t)dt-c21 al (t)4'1 (t)dtt f2(t)'Y1 (t)dt

a

b -c21} f2(t)`Y1(t)dt - 0 ,b so: c21-

-a

f2(t)Y'1 (t)dt

The variance D2 and the function 4'2(t) can be determined according to (18) and (19).

When the functions a.(t), `Y.(t), j-1,2,...(n-1), are

deter-J J

mined satisfying the equations (]9) and ( 21), and the varían-ces D1, D~,...,Dn-1 by formula ( 18), then we put

n-1

an(t) - ~ cnjaj(t) t fn(t) j-1

and define the coefficients c in such a way that the func-nj

(16)

This process, allowing us to determine ín succession the functions a.(t) a'nd ~Y.(t) and the variances D. can be

con-J J J

tinued indefinitely.

Depending on the choice of the system of functions f~(t), f2(t),.... we get different systems of pairs of functions. If we multiply the functions a.(t) by an arbítrary positive

J

number and divide the function Y`. (t) by the same number, J

then the variance D. is multiplied by the square of the J

same number, and the conditions (19) and (21) are not dis-turbed.

We can make use of these circumstances to make the variances D. equal to any positive number.

J Remark.

The completeness of the system of functions a.(t) is J

necessary and sufficient for the convergence in mean square of the canonical expansion (6) and the random function fo(t).

(17)

4. The Karhunen-Loève expansion as a special type of the canonical expans ion.

When we consider our problem, mentioned ín the introduction of this paper, to give a solution of the homogeneous inte-gral equation

b

Kf(t,T)`Yi(T)dT - ~i4'i(t) ~

where the set of functions {`Y.(t)} is orthonormal

i ,b

`Yi(t)Y~~ ( t)dt

- di~ ,

then we can see immediately that this problem is completely analogous to the problem of determining the functions a~(t) and Y'.(t) sati.sfying ( ]9) and (21),

]

where: 'f~ (t) - a~ (t)

a. - D.

J J

S. Conclusion.

It is a rather laborious task to determine the optimal coordinate functions `Y.(t), the eigenfunctionsolutions of

J

the integralequation (1).

The calculation of these optimal coordinate functions by means of the method of the canonical expansions of .random

(18)

State Variables and Communication Theory Research Monograph No. 61 The M.I.T. Press Cambridge, Massachusetts, and London, England.