Maximizing the growth of the utility vector in a dynamic programming model

Maximizing the growth of the utility vector in a dynamic

programming model

Citation for published version (APA):

Zijm, W. H. M. (1979). Maximizing the growth of the utility vector in a dynamic programming model. (Memorandum COSOR; Vol. 7911). Technische Hogeschool Eindhoven.

Document status and date: Published: 01/01/1979

Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)

Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:

www.tue.nl/taverne Take down policy

If you believe that this document breaches copyright please contact us at: openaccess@tue.nl

providing details and we will investigate your claim.

EINDHOVEN UNIVERSITY OF TECHNOLOGY

Department of Mathematics

PROBABILITY THEORY, STATISTICS AND OPERATIONS RESEARCH GROUP

Memorandum COSOR 79-11

Maximizing the growth of the utility vector

in a dynamic programming model

by

W.H.M. Zijm

Eindhoven, November 1979

DYNAMIC PROGRAMMING MODEL

by

W.H.M. Zijm

O. Abstract

In this paper we extend results of Sladky ([6J,[7J) for dynamic pro-gramming models. Consider a set of matrices M, which is generated by all possible interchanges of corresponding rows, taken from a fixed finite set of nonnegative square matrices. Let x(O) be a strictly positive (column)-vector and denote by x(n), n

=

1,2, .•• the utility vector at time point n, which obeys the following dynamic programming recursion

x(n + 1)

=

max P x(n)

PEM

n

=

1,2, ••••

In this paper the first order asymptot·ic. behaviour of thelilequence { x(n) ; n

=

0,1,2, •••• } is investigated. As a by-product of our analysis we get estimation procedures for max (a. (P); i

=

1, •..,s where s E ~

PEM ~

is fixed and a. (P) denotes the spectral radius of a specific sub-matrix

~

of P, for i

=

1, •••,s. In particular a

1(p) is equal to the spectral radius

2

-1. Introduction

Consider a set Mof matrices, which is generated by all possible inter-changes of corresponding rows, taken from a fixed finite set of nonnegative N x N - matrices. Motivated by the theory of Markov decision chains we refer to the indices 1, .•• ,N as states and we call I

=

{l, ••• ,N} the state space. Let at discrete time points n

=

0,1,2, ••• , the utility vector at time n, denoted by x(n) (a column N-vector), obey the following dynamic programming recursion

(1) x(n + 1)

=

max P x(n)

PEM

n

=

0,1,2, ••••

*

where x(O) is a fixed, strictly positive vector. From the structure of M it is obvious that we may take the maximum component-wise in (1).

In two papers ([6],G7J) Sladky investigated the asymptotic properties of the recursion (1) under some additional assumptions. In [6J he considered the case in which all matrices are

irreducible~

in [7J the matrices were al-lowed to be reducible, however, i t was assumed that every matrix possessed exactly one basic class (see for definitions below). Further~re, in both

papers the matrices had to be aperiodic, although, as Sladky already mentioned, this last assumption is not essential. In the follOWing we will drop almost all assumptions, except for the aperiodicity. We will need some material from the theory of nonnegative matrices, for which we refer to Sladky [6],[7J , Seneta [5J, Gantmacher [lJ, Zijm [9J and especially to a number of excellent papers by Rothblum ([2J,[3J,[4J).

For interpretations of the dynamic programming recursion we refer to Sladky [7J • In the following we will first give some definitions and nota-tional conventions. After summarizing some results concerning nonnegative ma-trices, we will give a short overview of the different steps, leading to a more or less complete analysis of the first order asymptotic behaviour of the dynamic programming recursion (1) (section 2). This analysis will be based on characterizations of nonnegative matrices in terms of its spectral radius and its index (Rothblum-[2J) and a decomposition result for sets of nonnega-tive matrices (Zijm [9J). In section 3 we establish the polynomially

ness of the sequence {x(n) ; n

=

0,1,2, .••• }. In section 4 we treat the ease in which the matrices with maximal spectral radius have index equal to one. Using the results of section 4 as a first step, we subsequently prove, in section 5, our main result by induction. This main result states that

lim

(n

+ vi(P) -n-+oo\v.(.P) - 1 ~ . -1

1)

- -n (0. (P» x. (n) ~ ~

~ists and is strictly positive. Here

P

is a particular matrix ~fM_with maximal spectral radius; 0. (P), resp. v. (P), denote the spectral radius, resp.

~ ~

the index, of certain submatrices of

P,

while x.(n) is the restriction of

~

x(n) to a certain subset of states (definitions will be given in section 2).

As a by-product of our analysis we get estimation procedures for max 0. (P);

PEM ~

denotes the spectral radius of i

=

1, •••,s, where s E IN is fixed and o. (P)

~

a specific subma~ixofP, for i

=

1, ••• ,s (section 6). In particular 0

1(P) is equal to the spectral radius of P.

2. -p-r-elimiuaries

We shall be working in the Euclidian space JRN • Matrices resp. (column)-vectors, are denoted by upper,resp. lower, case letters. We say that matrix A is nonnegative (positive) - denoted by A ~ 0 (A »0) - if all its coordinates are nonnegative (positive). We say that A is semi-positive - denoted by A > 0 -if A ~ 0 and A

'I-

O. We write A ~ B (A »B, A > B) if A - B ~ 0 (A - B »0, A - B >,0). Similar definitions apply to vectors.

For matrix A,[AJ. denotes the i-th. row of A,[AJ .. its ij-th element •

.- ~ - ~J

For a vector b,[bJ. again denotes its i-th element.

~

M is defined as a set of matrices,~whichis generated by all possible interchanges of corresponding rows, taken from a fixed finite set of non-negative N x N - matrices. In other words: if V is an arbitrary subset of

{l, •.••,N} , then P

1,P2 E M implies that P, defined by [pJi

=

[P1Ji ' for i E V, [pJ.

=

[P

2J. , for i

tv,

is also an element of M.

~ ~

We nekt summarize some results about nonnegative matrices. Let o(P) be the spectral radius of P. According to the well-lcnown Perron-Frobenius theo-rem o(P) equals the largest positive eigenvalue of P and we can choose the corresponding eigenvector ~(P) > O. Recall that if P is irreducible then

~ 4

-even ~(P) »0 and cr(P} is a simple eigenvalue (see e.g. Gantmacher [lJ or Seneta [5J). If P is reducible then by suitably permuting rows and corre-sponding columns Pmay be written in the following form

(2) P

=

P₂₂ --- P ... ,2r

"

" "

_"

"

I I ...

,

'P rr 1,.!...,p from p .. } if there ~~ i such that P k_{j-1' j}k > 0 for all j

=

=

Q;~•..,p} is called a chain. The sequence

fP

_k k -; j j ' j

We say that p .. is basic, resp. non-basic if cr. (P)

=

cr(P), resp.

~~ ~

where each p .. itself is an irreducible matrix with spectral radius cr. (P).

~~ ~

Motivated by the theory of Markov chains we say that p .. has access to

~~

exists a sequence of integers

P

i i (or Pi i is accessible

k

O

=

i < k1 < •••• < kp

=

a.

(P) < cr(P}. From Gantmacher [lJ it follows that ~(P) »0 if and only if

~

each non-basic class of P has access to some basic class and no basic class is accessible to any other irreducible class of P. In virtue of this fact diagonal blocks of (2) need not be the 'iarges~1 blocks having strictly po-sitive eigenvectors.

resp. final, if there does not exist a P .. , j

f

i,

JJ

accessible from, resp. to, p ..• Every square nonnegative

JJ

least one initial and one final irreducible class. We call P .. initial, ~~ is such that p .. ~~ matrix has at

The length of a chain is the number of basic classes i t contains. We say that p .. has access to P .. in n steps if the length of the longest chain

~~ JJ

from P .. to P .. is n. The height, resp. depth, of a basic class is the length

~~ JJ

of the longest chain of irreducible classes in which that class is final, resp. initial. Finally the index v(P} of P is defined as the length of its longest chain of irreducible classes.*

Having these concepts, we now are able to formulate two important de-composition. results about nonnegative matrices, which will prove to be essen-tial for the rest of the paper. It holds :

*

We remark that the given definition of v(P} is a rather unusual one. For non-negative matrices however, it can be shown that this definition is completely equivalent with the traditional one (compare Rothblum [2J).

Lemma 1 : By possibly permuting rows and corresponding columns of (2), we may write PH P 22 --- f2s (3) _P

₌

"

I I

..

,

... " I P ss where for i

=

1,2, •••• ,s (s depends on P)

(4) p .. ]..I~ (P)

=

cr. (P) ]..I. (P)

~~... ~ ~

with cr. (P) and ]..I. (P) »0 being the spectral radius and a corresponding right

~ ~

eigenvector of p .. (in general reducible) and

~~

(5) ~ cr (P)

s

Furthermore, cr. (P)

=

cr. l(P} implies:

~ ~+

Every irreducible class of p .. with spectral radius equal to cr. (P) has

~~ ~

access to some irreducible class of P. 1 '+1 with spectral radius equal to

~+ ,~

cr. (P) (= cr. 1(P) } •

~ ~+

Remark: A proof of lemma 1 can be found in Zijm [9J. Notice that for a matrix

for i

=

1, ••••,j. Analogous results radius.

P with cr(P}

=

cr

1(P)

=

the lemma implies v(P}

=

j,

=

cr.(P) J

since a

> cr. l(P} ; j < s the final statement of

J+

basic class of p .. has depth (j - i + 1),

~~

hold for the blocks with smaller spectral

To avoid trivialities we make the following assumption throughout the rest of this paper

Assumption: cr (P) > 0 for s

=

s(P} , P E M. s

An irreducible class of P .. will be called basic if its spectral radius

~~

equals cr. (P), else it is non-basic. Note that the concept of a basic class

~

is defined with respect to a particular matrix or submat:r:f"x,.hence a basic class of p .. need not be a basic class of P. Furthermore, define

~~

the set of states j, such that [pJ .. is an element of P .. , hence

JJ ~~ I. (P) as ~ ·s I

=

;Vi' i=4' I. (P) ~

6

-and Ii (P) n I

k(P)

=

tj> for i F k.

A nice property of the decomposition in lemma 1 is that it can be extended in a certain sense to the whole set of matrices M.

We have :

Lemma 2: There exists a matrix P E M such that (possibly after permuting rows and corresponding columns) •

(6 ) P

-

=

" "

,

, I

.-P ss

where for i

₌

1,2, •••• ,s

₌

s(P) and any P E M

(71 p .. ll. (P) ~ P .. ]1-, (P)

₌

o. (P) ll. (P) ll. (p) »

o.

J.J. J. J.J. J. J. J. J.

(here P .. contains the states of I. (P) ) , while

J.J. J.

(8 )

and

~ 0 (P)

s

(9) P_ik - 0 (matrix) for any P EM, k < i.

Furthermore 0i(P)

=

0i+1 (P) implies that every basic class of P_ii has access to some basic class of

P

_i+₁,i+1 •

Remark: The proof of lemma 2 again can be found inZijm [9J. Sladky [7J gives a version of lemma 2 under very special conditions. The lemma, espe~

cially (7), gives us some kind of "boundedness condition", which is uniform over the whole set of matrices M.

The last lermna of the section summarizes some well"';known. facts;-'~about

Lemma 3

a) o(P) is a continuous function of the elements of P. b) If P

1 > P2 then 0(P1) ~ 0(P2) with a sharp inequality if P1 is irre-ducible. c) If we define, for i =l, ....,s = s(P), 1 p .. ~~ then lim n"''''' and

-1

=

(cr. (P» P .. ~ ~~

n-1 ( )m

L

\p

~ ~

=:

n m=O ... P ..

*

~~ > 0 always exists

*

*

*

p.. p..

=

p .. p..

=

o. (P) P ..

~~ ~~ ~~ ~~ ~ ~~

Moreover, if P .. is aperiodic, then

~~

For a proof we refer to Seneta [5J, Gantmacher [1J and Sladky [7J.

We are now ready to formulate explicitly the main goal of this paper. It will be proved that for {x(n) ; n

=

0,1,2, •••• }, defined by (1), the following holds:

(10) lim

(n

+ vi(P) -

1)~1

(0. (P) )-n x. (n)

n-+so\ ..;: (P) - 1 ~ ~

~

o

exists and is strictly positive, for i

=

1, .••• ,s

=

s(P).

Here P is the matrix defined by lemma 2 (not necessarily unique); x. (n)

de-~

notes the restriction of the vector x(n) to I. (P). Finally v. (P) is defined

~ ~

8

-P.

~s

P..

P . . 1 ~~ ~,~+

P.

_{~+ ,~+}1 . 1---

P.

_.~+1_,s .... ....

.

"o-P ss

with respect to cr. (P). Note that from lemma 2 it follows that v. (P)

=

k

~ ~

implies: cri(P)

=

cr_i+

1(P)

=

=

cri+k-1(P) > cri+k(P). Hence a basic class of P .. has depth v.(P) with respect to

Q..•

~~ ~ ~~

Remark that the limit in (10) is strictly positive, hence (10) gives a precise description of the first order asymptotic behaviour of x(n). In order to prove (10) we proceed in the most natural way, i.e. we first try to establish the boundedness of

( . n. + v.(P) -

1)

~ (cr i

(P»

-n x. (n)

v:

(P) - 1 ~ ~ i

=

1, •••.,s

=

s(P)

and it is here, where we can show the power of lemma 2. After that we show that (10) holds for those indices i with

v.

(P)

=

1, by proving that there

~

cannot exist two different limit points.

Finally, by induction, we complete the proof of (10) for all indices. As a by-product of our analysis we find good estimates for cr. (P) i i

=

1, •••• ,s(P).

~

3. Boundedness of the dynamic programming recursion

The main goal of this section will be to prove the following

Lemma 4 : Let P be defined as in lemma 2 and let, for i

=

1, .•••,s(P), x. (n)

~

be the subvector of x(n), whose components belong to I. (P). Then, for

suit-~

ably chosen ~. (P), we have

~ (11) ( n + v. (P) -

1)

~ - n x. (n) ~ (cr. (P» ~. (P) ~ . v. (P) - 1 ~ ~ ~

where (cr. (P), ~.(P» satisfy (7), for i

=

1, •••• ,s(P).

Proof By induction. For i

=

s

~ (P) 2: x (0) •

s s

=

s(P) we may choose ~ (P) such that

s

Then, by lemma 2

x (n)_s

=

max P_ss x (n - 1)_s

=

P_ss(n) x (n - 1), and by iteration_s

P

x (n)

=

P (n) •••• P (1) x (0) ::;

s ss ss s

::; P (n) ••.• P (1) ~ (P) ::; (0 (p»n ~ (P)

ss ss s s s

Since v (P)

=

1, (11) is proved for i

=

s. s

Suppose now, (11) is fulfilled, for certain ~. (P), for j

=

i + 1, •••• ,s(P)

J

and let v. (P)

=

k.

';c-1. Hence :

(12)

Because 0f (2) and the fact that

lim

n+ 00

Q,

=

1,2, ••••

it is possible to find constants C. (j

=

i + k, •••• ,s) such that for alln

J we have

(n

+ Vj(P) \ v.(P) -J Now define

-1 1)

(o.(p»n ~.(P) ::; J J ... n • (cr. (P» C.~.(P) 1. J J j

=

i + k, •... ,s • x~ (n) = x. (n) , ~~ (P) = ~. (P) J J J J j

=

i + 1 , ••••, i + k - 2

x_i +_{k - 1}(n) xi+k(n) x (n) s - 10 -111+k-l(P)

=

]li+k-1 (P)

c.

_J.+k ]l. k(P)_J.+ C ]l'CP) s s Then we may rewrite our induction hypothesis, as follows:

Now choose ]li CP) »0 in such a way that

and

(p._J.,J.'+k ·1'····'P._{- J.,s}) ]lJ.~+k-1(P) ::; o.(P)]l.(P), for all P_{J. J.} where

x. (0)

J.

and ]li(P) satisfies (7).

Then we find by iteration

x.(n) ::; k t ( n - 1) +(k -,Q,) -1\j(0.(p»n ]l.(P)

=

J. ,Q,=O (k - ,Q,) - 1 J. J.

=

(n

+

k- 1)

k - 1 '- n (0 ,·(P) ) ]l. (P) J. J. n

=

1,2, ••••

o

Remark : In the case that M contains precisely one matrix, the lemma follows almost immediately from theorem 3.2 in Rothblum [4J. Sladky [7J proves a version of lemma 4, again under special restrictions.

Once having lemma 4, the most natural way to establish (10) will be to prove that the sequence {~.(n) ; n

=

0,1,2, •••• } , defined by

z. (n)

~

a.

(:P)

-n x. (n)

~ ~ n = 0,1,2, ••••

cannot have more than one limit point. In the case that

v.

(P)

=

1 this

in-~

deed will be the way we proceed. Unfortunately difficulties arise when

v. (P) > 1. In order to overcome these difficulties we will use a

relation-~

ship with another dynamic programming recursion, defined in section 5, and then prove (10) by induction to V. (P) again. Technical details can be found

~

in the appendix; we will now first treat the case v. (P)

=

1.

~

4. Convergence results under special conditions

The main result of this section will be the proof of (10), restricted to those indices i with v. (P)

=

1. The proof will follow almost the same

~

lines as that of theorem 4.2 in Sladky,[7J, although he also assumes that every p .. possesses exactly one basic class. In theorem 5 we only make some

~~

aperiodicity assumption. Before formulating the theorem, we first need one technical lemma:

Lemma 5 : Let {k(n} n

=

0,1,2, ••• } be bounded and for any ,n ~ nO x(n + 1} ~ P x(n}

wher~ P is an irreducible aperiodic, nonnegative matrix with spectral radius equal to one. Then

lim x(n}

=

x

=

Px

n-+co

Furthermore x (n) > 0 for some n ~ nO implies x »0

o

For a proof we refer to Sladky [7J.

We are now ready to formulate the main result of this section. It holds:

Theorem 5 Then

Let P.. be aperiodic, for fixed i and any P E M, and let v. (P)

=

1

~~ ~

lim (a. (P) )-n x. (n)

~ ~

12

-For convenience define

X.

(n) = (a. (F» -n x. (n), then

i.

(n) »0

1. 1. 1. 1.

=

O,1,2, ... } is bounded. Furthermore, again by

and by lemma 4

{i.

(n) ; n 1.

lemma 4 and the fact that v. (F)

=

1 (hence d.(P) < a. (F) for j > i) we have

1. J 1. Proof : ( 13) lim n +00 (a. (p»-n x.(n) = 0 for j > i 1. J Define a = limsup x. (n) 1. n+oo (component-wise) b

=

liminf

X.

(n) 1. (component-wise) n+oo and suppose a > b ~ O.

Fix some j E li(F) and let {n k :IN = {O,1,2, •••• } such that

k

=

O,1,2, ••.• } be a subsequence of lim

[x.

(n k)

J.

= b. k+oo 1. J J and lim xi(n k - 1) =: Xi exists k +00 (obviously x. ~ b) 1.

Repeating this procedure for all j E I. (F) (with different subsequences and 1.

using (13) and the construction of M (interchangeable rows) we conclude that for some P E M, pit say

b ~ pit b

ii (here Pl~.1.1.

(a. (F) )-1 p1.1 • )

1. 1.1.

In the same way, for some P E M, pI say

- -1

= (cr. (P» P~.)

1. 1.1.

Furthermore we have, since for _eve_ry J' E I._1.(P) [pit],_J was found as an optimal row in some sequence of steps, that

b ~ pit b ~ Plb ii

Combining these results we find

(14) a - b ~ P~

.

(a - b) ~~

hence obviously a. (PI)

=

C1. CP), since a - b > O.

~ ~.

By lemma 4 we have

P~. ]l. (15) $; ]l~ (15) , where ]l~ (15) »0

~~ ~...

...

which implies'that every basic class of P~. is final with respect to any

~ ~ ­

chain in

Pif.__

H~nce, ~fter possibly permuting rows and corresponding columns, We

may

write'·

(15) P~ , ~~ =

o

o

where P(A.) ; i s 1, •••• ,k, correspond with the basic classes of P~ "

~ ~~

Let

x~(n)

(aj,bj) denote the part of

X.

(n) (a,b), corresponding with

~ ~

P(A.) ; j = O, •••• ,k. By (1) and (15) we get for any n

J

x~

(n + 1)

~ 2: peA,)J

x~(n)

~ j = 1, ••.. ,k

which implies by lemma 5 (recall that the spectral radius of P(A.) is equal

J

to one, for j

=

l, ..••,k) that

lim

x~(n)

~

n +00

exists and is strictly positive. Hence aj

=

bj , for j

=

1, •••• ,k, and (14) reduces to

a

a

-

a

a - b $; P(A

O) (a

aO >- bO would imply that P(A

";" 14

-the fact that P(A

O) possesses no basic class. Hence aO

=

bO and we finally conclude a

=

b.

We still have to prove a »0. From

X.

(n + 1)

~

(cr. (P»

-It.

x.

(n)

~ ~ ~~ ~ n

=

0,1,2, ••••

it follows that

(16) a ~ (cr. (P»

- -1

P .. a ~ P.;.;a

-*

~ ~~ ...

where the restriction of a to the basic classes of p .. is strictly positive

~~

(follows from lemma 5 again). Furthermore, P .. possesses a strictly positive

~~

eigenvector ~' (P), which implies that every non-basic class of P .. has access

~ ~~

to some basic class of P .. , hence by (16) we conclude a »0

0

~~

Once having theorem 5, we may complete the proof of (10) by induction. Some proofs of intermediate results may be found in the appendix.

5. First order aSymptotic behaviour: the general case

In this section we will establish our main result (10), under one additional assumption. To be specific, the following theorem will be proved, by induction, using theorem 5 asa first step. It holds

Theorem 6 Let for i

=

1, •••• ,s

=

s(P) P .. be aperiodic for any P E M. ~~ Then (10) lim n +00

- 1)-1

+ v.(P) -~ v. (P) - 1 ~ -n cr. (P) x. (n) ~ ~ exists, for i

=

1, ....,s .

Denoting this limit by Xi' we furthermore have

i

=

1, ....

,s.

Proof For i

=

s the result follows immediately from theorem 5, since

case v. (P)

=

1 has already been treated in the

pre-~

ceding section (again theorem 5), hence we may suppose

v.

(P)

=

k > 1.

~

v (P)

=

1. Now let i E_{l, •••• ,s(P) - 1} and suppose that the limit in (10) s

exists, for j

=

i + 1, •••• ,s

=

s(P). Denote this-limit by x. and suppose

J

furthermore x. »0, j ~= i.+ 1, •••• , s. We have to prove the result of theorem 6

J

For convenience we define (compare section 4) (17)

x.

(n) = (0. (P» -n x. (n) ~ ~ ~ i

=

1, •••.,s

=

s(P) p .. ~J _ -1 = (a.(P» P .. ~ ~J i,j = l , ..•• ,s=s(Ei) P E M (18) p. ~

-

-=

(P . ._~,~+l' . . . . ,P._{. ~s}) i = l , ..•• , s - l P E M (19)

y.

(n)

=

(0. (P) )-n ~ ~ x. 1(n) ~+ x_i+₂(n) x (n) s i = l , •..• s - l n = 0,1~2,•..

Then from recursion (l)_we_have, for i 1, .•••,s - 1

(20)

X.

(n,.+ 1)

=

~ max

PEM

{P..

x. (n) +

P.

y.

(n) }

~~ ~ ~ ~ n = 0,1,2, .•.•

By lemma 2 i t follows that vi (P) k > 1 implies v

i+1(P) = k - 1 and 0i+1(P) = 0i(P), hence by the induction hypothesis

. . . -1

. (n

+k ;..., 2.' l~m k - 2 ) n+oo and by lemma 4 : X_i+ 1(n)

=

xi +1 »0 hence ( + k - 2,-1 4 - n lim n k _ 2 ) (0 i (P) ) x. (n) = 0 n+oo _J for j > i + 1. (21 ) 1 ._~m

(n

_{k - 2}+ k -n+oo If we now define (22) v.(n + 1) ~

=

max_PEM

{p..

_~~ v. (n)_~ + with v. (0) = x. (0) =

x.

(0) ~ ~ ~

- 16

-then by lemma A in the appendix we find

(23) lim

(n

n+oo

+ k

-k - 1 v. (n) exists,~

and by lemma B :

In other words: the first order asymptotic behaviour of {x. (n); n

=

0,1, .•• }

~

is exactly the same as the first order asymptotic behaviour of

{v.

(n); n = 0,1, •.• }. From (20),(23) and (24) we may immediately conclude to

~

the existence of x., defined by

~

(25) _{xi -}_ _l~m. (n + k - 1\-1_{k _ 1 ) (a i (P))}- - -n _{xi (n) -}_ ._l~m (n + k - 1) -_{\ k _ 1 xi (n)}

n+oo _n+oo

What remains is the proof of x. »0. For n

~

(22) that

1,2, •••• it follows easily from

v. (n + 1) ~ ~ n

L

m=O ( - -1 - \m_ (n -m+k- 2) -

-1-(oi(.~)) _{Pii ) -} k - 2 (oi (P)) Pi,i+l _{xi +1}

Since, by the aperiodicity of P .. (compare lemma 3)

~~ ' ( _ _ -1

_)m

_* lim \(0. (P)) P..

=

P .. ~ ~~ ~~ 111+ 00 and

I

(n -m

+

k- 2)

m=O \ k - 2

..

(n

+

k- 1)

\ k - 1 while lim «n + 1) + k_{k - 1} - 1)/(n + k - 1\_{k - 1}

₎

=

1 n+oo

it is easy to see that

lim «n+ 1) + k - 1)-1_{k - 1} v. (n + 1) ~ (cr. (P))- -1 P ..

-*

P.- . 1 _{xi +1}

By (24) and (25) we conclude

Since v

i+1(P)

=

k - 1, hence 0i(P)

=

0i+1(P), i t follows that every basic

class of

P

_ii is accessible to some basic class of Pi+1,i+1~ which means that at least one of the corresponding rows of

P;-.

1 must have positive elements.

J.,J.+ .

Furthermore, the restriction of

P:

i to some basic class is a str~ctlypositive matrix, hence x. 1 ~

o

implies tllat at least the restriction of x. to the basic

J.+ J.

<)l:asses of

P..

is strictly positive. Now from J.J.

- 1

-x. (n + 1) ;:: (0. (P)) P .. x. (n)

J. J. .. J.J. J.

and since lim

n+oo 1) + k -k - 1 we conclude Oy- f25~ • x. ;:: (o.(p))-l p .. J. J. J.J. X.J. X. J.

Recalling that every non-basic class of P .. is accessible to some basic J.J.

class of

P..

(since ]1. (P) »0) we finally conclude

J.J. J.

x· ₁ » O - x . » O .

J.+ J.

The proof can now be completed by induction.

o

With theorem 6 we have established completely the first order asymp-totic behaviou~of the sequence {x(n) ; n

=

O,1,2 ••• }. The only assumption actually made is an aperiodicity assumption. Although considerably simpli-fying our analysis, this assumption is not essential. We state without proof

Let P, satisfying lemma 2, be irreducible and periodic (with period d> 1) Then there exists:

lim n+oo Compare Sladky [7J. 1 n+d-1 d

L

xUc) k=n

- 18

-In the final section we show that relation (10) enables us to estimate

a. (P), for i

=

1, ••.. ,s. We end with some discussion.

~

6. The spectral radius of P ii

In this final section we will show that

lim n -+00 [x(n + l)J. [x(n)

J.

]

=

_{a i} (P) , for j E IiCP) ] i

=

1, ....

,s.

This would suggest an estimation procedure for a. (P), i

=

1, •••• ,s. However

~

determination of upper and lower bounds for a. (P), i

=

1, ••••,5 cannot be

~

done in an easy way, as will be shown in the discussion after the proof of theorem 7.

Theorem 7 Define

Let for i

=

1,2, •••• ,5

=

s(P) p .. be aperiodic for each P E M. ~~ then p(n,j)

=

[x(n)]-1 [x(n + l)J. ] for j E I.~(P), i

=

1, ••••,5 lim p(n,j) :::: _{a. (P) for j E I. (P), i}

₌

_{1, .... , s} ~ ~ n -+00 Proof lim p (n, j)

₌

n-+oo [_n-+lim. {(en_{oo ·. .} + 1) + v. (P) ~ ~ VlP) - 1

+

v. (P) -~ v.(P) - 1 .~

o

Remarko: We don't give the natural extension of theorem 4.4 in Sladky [7J, simply because the results, which have been stated there, are not correct. In [7J i t is asserted that, defining

p~ (n)

₌

min p(n,j) ~ jEI. (P) ~ p'~(n)

₌

max p (n, j) ~ jEI. (P) ~

ex> ex>

{pl. (n)} 0 ' resp. {p~(n)} 0 is nondecreasing, resp. nonincreasing, and

~ n= ~ n=

pi (n) :::;; cr. (P) :::;; p'.'(n)

i ~ ~

(Here it is assumed that cr. (P) > cr

2(P) > •••• > cr (~) while furthermore

~ s

every P .. possesses exactly one basic class, for i

=

1, •••• ,s

=

s(P), P EM).

~~

However, a simple counterexample is provided by the following suppose we have only one matrix

P

=

2

o

o

o

2

o

1 1 1

and let x(O)

=

1 1 1 {3} and v 1(P) = v2(P) = 1 whereas

pi

(n) = Pl(n)

=

2n+2 - 1)

I

(2n+1 - 1) > 2 (in fact, formulas 4.4.2 in [7J are incorrect).

n = 1,2, •...

o

In this paper we have presented some natural extensions of work of

Sladky [7J. The most restrictive assumption in [7J is the fact that chains of irreducible classes with the same spectral radius are not allowed (hence

v. (P)

=

1, i

=

1, •••• ,s(P) P

~

actly one basic class, for i

=

M). Furthermore every P .. may contain

ex-~~

1, •.•• ,s(P), P E M. The proof of theorem 5

stems mainly from [7J. The higher order asymptotic behaviour of the dynamic programming recursion (1) will be treated in a forthcoming paper.

- 20

-Appendix

In this appendix we discuss some aspects of the behaviour of

{v, (n) i n

=

0,1,2, •••• } , defined by (22), and its relation to {x, (n) i

~ ~

n

=

0,1,2, ••• } defined by (20). First we have Lemma A : lim (n

~ ~ ~

1)

vi (n) exists, for k

~

2.

n-+oo Proof If we define [w,(n)J, ~ J j E I,CP),~ n

=

1,2, ••• j,k E I , CP) ~ P E M

then we may rewrite (22) as follows

j E I,(P)

~ P E M,

in which the matrices Q" are (sub)stochastic now. For this recursion the

~~ existence of . ( n +

k- 1)-1.

lim . k _ 1 wi(n) n-+oo

is proved in Van der Wal and Zijm [8J, for k ~ 2. The lemma then follows

immediately.

o

In the next lemma we show the equivalence between the first order asymp-totic of {x, (n) i n

=

0,l,2, ••• } and {v, (n) i n

=

0,1,2, •.. } . The

formula-~ ~

tion will be in more general terms.

Lemma B: Let Z be a finite set of square nonnegative matrices of the same dimension with the interchangeability property, mentioned in the introduction. Let there exist a vector f.l »0 such that

(a) max P].l

=

].l PEZ

and let the sequences {x(n) ; n

=

0,1,2, •.• } and {v(n) be defined recursively by x(O) »0 and

n

=

0,1,2, ••• } n = 0,1,2, ••• ,k ~ 1,fixed. x(n + 1)

=

max {Px(n) + r(p,n)} ; n

=

0,1,2, ••. PEZ v(n + 1)

=

max {Pv(n) + (n

~ ~ ~

l)r(p)} PEZ

where reP), r(P,n), n

=

0,1,2, •..• are vectors with

Then

(b) lim

n-+oo (

n + k - 1)-1

k - 1 . r(P,n)

=

reP) for all P E Z.

lim

n-+oo (

+ k)-l

n k . (x(n) - v(n» O.

Proof x(n + 1)

=

max {Px(n) + r(p,n)}

=

pen + 1) x(n) + r(P(n + l),n) PEZ P (n + 1) •••. P (m + 2) J r (P (m + 1) , m) + [p (n + 1) ••• P (m O+ 1) J x (mO) and by iteration n (c) x(n + 1) '" [

I

m=m

_o

Almost analogous we find

n (d) v (n+ 1) ~

t

m=m O

(

m+

k-1)

[P(n+ 1) •.. P(m+ 2)J k-1 r(P(m+ 1» + [P(n+ 1) .•• p(m O+ l)Jv(m

Here we have chosen E > 0 and m

O - mO(E) in such a way that

(possible by (b) and since Z is finite). Using (a), combination of (c) and (d) gives

n

I

x(n + 1) - v(n + 1) ~ - 22

-(m

+ k -

1)£~

+

C(£)~ ~

(n

+

k\E~

+

C(E)~

k - 1 k } m=m O

(here C{e) is a constant, depending on E). Hence

lim (n +

~

+

k)-l

(x(n + 1) - v(n + 1» n -+ <Xl

and in the same way

-1

lim (n :

k)

(x (n+ 1) - v (n +:1»

~ E~

n-+<Xl

References

[lJ Gantmacher, F.R., Applications of the theory of matrices, translated

from Russian by J.L. Brenner, Interscience Publishers, Inc., New York(1959) [2J Rothblum, U.R., Algebraic Eigenspaces of nonnegative matrices, Linear

Algebra and its applications ~ (1975), 281 - 292.

[3J Rothblum, U.R., computation of the eigenprojection of a nonnegative matrix at its spectral radius, Mathematical Programming Study ~ (1976) , 188 - 201.

[4J Rothblum, U.R., Expansions of sums of matrix powers and resolvents, re-port Yale University (unpublished).

[5J Seneta, E., Nonnegative matrices, An introduction to theory and appli-cations, Wiley, New York (1973).

[6J Sladky, K., On dynamic programming recursions for multiplicative Markov decision chains, Mathematical Programming Study ~ (1976), 216 - 226. [7J Sladky, K., Successive approximation methods for dynamic programming models, Proceedings of the Third Formator Symposium on Mathematical

Methods for the analysis of Large-Scale Systems, Prague (1979), 171 - 189. [8J Van der Wal, J. and W.H.M. Zijm, Note on a dynamic programming

recur-sion, Memorandum Cosor M 79 - 12, Eindhoven University of Technology. [9J Zijm, W.H.M., On nonnegative matrices in dynamic programming I,