Asymptotic behaviour of the utility vector in a dynamic programming model

Asymptotic behaviour of the utility vector in a dynamic

programming model

Citation for published version (APA):

Zijm, W. H. M. (1980). Asymptotic behaviour of the utility vector in a dynamic programming model. (Memorandum COSOR; Vol. 8004). Technische Hogeschool Eindhoven.

Document status and date: Published: 01/01/1980 Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:

www.tue.nl/taverne Take down policy

If you believe that this document breaches copyright please contact us at: openaccess@tue.nl

providing details and we will investigate your claim.

Department of Mathematics

PROBABILITY THEORY, STATISTICS AND OPERATIONS RESEARCH GROUP

Memorandum COSOR 80-04

Asymptotic behaviour of the utility vector in a dynamic programming model

by

W.B.M. Zljm

Eindhoven, April 1980 The Netherlands

DYNAMIC PROGRAMMING .MODEL

by

W.H.M. Zijm

O. Abstract

In mathematical economics (e.g. Leontiefsubstitution systems) and in Markov decision theory we often deal with dynamic programming recursions of the following form

x(n + 1)

=

max Px(n) ; PEM

n

=

0,1,2 ••.

where x(O) is assumed to be a strictly positive vector. M is a set of matri-ces, generated by all possible interchanges of corresponding rows, taken from a fixed finite set of nonnegative square matrices (not necessarily stochastic). We investigate the asymptotic behaviour of the vector x(n) in terms of generalized eigenvectors of a particular matrix

P

E M, with respect to its spectral radius cr(P). This paper extends earlier results of Sladky [11J and Zijm [13J.

1. Introduction

Consider a set M of matrices, which is generated by all possible inter-changes of corresponding rows, taken from a fixed finite set of nonnegative N x N - matrices. In this paper we investigate the asymptotic behaviour of the utility vector x(n) (a column N-vector), defined by the following dynamic programming recursion

(1) x(n + 1) max Px(n)

Pe:M

n

=

0,1,2, •••.

where x(O) is a fixed, strictly positive vector. From the structure of M i t is obvious that we may take the maximum component-wise in (1).

Nonnegative matrices, and especially recursion (1), play an important role in mathematical economics, e.g. the analysis of (closed) Leontief sub-stitution systems. Also dynamic programming recursions in Markov decision theory and in open Leontief substitution models, which often have the follow-ing well-known form

v(n + 1) = max {r(Q) + Q v(n)} Qe:M

n

may be written as equation (1), by applying the following translation

P

[

:

1 r(Q) [ V1,n)

1

x(n)

=

n

=

0,1,2, ••••

as noted by Sladky [llJ. Here r(Q) denotes the immediate reward vector, or consumption vector, associated with the matrix Q, which will be (sub)stochas-tic in general. (Note that we do not assume anything about the matrices P ( M, in particular, they don't have to be (sub)stochastic or irreducible). For more details, compare e.g. Burmeister and Dobell [2J, and Bellman [1J.

The recursion {ll was investigated by Sladky ([10J, [11]) under very restrictive assumptions, whereas in [13J we studied the first order asympto-tic behaviour of x(n) in a more general context. In [10J all the matrices

*

*

were assumed to be irreducible , in [11J the matrices were allowed to be reducible, however every matrix contained exactly one basic class (see for defihitions below). These assumptions were dropped in [13J, in which the

first order behaviour of x(n) was characterized in terms of the maximal spectral radius and the maximal index associated with this spectral radius, using a number of decomposition results for the set of matrices M (Zijm [12J).

Finally, characterizations of these decomposition results in terms of gener-alized eigenvectors were given in [14J (compare also Rothblum [7J,[8J).

In this paper we describe the complete asymptotic behaviour of x(n) (not only its first order term). In the case that M contains exactly one matrix, P say, one may easily prove that we have fOr xen), n

=

1,2, ••..

(2)

*

where P is supposed to be aperiodic • Here a denotes the spectral radius of P, v the index associated with a, while p is the maximum of the absolute values of the eigenvalues with absolute value smaller thana (hence p < a)

and t is a fixed number. The vectors x., i

=

1, .••

,v

are called generalized

~

eigenvectors of P, associated with a· •. Formula (2) may be proved easily by using the so-called Jordan or normal form of P (compare e.g. Pease [6J).

In the case that M contains more than one matrix, we are still able to de-rive a result similar to (2); a then denotes the maximal spectral radius, taken over the whole set of matrices, whereas p is some number, defined by the set of matrices (again p < a). Since we do not have something like a simultaneous Jordan form for a whole set of matrices, the approach has to be different. The approach in this paper will be based on a number of decom-position results for the set of matrices M ([12], [14J), which will be men-tioned in the next section, after giving some definitions and notational conventions. After that we prove that x(n) is bounded by a polynomial, simi-lar to the right-hand side of (2); once having this result the polynomial expansion of xen) can be given immediately (section 3). TO be more specific

it is possible to decompose the state space S

=

{l, •.. ,N} in sets

(1) (r) (i)

C , ••• C , such that, if x denotes the restriction of a vector x to C(i)(i

=

l, .•• ,r), there exist strictly positive real numbers 0i'P

i , and; also positive, integers vi' together with finite sets of vectors

{ _Xj (i) _{; j}

=

_1""'V }

i ' for i

=

l, ... ,r, such that

(3 ) x (i) (n)

with 01 > 02 > ••• > or and Pi < 0ii i

=

l, ••• ,r. We will end with some con-cluding remarks. One technical detail will be proved in an appendix.

2. Preliminaries

N

We will work in the Euclidian space lR • Matrices, resp. (column) vectors, are denoted by upper, resp. lower, case letters. We say that matrix A is non-negative (positive)- denoted by A ~ 0 (A »0) - if all its coordinates are nonnegative (positive). We say that A is semi-positive - denoted by A > 0 -if A ~ 0 and A ~ O. Similar definitions apply to vectors. Finally, [AJ

i de-notes the i-th row of the matrix A, [AJ .. its ij-th element.

l.]

M is defined as a set of nonnegative N x N - matrices with the follow-ing property: if V is an arbitrary subset of {l, ••• ,N}, then P_l,P

2 E M implies that P, defined by [PJ

i

=

[PI]i' for i € V, [PJi

=

[P2]i' for i

I

V, is also an element of M.

We will refer to the indices 1 •••• ,N as states; S

=

{1, ••• ,N} will be called the state space.

Next we summarize some results about nonnegative matrices. Let o(P) be the spectral radius of P. According to the well-known Perron-Frobenius theo-rem o(p) equals the largest positive eigenvalue of P and we can choose the corresponding eigenvector ~(P) > O. Recall that if P is irreducible then even ~(P) »0 and o(P) is a simple eigenvalue. If P is reducible then, even-tually after permuting the states, we may write

*

P P 22,---

f

25 '-., , P ss where each P .. 11 p.. has access 1.1.

itself is irreducible with spectral radius cri(P). We say that to P

t t (or Pt t is accessible from Pii) if for some sequence of integers kO i < kl < < k~ = ! we have: P > 0, j

=

1, ... ,0.

\J k. 1,k.

]- J

The sequence{Pk.,k. ; j

J J

l, .•.

,o}

is called a chain. We say that PH is basic, resp. non-basic i f a.(P)

1

cr. (P) <

1. cr (P) • I t is well-known (compare e.g. Gantmacher

i f and only if each non-basic class of P has access to

=

a(P}, resp.

[4

J)

that j.I(P) some basic class no basic class is accessible to any other irreducible class of P.

»0 and

The length of a chain is the number of basic classes i t contains. The index v(P) of P is defined as the length of its longest chain of irreducible

*

classes

Having these concepts, we next formulate the decomposition result for sets of nonnegative matrices, which will be basic in this paper. It holds

Lemma 1 : There exists a matrix P E M, with spectral radius cr and index

V,

and a partition {DO,D1, .•. ,Dv}of the state space S, such that, after even-tually permuting the states, we may write, for all P E M:

(4) P ( P . . .

v,v

p.

v,v-1

4 - . - - - -p.

v,1

p- ';;' - - - . P. P. v-1,v-l ,v-l,l Iv-l,O

..

, I ,

,

where P . . is defined on D. x D. and P . .

=

a

for i < j, i,j

1 , ) 1 J 1 , ) 0, ••• \), and

We remark that the given definition of v(P) is a rather unusual one. For non-negative matrices however, i t can be shown that this definition is completely equivalent with the traditional one (compare Rothblum [7J)

for all P E M. Furthermore there exist vectors

O.

»0, such that 1 (5) while (6 ) max PEM max PEM p . . iJ 1, =: p. , O~ 1,1 1,1 .... 1, ...

,v

Finally, every basic class of P. i has access to some basic class of p, 1 . 1

1, 1-

,1-for i = 2, ••• ,\).

Proof: The proof of this lemma may be found in Zijm [12J. Note that (5) im-plies that max

PEM

cr (P . .)

1,1 O{Pi,i}

= a

for i = l, ...

,v,

since Pi »0. Remark: It will be clear that we may formulate a similar decomposition result for the set of matrices PO,O' defined on DO x DO' and with respect to 0

0

=

PEM max cr(PO , O}. Continuing in this way we obtain the complete

block-triangular decomposition as it was fomulated in [12J.

We have to mention onee~mentary result for nonnegative matrices.

Lemma 2: Let P be an N'X N-nonnegative matrix with spectral radius cr > 0

and strictly positive right eigenvector ~. Then

lim n~ 1 n n-1

I

m=O

o

*

exists and is again a nonnegative matrix. Denoting this matrix by P we have

*

P.P P .P

*

o.P

*

If P is aperiodic then even

*

-n pn

P lim 0

n~

Proof: Let [ll]i denote the i-th element of lJ.

[p]. , -1 -1 _{[P]ij [lJJ}

j

=

o [lJ]i

1J

then P is a stochastic matrix, for which the results are well-known (Kemeny and Snell [5J).

o

Having these preliminary results, we next turn to the dynamic programming recursion (1). In the following section we will determine the asymptotic

behaviour of xi(n), the restriction of x(n) to Di (i

=

l, •.. ,V) by an in-ductive argument.

3. Asymptotic behaviour of x(n)

In this section we suppose all matrices to be aperiodic. Denote by x. (n)

1 the restriction of x(n) to D. 1 (i = O, •••

,v),

as defined in lemma 1.

First we recall a convergence result for the sequence {xl (n) ; n = 0,1,2, .•. }, which was proved in Zijm [13J. After that we establish, step by step, the polynomial boundedness and the polynomial expansion of {x. (n) ; n

=

0,1, •.• },

1

for i

=

2, ••. ,\). For simplicity we assume in the following (j

=

1 and hence max a(PO,O) < 1 (e.g. by multiplying all matrices with

~-1)

First we have PEM

Lemma 3: Let {x (n) ; n = 0,1,2 •.•• } be defined by recursion (1), and let x. (n)

1

be the restriction of x(n) to D

i , as defined in lemma 1, for i = 0, ••• ,0; n = 0, 1 , 2, • •. • Then

lim Xl (n)

n-+<»

exists. Denote this limit by Xl' then Xl »0.

Proof: Compare Zijm [13J, theorem 5.

Lemma 4: Let {x(n) ; n

=

0,1,2, •.• } and {x,(n) ; n

=

0,1,2 •••. }

1

o

i O,l, ••.

,v

be defined as in lemma 3. Then there exists a 0,

a

< 0 < 1 and a constant A, such that

Proof: In Zijm [13J, lemma 4, i t was proved that

where B is some constant, a

O = max a(PO 0) < 1 and t the maximal index,

PEM '

associated with a

O. The result follows immediately for some 0 with aO < 0 < 1.

D

We are now ready to formulate the main theorem of this section. It holds

Theorem 5: For i 1, ... ,

v

there exist sequences of vectors {x. .; j

1.,J l, . . . ,i} such that (7) x. (n) 1. x . . 1.,1. with p < 1 and x . . »0 (i 1.,1. + .•• +

(~)

l, . . .

,v).

x. _1., 2 x. _1., 1

Proof: The proof will be given in 3 steps, the first one dealing with xl (n). From (1) we have for xl (n)

max {Pl,l xl (n - 1) + Pl,O xO(n - l)}

PEM

with Pl,O xb(n) converging to zero geometrically, for all P E M (lemma 4).

Furthermore lim xl (n) exists (lemma 3) and will be denoted by xl (xl »0). n-+oo

Finally xl (n) approaches xl geometrically fast; in fact this is a direct consequence of a result of Federgruen and Schweitzer [3J, as will be pointed out in the appendix (lemma A).

In the following two steps we suppose that (7) holds for i

=

1, .•• ,k <

v

and for some p < 1. In step 2 we prove that the difference between x

k+l (n) and a very specific polynomial is bounded.

20 Consider the following set of functional equations (S. 1 ) max

{p

Yk+l,k+l} PEM k+l,k+l Yk +1,k+l (8.2) max _{{Pk+l,k+l Yk+l,k}

+

Pk+1,k

~'k}

P'€A 1

.

.

, = _Yk+I ,k + Yk+1,k+l I

(8. (k+1) max {P_k+

I

,k+l: Yk+l,l + P_{k+1 ,k:}Xk , t ","+,P_{k+1 ,1 x1}

"I}

= Y

_k

_+1,1

+Yk+~,2

PE~

where {x . . 1 i,j = 1, ••• ,k

l,J j} are supposed to be known (by the

induc-tion hypothesis) and Yk 1 .

+ ,J j = 1, .•. ,k+l are the unknown vectors. Furnher-more A. denotes the set of matrices which are maximizing the

1

of (S.i), for i = 1, .•• ,k.

left-hand side

In [14J we proved that the equations (8) together possess a solution which determines {Y

k+1, j ; j = 2, .•• ,k+1} completely, with Yk+l,k+l »0. Further-more it is clear that with Y

k+1, 1 also yk+'l, 1 + a Yk+1,k+l satisfies equation (S. (k+l)) hence we may choose Yk+l,l so large that

(9) xk+l (O) ::;; Yk+1 1

_,

By the induction hypothesis we have for some p < 1 and some constant C

1 > 0, for all n , and for i 1, ••. ,k:

(10) x. (n) $

1 x . . 1.,1. + .•• +

(~)

xi ,2 +

(where e denotes the vector with all components equal to one), and

(11 )

Finally, define Y IImax P e" PEM

(lemma 4) •

Then, combining (S),(9),{10), (11) and

(12) x

k+1 (n)

which implies that

(13)

~+1

(n) -

{(~)Yk+1'k+1

+ .•. +

(~)Y-k+1'2}

is bounded from above.

In the same way we can establish a lower bound for the expression (13), since we may choose Yk+l, 1 (in the solution of (8. (k+l» and C₂ > 0 in such a way that (9), (10) and (11) hold with S replaced by ~, C

1 by C2 and Yk+l,l by

Yk+l,l' This completes the proof of step 2. For the rest of the proof we define:

j=2, .•• ,k+l

n

=

0,1,2, •••

30 In this step we finally prove (7) for i = k+1, which in our notations simply means that w

k+1 (n) must converge to some xk+1,1 geometrically fast. Note that the induction hypothesis already implies the geometric convergence of w. (n) to x. l ' for i

=

l, . . . ,k, whereas in step 2 the boundedness of

J. J.,

wk+1 (n) has been established. Since

lim n-l><>O

1, ••• ,k

i t is easy to see that in the right-hand side of (12) we will first maximize

terms of order (n

~

1) ,

then terms of order

(~ =

~)

,

etc., for n large enough. Reasoning in this way it can be derived immediately that for some nO and n ~ nO' (12) reduces to

xk+1,2 + wk+1 (n)

(by successive substitution of (8.1),(8.2) until (8.k». This equation can be written in the following form:

(14) w

k+1 (n) max {p PE~ k+l,k+l wk+1(n - 1) + rk+1(p with, for all PEAk

(15) n -1)

n -

1)}

Note that r

k+1 (p ; n) converges geometrically fast to rk+1(p), defined by

(16) _{r k+l (P)}

=

_{Pk+1,k x}

_k

_{,1 + ••• + Pk+I} ,1 x_{I ,l - xk+I},2 •

Furthermore, let PEAk maximize the left-hand side of (8.(k+1» for some

-*

Yk+l, 1 • Multiplying both sides of (8. (k+1) ) with P_k+₁,k+l gives, together with (16)

-*

_rk~1

(p) 0

P . =

k+1,k+1

and in general, for P E _Ak

*

r_{k+1 (P)} ~ D.

P k+1,k+1

Having these results the geometric convergence of w

k+1 (n) to some xk+1,l is again a consequence of the result of Federgruen and Schweitzer [3J, mentioned above. In our context the geometric convergence of w

k+1(n) in (14) follows immediately from lemma A in the Appendix. This completes the .proof of step 3. Combining 1°, 20 and 3° theorem 5 now follows immediately by induction.

_o

(1) and vectors Xj on C 1 by x ... v, J x .... 1 .

v-

,J (1) x. j 1, •... ,

V

J X 2 ,J . Xl . _{, J}

j

with x . .

=

0 for i < j, and x. j defined by (7) for i ~ j; i , j

=

1, •••

,v.

~,J ~, (1)

Then if we denote the restriction of x(n)to C

1 by X (n), we may write instead of (7): (17) X (1) (n)

= (

n )

v

1 (1) XA V + •...••• +

(~)

with {x

~

1)

J j 1, •••• ,v} satisfying the following equations:

P (1) (1) (1) max x .... = XA V v PEM (18) P (1) (1) (1) (1) max x. _Xj + x. 1 j = 1, .•• ,

v-l .

PEM J J+

(these equations can be derived immediately from (8), for k+1

=

1, ••• ,V).

In general we have (for

&

#

1) :

(19)

with p < &.

Recall that for all P, [PJ ..

=

0 for i E DO

=

s\C

1 and j E C1 (compare

lem-~J (0)

rna 1), which means in particular that x (n)(the restriction of x(n) to DO' for n

=

0,1,2, ••. ) is completely determined by the matrices PO,O • We may apply the same decomposition procedure to the set of matrices, defined on DO x DO' etc.,etc., (the ultimate result can be found in Zijm [12J). It will be clear however that we may find a second set C

number O

2 (the maximal spectral radius of the matrices, defined on DO x DO) and an integer v

2 (the associated maximal index) such that for x(2)(n), the restriction of x(n) to C

2, we have a result similar to (19). Continuing in this way we will find the following theorem.

Theorem 6: There exists a partition {Cl, ••. ,C

r } of the state space S, such that, after eventually permuting the states, we may write, for all P EM:

P p(l)Q(l) ______ R(l) p,(2) ______ ¥(2) , ' , ' (r) P

where p(i) is defined on C

i x Ci and [P]kt = 0 for k E Ci ' 2 E Cj and j < i (i,j = l, ••• ,r). Define o. and v. as follows:

1. 1.

max {o (P (i) )

I

P E M} _{i 1, ..• , r}

i = 1, .•. ,r

where v(p(i» denotes the index of p Ci ) with respect to o(p(i». Then there exist sequences of vectors of appropriate dimension, denoted by

{x;i); j

=

l, ••• ,v

i } , for i

=

l, ••. ,r , which satisfy the following func-tional equations: max PCM max PCM P (i) (i) Xj j = 1, •.• ,v.-1 _1.

while furthermore under the aperiodicity assumption, we have for x(n), de-fined by (1): (1) ( n ) n (i) (n) n x 2(i) + (nO-)xI(i) + n x (n)

=

1 0 . x + .•• + 1 0i O(P i ); n= 0,1,2, ••• Vi - 1. Vi 1

with xCi) (n) denoting the restriction of x(n) to C., and with Pi < cr. ). ). (i

=

1, .•• ,r). Furthermore it holds: cr 1 > cr 2 · > ••• > cr • r

o

With theorem 6 we completed the description of the asymptotic behav-iour of x(n), defined by (1), under the aperiodicity assumption, mentioned at the beginning of this section. It will be clear that we may weaken this assumption, e.g. by assuming aperiodicity only for those matrices which ap-pear infinitely often as the maximizing one in the dynamic programming re-cursion. Also notice that by a simple data-transformation we can obtain a system in which every matrix becomes aperiodic (compare Schweitzer [9J),

whereas the chain structure of the matrix remains completely the same. To be specific, define for all P E M

P o.P + (1 - a.) I (0 < a. < 1) •

Then P possesses the same eigenvectors as P, and cr(P) = o.cr(P) + (1 - a).

*

~*

*

~*

Furthermore, if P exists, then P exists and P = P .

We may even drop completely the aperiodicity assumption and obtain analo-gous results for certain subsequences of {x(n) ; n

=

O,l,2, ••• } (compare Sladky [11 J) •

Finally, the results obtained in this paper may be useful for estimating (i)

cr. = max cr(P. i) and x. for j

=

l"."V

i ; i

=

l, ••• ,r (compare Sladky

J. PEM J., J

Appendix

In this appendix we will treat some details, which were omitted in the proof of theorem 5, step 1 and step 3. The results will be based on a paper by Federgruen and Schweitzer [3], however we will treat a somewhat simpler case than they did. Suppose we have the following situation :

A Markov decision process with discrete time space, finite state space

S

=

{l, •..• ,N} and finite action space K. In each state i E S we may take action k which moves the system to state j at the next decision epoch with probability

p~.

while furthermore a reward

r~

is earned. A policy f is a

1J 1

function from the state space to the action space with which we associated the matrix P(f) and the column-vector ref), defined by

( f(i») P(f)

=

P.. . . S 1J 1, JE ( f(i») ref) = _{r i} iES Furthermore we assume

L

1 (i E S, k E K) jES

With each ref) we associate an approximating sequence {r(f,n) ; n = O,l, •.• }

such that lim r(f,n) = ref) geometrically, i.e. there exist a

p

< 1 and

n-t<X>

C > 0 such that, for all f

IIr(f,n) - r(f)1I $ C.p n

*

Finally define the maximal gain vector g by (a) g

*

max P (f) ref)

*

f

and assume all P(f) to be aperiodic. Let

Vo

be a fixed vector and define

(b) 1,2, • •• by v n == max f {r(f,n - 1 ) _{+ P(f)v} n-l }

From Federgruen and Schweitzer [3J we conclude:

*

There exists a vector w such that

* w*)

(v - ng

n

converges to zero geometrically fast, if n tends to infinity.

Notice that this Markov decision process describes precisely the situ-ation which we encountered in the proof of theorem 5, step 1 and step 3, except for the fact that the matrices Pl,l' resp. P _{k+l ,k+ .} l' were not neces-sarily (sub) stochastic. However, recall that there exist strictly positive vectors 0, resp. 0k+l' such that (5) in lemma 1 holds. A transformation like the one in the proof of the lemma 2 then yields immediately the situation in which the matrices are (sub) stochastic. Furthermore, both in step 1 and

*

in step 3 we have g , defined above, is equal to zero (in step 1 we have

*

lim xO(n) 0 geometrically, in step 3 g

=

0 follows from the last two n~

formulas in the proof). We finally summarize the result:

Lemma A: Suppose we have a Markov decision as described above, except for the fact that the transition matrices are not necessarily (sub)stochastici however for some ~ »0 we have

max P(f) ~ 5 ~ f

*

Let furthermore all the matrices be aperiodic and suppose g , defined by

*

(a), is equal to zero. Then there exists a w such that for {v in

=

O,1,2, ... } n defined by (b) we have lim v n n~

*

w geometrically. D

References

[1J

~ellman, R., Dynamic Programming, Princ.Univ. Press, Princeton, New Jersey, 1957.

[2J Burmeister, E. and R. Dobell, Mathematical Theories of Economic Growth, MacMillan, New York, 1970.

[3J Federgruen, A. and P.J. Schweitzer, Non-stationary Markov decision pro-blems with converging parameters, Math. Centre Report BW 90/78.

[4J Gantmacher, F.R., Applications of the theory of matrices, translated from Russian by J.L. Brenner, Interscience Publishers, Inc., New York, 1959.

[5] Kemeny, J.G. and J.L. Snell, Finite Markov chains, van Nostrand Company, Inc., Princeton, New Jersey, 1960.

[6J Pease, M.C., Methods of matrix algebra, Academic Press, New York, 1965. [7J Rothblum, U.G., Algebraic Eigenspaces of nonnegative matrices, Linear

Algebra and its applications ~ (1975), 281 - 292.

[8] Rothblum, U.G., Computation of the eigenprojection of a nonnegative matrix at its spectral radius, Mathematical Programming Study ~ (1976), 188 - 201.

[9J Schweitzer, P.J., Iterative solution of the functional equations for un-discounted Markov Renewal Programming, Journal of Math.An. and Appl. 34

(1971), 495 - 501.

[10J Sladky, K., On dynamic programming recursions formultlplicative Markov decision chains, Mathematical Programming Study ~ (1976), 216 - 226. [11J Sladky, K., Successive approximation methods for dynamic programming models, Proceedings of the Third Formator Symposium on mathematical

methods for the analysis of Large Scale Systems, Prague (1979), 171 - 189.

[12J Zijm, W.H.M., On nonnegative matrices in dynamic programming,

memoran-dum COSOR 79 - 10, Eindhoven University of Technology.

[1~ Zijm, W.H.M., Maximizing the growth of the utility vector in a dynamic programming model, memorandum COSOR 79 - 11, Eindhoven University of Technology.

[141 Zijm, W.H.M., Dynamic programming, nonnegative matrices and generalized eigenvectors, memorandum COSOR 80 - 03, Eindhoven University of Techno-logy.