On nonnegative matrices in dynamic programming I

On nonnegative matrices in dynamic programming I

Citation for published version (APA):

Zijm, W. H. M. (1979). On nonnegative matrices in dynamic programming I. (Memorandum COSOR; Vol. 7910). Technische Hogeschool Eindhoven.

Document status and date: Published: 01/01/1979 Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:

www.tue.nl/taverne Take down policy

If you believe that this document breaches copyright please contact us at: openaccess@tue.nl

providing details and we will investigate your claim.

Department of Mathematics

PROBABILITY THEORY, STATISTICS AND OPERATIONS RESEARCH GROUP

Memorandum COSOR 79-10

On nonnegative matrices in dynamic programming I

by W.H.M. Zijm

Eindhoven, November 1979 The Netherlands

On nonnegative matrices in dynamic programming I by

W.H.M. Zijm

o.

Abstract

In this paper we present a detailed analysis of the structure of sets of nonnegative matrices, which playa role in dynamic programming recursions. Consider a finite set M of (in general, reducible) matrices, which is generated by all possible interchanges of corresponding rows, selected from a fixed finite set of square nonnegative matrices. We first study a specific block-tr1angular representation of one matrix in terms of its spectral radius and the index of its spectral radius. Gene-ralizing results of Sladky [4J we next show that such a block-triangular representation also exists, in a certain way, for the whole set of non-negative matrices simultaneously. Denoting the subsets of states, asso-ciated with these diagonal blocks, by Cl' •.• 'C

n we prove the existence of a matrix

Po

E M and a strictly positive vector ~, such that for all

P E M we have

i

=

l, ...,n

C

i Ci

where P ,resp. ~ denotes the restriction of the matrix P, resp. the vector ~, to the subset C

i' for i z l, ••• ,n, and o. is defined as the

C. 1 spectral radius of P 0 1 ; i

=

l, ••• ,n. Furthermore we have: 0 1 ~ 02~•••~on

1. Introduction

Products of nonnegative matrices play an important role in the study of dynamic programming recursions. In order to describe the be-haviour of these products, we first need a detailed analysis of the structure of nonnegative matrices. Consider a finite set M of (in gene-ral, reducible) matrices, which is generated by all possible interchanges of corresponding rows, selected from a fixed finite set of square non-negative matrices. These matrices play an important role in e.g. Mar-kov decision processes, where the rows are· .associated with actions, taken in a certain state. (Note however, that we do not require the row-sums to be smaller than or equal to one, hence we have no proba-bilistic interpretation)

Following Rothblum ([3J) we first study a specific block-triangular representation of one matrix in terms of its spectral radius and the index of its spectral radius. After that we show that such a block-tri-angular representation also exists, in a certain way, for the whole set of nonnegative matrices simultaneously. More specifically, if we denote by C₁""'C

n the subsets of states, associated with these diagonal block~ we can prove the existence of a matrix Po E M and a strictly positive

vector ~, such that for all P E M we have

(1 ) C. C. C. C.

c.

p l . " l . : s p l . " l . = l. .. 0 " O'i ~ i:: 1, ...,n. C i Ci

where P ., resp. ~ , denotes the restriction of the matrix P, resp. the vector ~, to the subset C

i , for i

=

l, ...,n, and 0'. is defined as

C_i l.

the spectral radius of Po ' i

=

l, ••• ,n. It holds:

( 2) ~•••~ 0' •

n

This generalizes results of Sladky ([4J).

In the following we first summarize some notational conventions and a few spectral properties of square nonnegative matrices. After that we introduce a number of concepts, which describe the structure of a particular nonnegative matrix and present a block-triangular represen-tation of such a matrix. In the final section we prove our main result,

the fact that such a block-triangular representation also exists for the whole set of matrices simultaneously.

2. Notational conventions

N

We will work in the Euclidian space lR • Matrices, resp. (column)-vectors, are denoted by upper, resp. lower, case letters. We say that a matrix A is nonnegative (positive)- written A ~ O(A » 0)- if all its entries are nonnegative (positive). We say that A is semipositive -written A > 0 -if A ~ 0 and A , O. We write A ~ B (A » B, A > B), if A - B ~ 0

(A - B » 0, A - B > 0). Similar definitions apply to vectors. For matrices A, [AJ

i denotes the i-th row of A, [AJ.,~J

the

- iJ'-th element of A. For a vector c, [cJ, again denotes its i-th element.

~

If D is a subset of {1,2, ••• ,N}, and A, resp. c, an NXN-matrix resp. N-vector, then by AD, resp. cD, we denote the restriction of A, resp. c, to D. If D is a proper subset, then we call AD a principle mi-nor of A.

+

Finally, if c is an arbitrary vector, we denote by c the vector, aefined by

+

[c J.

=

[cJ" if [cJ. > 0

~ ~ ~

if [C],1 S O.

We next summarize a few spectral properties of square nonnegative matrices. Let cr(A) be the spectral radius of A, then cr(A) equals the largest positive eigenvalue of A and we can choose the corresponding

eigenvector ~(A) > 0, according to the well-known Perron-Frobenius theorem. Recall that if A is irreducible then even ~(A) » 0 and cr(A) is simple. Furthermore we have the following

Lemma 1: The spectral radius cr(B) of any principle minor B (of order < N)

Qf the nonnegative matrix A (of order N) does not exceed the .pectral radius a(A) of A:

If A is'irreducible, then the inequality in (3) is a strict in-equality.

If A is reducible, then the sign of equality holds in (3) for at least one irreducible principle minor.

For a proof we refer to Gantmacher ([2J).

Furthermore, define Jt'A) to be the null space of (A - O(A)I)k. Then the index of A, v(A) is the smallest nonnegative integer k such that Nk(A) = Jt+1(A). From Dunford & Schwartz ([ l

J),

it follows that V(A)

~

N and that Nk(A)

=

NV(A) (A) for k

~

V(A). NV(A) (A) is called the al9ebraic eigenspace of A and its elements are called generalized eigen-vectors (see Rothblum [3J).

3. Preliminaries; the "one matrix~' case

The aim of this section will be to present a block-triangular de-composition for a specific matrix. The terminology of Rothblum [3J will be followed here. We will refer to the indices l, ..•,N as states. Let P be an N x N nonnegative matrix. We say that state i has access to state

j (or state j is accessible from state i), if for some integer n ~ 0,

n

[p

J

ij > O. An irreducible class is then simply a set of states, such that each state has access to each other state. For this reason we may speak of having access to (from) an irreducible class if there is access to (from) some, or equivalently every, state in the irreducible class. Irreducible classes are partially ordered by the accessibility relation. If C is an irreducible class, then the matrix pC is called irreducible.

In the sequel a class will always mean an irreducible class. A class C of a square nonnegative matrix P is called basic if CJ(pC)

=

CJ(P), otherwise nonbasic (i.e. O(pC) < o(P); compare lemma 1). A class C is called initial, resp. final, if no other class has access to, resp. from, C.

By

a chain of classes we mean a collection of classes such that each

class in the collection has access to or from every other class in the collection. A chain of classes with initial class C and final class D is called a chain from C to D. The length of a chain is the number of basic classes it contains. We say class C has access to class D in n steps if the length of the longest chain from C to D is n. The height, resp. depth of a basic class is the length of the longest chain of classes in which this basic class is final, resp. initial.

*

For an illustration of the above definitions we refer to Rothblum ([3]} in which we also may find a proof of the following

Lemma 2: For a square nonnegative matrix P, the index of P equals the length of its longest chain.

Using these definitions we are now ready to present our first result concerning the decomposition of a particular square nonnegative matrix P. It holds:

Theorem 3: Let P be a square nonnegative matrix having spectral radius

a

and index

v.

Then, by possibly permuting the rows and corresponding columns of P, we may write

P

v,v

P

_v,v-

l---·--·· P

_v,

1 P_v,O (4) p ... P

_{v- ,v-}

1 1- -~-- - P

_{v- ,}

1 1 .. I ."'.. I

....~..

..

:

.' .. P1,1 P v-1,O

•

••

_•

I P1,O

Here Pi,i contains precisely all basic classes with depth i, and all nonbasic classes which have access to some basic class with depth

i and not to any basic class with depth j > ij i

=

1, ••• ,

v.

Furthermore, a(Pi,i)

=

a,

for i

=

l, •••

,v,

while a(PO,O) <

a

(if PO,O is not empty). Finally for i

=

1, •••

,v

there exists a ~. » 0, such that

1.

( 5) P. . )J.

=

a~. 1.,1. 1. 1.

Proof: By lemma 2, the maximal length of a chain equals

v,

hence there exist basic classes with depth i, for i

=

l, •••

,v.

Obviously a basic class with depth i has no access to a basic class with depth j > i. Because the construction is such that a nonbasic class, which has access to some basic class with dept i, and not to any basic class with depth

j > i, is contained in p. i' it follows that Pi . does not have access to

1., ,1.

P

j,J., for j > i, i

=

l, ••• ,V. What remains are eventually nonbasic

clas-ses which are not accessible to any basic class, these are contained in PO,O. From this the block-triangular form follows, while by lemma 1

a(Pi,i)

=

0, for i

=

1, •••

,v

and a(po,o) < a. Finally, basic classes in Pi

_,

1 have no access to each other, while a nonbasic class in P, , has

~,~

access to some basic class in Pi ,. Hence Pi ' possesses a strictly

,~ ,1

positive right eigenvector, associated with

a,

for i

=

1, .••

,v.

o

Remark: In the proof of theorem 3 we mean by "P, . is not accessible to

1,1

Pj,j" that Pi,j

=

°

(matrix).

From the construction of P, ., i

=

1, •••

,v

follows immediately:

1,1

Corollary 4: Every basic class in P, . is accessible to some basic class

1,1

in p._1-1_,i _- 1 ; i

=

2, •••

,v.

However, a basic class in Pi 1 . l' is not necessarily accessible from

-

,1-a b,1-asic cl,1-ass in P, ,.

~,~

The concepts "basic class" and "index" are defined with respect to the spectral radius of P. However, defining these concepts on PO,O and with respect to a(PO,O) now, we may repeat the whole procedure ~gain,

i.e. decompose PO,O in exactly the same way. It is immediately clear that, continuing in this way, we finally get a block-triangular decom-position of P, such that every blockmatrix on the diagonal possesses a strictly positive right eigenvector, associa~edwith the spectral

ra-dius of that blockmatrix. Using corollary 4, we have:

Theorem 5: Let P be a square nonnegative matrix. Then, eventually after permuting the states, we may write:

(6) P

=

P2,2--~~--~·P2,S

_..

.

~',

:

...

.

•

...

P s,s

where for i

=

1,2, ••• ,s there exists a ~. » 0 such that ~ and a(p. 1 . 1)· 1.+ ,~+

o

irreducLble class of

to a(P. i) has access to some irreducible

~,

spectral radius a(P . . )

=

1.,1.

while d(P . . )

=

a(P. 1 '+1) implies that every

~,~ ~+ ,1.

Pi . with spectral radius equal ,~

class of Pi 1 . 1 with the same₊

,~+

The decomposition described in theorem 5 will prove to be very useful for a complete analysis of the asymptotic behaviour of products of nonnegative matrices in dynamic programming recursions (see e.g. Zijm

[5J) •

4. Block-triangular decomposition for the set of matrices M.

We now return to the situation of the set of matrices M with the interchangeability property as described in the introduction. In this section the following will be proved:

Theorem 6: There exists a matrix P E Msuch that by possibly permuting rows and columns of P we may write

A A P

_u

P_{12 ·_·--·_· P1s} (9) P

=

A P22 -·--- P2s ... I ..._... I

..

,

.. I ... I

...

.

"P ss

where for i

=

1, •.• ,s

=

~(P) there exists a Pi » 0 such that for any

(10)

A

p. . ili ::s; p. i Pi = a(P. .) il.

~,~~, ~,~ ~

(here P is decomposed in p . . ; i,j = 1, ..•,s(P) according to the

de-~,J

composition of P, i.e. P . . is a matrix on the same states as P . . ).

~,~ ~,~ Furthermore A a(P_{1 ,1)} 2: a(P 2,2) A 2: ••• 2: a(P_s,s) A A

while a(P . . ) = a(P

i+1 . 1) implies that every irreducible class of

A ~,~ ,~+ A

Pi i with spectral radius equal to a(P . . ) has access to some

irreduc-, A ~,~ A

ible class ,of P. 1 . 1 with the same spectral radius a(P . . ). Finally

~+ ,~+ . . ~,~

Pi . = 0 for i,j = 1, ••••,s(P) ; j < i ; for all P E M.

,J

Proof: Because of the finiteness of the set M we may define:

(11) a = max a(P) PEM (12) K

=

{p

I

a(P)

=

a} and (13) v = max v(P) PEK

There exists a matrix, Po say, such that a(P

O

)

= a, v(Po) = v, which possesses a maximal number of basic classes, C(l), ••• ,C(k) say, with maximal depth v (cf. lemma 2).

Now let us construct (recursively) a sequence of matrices

{Pm; m

=

O,1,2, .•.~} and a sequence of sets {D(m)i m

=

O,l,2, ... }, such

that ( 14) ( 15) k 0(0) = U C(i) i=l [Pm+!

J

_j = [p

J.

,

for j E oem) m J and for j

t

Oem) :

(16) if [p 1J'k ~ 0 for some k E D(m)

m+ ]

(17) D(m+l) = D(m) U {j

I

[p IJ'k ~ 0 for some k E D(m)} m+ ]

As the state space is finite, in a finite number of steps we arrive at some p , and D(p), such that [pJ'k = 0 for all P E M, j ; D(p)

p )

k E D(p). Obviously o(P ) = 0 (lemma 2). C(1) , ••• ,C(k) are basic classes

p k

with respect to P too, while furthermore every state of D(p)\ UC(i)

k p i=1

has access to U C,. For this reason a chain in pD(p) with maximal

i=1 1. P

length will always end in one of the basic classes C(l), ••• ,C(k). The final basic class in such a chain will have depth v with respect to P ,

k P

since [p J, = [POJ, for j

t

D(p)\ U C(i). Because

v

is maximal, we find

p ] ] i=1

that every chain in pD(P) has length not greater than one, hence there

p

are no connected basic classes; C(l) , ••• ,C(k) all have depth v with

res-k

pect to p and D(p)\

U

C(i) possesses no basic class. This implies the

p i=l

existence of a strictly positive right eigenvector,

~D(P)

say, of pD(P) ,

P P

associated with o. By lemma B and lemma C in the appendix it is now easy to see that there exists a matrix

P

and a strictly positive vector

~D(P),

defined on D(p) such that

(18) pD(p) ~D(P) S pD(P) ~D(P) = o~D(P), for all P E M.

where

[pJ, ]

k

= [POJ j , for j

t

D(p)\ U C(i), and v(P)

i=l

=

v

Hence, also p possesses a maximal number of basic classes with maximal depth

v.

Since [PJjk = 0 for j

¢

D(P)',k E D(p), for all P E M, this

im-plies that for any matrix P, with o(pD (p»

=

0, we have*

V(pD' (P» s v - 1

Furthermore v(pD' (P» = v - 1, and, since the basic classes of pDfp) all -D(P)

have depth v, every basic , class of P has access to some basic class (of depth" - 1) of pDI (p) •

We now may cancel the rows and columns of

P,

corresponding to states in D

p and repeat the whole procedure. Continuing in this way,

the proof is completed.

o

By theorem 6 we see, that we may extend the block-triangular structure of a specific matrix (cf. theorem 5) to the whole set of matrices M, in a certain sense. One might also say that the set M is bounded in a cer-tain sense.

Discussion: Theorem 5 in section 3 looks very much like lemma 2.1 of Sladky [4J, with one important difference. In [4J the blocks

p. i; i

=

1, ••• ,s(P) are the "largest" submatrices, having strictly

po-1.,

sitive eigenvectors, corresponding to their spectral radius. Comparing with theorem 3, this means that for a matrix with index

v,

the first

~lock (P in theorem 3) contains all basic classes with height 1, the

v,v

second block all basic classes with height 2 etc. For example the matrix

P

=

1

o

o

o

1

o

1

a

1

will be decomposed by Sladky as follows:

P11

=(

~

~)

P₂₂

=

(1) P₁₂

=

(~

)

while theorem 5 gives

P11

=

(1) P

22

=

(~ ~)

P12

=

(0 1)

The advantage of theorem 5 is that the behaviour of powers of matrices can be described very well in terms of this last decomposition, as will be clear from the example. Theorem 5 can be extended to the whole set of matrices without making any further assumptions, as shown in section 4. In [4J only a very special case of theorem 6 has been proved. It is assumed that, for all P € M:

> ••• > IJ(P )

s,s

while furthermore every p. i ; i = l, ••• ,s may contain exactly one

1.,

irreducible class with spectral radius o(P . . ). 1.,1.

5. ApplWiix

For the sake of completeness we establish two technical lemmas. In this appendix we will deal several times with a recursion of the follo-wing form

k 0,1,2, •••

where ~k is a nonnegative vector, the eigenvector of P

k associated with

its spectral radius o. We make the following convention:

convention A: If, for some state i E {l, ••• ,N} we have

for all P

then we take

Lemma B: Suppose we have a finite set of N x N-matrices Z, generated by all possible interchanges of corresponding rows, selected from a fixed finite set of nonnegative matrices. Let 0 = max o(P) and K={P

I

a(P) =a}.

PEZ Suppose

\I(P)

=

1 for all P E K

20 3_Po

E K with a strictly positive right eigenvector,

associa-ted with o.

*

*

that:

*

* *

*

maxP~

=

P ~

=

O~

PEZ

Proof: Let ~O » 0 be the right eigenvector of PO' associated with a, hence:

Po ~₀ = a~

.

Define PI as follows (use convention A) :

PI ~O = max _{P~O ~ 0~0} PEZ

and suppose

(a) _{PI ~O ~ 0~0}

Obviously, since ~O » 0, we have 0(P

1)

=

0, while furthermore every

nonbasic class of PI is accessible to some basic class of Pl' Since V(P

1)

=

1, P1 again possesses a strictly positive right eigenvector, ~1 say.

Suppose B(l), ••• ,B(k) are the basic classes of Pl' Then

PB₁(!)_~OB(i»_{- v~O}~ B(i).i_mpl'_~es pB₁(!)_~OB(i) -_{- a~O}B(i) _' f_{or ~}. -_- 1_{" " ' k '}k Th~s_~ implies, since by convention A we have [P

1J,₎ = [POJ. for₎ j E U B(i),_i=l

o

1

o

2

B(l), ••• ,B(k) are basic classes with respect to Po too. We may choose ~1 in such a way that:

(b) _~1B(i)

₌

_~OB(i) _i

=

_{1, .•. ,k.}

Choosing ~1 in this way we next prove: ~1 ~ ~O

Suppose the contrary, i.e. for some j E {l, •••N}:

(c) ~1(j) < hl O(j)

k

Then, obViously JED = {l, •••,N}\ U B(i) i=l

hence certainly

B_Y ( )_{c « lJ lJl)}D)+ O

-with the fact that D

> 0, which would imply

a(p~) ~

a,

possesses no basic classes of Pl.

in contradiction

Now 1J

l

=

lJOwould imply: Pl lJO

=

alJOin contradiction with (a). Hence

1J

1 > 1JO• Iterating in this way, it is easy to see that we can get a

se-quence of matrices P

O,Pl'P2, •• , each with spectral radius equal to a,

and a sequence of vectors lJ

O,lJl,1J2, ••• , such that max

PEZ

k

=

0,1,2, •••

and 1J_k+_l > 1J_{k •}

This last relation implies that the procedure is not cycling, hence, since Z is finite, it must stop at some time, i.e. for a certain k we have max PEZ

=

p. for j ~ k. J

*

With P := P_k' 1J

*

:= 1J

k the lemma is proved

o

Remark: Obviously, 1~ is sufficient to assume v(P)

=

1 only for

Lemma C: Let Z, a and K be defined as in lemma B. Suppose v

=

max v(P) •

PEK

Suppose there exists a Po E Z and a subset of states D E {l, ••• ,N}, such that a(P

O

)

~ a, v(P

o)

=

v and Po possesses basic classes, C(l), ••. ,C(k)

k

U C(i), with res-i=l o 1 o 2 o 3 C(i) E D i i

=

1, ••• ,k k

Every state of D\ U C(i) is accessible to i=l

pect to the matrix Po

[PJjk = 0 for j

t

D, kED, for all P E M.

a nonnegative

D

llO » O. Now apply Then there exists a matrix

P

E M and a strictly positive vector jJD

(defined on D) such that

D -D -D -D -D

max P II = P II = all PEZ

k where [pJ. = [PoJ_j for j E U C

i .

J _i=l

k

Proof: Since v is maximal and every state of D\ U C(i) has access to

k k i=l

U C(i), it is immediately clear that D\ U C(i) possesses no basic

i=l i=l

classes (with respect to PO) and that Po possesses right eigenvector, llO say, associated with

a,

wiUh the same procedure as in the proof of lemma B, i.e.

Pi llO = max PEZ

D D

Since 110 » 0, again every nonbasic class of P

1 is accessible to some

D D

basic class of P

1" Suppose P1 possesses a chain of length greater

than one. Let F be a final basic class in a chain of

P~

with maximal

[

J

\

F F > F S'

length. Then P

1 jk = 0 for j E F, kED F, while P1 llO - allO. ~nce

P~

1s irreducible it follows that

P~ ll~

=

all~'

hence convention A im-plies that [P

1Jj = [PoJj for j E F. In other words: F = C(i) for some i E {l, ••• ,k}. Obviously [P

1Jj = [PoJj for j

t

D, hence F has depth v with respect to Pl. Now the supposition of a chain of

P~

with maximal length greater than one, with final basic class F, would imply that an ini tial basic class in such a chain would have depth greater than v. This is impossible, hence

P~

does not

con~ain

connected basic classes (v

(P~)

= 1), from which the existence of a strictly positive right eigenvector of

P~'

associated with

a,

immediately follows. Continuing in this way, by lemma B and the remark after the proof of lemma B, we finally arrive at some P E Z, and

~

such that

~D

» 0 and

max

PeZ

-D

0'1..1

-D

other words, once having l..I , we may choose

o

-C(i) . 0'1..1 , ~

=

1, ••• ,k, hence in k U C

i , which completes the proof.

i=l . l' pC(i) -C(i) < ~mp ~es 0 1..1 --C(i) 0'1..1 , D :":D "':D Now Po 1..1 ~ 0'1..1 pC(i)

"':c

(1)

=

o

1..1 [PJj

=

[POJ_j , for j € References

[1] Dunford, Nand J.T. Schwartz, Linear Operators, Part I,

Interscience, New York (1958).

[2] Gantmacher, F.R., Applications of the theory of matrices, translated from Russian by J.L. Brenner, Interscience, New York (1959).

[3J Rothblum, U.R., Algebraic Eigenspaces of nonnegative matrices, Linear Algebra and its applications ~' 281-292 (1975).

[4] Sladky, K., Successive approximation methods for dynamic pro-gramming models, Proceedings of the Third Formator Symposium on Mathematical Methods for the analysis of Large-Scale Sys-tems, 171-189, Prague (1979).

[5] Zijm, W.H.M., Maximizing the growth of the utility vector in a dynamic programming model, to appear.