On a variational formulation of the QSVD and the RSVD

(1)

www.elsevier.com/locate/laa

On a variational formulation of the QSVD and the RSVD

^聻

Delin Chu ^a ^, ^∗ , Bart De Moor ^b ^,1

a Department of Mathematics, National University of Singapore, Lower Kent Ridge Road, Singapore 119260, Singapore

b Department of Electrical Engineering (ESAT), Research Group SISTA, Katholieke Universiteit Leuven, Kardinaal Mercierlaan 94, B-3001 Leuven, Belgium

Received 16 December 1998; accepted 7 February 2000 Submitted by L. Elsner

Abstract

Recently, M.T. Chu, R.F. Funderlic and G.H. Golub [SIAM J. Matrix Anal. Appl., 18 (1997) 1082–1092] presented a variational formulation for the quotient singular value decomposition (QSVD) of two matrices A ∈ R ^n×m , C ∈ R ^p×m which is a generalization to two matrices of the ordinary singular value decomposition (SVD) and characterizes the role of

聻

This work is supported by several institutions:

1. The Flemish Government:

(a) Concerted Research Action GOA-MIPS (Model-based Information Processing Systems).

(b) The FWO (Fund for Scientific Research — Flanders) project G.0292.95: Matrix algorithms and differential geometry for adaptive signal processing, system identification and control.

(c) The FWO project G.0256.97: Numerical Algorithms for Subspace System Identificati on, extension to special cases.

(d) The FWO Research Communities: ICCoS (Identification and Control of Complex Systems) and Advanced Numerical Methods for Mathematical Modelling.

2. The Belgian State, Prime Minister’s Office — Federal Office for Scientific, Technical and Cultural Affairs: Interuniversity Poles of Attraction Programme (IUAP P4-02 (1997–2001): Modeling, Iden- tification, Simulation and Control of Complex Systems; and IUAP P4-24 (1997–2001): Intelligent Mechatronic Systems (IMechS)).

Delin Chu was a Visiting Research Fellow with the K.U. Leuven during the writing of this paper.

Bart De Moor is a Senior Research Associate with the F.W.O. and an Associate Professor with the K.U.

Leuven. The scientific responsibility is assumed by the authors.

∗

Corresponding author. Fax: +65-7795452.

E-mail adressess: matchudl@math.nus.edu.sg (D. Chu), Bart.Demoor@esat.kuleuven.ac.be (B. De Moor).

1 Tel.: +32-16-32-1970; fax: +32-16-32-1709.

0024-3795/00/$ - see front matter

2000 Elsevier Science Inc. All rights reserved.

PII: S 0 0 2 4 - 3 7 9 5 ( 0 0 ) 0 0 0 7 2 - 0

(2)

two orthogonal matrices in QSVD. In this paper, we give an alternative derivation of this variational formulation and extend it to establish an analogous variational formulation for the Re- stricted Singular Value Decomposition (RSVD) of a matrix triplet A ∈ R ^n×m , B ∈ R ^n×l , C ∈ R ^p×m , which provides a new understanding of the orthogonal matrices appearing in this decomposition. © 2000 Elsevier Science Inc. All rights reserved.

AMS classification: 65F15; 65H15

Keywords: SVD; QSVD; RSVD; Generalized singular value; Variational formulation; Stationary value;

Stationary point

1. Introduction

The ordinary singular value decomposition (OSVD) of a given matrix A ∈ R ^n×m is

U ^T AV =

r ^a m − r a

r a R 0

n − r a 0 0

, (1)

with V ∈ R ^n×n , V ∈ R ^m×m , R ∈ R ^r

^a

^×r

^a

and

U = r ^a n − r a

n U 1 U 2

, V = r ^a m − r a

m V 1 V 2

,

R = diag{σ 1 , . . . , σ _r

_a

}, σ 1 > σ 2 > · · · > σ _r

_a

> 0, r _a = rank(A), where U, V are orthogonal matrices. The σ 1 , . . . , σ _r

_a

are the non-trivial singular values of A, and the columns of U 1 and V 1 are, respectively, the non-trivial left and right singular vectors of A. In this paper, k · k denotes the two-norm of a vector. The following theorem is well known [4].

Theorem 1. Given A ∈ R ^n×m with OSVD (1).

(a) Consider the optimization problem

x∈R

max

m,y∈Rn y=Ax, y6=0

kyk

kxk . (2)

Then the non-trivial singular values σ 1 , . . . , σ _r

_a

of A are precisely the stationary values, i.e., the functional evaluations at the stationary points, of (2). Let the stationary points in (2) corresponding to the stationary values σ 1 , . . . , σ _r

_a

be

_x

y

11

, . . . , _x

_ra

y

_ra

. Then V 1 = h _x

kx

11

k · · · _kx ^x

^ra_ra

_k i

.

Moreover, if n = m = r a , then

(3)

U 1 = h _y

ky

11

k · · · _ky ^y

^ra_ra

_k i .

(b) Consider the dual optimization problem

x∈R

max

n,y∈Rm yT=xTA, y6=0

kyk

kxk . (3)

Then the non-trivial singular values σ 1 , . . . , σ r

a

of A are precisely the stationary values of (3). And, let the stationary points in (3) corresponding to the stationary values σ 1 , . . . , σ r

a

be _cx

y

11

, . . . , _x

_ra

y

_ra

. Then

U 1 = h _x

kx

11

k · · · _kx ^x

^ra_ra

_k i . Moreover, if n = m = r _a , then

V 1 = h _y

ky

11

k · · · _ky ^y

^ra_ra

_k i .

Recently, in [1] Theorem 1 has been generalized to the quotient singular value decomposition (QSVD) [3,5–11,13,14] of two matrices A ∈ R ^n×m , C ∈ R ^p×m . Naturally, it is very interesting to generalize the result in [1] to the restricted singular value decomposition (RSVD) [8–11,15,16] of a matrix triplet. After a study, we found that it is not trivial to get an analogous variational formulation for the RSVD via the approach in [1], this motivates us to re-derive the result in [1] via a different approach.

The purpose of this paper is twofold. Firstly, we present an alternative derivation of the variational formulation in [1] directly based on the QSVD of two matrices A, C. Then we extend this result to the RSVD [8–11,15,16] of a matrix triplet and obtain an analogous variational formulation that provides new understanding of the orthogonal matrices appearing in this decomposition.

Our approach is quite different from that one in [1]. We will show in Section 2 that an orthogonal reduction can be applied to the matrices A ∈ R ^n×m and C ∈ R ^p×m to get lower dimensional non-singular matrices A 22 and C 22 such that the non-trivial quotient singular values of the pair (A, C) are just the standard singular values of A 22 C ₂₂ ⁻¹ and there is a close relationship between the two orthogonal matrices in the QSVD of (A, C) and the (left and right) singular vectors of A 22 C ₂₂ ⁻¹ (see Lemma 2 and Theorem 3). Thus, the non-trivial quotient singular values of (A, C) can be characterized by Theorem 1 with matrix A 22 C ⁻¹ ₂₂ . Hence, Theorem 1 can be used as a bridge to re-derive the variational formulation in [1] for QSVD of (A, C). Moreover, same idea also works for the RSVD (see Lemma 5 and Theorem 6), so it offers a springboard to leap forward to the variational formulation of the RSVD.

In order to prove our main results, we will establish two condensed forms based

on orthogonal matrix transformations. The QSVD of two matrices and the RSVD of

matrix triplets can be obtained and the variational formulation for QSVD and RSVD

can be proved directly based on these two condensed forms.

(4)

In this paper, we use the following notation:

• S _∞ (M) denotes a matrix with orthogonal columns spanning the right nullspace of a matrix M.

• T _∞ (M) denotes a matrix with orthogonal columns spanning the right nullspace of a matrix M ^T .

• M ^⊥ denotes the orthogonal complement of the space spanned by the columns of

• For a matrix M, T M. ^T _∞ (M) and T ^⊥ _∞ (M) are defined by (T _∞ (M)) ^T and (T _∞ (M)) ^⊥ ,

respectively.

• Unless noted, we do not distinguish between a matrix with orthogonal columns and the space spanned by its columns.

We also use the following notation for any given matrices A, B, C with compatible sizes: denote

r _a = rank(A), r _b = rank(B), r _c = rank(C),

r _ab = rank A B

, ¯r _ac = rank

A C

, r _abc = rank

A B

C 0

,

k 1 = r _abc − r _b − r _c , k 2 = r _ab + r _c − r _abc , k 3 = ¯r ac + r b − r abc , k 4 = r a + r abc − r ab − ¯r ac .

2. A variational formulation for QSVD

Several generalizations of the OSVD have been proposed and analyzed. One that is well known is the generalized SVD as introduced by Paige and Saunders in [5], which was proposed by De Moor and Golub [11] to rename as the QSVD. Another one is the RSVD, introduced in its explicit form by Zha [16] and further developed and discussed by De Moor and Golub [8].

In this section, we will give an alternative proof of the variational formulation for the QSVD of [1] based directly on QSVD itself. Firstly, we present a condensed form to derive the QSVD of two matrices.

Lemma 2. Given matrices A ∈ R ^n×m , C ∈ R ^p×m . Then there exist three orthogonal matrices U _a ∈ R ^n×n , W ∈ R ^m×m , V _c ∈ R ^p×p such that

U _a ^T AW =





¯r ac − r c r a + r c − ¯r ac ¯r ac − r a m − ¯r ac

¯r ac − r c A 11 A 12 0 0

r _a + r _c − ¯r _ac 0 A 22 0 0

n − r _a 0 0 0 0



,

(4)

(5)

V _c ^T CW =





¯r ac − r c r a + r c − ¯r ac ¯r ac − r a m − ¯r ac

p − r _c 0 0 0 0

r a + r c − ¯r ac 0 C 22 0 0

¯r ac − r a C 31 C 32 C 33 0



,

where A 11 , A 22 , C 22 and C 33 are non-singular.

Proof. See Appendix A. Let the SVD of A 22 C ₂₂ ⁻¹ be

U ₂₂ ^T A 22 C ₂₂ ⁻¹ V 22 = diag{σ 1 , . . . , σ _s } =: S _AC , s = r _a + r _c − ¯r _ac , (5) where U 22 , V 22 are orthogonal matrices, σ 1 > σ 2 > · · · > σ _s > 0. Define

U :=U _a diag {I _¯r

_ac

_−r

_c

, U 22 , I _n−r

_a

}, (6) V :=V c diag {I p−r

c

, V 22 , I ¯r

ac

−r

a

}. (7)

X =



 



I 0 0 0

0 I 0 0

−C ₃₃ ⁻¹ C 31 −C ₃₃ ⁻¹ C 32 C ₃₃ ⁻¹ 0

0 0 0 I



 



×



 



A ⁻¹ ₁₁ −A ⁻¹ ₁₁ A 12 C ₂₂ ⁻¹ V 22 0 0

0 C ₂₂ ⁻¹ V 22 0 0

0 0 I 0

0 0 0 I



 

 . (8)

Then, as a direct consequence of the condensed form (4), we have the following well-known QSVD theorem.

Theorem 3 (QSVD theorem). Let A ∈ R ^n×m , C ∈ R ^p×m . There exist orthogonal matrices U ∈ R ^n×n , V ∈ R ^p×p and a non-singular matrix X such that

U ^T AX=





¯r _ac − r _c r _a + r _c − ¯r _ac ¯r _ac − r _a m − ¯r _ac

¯r ac − r c I 0 0 0

r _a + r _c − ¯r _ac 0 S _AC 0 0

n − r _a 0 0 0 0



,

V ^T CX=





¯r ac − r c r a + r c − ¯r ac ¯r ac − r a m − ¯r ac

p − r c 0 0 0 0

r a + r c − ¯r ac 0 I 0 0

¯r ac − r a 0 0 I 0



, (9)

(6)

where S _AC is of the form (5), and U, V and X can be chosen to be given by (6)–

(8), respectively. σ _i , i = 1, . . . , s, are defined to be the non-trivial quotient singular values of the matrix pair A, C.

According to the uniqueness theorem in [16], we only need to characterize the matrices U, V given by (6) and (7) in order to characterize the role of the orthogonal matrices in the QSVD. Let U, V be given by (6) and (7) and partition these two orthogonal matrices as

U = ¯r ^ac − r c r a + r c − ¯r ac n − r a

U 1 U 2 U 3

, (10)

V = p − r ^c r a + r c − ¯r ac ¯r ac − r a

V 1 V 2 V 3

. (11)

Then, from Lemma 2 we have

U 3 =T ∞ (A), U 1 = T ^⊥ _∞ (AS ∞ (C)), (12)

V 1 =T ∞ (C), V 3 = T ^⊥ _∞ (CS ∞ (A)). (13)

Hence, in order to characterize the role of the orthogonal matrices U, V in the QSVD, it should only characterize the role of U 2 , V 2 in QSVD.

The following variational formulation has been established in [1] to characterize U 2 and V 2 .

Theorem 4. Given A ∈ R ^n×m , C ∈ R ^p×m . Consider the optimization problem

x∈Rn,y∈

max

Rp, x6=0



 AT CT

TT

∞(A) 0

0 TT

∞(C)





−yx

= 0

kyk

kxk . (14)

Then the non-trivial quotient singular values σ 1 , . . . , σ _s of the matrix pair A, C are precisely the stationary values for the problem (14). Furthermore, let _x

−y

11

, . . . ,

_x

_s

−y

s

be stationary points of the problem (14) with corresponding stationary values σ 1 , . . . , σ _s , then

U 2 = h _x

kx

11

k · · · _kx ^x

^s_s

_k i

, V 2 = h _y

ky

11

k · · · _ky ^y

^s_s

_k i .

Proof. We prove Theorem 4 by the following three arguments.

• Argument 1. Firstly, we characterize the orthogonal matrices U 22 , V 22 in (5). Con-

sider the optimization problem

(7)

max

x2,y2∈^Rra+rc−¯rac xT2 A22 =yT

2 C22,x2 /=0

ky 2 k

kx 2 k . (15)

Since A 22 , C 22 are both non-singular, by Theorem 1 the σ 1 , . . . , σ _s , i.e., the singular values of the matrix A 22 C ₂₂ ⁻¹ are precisely the stationary values of the problem (15), and, if

h _x

1 2

−y

₂¹

i , . . . , h _x

_s

2

−y

₂^s

i

are the stationary points of the problem (15) with corresponding stationary values σ 1 , . . . , σ s , then

U 22 =

x

¹₂

kx

2¹

k · · · _kx ^x

²^ss 2

k

, V 22 =

y

¹₂

ky

¹2

k · · · _ky ^y

²^ss 2

k

.

• Argument 2. Secondly, define F=

 



x

−y

|x ∈ R ⁿ , y ∈ R ^p , U _a ^T x =



¯r _ac − r _c  0 r _a + r _c − ¯r _ac x 2

n − r _a 0



,

V _c ^T y =

 p − r _c  0 r _a + r _c − ¯r _ac y 2

¯r ac − r a 0



, x ^T A = y ^T C, x /= 0

 

 . Consider the optimization problem

max

x

−y

∈

F

kyk

kxk . (16)

Obviously, we have that _x

−y

is a stationary point of the problem (16) with sta-

tionary value σ if and only if _x

−y

22

is a stationary point of the problem (15) with the same stationary point σ and furthermore

U _a ^T x =



 



¯r ac − r c 0 r _a + r _c − ¯r _ac x 2

n − r _a

0 

 

, V _c ^T y =

 p − r c  0 r _a + r _c − ¯r _ac y 2

¯r _ac − r _a 0



.

• Argument 3. Finally, for any x ∈ R ⁿ , y ∈ R ^p , partition

U _a ^T x =



¯r _ac − r _c  x 1

r _a + r _c − ¯r _ac x 2

n − r a x 3



, V _c ^T y =

 p − r _c  y 1

r _a + r _c − ¯r _ac y 2

¯r ac − r a y 3



.

(8)

Since

U _a ^T T _∞ (A) =



¯r _ac − r _c  0 r _a + r _c − ¯r _ac 0

n − r _a I



,

V _c ^T T _∞ (C) =

 p − r _c  I r a + r c − ¯r ac 0

¯r ac − r a 0



, it is easy to know that _x

−y

∈ F if and only if



 A ^T C ^T

T ^T _∞ (A) 0 0 T ^T _∞ (C)





−y x

= 0, x /= 0.

Note that

U _a ^T U 2 =



¯r _ac − r _c  0 r _a + r _c − ¯r _ac U 22

n − r _a 0



, V _c ^T V 2 =

 p − r _c  0 r _a + r _c − ¯r _ac V 22

¯r _ac − r _a 0



, thus, Theorem 4 follows directly from the above Arguments 1–3.

3. A variational formulation for RSVD

In Section 2, we have derived the QSVD of two matrices A, C based on the condensed form (4). Now we will establish the RSVD of a matrix triplet (A, B, C) via an analogous condensed form.

Lemma 5. Given A ∈ R ^n×m , B ∈ R ^n×l , C ∈ R ^p×m . Then there exist orthogonal matrices P ∈ R ^n×n , Q ∈ R ^m×m , U _b ∈ R ^l×l , V _c ∈ R ^p×p such that

P AQ=



 



k 1 k 2 k 3 k 4 ¯r _ac − r _a m − ¯r _ac

k 1 A 11 A 12 0 0 0 0

k 2 0 A 22 0 0 0 0

k 3 A 31 A 32 A 33 A 34 0 0

k 4 A 41 A 42 0 A 44 0 0

r ab − r a 0 0 0 0 0 0

n − r ab 0 0 0 0 0 0



 



,

(9)

P BU _b =



 



l − r b k 3 k 4 r ab − r a

k 1 0 0 0 B 14

k 2 0 0 0 B 24

k 3 0 B 32 B 33 B 34

k 4 0 0 B 43 B 44

r ab − r a 0 0 0 B 54

n − r ab 0 0 0 0



 



, (17)

V _c ^T CQ=



 



k 1 k 2 k 3 k 4 ¯r _ac − r _a m − ¯r _ac

p − r _c 0 0 0 0 0 0

k 2 0 C 22 0 0 0 0

k 4 C 31 C 32 0 C 34 0 0

¯r _ac − r _a C 41 C 42 C 43 C 44 C 45 0



 

,

where A 11 , A 22 , A 33 , A 44 , B 32 , B 43 , B 54 , C 22 , C 34 and C 45 are non-singular.

Proof. See Appendix B. Let the SVD of B ₄₃ ⁻¹ A 44 C ₃₄ ⁻¹ be

U ₄₄ ^T B ₄₃ ⁻¹ A 44 C ₃₄ ⁻¹ V 44 = diag{σ 1 , . . . , σ _k

4

} =: S _ABC , (18) where U 44 , V 44 are orthogonal matrices, σ 1 > σ 2 > · · · > σ k

4

> 0. Define

U :=U _b



 

 I _l−r

_b

I _k

₃

U 44

I _r

_ab

_−r

_a



 

 , (19)

V :=V _c



 

 I _p−r

_c

I k

2

V 44

I ¯r

ac

−r

a



 

 . (20)

Similarly to Theorem 3, from Lemma 5 directly, we have the following theorem.

Theorem 6 (RSVD theorem). Given A ∈ R ^n×m , B ∈ R ^n×l , C ∈ R ^p×m . Then there

exist non-singular matrices X ∈ R ^n×n , Y ∈ R ^m×m and orthogonal matrices U ∈

R ^l×l , V ∈ R ^p×p such that

(10)

X ^T AY =



 



k 1 k 2 k 3 k 4 ¯r _ac − r _a m − ¯r _ac

k 1 I 0 0 0 0 0

k 2 0 I 0 0 0 0

k 3 0 0 I 0 0 0

k 4 0 0 0 S ABC 0 0

r ab − r a 0 0 0 0 0 0

n − r ab 0 0 0 0 0 0



 

 ,

X ^T BU =



 



l − r b k 3 k 4 r ab − r a

k 1 0 0 0 0

k 2 0 0 0 0

k 3 0 I 0 0

k 4 0 0 I 0

r _ab − r _a 0 0 0 I

n − r _ab 0 0 0 0



 



, (21)

V ^T CY =



 



k 1 k 2 k 3 k 4 ¯r _ac − r _a m − ¯r _ac

p − r _c 0 0 0 0 0 0

k 2 0 I 0 0 0 0

k 4 0 0 0 I 0 0

¯r _ac − r _a 0 0 0 0 I 0



 

,

where S _ABC is of the form (18), and U, V can be chosen to be given by (19) and (20), respectively, σ 1 , . . . , σ _k

₄

are defined to be the non-trivial restricted singular values of the matrix triplet A, B, C.

From the uniqueness theorem in [16], we only need to consider the matrices U, V given by (19) and (20) in order to characterize the role of the orthogonal matrices in the RSVD. Let U, V be defined by (19) and (20), respectively, and partition

U = l − r ^b k 3 k 4 r _ab − r _a

U 1 U 2 U 3 U 4

,

V = p − r ^c k 2 k 4 ¯r _ac − r _a

V 1 V 2 V 3 V 4

.

We have

U 1 =S _∞ (B), U 4 = S ^⊥ _∞ (T ^T _∞ (A)B), V 1 =T _∞ (C), V 4 = T ^⊥ _∞ (CS _∞ (A)).

Furthermore, if we define

W 1 :=CS _∞ (T ^T _∞ (B)A),

(11)

W 2 :=AS _∞ (T ^T _∞ (B)A), W 3 :=(T ∞ (W 2 S ∞ (W 1 ))) ^T B, then,

W 1 =V c ×



 



k 3 k 4 ¯r ac − r a m − ¯r ac

p − r c 0 0 0 0

k 2 0 0 0 0

k 4 0 C 34 0 0

¯r _ac − r _a C 43 C 44 C 45 0



 

,

W 2 =P ^T ×



 



k 3 k 4 ¯r ac − r a m − ¯r ac

k 1 0 0 0 0

k 2 0 0 0 0

k 3 A 33 A 34 0 0

k 4 0 A 44 0 0

r _ab − r _a 0 0 0 0

n − r _ab 0 0 0 0



 

 ,

W 3 =



 



l − r b k 3 k 4 r ab − r a

k 1 0 0 0 B 14

k 2 0 0 0 B 24

k 4 0 0 B 43 B 44

r ab − r a 0 0 0 B 54

n − r _ab 0 0 0 0



 

 U _b ^T . (22)

Hence, from (22) we have

U 1 U 2

= S ∞ (W 3 ).

Thus, in order to characterize the role of orthogonal matrices U, V in the RSVD we only need to characterize the role of U 3 , V 3 of U, V in the RSVD. This can be done by the following variational formulation.

Theorem 7. Given matrices A ∈ R ^n×m , B ∈ R ^n×l , C ∈ R ^p×m . Consider the optimization problem

max

x∈Rm, y∈Rl, x /=0







A B

ST

∞

A C

0 ST

∞(A)CT C 0

0 ST

∞(B)

0 TT

∞(B^S⊥∞(^W3))B







x

−y

= 0

kyk

kCxk . (23)

(12)

Then the stationary values for the problem (23) are precisely the non-trivial restricted singular values σ 1 , . . . , σ k

4

of the matrix triplet A, B, C. Moreover, if _x

−y

11

, . . . ,

_x

_k4

−y

_k4

are the stationary point of the problem (23) corresponding to the stationary values σ 1 , . . . , σ k

4

, respectively, then

U 3 = h _y

ky

11

k · · · _ky ^y

^k4_k4

_k i .

Proof. Same as the proof of Theorem 4, we prove part (a) by the following three arguments.

• Argument 1. Firstly, we characterize U 44 in (18). Consider the optimization problem

max

x

4

,y

3

∈

R^k4

,y

3

6=0 A

44

x

4

=B

43

y

3

ky 3 k

kC 34 x 4 k . (24)

Since A 44 , B 43 , C 34 are non-singular, so by Theorem 1, the stationary values of the problem (24) are precisely σ 1 , . . . , σ k

4

, i.e., the singular values of the matrix B ₄₃ ⁻¹ A 44 C ₃₄ ⁻¹ , and, if let the corresponding stationary points be _x

¹

−y

4₃¹

, . . . , _x

^k4₄

−y

₃^k4

,

then U 44 =

y

¹₃

ky

3¹

k · · · ^y

³^k4

ky

₃^k4

k

.

• Argument 2. Secondly, define

F :=

 

 

 

 



x

−y

kx ∈ R ^m , y ∈ R ^l , U _b ^T y =



 

 l − r _b 0

k 3 0

k 4 y 3

r _ab − r _a 0



 

, Q ^T x =



 



k 1 0

k 2 0

k 3 x 3

k 4 x 4

¯r _ac − r _a x 5 m − ¯r _ac 0



 

 ,

C 43 x 3 + C 44 x 4 + C 45 x 5 = 0, Ax = By, y /= 0

 

 

 

 

 .

Consider the optimization problem

max _x

−y

∈

F

kyk

kCxk . (25)

Since A 33 , A 44 , B 43 and C 45 are non-singular, so a simple calculation yields that

the problem (25) are equivalent to the problem (24) in the sense that the stationary

(13)

values of the problem (25) are precisely the stationary values of the problem (24), i.e., σ 1 , . . . , σ k

4

, and, _x

−y

is the stationary point of the problem (25) if and only if

_x

−y

43

is the stationary point of the problem (24) with same stationary value.

• Argument 3. Thirdly, for any x ∈ R ^m , y ∈ R ^l , denote

U _b ^T y =



 

 l − r _b y 1

k 3 y 2

k 4 y 3

r _ab − r _a y 4



 

, Q ^T x =



 



k 1 x 1

k 2 x 2

k 3 x 3

k 4 x 4

¯r _ac − r _a x 5

m − ¯r _ac x 6



 

 .

Since S ^T _∞

A C

x = 0 ⇐⇒ x 6 = 0,

Ax = By H⇒ x 1 = 0, x 2 = 0, y 4 = 0,

S ^T _∞ (A)C ^T Cx = 0 ⇐⇒ C 41 x 1 + C 42 x 2 + C 43 x 3 + C 44 x 4 + C 45 x 5 = 0, S ^T _∞ (B)y = 0 ⇐⇒ y 1 = 0.

From (22), we also know

T ^T _∞ (BS ^⊥ _∞ (W))By = 0 ⇐⇒ y 2 = 0.

Therefore, we have that _x

−y

∈ F if and only if



 

 

A B

S ^T _∞

A C

0 S ^T _∞ (A)C ^T C 0

0 S ^T _∞ (B)

0 T ^T _∞ (BS ^⊥ _∞ (W 3 ))B



 

 

x

−y

= 0, y /= 0.

Note that

U _b ^T U 3 =



 

 l − r _b 0

k 3 0

k 4 U 44

r _ab − r _a 0



 

,

so, Theorem 7 follows directly from the above Arguments 1–3.

(14)

Similarily, we also have the dual result of Theorem 7 which characterizes the non- trivial generalized singular values σ 1 , . . . , σ _k

₄

and the matrix V 3 in (20). For the sake of simplicity, we omit it here.

4. Conclusion

In this paper, we have studied generalized SVDs. We have given an alternative proof of the variational formulation for the QSVD in [1] and established an analogous variational formulation for the RSVD which provides a new understanding of the orthogonal matrices appearing in this decomposition.

The proofs of our main results (i.e., Theorems 4 and 7) provide deflation proce- dures for the numerical computation of two orthogonal matrices in the QSVD and RSVD. For example, we can compute the non-trivial restricted singular values and orthogonal matrix U in the RSVD (21) as follows:

• Compute the condensed form (17) by QR factorizations.

• Apply existing deflation process [4] to the optimization problem (24) to get the non-trivial restricted singular values σ 1 , . . . , σ _k

4

in (21) and the corresponding orthogonal matrix U 44 . The computational complexity of the optimization problem (24) is much lower than the variational formulation (23). This is one advantage of our deflation procedure.

• Compute the orthogonal matrix U by (19).

Acknowledgements

Some of Delin Chu’s work was done during his visit in the Department of Math- ematics at The University of Bielefeld in Germany in April 1998. He is grateful to Prof. L. Elsner for his kind hospitality and financial support. He also thanks Prof. L.

Elsner for his reading and revising the present paper.

We also wish to thank an anonymous referee for the valuable suggestions that significantly improved this paper.

Appendix A

In this appendix, we prove Lemma 2 constructively.

Proof. We prove Lemma 2 by four steps as follows:

Step 1: Perform simulaneous row and column compression:

U ₁ ^T AW 1 =

r ^a ¯r ac − r a m − ¯r ac

r a A ⁽¹⁾ ₁₁ 0 0

n − r a 0 0 0

,

(15)

V 1 ^T CW 1 =



 

r a ¯r ac − r a m − ¯r ac

p − r _c 0 0 0

r a + r c − ¯r ac C ₂₁ ⁽¹⁾ 0 0

¯r _ac − r _a C ₃₁ ⁽¹⁾ C 33 0



 

with A ⁽¹⁾ ₁₁ , C 33 non-singular and C ₂₁ ⁽¹⁾ full row rank.

Step 2: Perform a column compression:

C ₂₁ ⁽¹⁾ W 2 = ¯r ^ac − r _c r _a + r _c − ¯r _ac

0 C 22

with C 22 non-singular. Set

A ⁽¹⁾ ₁₁ W 2 = ¯r ^ac − r _c r _a + r _c − ¯r _ac A ⁽²⁾ ₁₁ A ⁽²⁾ ₁₂

,

C ₃₁ ⁽¹⁾ W 2 = ¯r ^ac − r _c r _a + r _c − ¯r _ac

C 31 C 32

.

Step 3: Perform a row compression:

U ₃ ^T A ⁽²⁾ ₁₁ = ¯r ^ac − r _c r _a + r _c − ¯r _ac

A 11

0 with A 11 non-singular. Set U ₃ ^T A ⁽²⁾ ₁₂ = ¯r ^ac − r _c

r _a + r _c − ¯r _ac

A 12

A 22 .

Step 4: Set

U _a = U 1

U 3

I

, W = W 1

W 2

I

, V _c = V 1 .

Then orthogonal matrices U a , V c and W satisfy (4).

Appendix B

Now we prove Lemma 5 constructively.

Proof. We prove Lemma 5 by five steps as follows:

Step 1: Compute orthogonal matrices P 1 , Q 1 and U 1 by applying Lemma 2 to

(A ^T , B ^T ) such that

(16)

P 1 AQ 1 =



 



r ab − r b r a + r b − r ab m − r a

r ab − r b H 11 0 0

r a + r b − r ab H 21 H 22 0

r ab − r a 0 0 0

n − r ab 0 0 0



 

,

P 1 BU 1 =



 



l − r _b r _a + r _b − r _ab r _ab − r _a

r _ab − r _b 0 0 U 14

r _a + r _b − r _ab 0 U 32 U 34

r _ab − r _a 0 0 B 54

n − r _ab 0 0 0



 



with H 11 , H 22 , U 32 and B 54 non-singular. Set

CQ 1 = r ^ab − r b r a + r b − r ab m − r a

C ₁ ⁽¹⁾ C ₂ ⁽¹⁾ C ⁽¹⁾ ₃ .

Step 2: Compute orthogonal matrices P 2 , Q 2 and V 2 based on Lemma 2 such that

P 2

H 22 0 Q 2 =

k ³ k 4 ¯r _ac − r _a m − ¯r _ac

k 3 A 33 A 34 0 0

k 4 0 A 44 0 0

,

V 2

h C ₂ ⁽¹⁾ C ₃ ⁽¹⁾ i Q 2 =





k 3 k 4 ¯r ac − r a m − ¯r ac

p − k 4 − ¯r ac − r a 0 0 0 0

k 4 0 C 34 0 0

¯r ac − r a C 43 C 44 C 45 0





with A 33 , A 44 and C 34 , C 45 non-singular. Set

V 2 C ₁ ⁽¹⁾ = p − k 4 − ¯r ac − r a

k 4

¯r ac − r a



 C ₁₁ ⁽²⁾ C ₃₁ ⁽²⁾ C ₄₁ ⁽²⁾ .





Step 3: Perform a simultaneous row and column compression:

V 3 C ₁₁ ⁽²⁾ Q 3 =

k ¹ k 2

p − r c 0 0 k 2 0 C 22

with C 22 non-singular. Set

"

C ₃₁ ⁽²⁾ C ₄₁ ⁽²⁾

# Q 3 =

k ¹ k 2

k 4 C 31 C 32

¯r _ac − r _a C 41 C 42

.

Step 4: Perform a row compression and a column compression:

(17)

P 4 H 11 Q 3 =

k ¹ k 2

k 1 A 11 A 12

k 2 0 A 22

, P 2 U 32 U 4 =

k ³ k 4

k 3 B 32 B 33

k 4 0 B 43

,

where A 11 , A 22 and B 32 , B 43 are non-singular. Set

P 2 H 21 Q 4 =

k ¹ k 2

k 3 A 31 A 32

k 4 A 41 A 42

,

P 4 U 14

P 2 U 34

=



 

 k 1 B 14

k 2 B 24

k 3 B 34

k 4 B 44



 

.

Step 5: Set

P =

P 4

I

  I _r

_ab

_−r

_b

P 2

I _n−r

_a



 P 1 , Q=Q 1

I Q 2

Q 3

I

,

U b =U 1



 I l−r

b

U 4

I r

ab

−r

a



 , V _c ^T =

V 3

I

V 2 .

Then, the orthogonal matrices P, Q, U _b and V _c satisfy the condensed form (17).

References

[1] M.T. Chu, R.E. Funderlic, G.H. Golub, On a variational formulation of the generalized singular value decomposition, SIAM J. Matrix Anal. Appl. 18 (1997) 1082–1092.

[2] E. Anderson, Z. Bai, C. Bischof, J.W. Demmel, J. Dongarra, J.D. Croz, A. Greenbaum, S. Hammar- ling, A. McKenney, S. Ostrouchov, D. Sorensen, LAPACK Users’ Guide. Society for Industrial and Applied Mathematics, Philadelphia, 1992.

[3] Z. Bai, J.W. Demmel, Computing the generalized singular value decomposition, SIAM J. Sci. Com- put. 14 (1993) 1464–1486.

[4] G.H. Golub, C.F. Van Loan, Matrix Computations, third ed., John Hopkins University Press, Balti- more, MD, 1996.

[5] C.C. Paige, M.A. Saunders, Towards a generalized singular value decomposition, SIAM J. Numer.

Anal. 18 (1981) 398–405.

[6] C.C. Paige, Computing the generalized singular value decomposition, SIAM J. Sci. Stat. Comput. 7 (1986) 1126–1146.

[7] B. De Moor, On the structure and geometry of the product singular value decomposition, Linear Algebra Appl. 168 (1992) 95–136.

[8] B. De Moor, G.H. Golub, The restricted singular value decomposition: properties and applications,

SIAM J. Matrix Anal. Appl. 12 (1991) 401–425.

(18)

[9] B. De Moor, On the structure of generalized singular value and QR decompositions, SIAM J. Matrix Anal. Appl. 15 (1994) 347–358.

[10] B. De Moor, P. Van Dooren, Generalizations of the QR and singular value decomposition, SIAM J.

Matrix Anal. Appl. 13 (1992) 993–1014.

[11] B. De Moor, G.H. Golub, Generalized singular value decompositions: a proposal for a standardized nomenclature. Internal Report 89-10, ESAT-SISTA, Leuven, Belgium, 1989.

[12] G.W. Stewart, Computing the CS-decomposition of a partitioned orthogonal matrix, Numer. Math.

40 (1982) 297–306.

[13] C.F. Van Loan, Generalizing the singular value decomposition, SIAM J. Numer. Anal. 13 (1976) 76–83.

[14] C.F. Van Loan, Computing the CS and the generalized singular value decomposition, Numer. Math.

46 (1985) 479–491.

[15] H. Zha, A numerical algorithm for computing the restricted singular value decomposition of matrix triplets, Linear Algebra Appl. 168 (1992) 1–25.

[16] H. Zha, The restricted singular value decomposition of matrix triplets, SIAM J. Matrix Anal. Appl.

12 (1991) 172–194.

On a variational formulation of the QSVD and the RSVD

www.elsevier.com/locate/laa