An Extension of the Generalized SVD for More than
Two Matrices
Lieven De Lathauwer Katholieke Universiteit Leuven
Group Science, Engineering and Technology, 8500 Kortrijk, Belgium and E.E. Dept. (ESAT) - SCD - SISTA, B-3001 Leuven, Belgium
Email: Lieven.DeLathauwer@kuleuven-kortrijk.be and Lieven.DeLathauwer@esat.kuleuven.be
Problem description
Given A1 ∈ RI1×J
, . . . , AK ∈ RIK ×J
, with Ik ≫ J for all k, find an
approx-imate decomposition of the type
A1 = U1· D1· MT .. . AK = UK· DK· MT, in which Uk ∈ RIk ×J
is column-wise orthonormal, Dk ∈ RJ ×J diagonal and
M ∈ RJ ×J not necessarily orthogonal (1 ≤ k ≤ K). For K = 2, this corre-sponds to the Generalized SVD. For K > 2, the problem is overdetermined (more equations than unknowns). We will solve it in least-squares sense, i.e., we will minimize f = K X k=1 kAk− Uk· Dk· MTk2, (1)
in which the norm is the Frobenius-norm.
Alternating Least Squares Algorithm
We can compute the solution by means of an Alternating Least Squares algorithm (ALS):
1. Update {Uk} for given estimates of {Dk} and M.
2. Update {Dk} and M for given estimates of {Uk}.
3. Go back to 1 until convergence.
Convergence to a local optimum is guaranteed, because in each step we will further decrease the cost function f . It may be necessary to reinitialize a number of times in order to find the global optimum. ALS is a very basic approach in comparison with the advanced techniques in current numerical linear algebra (for instance for the computation of the GSVD). However, it is quite common in multilinear algebra, where numerical aspects are still largely unexplored.
Updating {Dk} and M.
Since Uk are column-wise orthonormal, we have
f = K X k=1 kUT kAk− Dk· MTk2.
Call Bj ∈ RK×J the matrix in which the jth rows of UTkAk, 1 ≤ k ≤ K,
are stacked. Define Ej = (D1(j, j) . . . DK(j, j))T and let Mj denote the jth
column of M, 1 ≤ j ≤ J. Then we have f =
J
X
j=1
kBj − EjMjTk2.
This corresponds to a set of decoupled best rank-1 approximation problems. Ej is the dominant left singular vector of Bj, and Mj is the dominant right
singular vector times the dominant singular value of Bj, for all j. Actually it
does not matter which of the two vectors is multiplied by the singular value, since a scalar can be exchanged between Ej and Mj.
Updating {Uk}.
If h·, ·i denotes the inner product, then we have
f = K X k=1 hAk− Uk· Dk· MT, Ak− Uk· Dk· MTi = K X k=1 hAk, Aki − hAk, Uk· Dk· MTi − hUk· Dk· MT, Aki + hUk· Dk· MT, Uk· Dk· MTi = K X k=1 kAkk2+ kUk· Dk· MTk2− 2hUk· Dk· MT, Aki
Since Uk is column-wise orthonormal, kUk· Dk· MTk2 = kDk· MTk2 does
not depend on Uk. Hence, we have
f = c − 2 K X k=1 hUk· Dk· MT, Aki = c − 2 K X k=1 hUk, AkMDki
Up to a constant, the same expression would have been obtained if we had started from g = K X k=1 kAkMDk− Ukk2 (2)
The latter equation corresponds to a set of decoupled Procrustes problems (find the best column-wise orthonormal approximation of a given matrix) [1]. If the SVD of AkMDk is given by PkΣkQTk, then the optimal Uk is equal to
PkQTk, for all k. (Σk contains only the nonzero singular values of AkMDk;
the dimensions of Pk and Qk are defined accordingly.)
Note that the columns of the optimal Uk always lie in the column space of
AkMDk, hence in the column space of Ak. This means that prior to the
ALS algorithm, we may reduce the dimensions by projecting on the column space of Ak:
1. Compute the K SVD’s
Ak = ˜Uk· ˜Sk· ˜VkT
2. Substitute
Ak ← ˜UTkAk Uk ← ˜UTkUk
3. ALS for square matrices {Ak}, {Uk}, {Dk}, M.
4. Substitute Uk← ˜UkUk.
Acknowledgments
Research supported by: (1) Research Council K.U.Leuven: GOA-Ambiorics, GOA-MaNet, CoE EF/05/006 Optimization in Engineering (OPTEC), CIF1, STRT1/08/023, (2) F.W.O.: (a) project G.0321.06, (b) Research Commu-nities ICCoS, ANMMM and MLDM, (3) the Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, “Dynamical systems, control and optimiza-tion”, 2007–2011), (4) EU: ERNSI.
References
[1] Golub, G.H. and Van Loan, C.F. 1996. Matrix Computations, Baltimore, Maryland: Johns Hopkins University Press.
