Tilburg University
The classical multivariate regression model with singular covariance matrix
Neeleman, D.
Publication date:
1971
Document Version
Publisher's PDF, also known as Version of record
Link to publication in Tilburg University Research Portal
Citation for published version (APA):
Neeleman, D. (1971). The classical multivariate regression model with singular covariance matrix. (EIT
Research Memorandum). Stichting Economisch Instituut Tilburg.
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners
and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.
• You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal
Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.
~
'93
Bsitsma~in
9 SIf3L14THLEi~
~J~H~ENBUREAU
11Ts.
~CA'i~ ~~~L; t:~;E
~
~30GESCH(~OL
~
- --
TIhBURD
D. Neeleman
The classical multivariate
regression model with singular
covariance matrix
iiimiiiNhiiiiMiiiuiiimuiuiuimim
Research memorandum
-T- ~, ~~~v~a.~~.: ~e.
T~ ~ irr~~l,-.~~ ~
TILBURG INSTITUTE OF ECONOMICS
~~~'K.U.E3.
BIBLIC~THEEK
1. Introduction
In several publications the best lineax unbiased estimators in the clas-sical multivariate regression model are derived for the case of a non-singular covariance matrix. See for instance Anderson 1959~ Goldberger
1964 and Kendall and Stewart vol. III i966.
It is well known that in that case the least squares estimators are best linear unbiased.
With the help of the theory of estimable functions (section 2) and a theorem proved by Rao i968 (section 3) it is demonstrated in this paper that even if the covariance matrix is singular the least squares estima-tors are still best linear unbiased.
~ ~~
2
-2. The best linear unbiased estimators of estimable functions
Consider the single equation regression model,
y - Xb t e
where:
- y is a T x 1 vector of observations on the dependent variable - X is a T x k matrix of observations on the explanatory
varia-bles, with fixed elements and rank k(k ~ T) - b is a k x 1 vector of unknown parameters - E is a T x 1 vector of random disturbances
It is assumed that:
- E(E) - 0 (2)
- E(E E') - Si (3)
- Rank S2 - r ~ T ( 4)
In this paper we use the following definitions
Definition 1
A parameter function p'b is called estimable if there exists a linear function 1'y such that E(1'y) - p'b
Definition 2
3
-Using definition 1 the following theorem is easy to prove.
Theorem 1
All linear parameter functions are estimable, if and only if R(X) - k, where R(X) denotes the rank of X.
Proof:
If R(X') - k, then for every p there exists an 1 such that X'1 - p. If p'b is estimable, then there exists an 1 such that X'1 - p and p be-longs to the manifold spanned by the colur.uis of X'.
If p is arbitrary R(X') must be k.
The rest cf this section is divided into twc parts. Part A 3eals with the case R(S2) - T and is.based on Rao 1965.
Part B deals with the case R(S2) ~ T and is based on Mitra and Rao 1968. Part A
Theorem 2
,~ ~ ~ -~ r : ". '~
The nLUE of p'; is - y with 1-.~ X(X .~1 p
Proof :
The :roblem is one of seeking the minimum of the quadratic form 1' S2 1- 1' CC' 1
when 1 is subject to the consistent linear restrictions
X'1 - p
It is easy to see that
X'1} - P
(5)
(6)
(7)
b
To prove that 1~ y has minimum variance within the class of unbiased es-timators we define the vectors
v1 - C' 52-1 X(X' R-1X)-1 p
v2 - C'1 with 1 satisfying X'1 - p
Applying the Cauchy-Schwarz inequality to these vectors we find
(v2 v1)2 v2 v2 ' v1 vl or
(8)
(9)
1' S2 1~ L'x(x' ~-1x)-1 p12 - p'(x' ~-1x)-1 p- p~(X~ ~-1X)-1p (10) which is equivalent toV(i'y) ? V(1~~y)
,
In vieuw of theorem 1 it is easy to show that the BLUE of b is
ó- (X' S2 1X)-1 X' S2 1y
by taking for p the various unit vectors.
(12)
As a special case one finds that for S2 - IT the ordinary least squares estimators result
~ - (X'X)-1 X'y
Part B
Theorem 3
r
The BLUE of p'b is lt y with 1~ -(R-XC~ t NC2)p where:
N is a matrix of maximum rank such that
S2 N- 0 and N' N- I
6
F~ 1' (N~ ) e e' (F N)1] ~ 1~1~
So that the problem is one of seeking the minimum of
r
1~1~
when 1 is subject to the consistent linear restrictions
1 (X'F X'N) (1~) - P 2 (2~) (22)
(23)
The condition (23) implies that there exist vectors m and n such that
X' RXmt X'N n-p (24)
N'X m - 0
Then it is easy to verify that
1~ - F'X C~ p , 12 - C2 p
satisfies the condition (23) using the equations:
(25)
X' f2XC ~ X' Sl X t X' NC3X' Si X t X't2 X C2N' X t X' NC~N' X~ X' f2 X
(26)
N'XC~X'S2 X t N'XC2N'X - N'X
which are derived from the properties of the generalised inverse. It is
elso simple to derive that ~ 1~ 1~ - P'C~ X' S2 XC~ P: P'C~ P
Furthermore
(27)
r 1~ 1~ - P~ C~X'F1~ - P~C~p - P~C~ X'N12 - P~C~P (28)Cauchy-Schwarz inequality to the vectors '~ and lí we find (1~1 lí)2 ~ lí lí ~ ~, ~ lí lí l~lí ~ p'Cí p which is equivalent to -v( (l~F' t i2~V)y~ ' v~ (1~~F' t where l~ -( 2-X C~ t NC2)P
Because of theorem í the BLUE of b is
b - (CíX'2- t C2N') y
(29)
As a special case one finds for R(~2) - T
c - (X' s~-íX)-í X' S2-í ~ (33)
w~iich is the ~en~raiised least squares estimator of b.
3. A theor-~m ~-. `n - -- ~ir:ear ~lr.biase3 a-:d simple least sauares estimators
Before we state ~he f~in~amental theorem of this sectior. we give a weli known result frcm `.he theory on minimum variance estimation.'
ineorem 1
8
-and V(s) ~~ provided V(t) ~ m.
Proof:
Necessity:
Consider t t a s for arbitrary . E(t t a s) - E(t)
v(t t a s) - v(t) t 2 ~ c(t, s) t a2 v(s)
2c(t 5) Now V(t t a s) - V(t) for any a in the interval (0, - V(s) )
U.nless C(t, s) - 0. -
-Sufficiency:
Let s be another unbiased estimator such that V(s) ~~. Ther, E(t - s) - 0. Further by the condition gives above it follows that
-r'.[ t ( t - s ) ] - 0 that is V(t-) - C(t, s) with [ C(t,s)] 2 (2)
(3)
V(t)V(s) (2) gives V(t) - P2 V(s) ~ V(s) (k)The followiag theorem was first proved by Rao 1967 for non-singular matrices X'X and S2. T::e same thecrem holds more generall~r for si-ngular X'X and S?
as Rao proved in 1968.
Theorem 2
-9-estimator is that X` S2 Z s 0 where Z is a matrix of maximum rank such
that Z`X ~ 0
Proof-Let N be a matrix of maximum rank such that S2 N- 0. Then
E(N' y) - N' Xb (5)
and
E(N'(y - Xb)(y. - Xb)'N] - N' S2 N- 0 (6)
so that N'y is a vector of constants say c and the equations
N'Xb - c (7)
may be considered as restrictions on the unknown parameters b.
A necessary and sufficient condition that a linear function 1'y, has a constant expectation independent of b but subject to N'Xb - c is that X'1 belor,gs to the linear manifold of X'N, that is to say there exists a vector such that
X'1 - X'N m
So 1'y can then be written as
1'y - (1'y - m'N'y) t m'N'y
where
E(1'y - m'N'y) - 0
independent of the restrictions, that is to say
1 - Nm - Zd
(8)
(9)
10
-for scme i.
Now it is wel' kncwn that the simp~e least square:; estimator of an esti-mable fsncti~r. r'b is cf the ferm !'X'y.
in vieuw cf t:eore~: 1 of this section a necessary and sufficient condi-tion t'r.at ~'::y is a BLUE is that
C(;~};';i~ ï~~; - C~~'X'y~ 1'~ - m'N'y)
- i'X' :; ~d - 0 ( i2)
-- !~~) is .rue .,,r a11 ~ and d t-.en
X' :~.Z-~
(13)
~t!~.er ~:..~.~~,-~- and s1"ficier.t conditiora - 'or equivalence of best linear
;r,c:-.,,~. ,.. --.. . -;r~rrs -sti:n~-:rs are - i;:,, ~:' t ZP.2Z' ( Rao 1968 )
S2 - XB~X' t ZB2Z' } c2 I(Rao 1968)
where A1, A2, B1, B2 are arbitrary symmetric matrices, and a2 is an
arbi-trary scalar.
-There exists a subset of r(- rank X) orthogonal eigenvectors of S2 which forms a basis of the space spanned by the coloumns of X
(Zyskind 1967) The necessary and sufficient condition is Si X- XB for some B
(Kruskal 1968) 4. best 'inear unbiased estimaticn i- the multivariate classical linear
regression moàel
11
-yt - B'xt t~ t- 1,2... T
where
(1)
- y,t is a m x 1 vector of observations on the independent va-riables at time t
- xt is a k x 1 vector of observations on the k explanatory variables at time t. The vectors of explanatory variables (x11' x12....x1T)~ ... (xk' xk2~... xkT)
are linearly independent.
- B' is an m x k matrix of unknown parameters
-~ is an m x 1 vector of random disturbances
It is assumed that
- E(~) - 0
S2 (t - 1 )
- E(~ ~ ) - {0 (t ~ s )
- Rank S2 - r ~ m
Using the Kronecker product of matrices the model can be written as
vec Y-(Im ~ X) vec B t vec e
where Y -~1I ~l' f X - E c (2) E'I
:
(3)
~~2
-The model as written in (2) and (3) is the singie equation regression model of section 2 with r.(e e') -(Z ~ IT).
It is well known that the simple least squares estimators of vec B are BLUE of the rank of ~~ is m, as can be easily established by applying (12) of section 2. For
vec 3 - j ( im ~ X' ) ( ~- ~ ~ -,I, ) ( Im ~ X ) j - ~ ( Im ~ X' ) ( S2-~ ~ IT ) vec Y
-(?-~ ~ X'X)-~ (;.- ~ IT) vec Y-(Im ~(X'X)-~X') vec Y (4)or
3 - (X'X)-~X'Y (5)
With t'r.e aid of theorem 2 of section 3 it is easy to show that the simple least souares estir.:ators are slso BLU"r'. if the rank of SZ is smaller than m. :,et Z be a matrix. .,: r:a:{irr:.un rank such that
X'Z - 0
~hen aiso
(Im ~ X')(Im ~ Z) - (Im ~ X'Z) - 0 ar.d
(Im ~ x')(2 ~ IT)(Im ~ z) - (s~ ~ x'z) - o
(6)
(7)
(8)
5. Refereaces
1. Anderson, T.W. (1959): An Introductior. to Multivariate Statistical Analysis. Wiley, New York.
2. Goldberger, A.S. (1964): Exonometric Theory. Wiley, New York.
3. Kendall, M.G. and A.S. Stuart (1966): The Advanceà Theory of Statis-tics, Vol. III. Griffir., London.
4. Kruskal, W. (1968): When are Gauss-Markoff and least squares estima-tors identical? A coordinate free approach. The Annals of Mathematical
Statistics vol. 39, :p. 70 - 75.
5. Mitra, S.K. and C.R. í~ao ( 1968): Some resuïts ir, estimation and tests of linear hypotheses under the Gauss-Markoff mc3e-. Sar.khya A vol. 30 PP. 281 - 290.
6. Neudecker,H. (1968): Tre Kror.ecker Matrix Product and some cf its appli-cations in econometrics. Statistica Neerlandica, ~rol. 22 nc. ' 1968.
7. Rao C.R. (1965): Linear Statistical Inference and Its Applications. Wiley, .~dew York.
8. Rao C.R. (1967): Least squares using an estimated dispersior. matrix and its applications tc measurement of signals. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics andProbability
vol. 1 p. 263 - 280. Wi~ey, ,dew York.
9. Rao C.R. (1968): A note on a previous lemma ir. the thecry of least squares and some further results. Sankhya A vol. 30 pp. 259 - 266.
10. Zyskind G. (' r7): Cn canomical forms, negative covariance matrices
