A C T A U N I V E R S I T A T I S L O D Z I E N S I S FOLIA OECONOMICA 152, 2000 A n d r z e j G r z y b o w s k i * O N U N C E R T A IN T Y C L A S S E S A N D M IN IM A X E S T IM A T IO N IN T H E L IN E A R R E G R E S S IO N M O D E L S W IT H H E T E R O S C E D A S T IC IT Y A N D C O R R E L A T E D E R R O R S 1
Abstract. The problem of minimax estimation in the linear regression model is considered under the assumption that a prior information about the regression parameter and the covariance matrix of random component (error) is available for the decision-maker. Two models of the uncertainty of the prior knowledge (so called uncertainly classes) are proposed. The first one may represent the problem of estimation for heteroscedastic model, the other may reflect the uncertainty connected with the presence of the correlation among errors. Minimax estimators for considered classes are obtained. Some numerical examples are discussed as well.
L INTRODUCTION
Let us consider the o rd in ary linear regression m odel
Y = X/f + Z (1)
w here Y is an n-dim ensional vector o f observ ation s o f th e d ep e n d en t variable, X is a given n o nstoch astic (n x к) m atrix w ith the ra n k k , ß is a /с-dim ensional vector o f unknow n regression coefficients, Z is an n- -dim ensional vector o f ran d o m erro rs (ran d o m co m pon ents o f th e m odel). W e assum e E(Y ) = X ß and cov (Y) = E.
V arious papers deal with the problem o f the regression estim atio n in the presence o f p rio r know ledge ab o u t the p aram eter ß . Som e o f them study the problem s where the prio r inform ation is o f the form o f a restricted
* Technical University of Częstochowa, Chair of Applied Mathematics. 1 Support by KBN. Grant No. 1 H02B 013 15 is gratefully acknowledged.
p a ram eter space (see e.g.: D r y g a s , 1996; D r y g a s , P i l z , 1996; G i r к о , 1996; H o f f m a n , 1996; P i l z , 1996). O th er papers arc devoted to the problem s where the prio r inform ation is expressed in term s o f th e probability d istrib u tio n o f the p aram eter ß. In such a case the d istrib u tio n o f the p aram eter is often assum ed to belong to a given class o f d istrib u tio n s, (see: B e r g e r , 1982; B e r g e r , C h e n , 1987; B e r g e r , 1990; G r z y b o w s k i , 1997; V e r d u , P o o r , 1984). T h is class m odels the p rio r know ledge as well as its uncertainty, so we call it an uncertainty class, see V e r d u , P o o r (1984). Som etim es this ap p ro ach leads to gam e th eoretic fo rm u latio n o f the original decision problem (see: V e r d u , P o o r , 1984; G r z y b o w s k i , 1997). In this p aper we a d o p t the latter ap p ro ach . We in tro d u ce tw o u ncertainty classes reflecting the uncertainty in tw o co m m o n situ atio n s. Section 3 is concerned with th e regression estim atio n in heteroscedastic m odels. Section 4 deals with the problem s associated w ith th e presence o f co rrelatio n between errors. In each case we solve the gam e connected with the introduced uncertain ty class, i.e. we find m inim ax estim ato rs and the least favourable states o f natu re. Some num erical exam ples are also presented to illustrate im p o rta n t features o f the obtained solutions.
2. PRELIMINARY DEFINITIONS AND NOTATION
Let L (. , .) be a q u ad ratic loss function, i.e. L ( ß , a) = (ß — a)TH ( ß — a), fo r a given nonnegative definite (k x k) m atrix II. F o r a given estim ato r d the risk function R ( . , .) is defined by R (ß, d) = E pL ( ß , d(Y)). L et the p aram eter ß have a d istrib u tio n n. T h e Bayes risk r(. , .) co nnected w ith the estim ato r d is defined as usual: r(n, d) = E nR(Jl, d). Let D d en o te the set o f all allow able for the decision m aker estim ators. W e assum e th a t all o f them have the Bayes risk finite. T he estim ato r d* e l ) w hich satisfies the co n d itio n
r(n , d#) = inf r(n, d)
deD
is called a Bayes estimator (with respect to n).
L et us co n sid er a gam e ( Г , D, r ) , w here Г is a given class o f d istrib u tio n s o f the p aram eter ß. A ny d istrib u tio n n* e I \ satisfying the co n d itio n
inf г(я*, d) = sup inf r (it, d)
del) леГ deD
is called the least favourable d istrib u tio n (state o f natu re). T h e m inim ax estimator is defined as the estim ato r d * e l ) w hich satisfies th e follow ing conditio n:
sup r (n , (1*) = inf sup r(n, d). IT e r d e l) п е Г
Som etim es the robustness o f estim ators is described in term s o f the supremum o f the Bayes risks, see e.g. B e r g e r (1982, 1990), V e r d u , P o o r (1984). 'I hen the above estim ato r is called minimax-robust.
V arious problem s o f m inim ax -ro b u st regression were discussed e.g. in B e r g e r (1982, 1990), B e r g e r , C h e n (1987), C h a t u r v e d i , S r i v a s - t a w a (1992), G r z y b o w s k i (1997), P i n e l i s (1991). T h e relation s betw een B ayesian analysis and m in im ax estim atio n w ere exam ined in B e r g e r , C h e n (1987), D r y g a s , P i l z (1996), H o f f m a n (1996), P i l z (1986). Review o f recent results on ro b u st Bayesian analysis an d in terestin g references can be found in M ę c z a r s k i (1998). V arious classes o f stable (ro b u st, m inim ax and oth er) estim ators are also discussed in M i l o (1995).
Let us consider the case w here o u r in fo rm atio n a b o u t the p a ra m e te r ß is described w ith the help o f the follow ing class r SiK o f d istrib u tio n s л:
r SiK = {n: E J i = 9, cov (//) = A e K c M*},
w here К is a given subset o f a space M k o f all positive defin ite (k x k) m atrices. I he p o in t 9, fixed th ro u g h o u t this pap er, can be th o u g h t o f as a p rio r guess for //, while the set К reflects o u r u n certain ty con nected with the guess. Let us assum e the covariance m atrix E o f th e ra n d o m d istu rb an ce Z belongs to a given subset Q o f the space M„. P roblem s o f m inim ax regression estim ation in the presence o f such a p rior in fo rm atio n a b o u t E were considered in G r z y b o w s k i (1997), H o f f m a n (1996), P i n e l i s (1991). T h e set G = K x Í2 is the uncertain ty class in o u r problem .
Let us co nsider the situ atio n when the set o f allow able estim ato rs L consists o f all affine linear estim ators d, i.e. estim ators hav ing th e form d(Y) = AY + B, with Л and В being a m atrix an d a vector o f th e a p p ro p riate dim ensions. T h e original problem o f estim ation o f the p a ram eter ß now can be trea ted as a gam e <G’, L, r ) . T he solution o f the gam e w as fou nd in G r z y b o w s k i (1997). It was proved th a t if the u n certainty class G is convex then any afTinc linear estim ato r d* which is Bayes w ith respect to the least favourable pair o j matrices is m inim ax -ro b u st (no te th a t the Bayes risk fo r affine linear estim ators is determ ined by the first tw o m o m en ts o f the p rio r d istrib u tio n ). T he least favourable m atrices A* and E* satisfy the conditio n:
tr(C(A*, E*)H = sup tr(C (A , E )H ) (2) (A, L)eG
w here C(A, E) = ( X ' E ~ LX -f A - *)” whi l e the estim ato r d* is given by the follow ing fam iliar form ula:
d*(A*, £*) = C(A*, Z*)X r (E *)_ ł Y + С(А Ф, (3) L ast year, d u rin g M SA ’97, we exam ined an un certain ty class w ith the sets К and Q defined as follows: K = {A: A = d l k, d e ( 0, d0]}, Í2 = {E: E = s ln, s e (0, s 0]}, with given real values d0, s0 an d I*, I„ being the identity m atrices in the spaces M k and M n, respectively, sec G r z y b o w s k i (1997), H o f f m a n (1996), P i n e l i s (1991). P ractically, the class m ay rep resen t the case where the uncertainties connected with each coefficient ß t arc independent and the same while the regression m odel satisfies two assum ptions: homoscedasticity and independence o f errors Z,.
In the sequel o f the p aper we p ro p o se u n certain ty classes rep resen ting the situ atio n when the u ncertainties connected w ith each coefficient o f regression p aram eter ß m ay be different and the ab o v e tw o assu m p tio n s a b o u t the regression m odel m ay n o t be satisfied.
F o r convenience we a d o p t th e follow ing n o tatio n . F o r any n-dim cnsional vectors a, b we w rite a > b if ai ^ h i , í = 1, n. W e w rite a > 0 if all co m p o n en ts o f the vector a are positive. F o r any m atrix A we w rite A > 0 (A > 0) if the m atrix is positive (nonnegative) definite. F o r any vector a the sym bol d iag(a) stands fo r a d iagon al m atrix with th e co m p o n en ts o f a on the m ain diagonal.
3. MINIMAX ESTIMATION IN HETEROSCEDASTIC MODELS
Let S e R K and o e R " be given vectors. Let G(ô, a) = K x Í2 d en o te an uncertainty class where the sets К and Ü are defined as follows: К = {A e M k: A = diag(d), 0 < d ^ < 5 } , Q = { Z e M n: E = diag(s), 0 < s < c r } . T h e class m ay represent the case w here the uncertainty connected with coefficients ß. o f th e regression p a ra m e te r are d ifferen t an d th e ra n d o m e rro rs Z i are in depen dent b u t they m ay have different stan d ard deviation. T h e follow ing p ro p o sitio n provides the m in im ax estim ators for such problem s.
P roposition 1. Let A s = diag(c5) and = diag(<x). T he e stim ato r d*(Ai; given by (3) is the m inim ax estim ato r for the gam e <G(<5, a), L, r. □ In view o f the above m entioned results, in ord er to prove the P ro p o sitio n it is sufficient to show th a t the tw o m atrices (A6, E J satisfy th e co n d itio n (2). F o r this p u rp o se we need the follow ing lem m as.
Lemm a 1. Let A = [fly]* „ * > 0 and H = [hu]k „ * > 0. Let A „ and H n be the subm atrices o f A and H , respectively, o btained by deleting the first row and the first colum n. T hen
t r í A ^ H J - t r í A f ^ H n ^ O (4)
L et us w rite dow n the m atrix A in the follow ing form :
O ne can verify th a t
tr(A - 4 1 ) - t r í A í / H n ) = tr(M H ), w here
w ith с = (flu — wTA 111w )- 1 .
c- T . _ , det(A ) Since a u — w А ц w = \ we see th a t c > 0 . det(A u ) It a p p e a rs th a t M > 0 . In d e ed , fo r an arbitrary ^ -d im en sio n al v ec to r x T = ( x l , x l ) , * 1 e R, x 2 e R * -1 we have x r M x = c ( x x - b)2, w ith b = w tAi i1x2.
N ow , let e 7 = (1, 1, 1) and let M * H den o te the H a d a m a rd p ro d u c t o f the m atrices M , H . T h en tr(M H ) = e r (M * H )e, see R a o (1973). O n the o th e r h a n d , from the S chur lem m a we know th a t ( M * H ) > 0 fo r any m atrices M > 0 and H ^ 0. It follows th a t e r (M * H )c ^ 0 and th e p ro o f is com pleted. □
Lemma 2. Let A > 0 , H ^ O be given k x к m atrices. L et fo r i = 1, ..., к and x > 0 functions f t be defined as follows: / ,( x ) = tr(A " ‘H ), w here
a ri. i , \ a u + - f or j = l = i A* = [ M t x t w ith bjt = i x
aJt otherw ise.
Then fo r each i = 1, ..., к the f u n c ti o n / , is non-decreasing. P ro o f o f L em m a 2.
W ith o u t loss o f generality we m ay consider i = 1. A little calcu latio n show s th a t fo r each x > 0 the derivative o f f l does exist and
= d et(A n )det(A )[tr(A - lH ) - t ^ A ^ H n ) ] [det(A u ) + x d e t( A ) ] 2
In view o f Lem m a I and o u r assum ption s ab o u t the m atrices Л an d B, the derivative is nonnegativc, which com pletes the proo f. □
P ro o f o f the P ro p o sitio n 1.
F o r any A = d i a g ^ , , d 2, ..., d*) and L = diag(.st , s2, ..., s„), A > 0 , E > 0, let the function g (d lt ..., dk, s t , s j be defined as follows: g ( d t , dk, s u ..., s„) = T(A, I ) =
tr{[X r d i a g ( —, - ) X + diag (J , - ) ] ' 1 H (5)
Sj Sn “ 1 & к
It is easy to check th a t the function g can also be expressed in the follow ing way: g (d x, dk, su ..., sn) =
tr[d iag (d p dk)H \ - tr[X diag(</j, dt )X r + d ia g ( s 1, s„)]_1N (6) w here M = X d ia g ( d 1, dk) H d ia g (ť /j, dk) \ ' . N o te th a t M > 0 . I t can be seen from (5) th a t for each i = 1, к the fu n c tio n g as a function o f dt is o f the sam e form as the fu nction s / , from L em m a 2. T h e relatio n (6) show s th a t g as a function o f s, has got the form : const - / j f 1 ). T h u s, in view o f (5), (6) and L em m a 2, th e fu n c tio n g is
\ s i J
no n-decreasing w .r.t. each variable d lt dk, s it ..., s„. It results th a t on the set {(d, s ) e R ‘ x R ': 0 < d ^ < 5 , 0 < s ^ o - } it achieves its m axim um a t the p o in t (ô , и). Since the set is convex the cond ition (2) yields the desired result. □
O ne m ay notice th a t in the considered case the least fa v o u rab le states o f n a tu re (and associated w ith them m inim ax estim ators) are intuitive - the m atrices A s, £„ are connected w ith greatest values o f variances o f the regression p aram eter and e rro r, respectively. In th e next section we discuss a problem where th ere is no such “ p red ictab le” value o f th e least fa v o u rab le state o f nature.
4. MINIMAX ESTIMATION IN SOME PROBLEMS CONNECTED WITH CORRELATION BETWEEN ERRORS
рН-л
Let P (p ) d en o te a m atrix with elem ents p u = y —— 2 , |/j| < 1. Such m atrices a p p e ar in a n atu ra l way in the case w here the depen den ce betw een erro rs can be described by the follow ing first o rd e r a u to c o rre la tio n process:
where V = (Kj, V2, Vn) is a random vector with E(V) = 0 and D 2(V) = a>I„, 0 < c o < c o . It is w ell-know n th a t then C ov(Z ) = a)P(p).
I'o r given constants < u > 0, — \ < . p l <.p2 <. \ let us consider an uncertainty class G(S, о), p u p 2) = K x £2 with the set К defined as previously while Í2 = { £ e M , : I = E(w, a) = w f a P ^ j) + (1 - a)P(/?2)], 0 < w < w, 0 < ос < 1}.
T h e follow ing p ro p o sitio n provides the m inim ax estim ato rs w hen the uncertain ty class is G(<5, со, p , , p 2).
Proposition 2. T here exists a num ber a o G[0> 1] such th a t th e p air o f m atrices A s, E(ro, a 0) is the least favourable state o f n atu re and the estim ato r d*[A)> a o)] *s the m inim ax estim ato r in the gam e <G(á, со, p u p 2)t L , r ) .
I he n u m ber a o e[0 , 1] depend s on the m atrices X an d H . □
P ro p o sitio n 2 states th a t, as in the previous case, the least fa v o u rab le m atrix A is associated with the greatest variances o f /У,. O n the o th e r h and the p ro p o sitio n asserts th a t the least favourab le value o f the p aram eter a depends on the m atrices X, H and, in th a t sense, the least fa v o u rab le covariancc m atrix o f the vector Z is “ u n p red ictab le” .
I he p ro o f oi P ro p o sitio n 2 is based on L em m a 2 an d will be o m itted. In the sequel we present som e num erical exam ples to show how a 0 dep en d s o n the m atrix II determ ining the loss. T h e dependence yields th a t o u r solu tio n does n o t have the feature: “m inim ax p red ictio n ” equ als “ p redictio n based on m inim ax estim ate o f the regression p a ra m e te r” . T h e so lu tio n o f the problem s considered in the previous Section has g o t such a pro p erty .
In o u r exam ples we consider the m odel (1) with the follow ing fixed values o f its characteristics:
к = 3, n ш 7, X = 5 10 - 5 5 - 15 - 5 . 10 10 5 10 15 10 15 - 15 15 15 - 1 5 10 5 5 - 5
In all exam ples we consider the class G(<5, w, 0, 0.5) w ith fixed values <5 = (10, ..., 10) and со = 10. T his is because we alread y k no w how the estim ato rs depend upon these values. T o sim plify the n o ta tio n we w rite Tr(ot) instead o f tr{C[Ai; E(w, a)] H}.
Example 1. The classical problem of estim ation o f the regression param eter. In this exam ple we consider the case w here H = I, i.e. th e classical problem o f estim ation o f the p aram eter /?. F igu re 1 show s the g ra p h o f the function I r . W e can see th a t it has got one m axim um .
Fig. 1. The function Tr for the problem of estimation of [i.
It can be n um erically verified th a t the m axim um is ta k e n o n for
<*max = 0,728304.
Exam ple 2. The prediction o f the dependent variable.
N ow let us consider a problem o f prediction o f the value o f the d ep en d en t variable Y w hen the independent variables take on the follow ing values: = 1, x 2 = 5, x 3 = 9. T h e correspo nd in g m atrix d eterm in in g the
' l 5 9 '
loss function is o f the follow ing form : H = 5 25 45 . T h e g ra p h o f the 9 45 81
fu n ctio n T r in this case is presented in Fig. 2. N um erical calcu latio n show s th a t the only m axim um o f T r is achieved for a mttx= 0,812536.
T h e tw o above exam ples show th a t th e least fav o u rab le value o f the p a ra m e te r a (an d the associated co v arian ce m a trix £ ) ca n h a rd ly be considered as intuitive. T h e value changes for differen t m atrices H . It seems th a t in such situ atio n s we have different m inim ax estim ato rs fo r v ario us purp oses (such as the estim atio n o f regression p aram eter, p red ictio n for different values o f independent variables etc.) even in the sam e m o del. So, in th e case o f correlated erro rs one should be p artiu larly aw are o f the p u rp o se o f the m inim ax estim ation.
REFERENCES
B e r g e r J. (1982), Bayesian Robustness and the Stein Effect, J. Amer. Stat. Assoc. 358-368. B e r g e r J., C h e n S. (1987), Minimaxity o f Empirical Bayes Estimators Derived from Subjective Hyperpriors, [in:] Advances of the Multivariate Statistical Analysis, ed. A. K. G upta, 1-12.
B e r g e r J. (1990), Robust Bayesian Analysis: Sensitivity to the Prior, J. Statist. Plann Inference, 25, 303-328.
C h a t u r v e d i A., S r i v a s t a w a A. K. (1992), Families o f Minimax Estimators in Linear Regression Model, “Sankhya” , 54, Ser. B, 278-288.
D r y g a s H. (1996), Spectral Methods in Linear Minimax Estimation, Acta Appl. Math 43 17-42.
D r y g a s H., P i l z J. (1996), On the Equivalence o f the Spectral Theory and Bayesian Analysis in Minimax Linear Estimation, Acta Appl. Math., 43, 43-57.
G i r k o V. L. (1996), Spectral Theory o f Minimax Estimation, Acta Appl. Math., 43, 59-69. G r z y b o w s k i A. (1997), Minimaksowo-odporna estymacja parametrów regresji, “Przegląd
Statystyczny", 3, 427-435.
G r z y b o w s k i A. (1997), On Some Model o f Uncertainty in the Minimax-Robust Multivariate Linear Regression, Proceedings of Iď“ International Conference on Multivariate Statistical Analysis MSA‘97, 203-212.
H o f f m a n K. (1996), A Subclass o f Bayes Linear Estimators that are Minimax, Acta Appl Math., 43, 87-95.
M ę c z a r s k i M. (1998), Problemy odporności w bayesowskiej analizie statystycznej, ser. “ Monografie i Opracowania” , 446, SGH, Warszawa.
M i l o W. (1995), Stabilność i wrażliwość metod ekonometrycznych, Wydawnictwo Uniwersytetu Łódzkiego, Łódź.
P i l z J. (1986), Minimax Linear Regression Estimation with Symmetric Parameter Restrictions, J. Statist. Plann. Inference, 13, 297-318.
P i n e l i s I. F. (1991), O minimaksnom ocenivanii regressii, Teorija Verojatnostej i ее primenenija 35 (3), s. 494-505.
R a o C. R. (1973), Linear Statistical Inference and its Applications, John Wiley and Sons, New York. (Polish translation: Modele liniowe statystyki matematycznej, PWN, Warszawa 1984).
V e r d u S., P o o r H. V. (1984), On Minimax Robustness: A General Approach and Applications, IEEE Trans. Inform. Theory IT-30, 328-340.
