A NOTE ON THE STRONG CONSISTENCY OF LEAST SQUARES ESTIMATES

Probability and Statistics 29 (2009 ) 223-231

A NOTE ON THE STRONG CONSISTENCY OF LEAST SQUARES ESTIMATES

Joˇ ao Lita da Silva Faculdade de Ciˆ encias e Tecnologia

Universidade Nova de Lisboa

Quinta da Torre, 2829–516 Caparica, Portugal e-mail: jfls@fct.unl.pt

Abstract

The strong consistency of least squares estimates in multiples re- gression models with i.i.d. errors is obtained under assumptions on the design matrix and moment restrictions on the errors.

Keywords: least squares estimates, linear models, strong consistency.

2000 Mathematics Subject Classification: 60F15.

1. Introduction

Many statisticians have considered the problem of strong consistency of the least squares estimates in multiple regression models. In the seventies, this problem was completely solved under weak moment conditions on the errors, namely, assuming their finite variance (see [5, 6] and [9]). More recently, some other authors had studied this same problem for the case where the variance of the errors is infinite. Nevertheless, these works reveals very restrictive conditions on the design matrix (see [8]) or particular scenarios for the errors (see [11, 12] or [13]).

In this paper, we establish the strong consistency of the least squares estimates for the parameters β _j of the multiple regression model

(1.1) y _i = β ₁ x _i1 + . . . + β _p x _ip + ε _i (i = 1, 2, . . .)

under suitable assumptions on the design matrix x _ij when the error variance is infinite. Specially, we shall assume that

(1.2)

ε _i are i.i.d. with E |ε 1 | ^r < ∞ for some r ∈ (0, 2) and E ε 1 = 0 whenever r ∈ (1, 2)

admitting cases where the errors don’t have mean value. Let us stress that in [6] or [7] only the errors ε i with sup _i E |ε i | ^r < ∞ for some 1 6 r < 2 are considered leaving the cases where 0 < r < 1 unsolved. In particular, on the paper [6], the authors establish the strong consistency of the least squares estimates for the case 1 6 r < 2 using the H¨older inequality (see Corollary 3 and Lemma 4), which is no more useful when 0 < r < 1.

Throughout this work, we shall let X _n denote the design matrix x _ij

16i6n,16j6p

and let y _n = (y ₁ , . . . , y _n ) ⁰ and β = (β ₁ , . . . , β _p ) ⁰ , where prime denotes trans- pose. For n > p, the least squares estimate b _n = (b _n1 , . . . , b _np ) ⁰ of the vector β based on the design matrix X n and the response vector y n is given by (1.3) b _n = (X ⁰ _n X _n ) ⁻¹ X ⁰ _n y _n

provided that

X ⁰ _n X _n =









































n

X

i=1

x ² _i1

n

X

i=1

x _i1 x _i2 . . .

n

X

i=1

x _i1 x _ip

n

X

i=1

x i1 x i2 n

X

i=1

x ² _i2 . . .

n

X

i=1

x i2 x ip

.. . .. . . .. .. .

n

X

i=1

x _i1 x _ip

n

X

i=1

x _i2 x _ip . . .

n

X

i=1

x ² _ip









































is nonsingular for all n > n ₀ . From the expression of b _n it follows that the strong consistency of the least squares estimates is equivalent to

(X ⁰ _n X _n ) ⁻¹

n

X

i=1

x _i ε _i −→ 0 ^a.s.

where x _i = (x _i1 , . . . , x _ip ) ⁰ .

2. Auxiliary tools

It is well-known that every positive definite matrix A = a _ij

16i,j6p satisfies det(A) 6 a ₁₁ . . . a _pp

and the equality holds if and only if A is diagonal (see [3], page 477). This classical result due to Hadamard leads us to the following definition.

Definition. A sequence of p×p positive definite matrices A n = a ⁽ⁿ⁾ _ij

16i,j6p

is said asymptotically diagonal dominant if det(A _n )  a ⁽ⁿ⁾ ₁₁ . . . a ⁽ⁿ⁾ _pp , n → ∞. ^∗

An important tool in proving the strong consistency of b _n for error structures satisfying (1.2) is the next lemma which is an extension of Marcinkiewicz- Zygmund theorem presented in [1] (page 118).

Lemma 1. If {X _n } are i.i.d. r.v.’s with E |X 1 | ^r < ∞ and {a _n } are real numbers such that a _n = O n ^−1/r
for some 0 < r < 2 then

∞

X

n=1

(a n X n − E Y n )

converges a.s., where Y _n = a _n X _n I _{|X

_|≤n

} . Furthermore, if either (i) 0 < r < 1 or (ii) 1 < r < 2 and E X 1 = 0, then P _∞

n=1 a _n X _n converges a.s.

∗

a

 b

, n → ∞ means that a

= O(b

) and b

= O(a

) as n → ∞.

P roof. Set A _j = (j − 1) ^1/r ≤ |X ₁ | ≤ j ^1/r , j ≥ 1. Then for α > r > 0

∞

X

n=1

E |Y n | ^α ≤

∞

X

n=1 n

X

j=1

|a _n | ^α Z

A

|X ₁ | ^α

≤ C

∞

X

n=1 n

X

j=1

n ^−α/r Z

A

|X ₁ | ^α

≤ C

∞

X

n=1



j ^−α/r + r

α − r j ^(r−α)/r

 Z

A

|X ₁ | ^α

≤

∞

X

n=1

α α − r

Z

A

|X ₁ | ^r

≤ α

α − r E |X 1 | ^r < ∞, (2.1)

whence (α = 2) P _∞

n=1 (Y _n − E Y n ) converges a.s. by Khintchine’s theorem (see [1], page 113). Since

∞

X

n=1

P {a n X _n 6= Y _n } =

∞

X

n=1

P n

|X ₁ | > n ^1/r o

≤ E |X 1 | ^r < ∞

the sequences {a _n X _n }, {Y _n } are equivalent and so P _∞

n=1 (a _n X _n − E Y n ) con- verges a.s.

In case (i), where 0 < r < 1, P _∞

n=1 |E Y n | < ∞ via (1.1) with α = 1. In

case (ii), where 1 < r < 2 and E X 1 = 0, we have

∞

X

n=1

|E Y n | ≤

∞

X

n=1

|a n | Z

{ |X

|>n

} |X n |

≤ C

∞

X

n=1

∞

X

j=n+1

n ^−1/r Z

A

|X ₁ |

= C

∞

X

j=2 j−1

X

n=1

n ^−1/r Z

A

|X ₁ |

≤ C r r − 1

∞

X

j=1

(j − 1) ^(r−1)/r Z

A

|X ₁ | ^r

≤ C r r − 1

∞

X

j=1

Z

A

|X ₁ | ^r

= C r

r − 1 E |X 1 | ^r < ∞.

Thus, the second part of Lemma 1 follows from the first.

Remark. If {X n , n ≥ 1} is a i.i.d. sequence of r.v.’s with E |X 1 | < ∞ and {a _n , n ≥ 1} are real numbers such that a _n = O n ⁻¹
and P ^∞ _n=1 a _n converges then P ∞

n=1 a _n X _n converges a.s.

3. Strong consistency

In this section we shall prove the main result of this paper.

Theorem 1. Suppose that in model (1.1), ε ₁ , ε ₂ , . . . are random variables

satisfying (1.2) and {x _ij } (i = 1, 2, . . . ; j = 1, . . . , p) is an arbitrary double

array of constants. If X ⁰ _n X _n is nonsingular for all n > n ₀ and asymptotically diagonal dominant with constants x ij satisfying

n

X

k=1

x ² _kj → ∞ for all j and

(i) x _nj

n

X

k=1

x ² _ki

n

X

k=1

x ² _kj

! 1/2 = O  n ^−1/r 

for all i, j = 1, . . . , p when r 6= 1

or

(ii) x nj

n

X

k=1

x ² _ki

n

X

k=1

x ² _kj

! 1/2 = O n ⁻¹
and

∞

X

n=1

x nj n

X

k=1

x ² _ki

n

X

k=1

x ² _kj

! 1/2

converges for all i, j = 1, . . . , p whenever r = 1, then b _n −→ β. ^a.s.

P roof. Setting C _n = c ⁽ⁿ⁾ _ij

16i,j6p = (X ⁰ _n X _n ) ⁻¹ we have det(X ⁰ _n X _n ) c ⁽ⁿ⁾ _ij = (−1) ^i+j X

σ∈S

sgn(σ)

p

Y

m=1 m6=i σ(m)6=j

n

X

k=1

x _km x _kσ(m)

where the sum is computed over all bijections σ of {1, . . . , i − 1, i + 1, . . . , p}

into {1, . . . , j − 1, j + 1, . . . , p}. Thus, using Cauchy-Schwarz inequality we get

det(X ⁰ _n X _n )   c

(n) ii

 6 (p − 1)!

p

Y

m=1 m6=i n

X

k=1

x ² _km

and

det(X ⁰ _n X _n )   c

(n) ij

   6

6 (p − 1)!

n

X

k=1

x ² _ki

! 1/2 n

X

k=1

x ² _kj

! 1/2 p

Y

m6=i,j m=1 n

X

k=1

x ² _km , i 6= j.

Since det(X ⁰ _n X _n )  P _n

k=1 x ² _k1 . . . P _n

k=1 x ² _kp as n → ∞ it is sufficient to prove that

(3.1) 1

n

X

k=1

x ² _ki

n

X

k=1

x ² _kj

! 1/2 n

X

m=1

x _mj ε _m −→ 0 ^a.s. (i, j = 1, . . . , p).

By Lemma 1

n

X

m=1

x mj m

X

k=1

x ² _ki

m

X

k=1

x ² _kj

! 1/2 ε _m converges a.s. (i, j = 1, . . . , p)

and Kronecker’s lemma permit us to conclude (3.1) which establish the thesis.

Example. Suppose that in model (1.1), ε ₁ , ε ₂ , . . . are random variables sa- tisfying (1.2) and let {x _ij } (i = 1, 2, . . . ; j = 1, . . . , p) be a double array of constant given by x _ki = k ^α

with α _i > 1/r − 1 (i = 1, . . . , p) all different.

Therefore,

n

X

k=1

x ² _ki ∼ n ^2α

⁺¹

2α _i + 1 and

n

X

k=1

x _ki x _kj ∼ n ^α

^+α

⁺¹

α _i + α _j + 1 , i 6= j

which implies that X ⁰ _n X _n is asymptotically diagonal dominant. Indeed, using induction on p we have

det(X ⁰ _n X _n ) =

p

X

j=1 n

X

k=1

x _kp x _kj · (−1) ^p+j det X ⁰ _n X _n (p|j)

where the modulus of each term of (−1) ^p+j det X ⁰ _n X _n (p|j)
, j = 1, . . . , p−1 is bounded by

n

X

k=1

x ² _kp

! 1/2 n

X

k=1

x ² _kj

! 1/2 p

Y

m6=j,p m=1 n

X

k=1

x ² _km .

The conclusion now follows since

n

X

k=1

x _kp x _kj

n

X

k=1

x ² _kp

! 1/2 n

X

k=1

x ² _kj

! 1/2 ∼ p2α p + 1p2α j + 1

α _p + α _j + 1 < 1, j = 1, . . . , p − 1

and P _n

k=1 x ² _kp · det X ⁰ _n X _n (p|p)
∼ P ⁿ _k=1 x ² _k1 . . . P _n

k=1 x ² _kp as n → ∞. The assumptions (i) or (ii) of Theorem 1 are also satisfied.

References

[1] Y.S. Chow and H. Teicher, Probability Theory: Independence, Interchangeability, Martingales (third edition) Springer 1997.

[2] Chen Xiru, Consistency of LS estimates of multiple regression under a lower order moment condition, Sci. Chin. 38 (12) (1995), 1420–1431.

[3] R.A. Horn and C.R. Johnson, Matrix Analysis, Cambridge University Press 1985.

[4] H. Drygas, Consistency of the least squares and Gauss-Markov estimators in regression models, Z. Wahrscheinlichkeitstheorie Verw. Gebiete 17 (1971), 309–326.

[5] H. Drygas, Weak and strong consistency of the least squares estimators in regression model, Z. Wahrscheinlichkeitstheorie Verw. Gebiete 34 (1976), 119–127.

[6] C. Gui-Jing, T.L. Lai and C.Z. Wei, Convergence systems and strong, con- sistency of least squares estimates in regression models, J. Multivariate Anal.

11 (1981), 319–333.

[7] C. GuiJing, Extension of Lai-Robbins-Wei’s theorem, Acta Mathematicae Applicatae Sinica 1 (1) (1984), 2–7.

[8] J. Mingzhong, Some new results of the strong consistency of multiple re- gression coefficients, in: S. Tangmanee & E. Schulz, eds. World Scientific, Proceedings of the Second Asian Mathematical Conference 1995 (R. Nakhon 1995), 514–519.

[9] T.L. Lai, H. Robbins and C.Z. Wei, Strong consistency of least squares es-

timates in multiple regression II, J. Multivariate Anal. 9 (1979), 343–362.

[10] B.M. Makarov, M.G. Goluzina, A.A. Lodkin and A.N. Podkorytov, Selected Problems in Real Analysis (American Mathematical Society, Providence R.I.

1992).

[11] J.T. Mexia, P. Corte Real, M.L. Esqu´ıvel e J. Lita da Silva, Convergˆencia do estimador dos m´ınimos quadrados em modelos lineares, Estat´ıstica Jubilar.

Actas do XII Congresso da Sociedade Portuguesa de Estat´ıstica, Edi¸c˜ oes SPE (2005), 455–466.

[12] J.T. Mexia e J. Lita da Silva, A consistˆencia do estimador dos m´ınimos quadrados em dom´ınios de atrac¸c˜ ao maximais, Ciˆencia Estat´ıstica. Actas do XIII Congresso Anual da Sociedade Portuguesa de Estat´ıstica, Edi¸c˜ oes SPE (2006), 481–492.

[13] J.T. Mexia and J. Lita da Silva, Sufficient conditions for the strong consis- tency of least squares estimator with α-stable errors, Discussiones Mathe- maticae - Probability and Statistics 27 (2007), 27–45.

Received 15 November 2009