2. Asymptotic properties of the LR test

Mathematical Statistics

Anna Janicka

Lecture XII, 13.05.2019

HYPOTHESIS TESTING IV:

P^ARAMETRIC T^ESTS: C^OMPARING T^{WO OR} M^ORE P^OPULATIONS

Plan for today

1. Parametric LR tests for one population – cont.

2. Asymptotic properties of the LR test

3. Parametric LR tests for two populations 4. Comparing more than two populations

ANOVA

Notation

x

always means a quantile of rank

something

Model III: comparing the mean

Asymptotic model: X₁, X₂, ..., X_n are an IID sample from a distribution with mean

µ

(unknown), n – large.

H₀:

µ

µ

Test statistic:

has, for large n, an approximate distribution N(0,1) H₀:

µ

µ

µ

µ

critical region

H₀:

µ

µ

µ

µ

critical region

H₀:

µ

µ

µ ≠ µ

critical region

S n T X −

µ

=

} )

( :

{

* = x T x > u

K

} )

( :

{

* = x T x < u

= − u

K

}

| ) ( :|

{

* = x T x > u

K

Model IV: comparing the fraction

Asymptotic model: X₁, X₂, ..., X_n are an IID sample from a two-point distribution, n – large.

H₀: p = p₀

Test statistic:

has an approximate distribution N(0,1) for large n H₀: p = p₀ against H₁: p > p₀

critical region

H₀: p = p₀ against H₁: p < p₀ critical region

H₀: p = p₀ against H₁: p

≠

) 0 (

1 )

1

( X = = p = − P X =

P

p n p

p n p

p p

p U X

) 1

( ˆ )

1

* (

0 0

0 0

0

0

−

= −

−

= −

} )

( :

{

* = x U x > u

K

} )

( :

{

* = x U x < u

= − u

K

}

| ) ( :|

{

* = x U x > u

K

Model IV: example

We toss a coin 400 times. We get 180 heads. Is the coin symmetric?

H₀: p = ½

for α = 0.05 and H₁: p ≠ ½ we have u_0.975 =1.96 → we reject H₀ for α = 0.05 and H₁: p < ½ we have u_0.05 = -u_0.95 =-1.64

→ we reject H₀

for α = 0.01 and H₁: p ≠ ½ we have u_0.995 =2.58

→ we do not reject H₀

for α = 0.01 and H₁: p < ½ we have u_0.01 = -u_0.99 =-2.33

→ we do not reject H₀

p-value for H₁: p ≠ ½: 0.044 p-value for H₁: p < ½: 0.022

2 ) 400

2 / 1 1

( 2 / 1

) 2 / 1 400

/ 180

* ( = −

−

= − U

Likelihood ratio test for composite hypotheses – reminder

X ~ P

, {P

: θ ∈ Θ} – family of distributions We are testing H

: θ ∈ Θ

against H

: θ ∈ Θ

such that Θ

∩ Θ

= ∅, Θ

∪ Θ

= Θ Let

H

: X ~ f

( θ

,⋅) for some θ

∈ Θ

H

: X ~ f

( θ

, ⋅) for some θ

∈ Θ

,

where f

and f

are densities (for θ ∈ Θ

and θ

∈ Θ

, respectively)

Likelihood ratio test for composite hypotheses – reminder (cont.)

Test statistic:

or

where are the ML estimators for the model without restrictions and for the null model, respectively.

We reject H

if for a constant .

) ,

( sup

) ,

(

~ sup

0

0 0

0

f X

X f

θ λ θ

θ θ

Θ

∈ Θ

=

) ˆ ,

(

) ˆ ,

~ (

0

0

X

f

X f

θ λ = θ

ˆ

ˆ , θ θ

~ c~

λ > ^c~

Asymptotic properties of the LR test

We consider two nested models, we test H₀: h(

θ

θ

Under the assumption that h is a nice function

Θ is a d-dimensional set

Θ₀ = {

θ

θ

Theorem: If H₀ is true, then for n→∞ the distribution of the statistic converges to a chi-squared

distribution with p degrees of freedom

λ

degrees of freedom = number of restrictions

Asymptotic properties of the LR test – example Exponential model: X₁, X₂, ..., X_n are an IID sample from Exp(

θ

We test H₀:

θ

θ

then:

from Theorem:

for a sign. level

α

X MLE(θ ) = θˆ = 1/

(

)

1 exp )

exp(

) exp(

) (

)

~ ( ^{1 1}

1

ˆ = −

Σ

−

Σ

= − Π

= Π n X

X x

x x

f

x f

n i

X i X

i

i ⁿ

λ

) 1 ( )

ln )

1 ((

~ 2 ln

2 λ = n X − − X →^D χ ² c

c ~ 2ln ~ ln

~ 2

~ > ⇔

λ

λ

c~

ln 2 84

. 3 )

1

2 (

95 .

0 ≈ ≈

χ

2 / 84 .

~ 3

> e

λ

Comparing two or more populations

We want to know if populations studied are

“the same” in certain aspects:

parametric tests: we check the equality of certain distribution parameters

nonparametric tests: we check whether

distributions are the same

Model I: comparison of means, variance known, significance level

α

X₁, X₂, ..., X_nX are an IID sample from distr N(

µ

σ

µ

σ

σ

σ

µ

µ

Test statistic:

H₀:

µ

µ

µ

µ

critical region

H₀:

µ

µ

µ

≠ µ

critical region

) 1 , 0 (

2 ~

2 Y N

U X

Y Y X

X

n n

σ σ +

= −

} )

( :

{

* = x U x > u

K

}

| ) ( :|

{

* = x U x > u

K

assuming H₀ is true

Model I – comparison of means. Example

X₁, X₂, ..., X₁₀ are an IID sample from distr N(

µ

µ

Are the means equal, for significance level 0.05?

H₀:

µ

µ

µ

≠ µ

we have: u_0.975 ≈ 1.96.

|0.557| < 1.96 → no grounds to reject H₀

498 ,

501 =

= Y

X

557 .

498 0 501

10 11 10

13^{2 2}

≈ +

= − U

Model II: comparison of means, variance

unknown but assumed equal, significance level

α

X₁, X₂, ..., X_nX are an IID sample from distr N(

µ

σ

µ

σ

σ

H₀:

µ

µ

H₀:

µ

µ

µ

µ

critical region

H₀:

µ

µ

µ

≠ µ

critical region

)}

2 (

) ( :

{

* = x T x > t

n

+ n

−

K

)}

2 (

| ) ( :|

{

* = x T x > t_{1− /2} n_x + n_y −

K _α

Assuming H₀ is true

2 1 1

2 1 2 1 1

2 1

) (

, )

( ∑

∑ = − =

− − = −

= ^Y

Y X

X

n

i i

Y n n

i i

X n X X S Y Y

S

) 2 (

~ ) 2 (

) 1 (

) 1

( ^{2 2}

− +

− + +

− +

−

= − _{X Y X Y}

Y X

Y X

Y Y

X x

n n

t n

n n n

n n S

n S

n

Y T X

Model II: comparison of means, variance unknown but assumed equal, cont.

can be rewritten as

where

is an estimator of the variance

σ

) 2 (

~ ) 2 (

) 1 (

) 1

( ^{2 2}

− +

− + +

− +

−

= − _{X Y X Y}

Y X

Y X

Y Y

X x

n n

t n

n n n

n n S

n S

n

Y T X

) 2 (

1 ~

1 + −

+

= −

∗

Y X

Y X

n n

t n

S n

Y T X

2 ) 1 (

) 1

( ^{2 2}

2

− +

− +

= −

∗

y x

Y Y

X x

n n

S n

S S n

Model II: comparison of variances, significance level

α

X₁, X₂, ..., X_nX are an IID sample from distr N(

µ

σ

µ

σ

σ

σ

σ

σ

Test statistic:

H₀:

σ

σ

σ

σ

critical region

H₀:

σ

σ

σ

≠ σ

critical region

) 1 ,

1 (

2 ~

2

−

−

= _{X Y}

Y

X F n n

S F S

)}

1 ,

1 (

) ( :

{

* = x F x > F₁₋ n_X − n_Y −

K _α

)}

1 ,

1 (

) (

) 1 ,

1 (

) ( : {

*

2 / 1

2 /

−

−

>

∨

−

−

<

=

− X Y

Y X

n n

F x

F

n n

F x

F x

K

α α

assuming H₀ is true

2 1 1

2 1 2 1 1

2 1

) (

, )

( ∑

∑ = − =

− − = −

= ^Y

Y X

X

n

i i

Y n n

i i

X n X X S Y Y

S

Model II: comparison of means, variances unknown and no equality assumption

X₁, X₂, ..., X_nX are an IID sample from distr N(

µ

σ

µ

σ

σ

σ

µ

µ

The test statistic would be very simple, but:

It isn’t possible to design a test statistic such that the distribution does not depend on

σ

σ

2 1 1

2 1 2 1 1

2 1

) (

, )

( ∑

∑ = − =

− − = −

= ^Y

Y X

X

n

i i

Y n n

i i

X n X X S Y Y

S

?

2 ~

2

Y Y X

X

n S n

S

Y X

+

−

Model III: comparison of means for large samples, significance level

α

X₁, X₂, ..., X_nX are an IID sample from distr. with mean µ_X, Y₁, Y₂, ..., Y_nY are an IID sample from distr. with mean µ_Y , both distr. have unknown variances, samples are independent,

n_X, n_Y – large.

H₀: µ_x = µ_Y Test statistic:

H₀: µ_x = µ_Y against H₁: µ_x > µ_Y

critical region

H₀: µ_x = µ_Y against H₁: µ_x ≠ µ_Y

critical region

) 1 , 0 (

2 ~

2 Y N

U X

Y Y X

X

n S n

S +

= −

assuming H_0. is true, for large samples

approximately

} )

( :

{

* = x U x > u

K

}

| ) ( :|

{

* = x U x > u

K

2 1 1

2 1 2 1 1

2 1

) (

, )

( ∑

∑ = − =

− − = −

= ^Y

Y X

X

n

i i

Y n n

i i

X n X X S Y Y

S

Model III – example (equality of means?)

Model IV: comparison of fractions for large samples, significance level

α

Two IID samples from two-point distributions. X – number of successes in n_X trials with prob of success p_X, Y – number of successes in n_Y trials with prob of success p_Y. p_X and p_Y

unknown, n_X and n_Y large.

H₀: p_X = p_Y Test statistic:

where

H₀: p_X = p_Y against H₁: p_X > p_Y critical region

H₀: p_X = p_Y against H₁: p_X ≠ p_Y critical region

( )

) 1

* (

1

1 N

p p

n Y n

X U

Y

X n

n Y X

+

−

−

=

∗

∗

} )

( :

{

* = x U

x > u

K

}

| ) ( :|

{

* = x U

x > u

K

y

x n

n

Y p X

+

= +

∗

assuming H_0. is true, for large samples

approximately

Model IV – example (equality of probabilities?)

Tests for more than two populations

A naive approach:

pairwise tests for all pairs But:

in this case, the type I error is higher than

the significance level assumed for each

simple test...

More populations

Assume we have k samples:

, and

all X

are independent (i=1,...,k, j=1,.., n

) X

~N(m

, σ

⁾

we do not know m

, m

, ..., m

, nor σ

let n=n

+n

+...+n

nk

k k

k

n n

X X

X

X X

X

X X

X

, 2

, 1

,

, 2 2

, 2 1

, 2

, 1 2

, 1 1

, 1

,..., ,

...

, ,...,

,

, ,...,

,

2 1

Test of the Analysis of Variance (ANOVA) for significance level

α

H

: µ

⁼ µ

^{=... =} µ

H

: ¬ ^H

(i.e. not all µ

^{are equal)} A LR test; we get a test statistic:

with critical region

for k=2 the ANOVA is equivalent to the two-sample t-test.

) ,

1 (

~ ) /(

) (

) 1 /(

) (

1 1

2 ,

1

2

k n

k F k

n X

X

k X

X F _k n

i

n

j i j i

k

i i i

i − −

−

−

−

= −

∑ ∑

∑

= =

=

∑

∑ ∑

∑

= ^k

i i i

k i

n

j i j

n

j i j

i

i n X

X n X n

n X

X ^{i i}

1

1 1 ,

1 ,

1 , 1

1

)}

, 1 (

) ( :

{

* x F x F

k n k

K = >

− −

ANOVA – interpretation

we have

– between group variance estimator

– within group variance estimator

∑ ∑

−

k i

n

j i j i

i X X

k

n ^{1 1}

2

, )

1 (

Sum of Squares (SS)

Sum of Squares Between (SSB)

Sum of Squares Within (SSW)

∑

−

k

i ni Xi X

k ¹

)2

1 ( 1

∑ ∑ ∑

∑ ∑

i

k i

n

j i j i

i i

k i

n

j i j

i

i X X n X X X X

1 1 1

2 ,

2

1 1

2

, ) ( ) ( )

(

ANOVA test – table

source of

variability sum of squares degrees of freedom

value of the test statistic F between

groups SSB k-1 –

within groups SSW n-k –

total SS n-1 F

ANOVA test – example

Yearly chocolate consumption in three cities: A, B, C based on random samples of n_A = 8, n_B = 10, n_C = 9 consumers. Does consumption depend on the city?

α=0.01

→ reject H₀ (equality of means), consumption depends on city

A B C

sample mean 11 10 7

sample variance 3.5 2.8 3

61 . 5 )

24 . 2 (

and

31 . 24 12

/ 7 . 73

2 / 63 . 75

7 . 73 8

3 9 8 . 2 7 5 . 3

63 . 75 9

) 3 . 9 7 ( 10 )

3 . 9 10 ( 8 ) 3 . 9 11 (

3 . 9 )

9 7 10 10

8 11 (

99 . 0

2 2

2 27

1

≈

≈

=

=

⋅ +

⋅ +

⋅

=

=

⋅

− +

⋅

− +

⋅

−

=

=

⋅ +

⋅ +

⋅

=

F F

SSW SSB

X

ANOVA test – table – example

source of

variability sum of squares degrees of freedom

value of the test statistic F between

groups 75.63 2 –

within groups 73.7 24 –

total 149.33 26 12.31