Mathematical Statistics Anna Janicka

Mathematical Statistics

Anna Janicka

Lecture XII, 18.05.2020

HYPOTHESIS TESTING IV:

PARAMETRIC TESTS: COMPARING TWO OR MORE

POPULATIONS

Plan for today

1. Parametric LR tests for one population – cont.

2. Asymptotic properties of the LR test

3. Parametric LR tests for two populations 4. Comparing more than two populations

 ANOVA

Notation

x

always means a quantile of rank

something

Model IV: comparing the fraction – reminder Asymptotic model: X

, X

, ..., X

are an IID sample from a two-point distribution, n – large.

H

: p = p

Test statistic:

has an approximate distribution N(0,1) for large n H

: p = p

against H

: p > p

critical region

H

: p = p

against H

: p < p

critical region

H

: p = p

against H

: p ≠ p

critical region

𝑃𝑃_𝑝𝑝(𝑋𝑋 = 1) = 𝑝𝑝 = 1 − 𝑃𝑃_𝑝𝑝(𝑋𝑋 = 0) 𝑈𝑈^∗ = ̄𝑋𝑋 − 𝑝𝑝₀

𝑝𝑝₀(1 − 𝑝𝑝₀) 𝑛𝑛 = ̂𝑝𝑝 − 𝑝𝑝₀

𝑝𝑝₀(1 − 𝑝𝑝₀) 𝑛𝑛

𝐶𝐶^∗ = {𝑥𝑥 : 𝑈𝑈^∗ (𝑥𝑥) > 𝑢𝑢_1−𝛼𝛼}

𝐶𝐶^∗ = {𝑥𝑥 : 𝑈𝑈^∗ (𝑥𝑥) < 𝑢𝑢_𝛼𝛼 = −𝑢𝑢_1−𝛼𝛼} 𝐶𝐶^∗ = {𝑥𝑥 : | 𝑈𝑈^∗(𝑥𝑥)| > 𝑢𝑢_{1−𝛼𝛼/2}}

Model IV: example

We toss a coin 400 times. We get 180 heads. Is the coin symmetric?

H

: p = ½

for α = 0.05 and H₁: p ≠ ½ we have u_0.975 =1.96 → we reject H₀ for α = 0.05 and H₁: p < ½ we have u_0.05 = -u_0.95 =-1.64

→ we reject H₀

for α = 0.01 and H₁: p ≠ ½ we have u_0.995 =2.58

→ we do not reject H₀

for α = 0.01 and H₁: p < ½ we have u_0.01 = -u_0.99 =-2.33

→ we do not reject H₀

p-value for H₁: p ≠ ½: 0.044 p-value for H₁: p < ½: 0.022 𝑈𝑈^∗ = (180/400 − 1/2)

1/2(1 − 1/2) 400 = −2

Likelihood ratio test for composite hypotheses – reminder

X ~ P

, {P

: θ ∈ Θ} – family of distributions We are testing H

: θ ∈ Θ

against H

: θ ∈ Θ

such that Θ

∩ Θ

= ∅, Θ

∪ Θ

= Θ Let

H

: X ~ f

( θ

,⋅) for some θ

∈ Θ

H

: X ~ f

( θ

, ⋅) for some θ

∈ Θ

,

where f

and f

are densities (for θ ∈ Θ

and θ

∈ Θ

, respectively)

Likelihood ratio test for composite hypotheses – reminder (cont.)

Test statistic:

or

where are the ML estimators for the model without restrictions and for the null model.

We reject H

if for a constant .

̃𝜆𝜆 = sup

𝑓𝑓 (𝜃𝜃, 𝑋𝑋) sup

𝑓𝑓

(𝜃𝜃

, 𝑋𝑋)

̃𝜆𝜆 = 𝑓𝑓( �𝜃𝜃, 𝑋𝑋) 𝑓𝑓

( �𝜃𝜃

, 𝑋𝑋)

�𝜃𝜃, �𝜃𝜃

̃𝜆𝜆 > ̃𝑐𝑐 ̃𝑐𝑐

more convenient if the null is simple or if models are nested

Asymptotic properties of the LR test We consider two nested models, we test H

: h( θ ) = 0 against H

: h( θ ) ≠ 0

Under the assumption that

 h is a nice function

 Θ is a d-dimensional set

 Θ

= { θ : h( θ ) = 0} is a d – p dimensional set

Theorem: If H

is true, then for n →∞ the distribution of the statistic converges to a chi-squared

distribution with p degrees of freedom

2ln ̃𝜆𝜆

degrees of freedom = number of restrictions

Asymptotic properties of the LR test – example Exponential model: X

, X

, ..., X

are an IID sample from Exp( θ ).

We test H

: θ = 1 against H

: θ ≠ 1

then:

from Theorem:

for a sign. level α =0.05 we have so we reject H

in favor of H

if

𝑀𝑀𝑀𝑀𝑀𝑀(𝜃𝜃) = �𝜃𝜃 = 1/ ̄𝑋𝑋

̃𝜆𝜆 = Π𝑓𝑓^�𝜃𝜃(𝑥𝑥_𝑖𝑖) Π𝑓𝑓₁(𝑥𝑥_𝑖𝑖) =

̄𝑋𝑋1^𝑛𝑛 exp( − 1̄𝑋𝑋Σ𝑥𝑥^𝑖𝑖)

exp( − Σ𝑥𝑥_𝑖𝑖) = 1

̄𝑋𝑋^𝑛𝑛 exp 𝑛𝑛( ̄𝑋𝑋 − 1)

2ln ̃𝜆𝜆 = 2𝑛𝑛(( ̄𝑋𝑋 − 1) − ̄ln 𝑋𝑋) ^𝐷𝐷 𝜒𝜒²(1)

̃𝜆𝜆 > ̃𝑐𝑐 ⇔ 2ln ̃𝜆𝜆 > 2ln ̃𝑐𝑐

𝜒𝜒_0.95² (1) ≈ 3.84 ≈ 2ln ̃𝑐𝑐

̃𝜆𝜆 > 𝑒𝑒^3.84/2

Comparing two or more populations

We want to know if populations studied are

“the same” in certain aspects:

 parametric tests: we check the equality of certain distribution parameters

 nonparametric tests: we check whether

distributions are the same

Model I: comparison of means, variance known, significance level α

X

, X

, ..., X

are an IID sample from distr N( µ

, σ

), Y

, Y

, ..., Y

are an IID sample from distr N( µ

, σ

),

σ

, σ

are known, samples are independent H

: µ

= µ

Test statistic:

H

: µ

= µ

against H

: µ

> µ

critical region

H

: µ

= µ

against H

: µ

≠ µ

critical region

𝑈𝑈 = ̄𝑋𝑋 − ̄𝑌𝑌 𝜎𝜎

�

𝑛𝑛

+ � 𝜎𝜎

𝑛𝑛

~ 𝑁𝑁 (0,1)

𝐶𝐶^∗ = {𝑥𝑥 : 𝑈𝑈 (𝑥𝑥) > 𝑢𝑢_1−𝛼𝛼} 𝐶𝐶^∗ = {𝑥𝑥 : | 𝑈𝑈(𝑥𝑥)| > 𝑢𝑢_{1−𝛼𝛼/2}}

assuming H₀ is true

Model I – comparison of means. Example

X

, X

, ..., X

are an IID sample from distr N( µ

,11

), Y

, Y

, ..., Y

are an IID sample from distr N( µ

,13

) Based on the sample:

Are the means equal, for significance level 0.05?

H

: µ

= µ

against H

: µ

≠ µ

we have: u

≈ 1.96.

|0.557| < 1.96 → no grounds to reject H

̄𝑋𝑋 = 501, ̄𝑌𝑌 = 498

𝑈𝑈 = 501 − 498 13²

10 + 11² 10

≈ 0.557

Model II: comparison of means, variance

unknown but assumed equal, significance level α

X

, X

, ..., X

are an IID sample from distr N( µ

, σ

), Y

, Y

, ..., Y

are an IID sample from distr N( µ

, σ

) with σ

unknown, samples are independent

H

: µ

= µ

Test statistic:

H

: µ

= µ

against H

: µ

> µ

critical region

H

: µ

= µ

against H

: µ

≠ µ

critical region

𝐾𝐾 ∗= {𝑥𝑥 : 𝑇𝑇 (𝑥𝑥) > 𝑡𝑡_1−𝛼𝛼(𝑛𝑛_𝑥𝑥 + 𝑛𝑛_𝑦𝑦 − 2)}

𝐶𝐶^∗ = {𝑥𝑥 : | 𝑇𝑇(𝑥𝑥)| > 𝑡𝑡_{1−𝛼𝛼/2}(𝑛𝑛_𝑥𝑥 + 𝑛𝑛_𝑦𝑦 − 2)}

Assuming H₀ is true

𝑆𝑆_𝑋𝑋² = 1

𝑛𝑛_𝑋𝑋−1�

𝑖𝑖=1 𝑛𝑛_𝑋𝑋

(𝑋𝑋_𝑖𝑖 − ̄𝑋𝑋)², 𝑆𝑆_𝑌𝑌² = 1

𝑛𝑛_𝑌𝑌−1�

𝑖𝑖=1 𝑛𝑛_𝑌𝑌

(𝑌𝑌_𝑖𝑖 − ̄𝑌𝑌)²

𝑇𝑇 = ̄𝑋𝑋 − ̄𝑌𝑌

(𝑛𝑛_𝑥𝑥 − 1)𝑆𝑆_𝑋𝑋² + (𝑛𝑛_𝑌𝑌 − 1)𝑆𝑆_𝑌𝑌²

𝑛𝑛_𝑋𝑋𝑛𝑛_𝑌𝑌

𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 (𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 − 2) ~ 𝑡𝑡 (𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 − 2)

Model II: comparison of means, variance unknown but assumed equal, cont.

can be rewritten as

where

is an estimator of the variance σ

based on the two samples

𝑇𝑇 = ̄𝑋𝑋 − ̄𝑌𝑌

(𝑛𝑛_𝑥𝑥 − 1)𝑆𝑆_𝑋𝑋² + (𝑛𝑛_𝑌𝑌 − 1)𝑆𝑆_𝑌𝑌²

𝑛𝑛_𝑋𝑋𝑛𝑛_𝑌𝑌

𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 (𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 − 2) ~ 𝑡𝑡 (𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 − 2)

𝑇𝑇 = ̄𝑋𝑋 − ̄𝑌𝑌 𝑆𝑆_∗ 1

𝑛𝑛_𝑋𝑋 + 1𝑛𝑛_𝑌𝑌

~ 𝑡𝑡 (𝑛𝑛_𝑋𝑋 + 𝑛𝑛_𝑌𝑌 − 2)

𝑆𝑆_∗² = (𝑛𝑛_𝑥𝑥 − 1)𝑆𝑆_𝑋𝑋² + (𝑛𝑛_𝑌𝑌 − 1)𝑆𝑆_𝑌𝑌² 𝑛𝑛_𝑥𝑥 + 𝑛𝑛_𝑦𝑦 − 2

Model II: comparison of variances, significance level α

X

, X

, ..., X

are an IID sample from distr N( µ

, σ

), Y

, Y

, ..., Y

are an IID sample from distr N( µ

, σ

),

σ

, σ

are unknown, samples are independent H

: σ

= σ

Test statistic:

H

: σ

= σ

against H

: σ

> σ

critical region

H

: σ

= σ

against H

: σ

≠ σ

critical region

𝐹𝐹 = 𝑆𝑆_𝑋𝑋²

𝑆𝑆_𝑌𝑌² ~ 𝐹𝐹 (𝑛𝑛_𝑋𝑋 − 1, 𝑛𝑛_𝑌𝑌 − 1)

𝐶𝐶^∗ = {𝑥𝑥 : 𝐹𝐹 (𝑥𝑥) > 𝐹𝐹_1−𝛼𝛼(𝑛𝑛_𝑋𝑋 − 1, 𝑛𝑛_𝑌𝑌 − 1)}

𝐶𝐶^∗ = {𝑥𝑥 : 𝐹𝐹 (𝑥𝑥) < 𝐹𝐹_𝛼𝛼/2(𝑛𝑛_𝑋𝑋 − 1, 𝑛𝑛_𝑌𝑌 − 1)

∨ 𝐹𝐹(𝑥𝑥) > 𝐹𝐹_{1−𝛼𝛼/2}(𝑛𝑛_𝑋𝑋 − 1, 𝑛𝑛_𝑌𝑌 − 1)}

assuming H₀ is true

𝑆𝑆𝑋𝑋2= 1 𝑛𝑛_𝑋𝑋−1�

𝑖𝑖=1 𝑛𝑛_𝑋𝑋

(𝑋𝑋𝑖𝑖− ̄𝑋𝑋)², 𝑆𝑆𝑌𝑌2= 1 𝑛𝑛_𝑌𝑌−1�

𝑖𝑖=1 𝑛𝑛_𝑌𝑌

(𝑌𝑌𝑖𝑖− ̄𝑌𝑌)²

Model II: comparison of means, variances unknown and no equality assumption

X

, X

, ..., X

are an IID sample from distr N( µ

, σ

), Y

, Y

, ..., Y

are an IID sample from distr N( µ

, σ

),

σ

, σ

are unknown, samples independent H

: µ

= µ

The test statistic would be very simple, but:

It isn’t possible to design a test statistic such that the distribution does not depend on σ

and σ

(values)...

𝑆𝑆_𝑋𝑋²= 1 𝑛𝑛_𝑋𝑋−1�

𝑖𝑖=1 𝑛𝑛_𝑋𝑋

(𝑋𝑋_𝑖𝑖− ̄𝑋𝑋)², 𝑆𝑆_𝑌𝑌²= 1 𝑛𝑛_𝑌𝑌−1�

𝑖𝑖=1 𝑛𝑛_𝑌𝑌

(𝑌𝑌_𝑖𝑖− ̄𝑌𝑌)²

̄𝑋𝑋 − ̄𝑌𝑌 𝑆𝑆_𝑋𝑋²

𝑛𝑛_𝑋𝑋 + 𝑆𝑆𝑛𝑛^𝑌𝑌_𝑌𝑌²

~?

Model III: comparison of means for large samples, significance level α

X₁, X₂, ..., X_nX are an IID sample from distr. with mean µ_X, Y₁, Y₂, ..., Y_nY are an IID sample from distr. with mean µ_Y , both distr. have unknown variances, samples are independent,

n_X, n_Y – large.

H₀: µ_x = µ_Y Test statistic:

H₀: µ_x = µ_Y against H₁: µ_x > µ_Y critical region

H₀: µ_x = µ_Y against H₁: µ_x ≠ µ_Y critical region

𝑈𝑈 = ̄𝑋𝑋 − ̄𝑌𝑌 𝑆𝑆_𝑋𝑋²

𝑛𝑛_𝑋𝑋 + 𝑆𝑆𝑛𝑛^𝑌𝑌_𝑌𝑌²

~ 𝑁𝑁 (0,1)

assuming H_0. is true, for large samples

approximately

𝐶𝐶^∗ = {𝑥𝑥 : 𝑈𝑈 (𝑥𝑥) > 𝑢𝑢_1−𝛼𝛼}

𝐶𝐶^∗ = {𝑥𝑥 : | 𝑈𝑈(𝑥𝑥)| > 𝑢𝑢_{1−𝛼𝛼/2}}

𝑆𝑆_𝑋𝑋²= 1 𝑛𝑛_𝑋𝑋−1�

𝑖𝑖 1 𝑛𝑛𝑋𝑋

(𝑋𝑋_𝑖𝑖− ̄𝑋𝑋)², 𝑆𝑆_𝑌𝑌²= 1 𝑛𝑛_𝑌𝑌−1�

𝑖𝑖 1 𝑛𝑛𝑌𝑌

(𝑌𝑌_𝑖𝑖− ̄𝑌𝑌)²

Model III – example (equality of means?)



Model IV: comparison of fractions for large samples, significance level α

Two IID samples from two-point distributions. X – number of successes in n_X trials with prob of success p_X, Y – number of successes in n_Y trials with prob of success p_Y. p_X and p_Y

unknown, n_X and n_Y large.

H₀: p_X = p_Y Test statistic:

where

H₀: p_X = p_Y against H₁: p_X > p_Y critical region

H₀: p_X = p_Y against H₁: p_X ≠ p_Y critical region

𝑈𝑈^∗ =

𝑛𝑛𝑋𝑋_𝑋𝑋 − 𝑌𝑌𝑛𝑛_𝑌𝑌

𝑝𝑝_∗(1 − 𝑝𝑝_∗) 1𝑛𝑛_𝑋𝑋 + 1𝑛𝑛_𝑌𝑌

~ 𝑁𝑁 (0,1)

𝐶𝐶^∗ = {𝑥𝑥 : 𝑈𝑈^∗( 𝑥𝑥) > 𝑢𝑢_1−𝛼𝛼}

𝐶𝐶^∗ = {𝑥𝑥 : | 𝑈𝑈^∗(𝑥𝑥)| > 𝑢𝑢_{1−𝛼𝛼/2}}

𝑝𝑝^∗ = 𝑋𝑋 + 𝑌𝑌 𝑛𝑛_𝑥𝑥 + 𝑛𝑛_𝑦𝑦

assuming H_0. is true, for large samples

approximately

Model IV – example (equality of probabilities?)



Tests for more than two populations

A naive approach:

pairwise tests for all pairs But:

in this case, the type I error is higher than

the significance level assumed for each

simple test...

More populations

Assume we have k samples:

, and

 all X

are independent (i=1,...,k, j=1,.., n

)

 X

~N(m

, σ

⁾

 we do not know m

, m

, ..., m

, nor σ

let n=n

+n

+...+n

𝑋𝑋

, 𝑋𝑋

, . . . , 𝑋𝑋

, 𝑋𝑋

, 𝑋𝑋

, . . . , 𝑋𝑋

, . . .

𝑋𝑋

, 𝑋𝑋

, . . . , 𝑋𝑋

Test of the Analysis of Variance (ANOVA) for significance level α

H

: µ

⁼ µ

^{=... =} µ

H

: ¬ ^H

(i.e. not all µ

^{are equal)} A LR test; we get a test statistic:

with critical region

for k=2 the ANOVA is equivalent to the two-sample t-test.

𝐹𝐹 = ∑_𝑖𝑖=1^𝑘𝑘 𝑛𝑛_𝑖𝑖( ̄𝑋𝑋_𝑖𝑖 − ̄𝑋𝑋)²/(𝑘𝑘 − 1)

∑_𝑖𝑖=1^𝑘𝑘 ∑_𝑗𝑗=1^{𝑛𝑛𝑖𝑖} (𝑋𝑋_{𝑖𝑖,𝑗𝑗} − ̄𝑋𝑋_𝑖𝑖)²/(𝑛𝑛 − 𝑘𝑘) ~ 𝐹𝐹 (𝑘𝑘 − 1, 𝑛𝑛 − 𝑘𝑘)

̄𝑋𝑋_𝑖𝑖 = 1 𝑛𝑛_𝑖𝑖 �

𝑗𝑗=1 𝑛𝑛_𝑖𝑖

𝑋𝑋_{𝑖𝑖,𝑗𝑗} , ̄𝑋𝑋 = 1 𝑛𝑛 �_𝑖𝑖=1

𝑘𝑘

�

𝑗𝑗=1 𝑛𝑛_𝑖𝑖

𝑋𝑋_{𝑖𝑖,𝑗𝑗} = 1 𝑛𝑛 �_𝑖𝑖=1

𝑘𝑘

𝑛𝑛_𝑖𝑖 ̄𝑋𝑋_𝑖𝑖

𝐶𝐶

= {𝑥𝑥 : 𝐹𝐹 (𝑥𝑥) > 𝐹𝐹

(𝑘𝑘 − 1, 𝑛𝑛 − 𝑘𝑘)}

ANOVA – interpretation

we have

– between group variance estimator – within group variance estimator 1

𝑛𝑛 − 𝑘𝑘 �_𝑖𝑖=1

𝑘𝑘

�

𝑗𝑗=1 𝑛𝑛_𝑖𝑖

(𝑋𝑋_{𝑖𝑖,𝑗𝑗} − ̄𝑋𝑋_𝑖𝑖)²

Sum of Squares (SS)

Sum of Squares Between (SSB)

Sum of Squares Within (SSW)

1

𝑘𝑘 − 1 �_𝑖𝑖=1

𝑘𝑘

𝑛𝑛_𝑖𝑖( ̄𝑋𝑋_𝑖𝑖 − ̄𝑋𝑋)²

�

𝑖𝑖=1 𝑘𝑘

�

𝑗𝑗=1 𝑛𝑛_𝑖𝑖

(𝑋𝑋_{𝑖𝑖,𝑗𝑗} − ̄𝑋𝑋)² = �

𝑖𝑖=1 𝑘𝑘

𝑛𝑛_𝑖𝑖( ̄𝑋𝑋_𝑖𝑖 − ̄𝑋𝑋)² + �

𝑖𝑖=1 𝑘𝑘

�

𝑗𝑗=1 𝑛𝑛_𝑖𝑖

(𝑋𝑋_{𝑖𝑖,𝑗𝑗} − ̄𝑋𝑋_𝑖𝑖)²

ANOVA test – table

source of

variability sum of squares degrees of freedom

value of the test statistic F between

groups SSB k-1 –

within groups SSW n-k –

total SS n-1 F

ANOVA test – example

Yearly chocolate consumption in three cities: A, B, C based on random samples of n

= 8, n

= 10, n

= 9 consumers. Does consumption depend on the city?

α=0.01

→ reject H

(equality of means), consumption depends on city

A B C

sample mean 11 10 7

sample variance 3.5 2.8 3

̄𝑋𝑋 = 127 (11 ⋅ 8 + 10 ⋅ 10 + 7 ⋅ 9) = 9.3

𝑆𝑆𝑆𝑆𝑆𝑆 = (11 − 9.3)² ⋅ 8 + (10 − 9.3)² ⋅ 10 + (7 − 9.3)² ⋅ 9 = 75.63 𝑆𝑆𝑆𝑆𝑆𝑆 = 3.5 ⋅ 7 + 2.8 ⋅ 9 + 3 ⋅ 8 = 73.7

𝐹𝐹 = 75.63/2

73.7/24 ≈ 12.31 and 𝐹𝐹^0.99(2,24) ≈ 5.61

ANOVA test – table – example

source of

variability sum of squares degrees of freedom

value of the test statistic F between

groups 75.63 2 –

within groups 73.7 24 –

total 149.33 26 12.31