Mathematical Statistics Anna Janicka

Mathematical Statistics Anna Janicka

Lecture V, 22.03.2021

PROPERTIES OF ESTIMATORS, PART I

Plan for today

1. Maximum likelihood estimation examples – cont.

2. Basic estimator properties:

◼ estimator bias

◼ unbiased estimators

3. Measures of quality: comparing estimators

◼ mean square error

◼ incomparable estimators

◼ minimum-variance unbiased estimator

MLE – Example 1.

 Quality control, cont. We maximize

or equivalently maximize

i.e. solve solution:

x n x

x x n

X P

L  − ⁻



 



= 

=

= ( ) (1 )

)

( _  

) 1

ln(

) (

) ln(

ln )

) 1

ln((

) ln(

ln )

(   +  + − −



 



= 

− +

+



 



=  ⁻ x n x

x n x

l n ^{x n x}

1 0 )

(

' =

−

− −

=  

 ^{x n x}

l

n MLE

⁽  ⁾ =  ^ˆ

=

MLE – Example 3.

 Normal model: X₁, X₂, ..., X_n are a sample from N(









we solve

we get:

( ) ( )

( )

(

)

2 1 2

2 2

1 2

1

2 ln

) 2 ln(

) (

exp ln

) , (

2 2























n x

x n

x l

i i

n

i n

+



−



−

−

−

=

−

−

=









=

−



=

= +



−

 +

−

=





0

0 )

2 (

2 2

3

1

2 1 2





 

 

  

n i

l

i i

l n

x n x

x

1 2

2

( )

ˆ

ˆ

⁼

,

^

⁼



⁻



Estimator properties

 Aren’t the errors too large? Do we estimate what we want?

 is supposed to approximate



In general: is to approximate g(



 What do we want? Small error. But:

◼ errors are random variables (data are RV)

→ we can only control the expected value

◼ the error depends on the unknown .

→ we can’t do anything about it...

^ˆ

) ˆ X( g

Estimator bias

If is an estimator of



bias of the estimator is equal to

If is an estimator of g(



bias of the estimator is equal to

/ is unbiased, if

other notations, e.g.:

) ˆ X(



) ( )

ˆ ( ))

( )

ˆ ( ( )

(  E

g X g  E

g X g 

b = − = −









 ) =

( ˆ ( ) − ) =

ˆ ( ) −

(

b

) ˆ X( g

) ˆ X( ) g

ˆ X(

 b

(  ) = 0 for    

ˆ) (g B_

The normal model: reminder

Normal model: X₁, X₂, ..., X_n are a sample from distribution N(









Theorem. In the normal model, and S² are independent random variables, such that

In particular:

X

) ,

(

~ N

X 

) 1 (

~

1 2

2

 −

−

S n

n



) 1 2 (

Var and

, _, ^{2 4}

2 2

, = = −

S n S

E_{ }  _{ } 

X n X

E_, = ,and Var_, =  ²

Estimator bias – Example 1

In a normal model:







= X

 ˆ

1

ˆ

= X



ˆ

= 5



Estimator bias – Example 1

In a normal model:

 is an unbiased estimator of



 is an unbiased estimator of



 is biased:

bias:

= X

 ˆ











X = E X = E 

X = n =

E

i i

n

1 1

1 ,

, ,

ˆ ( )

1

ˆ

= X

 E

 ˆ

( X ) = E

X

= 

ˆ

= 5



2 for

eg

5 5

)

ˆ

(

,_

 =

=    =



X E

E



 ) = 5 − (

b

Estimator bias – Example 1

In a normal model:

 is an unbiased estimator of



 is an unbiased estimator of



 is biased:

bias:

= X

 ˆ











X = E X = E 

X = n =

E

i i

n

1 1

1 ,

, ,

ˆ ( )

1

ˆ

= X

 E

 ˆ

( X ) = E

X

= 

ˆ

= 5



2 for

eg

5 5

)

ˆ

(

,_

 =

=    =



X E

E



 ) = 5 − (

b

any model with unknown mean :

Estimator bias – Example 1 cont.

 is a biased

estimator of



 is an unbiased

estimator of





−

=

i i

n

X X

S

1 2

2

( )

ˆ

(

)

1

2 2

, 1

1 1 2

, 2

,

2 2 )

( )

(

) (

) (

) ˆ (









 ^{ }















−

= +

− +

=

−

=

−

=





n n n

n i n

i i

n

n n

X n X

E X

X E

X S

E



−

−

=

i i

n

X X

S

2 1

2 1

) (

(

)

(

)

1 1

2 2

1 , 1 1

2 1

1 ,

2 ,

) 1 (

) (

) (

) (

) (

) (

2  





 ^













=

−

= +

− +

=

−

=

−

=

−

−

= −

−

 

n n

n

X n X

E X

X E

X S

E

n n n

n i n

i i

n

Estimator bias – Example 1 cont.

 is a biased

estimator of



 is an unbiased

estimator of





−

=

i i

n

X X

S

1 2

2

( )

ˆ

(

)

1

2 2

, 1

1 1 2

, 2

,

2 2 )

( )

(

) (

) (

) ˆ (









 ^{ }















−

= +

− +

=

−

=

−

=





n n n

n i n

i i

n

n n

X n X

E X

X E

X S

E



−

−

=

i i

n

X X

S

2 1

2 1

) (

(

)

(

)

1 1

2 2

1 , 1 1

2 1

1 ,

2 ,

) 1 (

) (

) (

) (

) (

) (

2  





 ^













=

−

= +

− +

=

−

=

−

=

−

−

= −

−

 

n n

n

X n X

E X

X E

X S

E

n n n

n i n

i i

n

not necessarily the normal model!

Estimator bias – Example 1 cont. (2)

Bias of estimator is equal to

for n → , bias tends to 0, so this estimator is also OK for large samples



−

=

i i

n

X X

S

1 2

2

( )

ˆ

b n

2

)

(  = − 

for any distribution with a variance

Asymptotic unbiased estimator

 An estimator of g(



0 )

( lim

: =





 



n

) ˆ X( g

How to compare estimators?

 We want to minimize the error of the estimator; the estimator which makes smaller mistakes is better.

 The error may be either + or -, so usually we look at the square of the error (the

mean difference between the estimator and the estimated value)

Mean Square Error

If is an estimator of



Mean Square Error of estimator is the function

If is an estimator of g(



MSE of estimator is the function

We will only consider the MSE. Other measures are also possible (eg with absolute value)

) ˆ X(



))

( )

ˆ ( ( ˆ )

,

(  g E

g X g 

MSE = −

)

) ˆ (

( ˆ )

,

(   =



− 

MSE

) ˆ X( g

) ˆ X(



) ˆ X( g

Properties of the MSE

We have:

For unbiased estimators, the MSE is equal to the variance of the estimator

ˆ ) ( Var )

( ˆ )

,

( g b

g

MSE  =  +

MSE – Example 1

X₁, X₂, ..., X_n are a sample from a distribution with mean









 MSE of (unbiased):

 MSE of (unbiased):

 MSE of (biased):

= X

 ˆ

X n Var

X E

X MSE

2 ,

2

,

( )

) ,

,

(   =

−  =

= 

1

ˆ

= X



ˆ

= 5



2 1

, 2

1 ,

1

) ( )

, ,

(   X = E

X −  = Var

X = 

MSE

2 2

,

( 5 ) ( 5 )

) 5 , ,

(   = E

−  = − 

MSE

MSE – Example 2 Normal model

 MSE of

 MSE of

= 

−

i i

n

X X

S

1 2

2

( )

ˆ



−

−

=

i i

n

X X

S

2 1

2 1

) (

1 ) 2

( )

, , (

4 2

, 2

2 2

, 2

= −

=

−

= E S Var S n

S

MSE

 





4 2

4 2

2 2

4

2 ,

2 2

2 2

, 2

1 2

1 2

) 1 (

) ˆ (

ˆ ) ( ˆ )

, , (

 









 _{   }

n n n

n n n

S Var

b S

E S

MSE

= −

− + −

=

+

=

−

=

ˆ ) , , ( )

, ,

( S² MSE S²

MSE     

MSE – Example 2 Normal model

 MSE of

 MSE of

= 

−

i i

n

X X

S

1 2

2

( )

ˆ



−

−

=

i i

n

X X

S

2 1

2 1

) (

1 ) 2

( )

, , (

4 2

, 2

2 2

, 2

= −

=

−

= E S Var S n

S

MSE

 





4 2

4 2

2 2

4

2 ,

2 2

2 2

, 2

1 2

1 2

) 1 (

) ˆ (

ˆ ) ( ˆ )

, , (

 









 _{   }

n n n

n n n

S Var

b S

E S

MSE

= −

− + −

=

+

=

−

=

ˆ ) , , ( )

, ,

( S² MSE S²

MSE     

in any model: similarly, just with different expressions

MSE and bias – Example 2.

Poisson Model: X₁, X₂, ..., X_n are a sample from a Poisson distribution with unknown parameter



ML

= ... = X

 ˆ

0 )

( = b

X n X

X

MSE ⁿ

i i

n







Var 1

Var )

, (

Comparing estimators

is better than (dominates) , if

and

an estimator will be better than a different estimator only if its plot of the MSE never lies above the MSE plot of the other estimator; if the plots intersect, estimators are

incomparable

) ˆ₁(X

g gˆ₂( X )

ˆ ) , ( ˆ )

, (

MSE  g

MSE  g

   



ˆ ) , ( ˆ )

, (

MSE  g

MSE  g

   



MSE – Example 1 again

X₁, X₂, ..., X_n are a sample from a distribution with mean









 (unbiased)

 (unbiased)

 (biased)

 S² (biased)

 (unbiased)

= X

 ˆ

1

ˆ

= X



ˆ

= 5



ˆ 2

S

Comparing estimators – Example 1 cont.

We have

 From among

is better (for n>1)

 are incomparable,

just like

 From among is better

1

ˆ1

ˆ = X and







ˆ 5

ˆ = and





ˆ 5

ˆ₁ = ₁ and





2 2 and Sˆ S

ˆ 2

S

Comparing estimators – cont.

A lot of estimators are incomparable →

comparing any old thing is pointless; we need to constrain the class of estimators If we compare two unbiased estimators,

the one with the smaller variance will be better

Minimum-variance unbiased estimator

We constrain comparisons to the class of unbiased estimators. In this class, one can usually find the best estimator:

g*(X) is a minimum-variance unbiased estimator (MVUE) for g(



◼ g*(X) is an unbiased estimator of g(),

◼ for any unbiased estimator we have for 

) ˆ X( g

) ˆ(

) (

* X Var g X g

Var_  _

How can we check if the estimator has a minimum variance?

 In general, it is not possible to freely minimize the variance of unbiased

estimators – for many statistical models there exists a limit of variance

minimization. It depends on the

distribution and on the sample size.